aboutsummaryrefslogtreecommitdiff
path: root/usr.bin/lex/NEWS
blob: 3e23e7d9f62327021a99d105ff10327c10aeccdd (plain) (blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
Changes between release 2.5.4 (11Sep96) and release 2.5.3:

	- Fixed a bug introduced in 2.5.3 that blew it when a call
	  to input() occurred at the end of an input file.

	- Fixed scanner skeleton so the example in the man page of
	  scanning strings using exclusive start conditions works.

	- Minor Makefile tweaks.


Changes between release 2.5.3 (29May96) and release 2.5.2:

	- Some serious bugs in yymore() have been fixed.  In particular,
	  when using AT&T-lex-compatibility or %array, you can intermix
	  calls to input(), unput(), and yymore().  (This still doesn't
	  work for %pointer, and isn't likely to in the future.)

	- A bug in handling NUL's in the input stream of scanners using
	  REJECT has been fixed.

	- The default main() in libfl.a now repeatedly calls yylex() until
	  it returns 0, rather than just calling it once.

	- Minor tweak for Windows NT Makefile, MISC/NT/Makefile.


Changes between release 2.5.2 (25Apr95) and release 2.5.1:

	- The --prefix configuration option now works.

	- A bug that completely broke the "-Cf" table compression
	  option has been fixed.

	- A major headache involving "const" declarators and Solaris
	  systems has been fixed.

	- An octal escape sequence in a flex regular expression must
	  now contain only the digits 0-7.

	- You can now use "--" on the flex command line to mark the
	  end of flex options.

	- You can now specify the filename '-' as a synonym for stdin.

	- By default, the scanners generated by flex no longer
	  statically initialize yyin and yyout to stdin and stdout.
	  This change is necessary because in some ANSI environments,
	  stdin and stdout are not compile-time constant.  You can
	  force the initialization using "%option stdinit" in the first
	  section of your flex input.

	- "%option nounput" now correctly omits the unput() routine
	  from the output.

	- "make clean" now removes config.log, config.cache, and the
	  flex binary.  The fact that it removes the flex binary means
	  you should take care if making changes to scan.l, to make
	  sure you don't wind up in a bootstrap problem.

	- In general, the Makefile has been reworked somewhat (thanks
	  to Francois Pinard) for added flexibility - more changes will
	  follow in subsequent releases.

	- The .texi and .info files in MISC/texinfo/ have been updated,
	  thanks also to Francois Pinard.

	- The FlexLexer::yylex(istream* new_in, ostream* new_out) method
	  now does not have a default for the first argument, to disambiguate
	  it from FlexLexer::yylex().

	- A bug in destructing a FlexLexer object before doing any scanning
	  with it has been fixed.

	- A problem with including FlexLexer.h multiple times has been fixed.

	- The alloca() chud necessary to accommodate bison has grown
	  even uglier, but hopefully more correct.

	- A portability tweak has been added to accommodate compilers that
	  use char* generic pointers.

	- EBCDIC contact information in the file MISC/EBCDIC has been updated.

	- An OS/2 Makefile and config.h for flex 2.5 is now available in
	  MISC/OS2/, contributed by Kai Uwe Rommel.

	- The descrip.mms file for building flex under VMS has been updated,
	  thanks to Pat Rankin.

	- The notes on building flex for the Amiga have been updated for
	  flex 2.5, contributed by Andreas Scherer.


Changes between release 2.5.1 (28Mar95) and release 2.4.7:

	- A new concept of "start condition" scope has been introduced.
	  A start condition scope is begun with:

		<SCs>{

	  where SCs is a list of one or more start conditions.  Inside
	  the start condition scope, every rule automatically has the
	  prefix <SCs> applied to it, until a '}' which matches the
	  initial '{'.  So, for example:

		<ESC>{
			"\\n"	return '\n';
			"\\r"	return '\r';
			"\\f"	return '\f';
			"\\0"	return '\0';
		}

	  is equivalent to:

		<ESC>"\\n"	return '\n';
		<ESC>"\\r"	return '\r';
		<ESC>"\\f"	return '\f';
		<ESC>"\\0"	return '\0';

	  As indicated in this example, rules inside start condition scopes
	  (and any rule, actually, other than the first) can be indented,
	  to better show the extent of the scope.

	  Start condition scopes may be nested.

	- The new %option directive can be used in the first section of
	  a flex scanner to control scanner-generation options.  Most
	  options are given simply as names, optionally preceded by the
	  word "no" (with no intervening whitespace) to negate their
	  meaning.  Some are equivalent to flex flags, so putting them
	  in your scanner source is equivalent to always specifying
	  the flag (%option's take precedence over flags):

		7bit	-7 option
		8bit	-8 option
		align	-Ca option
		backup	-b option
		batch	-B option
		c++	-+ option
		caseful	opposite of -i option (caseful is the default);
		case-sensitive	same as above
		caseless	-i option;
		case-insensitive	same as above
		debug	-d option
		default	opposite of -s option
		ecs	-Ce option
		fast	-F option
		full	-f option
		interactive	-I option
		lex-compat	-l option
		meta-ecs	-Cm option
		perf-report	-p option
		read	-Cr option
		stdout	-t option
		verbose	-v option
		warn	opposite of -w option (so use "%option nowarn" for -w)

		array	equivalent to "%array"
		pointer	equivalent to "%pointer" (default)

	  Some provide new features:

		always-interactive	generate a scanner which always
			considers its input "interactive" (no call to isatty()
			will be made when the scanner runs)
		main	supply a main program for the scanner, which
			simply calls yylex().  Implies %option noyywrap.
		never-interactive	generate a scanner which never
			considers its input "interactive" (no call to isatty()
			will be made when the scanner runs)
		stack	if set, enable start condition stacks (see below)
		stdinit	if unset ("%option nostdinit"), initialize yyin
			and yyout statically to nil FILE* pointers, instead
			of stdin and stdout
		yylineno	if set, keep track of the current line
			number in global yylineno (this option is expensive
			in terms of performance).  The line number is available
			to C++ scanning objects via the new member function
			lineno().
		yywrap	if unset ("%option noyywrap"), scanner does not
			call yywrap() upon EOF but simply assumes there
			are no more files to scan

	  Flex scans your rule actions to determine whether you use the
	  REJECT or yymore features (this is not new).  Two %options can be
	  used to override its decision, either by setting them to indicate
	  the feature is indeed used, or unsetting them to indicate it
	  actually is not used:

		reject
		yymore

	  Three %option's take string-delimited values, offset with '=':

		outfile="<name>"	equivalent to -o<name>
		prefix="<name>"		equivalent to -P<name>
		yyclass="<name>"	set the name of the C++ scanning class
					(see below)

	  A number of %option's are available for lint purists who
	  want to suppress the appearance of unneeded routines in
	  the generated scanner.  Each of the following, if unset,
	  results in the corresponding routine not appearing in the
	  generated scanner:

		input, unput
		yy_push_state, yy_pop_state, yy_top_state
		yy_scan_buffer, yy_scan_bytes, yy_scan_string

	  You can specify multiple options with a single %option directive,
	  and multiple directives in the first section of your flex input file.

	- The new function:

		YY_BUFFER_STATE yy_scan_string( const char *str )

	  returns a YY_BUFFER_STATE (which also becomes the current input
	  buffer) for scanning the given string, which occurs starting
	  with the next call to yylex().  The string must be NUL-terminated.
	  A related function:

		YY_BUFFER_STATE yy_scan_bytes( const char *bytes, int len )

	  creates a buffer for scanning "len" bytes (including possibly NUL's)
	  starting at location "bytes".

	  Note that both of these functions create and scan a *copy* of
	  the string/bytes.  (This may be desirable, since yylex() modifies
	  the contents of the buffer it is scanning.)  You can avoid the
	  copy by using:

		YY_BUFFER_STATE yy_scan_buffer( char *base, yy_size_t size )

	  which scans in place the buffer starting at "base", consisting
	  of "size" bytes, the last two bytes of which *must* be
	  YY_END_OF_BUFFER_CHAR (these bytes are not scanned; thus, scanning
	  consists of base[0] through base[size-2], inclusive).  If you
	  fail to set up "base" in this manner, yy_scan_buffer returns a
	  nil pointer instead of creating a new input buffer.

	  The type yy_size_t is an integral type to which you can cast
	  an integer expression reflecting the size of the buffer.

	- Three new routines are available for manipulating stacks of
	  start conditions:

		void yy_push_state( int new_state )

	  pushes the current start condition onto the top of the stack
	  and BEGIN's "new_state" (recall that start condition names are
	  also integers).

		void yy_pop_state()

	  pops the top of the stack and BEGIN's to it, and

		int yy_top_state()

	  returns the top of the stack without altering the stack's
	  contents.

	  The start condition stack grows dynamically and so has no built-in
	  size limitation.  If memory is exhausted, program execution
	  is aborted.

	  To use start condition stacks, your scanner must include
	  a "%option stack" directive.

	- flex now supports POSIX character class expressions.  These
	  are expressions enclosed inside "[:" and ":]" delimiters (which
	  themselves must appear between the '[' and ']' of a character
	  class; other elements may occur inside the character class, too).
	  The expressions flex recognizes are:

		[:alnum:] [:alpha:] [:blank:] [:cntrl:] [:digit:] [:graph:]	
		[:lower:] [:print:] [:punct:] [:space:] [:upper:] [:xdigit:]

	  These expressions all designate a set of characters equivalent to
	  the corresponding isXXX function (for example, [:alnum:] designates
	  those characters for which isalnum() returns true - i.e., any
	  alphabetic or numeric).  Some systems don't provide isblank(),
	  so flex defines [:blank:] as a blank or a tab.

	  For example, the following character classes are all equivalent:

		[[:alnum:]]
		[[:alpha:][:digit:]
		[[:alpha:]0-9]
		[a-zA-Z0-9]

	  If your scanner is case-insensitive (-i flag), then [:upper:]
	  and [:lower:] are equivalent to [:alpha:].

	- The promised rewrite of the C++ FlexLexer class has not yet
	  been done.  Support for FlexLexer is limited at the moment to
	  fixing show-stopper bugs, so, for example, the new functions
	  yy_scan_string() & friends are not available to FlexLexer
	  objects.

	- The new macro

		yy_set_interactive(is_interactive)

	  can be used to control whether the current buffer is considered
	  "interactive".  An interactive buffer is processed more slowly,
	  but must be used when the scanner's input source is indeed
	  interactive to avoid problems due to waiting to fill buffers
	  (see the discussion of the -I flag in flex.1).  A non-zero value
	  in the macro invocation marks the buffer as interactive, a zero
	  value as non-interactive.  Note that use of this macro overrides
	  "%option always-interactive" or "%option never-interactive".

	  yy_set_interactive() must be invoked prior to beginning to
	  scan the buffer.

	- The new macro

		yy_set_bol(at_bol)

	  can be used to control whether the current buffer's scanning
	  context for the next token match is done as though at the
	  beginning of a line (non-zero macro argument; makes '^' anchored
	  rules active) or not at the beginning of a line (zero argument,
	  '^' rules inactive).

	- Related to this change, the mechanism for determining when a scan is
	  starting at the beginning of a line has changed.  It used to be
	  that '^' was active iff the character prior to that at which the
	  scan started was a newline.  The mechanism now is that '^' is
	  active iff the last token ended in a newline (or the last call to
	  input() returned a newline).  For most users, the difference in
	  mechanisms is negligible.  Where it will make a difference,
	  however, is if unput() or yyless() is used to alter the input
	  stream.  When in doubt, use yy_set_bol().

	- The new beginning-of-line mechanism involved changing some fairly
	  twisted code, so it may have introduced bugs - beware ...

	- The macro YY_AT_BOL() returns true if the next token scanned from
	  the current buffer will have '^' rules active, false otherwise.

	- The new function

		void yy_flush_buffer( struct yy_buffer_state* b )

	  flushes the contents of the current buffer (i.e., next time
	  the scanner attempts to match a token using b as the current
	  buffer, it will begin by invoking YY_INPUT to fill the buffer).
	  This routine is also available to C++ scanners (unlike some
	  of the other new routines).

	  The related macro

		YY_FLUSH_BUFFER

	  flushes the contents of the current buffer.

	- A new "-ooutput" option writes the generated scanner to "output".
	  If used with -t, the scanner is still written to stdout, but
	  its internal #line directives (see previous item) use "output".

	- Flex now generates #line directives relating the code it
	  produces to the output file; this means that error messages
	  in the flex-generated code should be correctly pinpointed.

	- When generating #line directives, filenames with embedded '\'s
	  have those characters escaped (i.e., turned into '\\').  This
	  feature helps with reporting filenames for some MS-DOS and OS/2
	  systems.

	- The FlexLexer class includes two new public member functions:

		virtual void switch_streams( istream* new_in = 0,
						ostream* new_out = 0 )

	  reassigns yyin to new_in (if non-nil) and yyout to new_out
	  (ditto), deleting the previous input buffer if yyin is
	  reassigned.  It is used by:

		int yylex( istream* new_in = 0, ostream* new_out = 0 )

	  which first calls switch_streams() and then returns the value
	  of calling yylex().

	- C++ scanners now have yy_flex_debug as a member variable of
	  FlexLexer rather than a global, and member functions for testing
	  and setting it.

	- When generating a C++ scanning class, you can now use

		%option yyclass="foo"

	  to inform flex that you have derived "foo" as a subclass of
	  yyFlexLexer, so flex will place your actions in the member
	  function foo::yylex() instead of yyFlexLexer::yylex().  It also
	  generates a yyFlexLexer::yylex() member function that generates a
	  run-time error if called (by invoking yyFlexLexer::LexerError()).
	  This feature is necessary if your subclass "foo" introduces some
	  additional member functions or variables that you need to access
	  from yylex().

	- Current texinfo files in MISC/texinfo, contributed by Francois
	  Pinard.

	- You can now change the name "flex" to something else (e.g., "lex")
	  by redefining $(FLEX) in the Makefile.

	- Two bugs (one serious) that could cause "bigcheck" to fail have
	  been fixed.

	- A number of portability/configuration changes have been made
	  for easier portability.

	- You can use "YYSTATE" in your scanner as an alias for YY_START
	  (for AT&T lex compatibility).

	- input() now maintains yylineno.

	- input() no longer trashes yytext.

	- interactive scanners now read characters in YY_INPUT up to a
	  newline, a large performance gain.

	- C++ scanner objects now work with the -P option.  You include
	  <FlexLexer.h> once per scanner - see comments in <FlexLexer.h>
	  (or flex.1) for details.

	- C++ FlexLexer objects now use the "cerr" stream to report -d output
	  instead of stdio.

	- The -c flag now has its full glorious POSIX interpretation (do
	  nothing), rather than being interpreted as an old-style -C flag.

	- Scanners generated by flex now include two #define's giving
	  the major and minor version numbers (YY_FLEX_MAJOR_VERSION,
	  YY_FLEX_MINOR_VERSION).  These can then be tested to see
	  whether certain flex features are available.

	- Scanners generated using -l lex compatibility now have the symbol
	  YY_FLEX_LEX_COMPAT #define'd.

	- When initializing (i.e., yy_init is non-zero on entry to yylex()),
	  generated scanners now set yy_init to zero before executing
	  YY_USER_INIT.  This means that you can set yy_init back to a
	  non-zero value in YY_USER_INIT if you need the scanner to be
	  reinitialized on the next call.

	- You can now use "#line" directives in the first section of your
	  scanner specification.

	- When generating full-table scanners (-Cf), flex now puts braces
	  around each row of the 2-d array initialization, to silence warnings
	  on over-zealous compilers.

	- Improved support for MS-DOS.  The flex sources have been successfully
	  built, unmodified, for Borland 4.02 (all that's required is a
	  Borland Makefile and config.h file, which are supplied in
	  MISC/Borland - contributed by Terrence O Kane).

	- Improved support for Macintosh using Think C - the sources should
	  build for this platform "out of the box".  Contributed by Scott
	  Hofmann.

	- Improved support for VMS, in MISC/VMS/, contributed by Pat Rankin.

	- Support for the Amiga, in MISC/Amiga/, contributed by Andreas
	  Scherer.  Note that the contributed files were developed for
	  flex 2.4 and have not been tested with flex 2.5.

	- Some notes on support for the NeXT, in MISC/NeXT, contributed
	  by Raf Schietekat.

	- The MISC/ directory now includes a preformatted version of flex.1
	  in flex.man, and pre-yacc'd versions of parse.y in parse.{c,h}.

	- The flex.1 and flexdoc.1 manual pages have been merged.  There
	  is now just one document, flex.1, which includes an overview
	  at the beginning to help you find the section you need.

	- Documentation now clarifies that start conditions persist across
	  switches to new input files or different input buffers.  If you
	  want to e.g., return to INITIAL, you must explicitly do so.

	- The "Performance Considerations" section of the manual has been
	  updated.

	- Documented the "yy_act" variable, which when YY_USER_ACTION is
	  invoked holds the number of the matched rule, and added an
	  example of using yy_act to profile how often each rule is matched.

	- Added YY_NUM_RULES, a definition that gives the total number
	  of rules in the file, including the default rule (even if you
	  use -s).

	- Documentation now clarifies that you can pass a nil FILE* pointer
	  to yy_create_buffer() or yyrestart() if you've arrange YY_INPUT
	  to not need yyin.

	- Documentation now clarifies that YY_BUFFER_STATE is a pointer to
	  an opaque "struct yy_buffer_state".

	- Documentation now stresses that you gain the benefits of removing
	  backing-up states only if you remove *all* of them.

	- Documentation now points out that traditional lex allows you
	  to put the action on a separate line from the rule pattern if
	  the pattern has trailing whitespace (ugh!), but flex doesn't
	  support this.

	- A broken example in documentation of the difference between
	  inclusive and exclusive start conditions is now fixed.

	- Usage (-h) report now goes to stdout.

	- Version (-V) info now goes to stdout.

	- More #ifdef chud has been added to the parser in attempt to
	  deal with bison's use of alloca().

	- "make clean" no longer deletes emacs backup files (*~).

	- Some memory leaks have been fixed.

	- A bug was fixed in which dynamically-expanded buffers were
	  reallocated a couple of bytes too small.

	- A bug was fixed which could cause flex to read and write beyond
	  the end of the input buffer.

	- -S will not be going away.


Changes between release 2.4.7 (03Aug94) and release 2.4.6:

	- Fixed serious bug in reading multiple files.

	- Fixed bug in scanning NUL's.

	- Fixed bug in input() returning 8-bit characters.

	- Fixed bug in matching text with embedded NUL's when
	  using %array or lex compatibility.

	- Fixed multiple invocations of YY_USER_ACTION when using '|'
	  continuation action.

	- Minor prototyping fixes.

Changes between release 2.4.6 (04Jan94) and release 2.4.5:

	- Linking with -lfl no longer required if your program includes
	  its own yywrap() and main() functions.  (This change will cause
	  problems if you have a non-ANSI compiler on a system for which
	  sizeof(int) != sizeof(void*) or sizeof(int) != sizeof(size_t).)

	- The use of 'extern "C++"' in FlexLexer.h has been modified to
	  get around an incompatibility with g++'s header files.

Changes between release 2.4.5 (11Dec93) and release 2.4.4:

	- Fixed bug breaking C++ scanners that use REJECT or variable
	  trailing context.

	- Fixed serious input problem for interactive scanners on
	  systems for which char is unsigned.

	- Fixed bug in incorrectly treating '$' operator as variable
	  trailing context.

	- Fixed bug in -CF table representation that could lead to
	  corrupt tables.

	- Fixed fairly benign memory leak.

	- Added `extern "C++"' wrapper to FlexLexer.h header.  This
	  should overcome the g++ 2.5.X problems mentioned in the
	  NEWS for release 2.4.3.

	- Changed #include of FlexLexer.h to use <> instead of "".

	- Added feature to control whether the scanner attempts to
	  refill the input buffer once it's exhausted.  This feature
	  will be documented in the 2.5 release.


Changes between release 2.4.4 (07Dec93) and release 2.4.3:

	- Fixed two serious bugs in scanning 8-bit characters.

	- Fixed bug in YY_USER_ACTION that caused it to be executed
	  inappropriately (on the scanner's own internal actions, and
	  with incorrect yytext/yyleng values).

	- Fixed bug in pointing yyin at a new file and resuming scanning.

	- Portability fix regarding min/max/abs macros conflicting with
	  function definitions in standard header files.

	- Added a virtual LexerError() method to the C++ yyFlexLexer class
	  for reporting error messages instead of always using cerr.

	- Added warning in flexdoc that the C++ scanning class is presently
	  experimental and subject to considerable change between major
	  releases.


Changes between release 2.4.3 (03Dec93) and release 2.4.2:

	- Fixed bug causing fatal scanner messages to fail to print.

	- Fixed things so FlexLexer.h can be included in other C++
	  sources.  One side-effect of this change is that -+ and -CF
	  are now incompatible.

	- libfl.a now supplies private versions of the the <string.h>/
	  <strings.h> string routines needed by flex and the scanners
	  it generates, to enhance portability to some BSD systems.

	- More robust solution to 2.4.2's flexfatal() bug fix.

	- Added ranlib of installed libfl.a.

	- Some lint tweaks.

	- NOTE: problems have been encountered attempting to build flex
	  C++ scanners using g++ version 2.5.X.  The problem is due to an
	  unfortunate heuristic in g++ 2.5.X that attempts to discern between
	  C and C++ headers.  Because FlexLexer.h is installed (by default)
	  in /usr/local/include and not /usr/local/lib/g++-include, g++ 2.5.X
	  decides that it's a C header :-(.  So if you have problems, install
	  the header in /usr/local/lib/g++-include instead.


Changes between release 2.4.2 (01Dec93) and release 2.4.1:

	- Fixed bug in libfl.a referring to non-existent "flexfatal" function.

	- Modified to produce both compress'd and gzip'd tar files for
	  distributions (you probably don't care about this change!).


Changes between release 2.4.1 (30Nov93) and release 2.3.8:

	- The new '-+' flag instructs flex to generate a C++ scanner class
	  (thanks to Kent Williams).  flex writes an implementation of the
	  class defined in FlexLexer.h to lex.yy.cc.  You may include
	  multiple scanner classes in your program using the -P flag.  Note
	  that the scanner class also provides a mechanism for creating
	  reentrant scanners.  The scanner class uses C++ streams for I/O
	  instead of FILE*'s (thanks to Tom Epperly).  If the flex executable's
	  name ends in '+' then the '-+' flag is automatically on, so creating
	  a symlink or copy of "flex" to "flex++" results in a version of
	  flex that can be used exclusively for C++ scanners.

	  Note that without the '-+' flag, flex-generated scanners can still
	  be compiled using C++ compilers, though they use FILE*'s for I/O
	  instead of streams.

	  See the "GENERATING C++ SCANNERS" section of flexdoc for details.

	- The new '-l' flag turns on maximum AT&T lex compatibility.  In
	  particular, -l includes support for "yylineno" and makes yytext
	  be an array instead of a pointer.  It does not, however, do away
	  with all incompatibilities.  See the "INCOMPATIBILITIES WITH LEX
	  AND POSIX" section of flexdoc for details.

	- The new '-P' option specifies a prefix to use other than "yy"
	  for the scanner's globally-visible variables, and for the
	  "lex.yy.c" filename.  Using -P you can link together multiple
	  flex scanners in the same executable.

	- The distribution includes a "texinfo" version of flexdoc.1,
	  contributed by Roland Pesch (thanks also to Marq Kole, who
	  contributed another version).  It has not been brought up to
	  date, but reflects version 2.3.  See MISC/flex.texinfo.

	  The flex distribution will soon include G.T. Nicol's flex
	  manual; he is presently bringing it up-to-date for version 2.4.

	- yywrap() is now a function, and you now *must* link flex scanners
	  with libfl.a.

	- Site-configuration is now done via an autoconf-generated
	  "configure" script contributed by Francois Pinard.

	- Scanners now use fread() (or getc(), if interactive) and not
	  read() for input.  A new "table compression" option, -Cr,
	  overrides this change and causes the scanner to use read()
	  (because read() is a bit faster than fread()).  -f and -F
	  are now equivalent to -Cfr and -CFr; i.e., they imply the
	  -Cr option.

	- In the blessed name of POSIX compliance, flex supports "%array"
	  and "%pointer" directives in the definitions (first) section of
	  the scanner specification.  The former specifies that yytext
	  should be an array (of size YYLMAX), the latter, that it should
	  be a pointer.  The array version of yytext is universally slower
	  than the pointer version, but has the advantage that its contents
	  remain unmodified across calls to input() and unput() (the pointer
	  version of yytext is, still, trashed by such calls).

	  "%array" cannot be used with the '-+' C++ scanner class option.

	- The new '-Ca' option directs flex to trade off memory for
	  natural alignment when generating a scanner's tables.  In
	  particular, table entries that would otherwise be "short"
	  become "long".

	- The new '-h' option produces a summary of the flex flags.

	- The new '-V' option reports the flex version number and exits.

	- The new scanner macro YY_START returns an integer value
	  corresponding to the current start condition.  You can return
	  to that start condition by passing the value to a subsequent
	  "BEGIN" action.  You also can implement "start condition stacks"
	  by storing the values in an integer stack.

	- You can now redefine macros such as YY_INPUT by just #define'ing
	  them to some other value in the first section of the flex input;
	  no need to first #undef them.

	- flex now generates warnings for rules that can't be matched.
	  These warnings can be turned off using the new '-w' flag.  If
	  your scanner uses REJECT then you will not get these warnings.

	- If you specify the '-s' flag but the default rule can be matched,
	  flex now generates a warning.

	- "yyleng" is now a global, and may be modified by the user (though
	  doing so and then using yymore() will yield weird results).

	- Name definitions in the first section of a scanner specification
	  can now include a leading '^' or trailing '$' operator.  In this
	  case, the definition is *not* pushed back inside of parentheses.

	- Scanners with compressed tables are now "interactive" (-I option)
	  by default.  You can suppress this attribute (which makes them
	  run slightly slower) using the new '-B' flag.

	- Flex now generates 8-bit scanners by default, unless you use the
	  -Cf or -CF compression options (-Cfe  and -CFe result in 8-bit
	  scanners).  You can force it to generate a 7-bit scanner using
	  the new '-7' flag.  You can build flex to generate 8-bit scanners
	  for -Cf and -CF, too, by adding -DDEFAULT_CSIZE=256 to CFLAGS
	  in the Makefile.

	- You no longer need to call the scanner routine yyrestart() to
	  inform the scanner that you have switched to a new file after
	  having seen an EOF on the current input file.  Instead, just
	  point yyin at the new file and continue scanning.

	- You no longer need to invoke YY_NEW_FILE in an <<EOF>> action
	  to indicate you wish to continue scanning.  Simply point yyin
	  at a new file.

	- A leading '#' no longer introduces a comment in a flex input.

	- flex no longer considers formfeed ('\f') a whitespace character.

	- %t, I'm happy to report, has been nuked.

	- The '-p' option may be given twice ('-pp') to instruct flex to
	  report minor performance problems as well as major ones.

	- The '-v' verbose output no longer includes start/finish time
	  information.

	- Newlines in flex inputs can optionally include leading or
	  trailing carriage-returns ('\r'), in support of several PC/Mac
	  run-time libraries that automatically include these.

	- A start condition of the form "<*>" makes the following rule
	  active in every start condition, whether exclusive or inclusive.

	- The following items have been corrected in the flex documentation:

		- '-C' table compression options *are* cumulative.

		- You may modify yytext but not lengthen it by appending
		  characters to the end.  Modifying its final character
		  will affect '^' anchoring for the next rule matched
		  if the character is changed to or from a newline.

		- The term "backtracking" has been renamed "backing up",
		  since it is a one-time repositioning and not a repeated
		  search.  What used to be the "lex.backtrack" file is now
		  "lex.backup".

		- Unindented "/* ... */" comments are allowed in the first
		  flex input section, but not in the second.

		- yyless() can only be used in the flex input source, not
		  externally.

		- You can use "yyrestart(yyin)" to throw away the
		  current contents of the input buffer.

		- To write high-speed scanners, attempt to match as much
		  text as possible with each rule.  See MISC/fastwc/README
		  for more information.

		- Using the beginning-of-line operator ('^') is fairly
		  cheap.  Using unput() is expensive.  Using yyless() is
		  cheap.

		- An example of scanning strings with embedded escape
		  sequences has been added.

		- The example of backing-up in flexdoc was erroneous; it
		  has been corrected.

	- A flex scanner's internal buffer now dynamically grows if needed
	  to match large tokens.  Note that growing the buffer presently
	  requires rescanning the (large) token, so consuming a lot of
	  text this way is a slow process.  Also note that presently the
	  buffer does *not* grow if you unput() more text than can fit
	  into the buffer.

	- The MISC/ directory has been reorganized; see MISC/README for
	  details.

	- yyless() can now be used in the third (user action) section
	  of a scanner specification, thanks to Ceriel Jacobs.  yyless()
	  remains a macro and cannot be used outside of the scanner source.

	- The skeleton file is no longer opened at run-time, but instead
	  compiled into a large string array (thanks to John Gilmore and
	  friends at Cygnus).  You can still use the -S flag to point flex
	  at a different skeleton file.

	- flex no longer uses a temporary file to store the scanner's
	  actions.

	- A number of changes have been made to decrease porting headaches.
	  In particular, flex no longer uses memset() or ctime(), and
	  provides a single simple mechanism for dealing with C compilers
	  that still define malloc() as returning char* instead of void*.

	- Flex now detects if the scanner specification requires the -8 flag
	  but the flag was not given or on by default.

	- A number of table-expansion fencepost bugs have been fixed,
	  making flex more robust for generating large scanners.

	- flex more consistently identifies the location of errors in
	  its input.

	- YY_USER_ACTION is now invoked only for "real" actions, not for
	  internal actions used by the scanner for things like filling
	  the buffer or handling EOF.

	- The rule "[^]]" now matches any character other than a ']';
	  formerly it matched any character at all followed by a ']'.
	  This change was made for compatibility with AT&T lex.

	- A large number of miscellaneous bugs have been found and fixed
	  thanks to Gerhard Wilhelms.

	- The source code has been heavily reformatted, making patches
	  relative to previous flex releases no longer accurate.


Changes between 2.3 Patch #8 (21Feb93) and 2.3 Patch #7:

	- Fixed bugs in dynamic memory allocation leading to grievous
	  fencepost problems when generating large scanners.
	- Fixed bug causing infinite loops on character classes with 8-bit
	  characters in them.
	- Fixed bug in matching repetitions with a lower bound of 0.
	- Fixed bug in scanning NUL characters using an "interactive" scanner.
	- Fixed bug in using yymore() at the end of a file.
	- Fixed bug in misrecognizing rules with variable trailing context.
	- Fixed bug compiling flex on Suns using gcc 2.
	- Fixed bug in not recognizing that input files with the character
	  ASCII 128 in them require the -8 flag.
	- Fixed bug that could cause an infinite loop writing out
	  error messages.
	- Fixed bug in not recognizing old-style lex % declarations if
	  followed by a tab instead of a space.
	- Fixed potential crash when flex terminated early (usually due
	  to a bad flag) and the -v flag had been given.
	- Added some missing declarations of void functions.
	- Changed to only use '\a' for __STDC__ compilers.
	- Updated mailing addresses.


Changes between 2.3 Patch #7 (28Mar91) and 2.3 Patch #6:

	- Fixed out-of-bounds array access that caused bad tables
	  to be produced on machines where the bad reference happened
	  to yield a 1.  This caused problems installing or running
	  flex on some Suns, in particular.


Changes between 2.3 Patch #6 (29Aug90) and 2.3 Patch #5:

	- Fixed a serious bug in yymore() which basically made it
	  completely broken.  Thanks goes to Jean Christophe of
	  the Nethack development team for finding the problem
	  and passing along the fix.


Changes between 2.3 Patch #5 (16Aug90) and 2.3 Patch #4:

	- An up-to-date version of initscan.c so "make test" will
	  work after applying the previous patches


Changes between 2.3 Patch #4 (14Aug90) and 2.3 Patch #3:

	- Fixed bug in hexadecimal escapes which allowed only digits,
	  not letters, in escapes
	- Fixed bug in previous "Changes" file!


Changes between 2.3 Patch #3 (03Aug90) and 2.3 Patch #2:

	- Correction to patch #2 for gcc compilation; thanks goes to
	  Paul Eggert for catching this.


Changes between 2.3 Patch #2 (02Aug90) and original 2.3 release:

	- Fixed (hopefully) headaches involving declaring malloc()
	  and free() for gcc, which defines __STDC__ but (often) doesn't
	  come with the standard include files such as <stdlib.h>.
	  Reordered #ifdef maze in the scanner skeleton in the hope of
	  getting the declarations right for cfront and g++, too.

	- Note that this patch supercedes patch #1 for release 2.3,
	  which was never announced but was available briefly for
	  anonymous ftp.


Changes between 2.3 (full) release of 28Jun90 and 2.2 (alpha) release:

    User-visible:

	- A lone <<EOF>> rule (that is, one which is not qualified with
	  a list of start conditions) now specifies the EOF action for
	  *all* start conditions which haven't already had <<EOF>> actions
	  given.  To specify an end-of-file action for just the initial
	  state, use <INITIAL><<EOF>>.

	- -d debug output is now contigent on the global yy_flex_debug
	  being set to a non-zero value, which it is by default.

	- A new macro, YY_USER_INIT, is provided for the user to specify
	  initialization action to be taken on the first call to the
	  scanner.  This action is done before the scanner does its
	  own initialization.

	- yy_new_buffer() has been added as an alias for yy_create_buffer()

	- Comments beginning with '#' and extending to the end of the line
	  now work, but have been deprecated (in anticipation of making
	  flex recognize #line directives).

	- The funky restrictions on when semi-colons could follow the
	  YY_NEW_FILE and yyless macros have been removed.  They now
	  behave identically to functions.

	- A bug in the sample redefinition of YY_INPUT in the documentation
	  has been corrected.

	- A bug in the sample simple tokener in the documentation has
	  been corrected.

	- The documentation on the incompatibilities between flex and
	  lex has been reordered so that the discussion of yylineno
	  and input() come first, as it's anticipated that these will
	  be the most common source of headaches.


    Things which didn't used to be documented but now are:

	- flex interprets "^foo|bar" differently from lex.  flex interprets
	  it as "match either a 'foo' or a 'bar', providing it comes at the
	  beginning of a line", whereas lex interprets it as "match either
	  a 'foo' at the beginning of a line, or a 'bar' anywhere".

	- flex initializes the global "yyin" on the first call to the
	  scanner, while lex initializes it at compile-time.

	- yy_switch_to_buffer() can be used in the yywrap() macro/routine.

	- flex scanners do not use stdio for their input, and hence when
	  writing an interactive scanner one must explictly call fflush()
	  after writing out a prompt.

	- flex scanner can be made reentrant (after a fashion) by using
	  "yyrestart( yyin );".  This is useful for interactive scanners
	  which have interrupt handlers that long-jump out of the scanner.

	- a defense of why yylineno is not supported is included, along
	  with a suggestion on how to convert scanners which rely on it.


    Other changes:

	- Prototypes and proper declarations of void routines have
	  been added to the flex source code, courtesy of Kevin B. Kenny.

	- Routines dealing with memory allocation now use void* pointers
	  instead of char* - see Makefile for porting implications.

	- Error-checking is now done when flex closes a file.

	- Various lint tweaks were added to reduce the number of gripes.

	- Makefile has been further parameterized to aid in porting.

	- Support for SCO Unix added.

	- Flex now sports the latest & greatest UC copyright notice
	  (which is only slightly different from the previous one).

	- A note has been added to flexdoc.1 mentioning work in progress
	  on modifying flex to generate straight C code rather than a
	  table-driven automaton, with an email address of whom to contact
	  if you are working along similar lines.


Changes between 2.2 Patch #3 (30Mar90) and 2.2 Patch #2:

	- fixed bug which caused -I scanners to bomb


Changes between 2.2 Patch #2 (27Mar90) and 2.2 Patch #1:

	- fixed bug writing past end of input buffer in yyunput()
	- fixed bug detecting NUL's at the end of a buffer


Changes between 2.2 Patch #1 (23Mar90) and 2.2 (alpha) release:

	- Makefile fixes: definition of MAKE variable for systems
	  which don't have it; installation of flexdoc.1 along with
	  flex.1; fixed two bugs which could cause "bigtest" to fail.

	- flex.skel fix for compiling with g++.

	- README and flexdoc.1 no longer list an out-of-date BITNET address
	  for contacting me.

	- minor typos and formatting changes to flex.1 and flexdoc.1.


Changes between 2.2 (alpha) release of March '90 and previous release:

    User-visible:

	- Full user documentation now available.

	- Support for 8-bit scanners.

	- Scanners now accept NUL's.

	- A facility has been added for dealing with multiple
	  input buffers.

	- Two manual entries now.  One which fully describes flex
	  (rather than just its differences from lex), and the
	  other for quick(er) reference.

	- A number of changes to bring flex closer into compliance
	  with the latest POSIX lex draft:

		%t support
		flex now accepts multiple input files and concatenates
		    them together to form its input
		previous -c (compress) flag renamed -C
		do-nothing -c and -n flags added
		Any indented code or code within %{}'s in section 2 is
		    now copied to the output

	- yyleng is now a bona fide global integer.

	- -d debug information now gives the line number of the
	  matched rule instead of which number rule it was from
	  the beginning of the file.

	- -v output now includes a summary of the flags used to generate
	  the scanner.

	- unput() and yyrestart() are now globally callable.

	- yyrestart() no longer closes the previous value of yyin.

	- C++ support; generated scanners can be compiled with C++ compiler.

	- Primitive -lfl library added, containing default main()
	  which calls yylex().  A number of routines currently living
	  in the scanner skeleton will probably migrate to here
	  in the future (in particular, yywrap() will probably cease
	  to be a macro and instead be a function in the -lfl library).

	- Hexadecimal (\x) escape sequences added.

	- Support for MS-DOS, VMS, and Turbo-C integrated.

	- The %used/%unused operators have been deprecated.  They
	  may go away soon.


    Other changes:

	- Makefile enhanced for easier testing and installation.
	- The parser has been tweaked to detect some erroneous
	  constructions which previously were missed.
	- Scanner input buffer overflow is now detected.
	- Bugs with missing "const" declarations fixed.
	- Out-of-date Minix/Atari patches provided.
	- Scanners no longer require printf() unless FLEX_DEBUG is being used.
	- A subtle input() bug has been fixed.
	- Line numbers for "continued action" rules (those following
	  the special '|' action) are now correct.
	- unput() bug fixed; had been causing problems porting flex to VMS.
	- yymore() handling rewritten to fix bug with interaction
	  between yymore() and trailing context.
	- EOF in actions now generates an error message.
	- Bug involving -CFe and generating equivalence classes fixed.
	- Bug which made -CF be treated as -Cf fixed.
	- Support for SysV tmpnam() added.
	- Unused #define's for scanner no longer generated.
	- Error messages which are associated with a particular input
	  line are now all identified with their input line in standard
	  format.
	- % directives which are valid to lex but not to flex are
	  now ignored instead of generating warnings.
	- -DSYS_V flag can now also be specified -DUSG for System V
	  compilation.


Changes between 2.1 beta-test release of June '89 and previous release:

    User-visible:

	- -p flag generates a performance report to stderr.  The report
	  consists of comments regarding features of the scanner rules
	  which result in slower scanners.

	- -b flag generates backtracking information to lex.backtrack.
	  This is a list of scanner states which require backtracking
	  and the characters on which they do so.  By adding rules
	  one can remove backtracking states.  If all backtracking states
	  are eliminated, the generated scanner will run faster.
	  Backtracking is not yet documented in the manual entry.

	- Variable trailing context now works, i.e., one can have
	  rules like "(foo)*/[ \t]*bletch".  Some trailing context
	  patterns still cannot be properly matched and generate
	  error messages.  These are patterns where the ending of the
	  first part of the rule matches the beginning of the second
	  part, such as "zx*/xy*", where the 'x*' matches the 'x' at
	  the beginning of the trailing context.  Lex won't get these
	  patterns right either.

	- Faster scanners.

	- End-of-file rules.  The special rule "<<EOF>>" indicates
	  actions which are to be taken when an end-of-file is
	  encountered and yywrap() returns non-zero (i.e., indicates
	  no further files to process).  See manual entry for example.

	- The -r (reject used) flag is gone.  flex now scans the input
	  for occurrences of the string "REJECT" to determine if the
	  action is needed.  It tries to be intelligent about this but
	  can be fooled.  One can force the presence or absence of
	  REJECT by adding a line in the first section of the form
	  "%used REJECT" or "%unused REJECT".

	- yymore() has been implemented.  Similarly to REJECT, flex
	  detects the use of yymore(), which can be overridden using
	  "%used" or "%unused".

	- Patterns like "x{0,3}" now work (i.e., with lower-limit == 0).

	- Removed '\^x' for ctrl-x misfeature.

	- Added '\a' and '\v' escape sequences.

	- \<digits> now works for octal escape sequences; previously
	  \0<digits> was required.

	- Better error reporting; line numbers are associated with rules.

	- yyleng is a macro; it cannot be accessed outside of the
	  scanner source file.

	- yytext and yyleng should not be modified within a flex action.

	- Generated scanners #define the name FLEX_SCANNER.

	- Rules are internally separated by YY_BREAK in lex.yy.c rather
	  than break, to allow redefinition.

	- The macro YY_USER_ACTION can be redefined to provide an action
	  which is always executed prior to the matched rule's action.
	
	- yyrestart() is a new action which can be used to restart
	  the scanner after it has seen an end-of-file (a "real" one,
	  that is, one for which yywrap() returned non-zero).  It takes
	  a FILE* argument indicating a new file to scan and sets
	  things up so that a subsequent call to yylex() will start
	  scanning that file.

	- Internal scanner names all preceded by "yy_"

	- lex.yy.c is deleted if errors are encountered during processing.

	- Comments may be put in the first section of the input by preceding
	  them with '#'.



    Other changes:

	- Some portability-related bugs fixed, in particular for machines
	  with unsigned characters or sizeof( int* ) != sizeof( int ).
	  Also, tweaks for VMS and Microsoft C (MS-DOS), and identifiers all
	  trimmed to be 31 or fewer characters.  Shortened file names
	  for dinosaur OS's.  Checks for allocating > 64K memory
	  on 16 bit'ers.  Amiga tweaks.  Compiles using gcc on a Sun-3.
	- Compressed and fast scanner skeletons merged.
	- Skeleton header files done away with.
	- Generated scanner uses prototypes and "const" for __STDC__.
	- -DSV flag is now -DSYS_V for System V compilation.
	- Removed all references to FTL language.
	- Software now covered by BSD Copyright.
	- flex will replace lex in subsequent BSD releases.