freebsd-src/bugs-fixed
Warner Losh 2e406c584f ota: Import One True Awk from 20231102 (254b979f32df)
This is a rollup of a lot of changes. In summary, lots of bug fixes,
Unicode support and CSV support to match the 2nd Edition of the Awk
Book.

In detail, from AWK's FIXES and FIXES.1e:

From FIXES:
Oct 30, 2023:
	multiple fixes and a minor code cleanup.
	disabled utf-8 for non-multibyte locales, such as C or POSIX.
	fixed a bad char * cast that causes incorrect results on big-endian
	systems. also fixed an out-of-bounds read for empty CCL.
	fixed a buffer overflow in substr with utf-8 strings.
	many thanks to Todd C Miller.

Sep 24, 2023:
	fnematch and getrune have been overhauled to solve issues around
	unicode FS and RS. also fixed gsub null match issue with unicode.
	big thanks to Arnold Robbins.

Sep 12, 2023:
	Fixed a length error in u8_byte2char that set RSTART to
	incorrect (cannot happen) value for EOL match(str, /$/).

-----------------------------------------------------------------

[This entry is a summary, not a precise list of changes.]

	Added --csv option to enable processing of comma-separated
	values inputs.  When --csv is enabled, fields are separated
	by commas, fields may be quoted with " double quotes, fields
	may contain embedded newlines.

	If no explicit separator argument is provided, split() uses
	the setting of --csv to determine how fields are split.

	Strings may now contain UTF-8 code points (not necessarily
	characters).  Functions that operate on characters, like
	length, substr, index, match, etc., use UTF-8, so the length
	of a string of 3 emojis is 3, not 12 as it would be if bytes
	were counted.

	Regular expressions are processes as UTF-8.

	Unicode literals can be written as \u followed by one
	to eight hexadecimal digits.  These may appear in strings and
	regular expressions.

From FIXES.1e:

Sep 06, 2023:
	Fix edge case where FS is changed on commandline. Thanks to
	Gordon Shephard and Miguel Pineiro Jr.

	Fix regular expression clobbering in the lexer, where lexer does
	not make a copy of regexp literals. also makedfa memory leaks have
	been plugged. Thanks to Miguel Pineiro Jr.

Dec 15, 2022:
	Force hex escapes in strings to be no more than two characters,
	as they already are in regular expressions. This brings internal
	consistency, as well as consistency with gawk. Thanks to
	Arnold Robbins.

Sep 12, 2022:
	adjbuf minlen error (cannot be 0) in cat, resulting in NULL pbuf.
	discovered by todd miller. also use-after-free issue with
	tempfree in cat, thanks to Miguel Pineiro Jr and valgrind.

Aug 30, 2022:
	Various leaks and use-after-free issues plugged/fixed.
	Thanks to Miguel Pineiro Jr. <mpj@pineiro.cc>.

May 23, 2022:
	Memory leak when assigning a string to some of the built-in
	variables. allocated string erroneously marked DONTFREE.
	Thanks to Miguel Pineiro Jr. <mpj@pineiro.cc>.

Mar 14, 2022:
	Historic bug: command-line "name=value" assignment had been
	truncating its entry in ARGV. (circa 1989) Thanks to
	Miguel Pineiro Jr. <mpj@pineiro.cc>.

Mar 3, 2022:
	Fixed file management memory leak that appears to have been
	there since the files array was first initialized with stdin,
	stdout, and stderr (circa 1992). Thanks to Miguel Pineiro Jr.
	<mpj@pineiro.cc>.

December 8, 2021:
	The error handling in closefile and closeall was mangled. Long
	standing warnings had been made fatal and some fatal errors went
	undetected. Thanks to Miguel Pineiro Jr. <mpj@pineiro.cc>.

Nov 03, 2021:
        getline accesses uninitialized data after getrec()
	returns 0 on EOF and leaves the contents of buf unchanged.
	Thanks to Volodymyr Gubarkov, and Todd C Miller.

Oct 12, 2021:
	The fix for #83 changed the code to insert 2 chars, but the
	call to adjbuf just above it only allows for 1 char. This can
	cause a heap buffer overflow.

Sponsored by:		Netflix
2023-11-02 10:20:09 -06:00
..
a-format.awk Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
a-format.bad Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
a-format.ok Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
concat-assign-same.awk Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
concat-assign-same.bad Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
concat-assign-same.ok Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
decr-NF.awk Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
decr-NF.bad Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
decr-NF.ok Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
fmt-overflow.awk Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
fmt-overflow.ok Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
fs-overflow.awk Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
fs-overflow.ok awk: bring in vendor branch from upstream 20210727 2021-08-01 10:04:30 -06:00
getline-corruption.awk ota: Import One True Awk from 20231102 (254b979f32df) 2023-11-02 10:20:09 -06:00
getline-corruption.in ota: Import One True Awk from 20231102 (254b979f32df) 2023-11-02 10:20:09 -06:00
getline-corruption.ok ota: Import One True Awk from 20231102 (254b979f32df) 2023-11-02 10:20:09 -06:00
getline-numeric.awk Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
getline-numeric.bad Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
getline-numeric.in Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
getline-numeric.ok Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
inf-nan-torture.awk awk: bring in vendor branch from upstream 20210727 2021-08-01 10:04:30 -06:00
inf-nan-torture.in awk: bring in vendor branch from upstream 20210727 2021-08-01 10:04:30 -06:00
inf-nan-torture.ok awk: bring in vendor branch from upstream 20210727 2021-08-01 10:04:30 -06:00
missing-precision.awk Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
missing-precision.ok one-true-awk: import 20210221 (1e4bc42c53a1) which fixes a number of bugs 2021-07-07 17:09:41 -06:00
negative-nf.awk Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
negative-nf.ok one-true-awk: import 20210221 (1e4bc42c53a1) which fixes a number of bugs 2021-07-07 17:09:41 -06:00
nf-self-assign.awk Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
nf-self-assign.bad Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
nf-self-assign.ok Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
numeric-fs.awk Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
numeric-fs.ok Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
numeric-output-seps.awk Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
numeric-output-seps.bad Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
numeric-output-seps.ok Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
numeric-rs.awk Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
numeric-rs.bad Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
numeric-rs.ok Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
numeric-subsep.awk Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
numeric-subsep.bad Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
numeric-subsep.ok Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
ofs-rebuild.awk Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
ofs-rebuild.bad Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
ofs-rebuild.ok Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
pfile-overflow.awk awk: bring in vendor branch from upstream 20210727 2021-08-01 10:04:30 -06:00
pfile-overflow.ok awk: bring in vendor branch from upstream 20210727 2021-08-01 10:04:30 -06:00
README Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
REGRESS awk: bring in vendor branch from upstream 20210727 2021-08-01 10:04:30 -06:00
rs_underflow.awk awk: bring in vendor branch from upstream 20210727 2021-08-01 10:04:30 -06:00
rs_underflow.in awk: bring in vendor branch from upstream 20210727 2021-08-01 10:04:30 -06:00
rs_underflow.ok awk: bring in vendor branch from upstream 20210727 2021-08-01 10:04:30 -06:00
rstart-rlength.awk ota: Import One True Awk from 20231102 (254b979f32df) 2023-11-02 10:20:09 -06:00
rstart-rlength.ok ota: Import One True Awk from 20231102 (254b979f32df) 2023-11-02 10:20:09 -06:00
space.awk Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
space.bad Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
space.ok Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
split-fs-from-array.awk Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
split-fs-from-array.ok Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
string-conv.awk Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
string-conv.bad Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
string-conv.ok Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
subsep-overflow.awk Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
subsep-overflow.ok Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
system-status.awk Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
system-status.bad Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
system-status.ok Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
unary-plus.awk Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
unary-plus.bad Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
unary-plus.ok Import latest one-true-awk from upstream 2019-06-02 04:23:56 +00:00
unicode-fs-rs-1.awk ota: Import One True Awk from 20231102 (254b979f32df) 2023-11-02 10:20:09 -06:00
unicode-fs-rs-1.in ota: Import One True Awk from 20231102 (254b979f32df) 2023-11-02 10:20:09 -06:00
unicode-fs-rs-1.ok ota: Import One True Awk from 20231102 (254b979f32df) 2023-11-02 10:20:09 -06:00
unicode-fs-rs-2.awk ota: Import One True Awk from 20231102 (254b979f32df) 2023-11-02 10:20:09 -06:00
unicode-fs-rs-2.in ota: Import One True Awk from 20231102 (254b979f32df) 2023-11-02 10:20:09 -06:00
unicode-fs-rs-2.ok ota: Import One True Awk from 20231102 (254b979f32df) 2023-11-02 10:20:09 -06:00
unicode-null-match.awk ota: Import One True Awk from 20231102 (254b979f32df) 2023-11-02 10:20:09 -06:00
unicode-null-match.bad ota: Import One True Awk from 20231102 (254b979f32df) 2023-11-02 10:20:09 -06:00
unicode-null-match.ok ota: Import One True Awk from 20231102 (254b979f32df) 2023-11-02 10:20:09 -06:00

List of bugs fixed.

1. ofs-rebuild: OFS value used to rebuild the record was incorrect.
Fixed August 19, 2014. Revised fix August 2018.

2. system-status: Instead of a floating-point division by 256, use
the wait(2) macros to create a reasonable exit status. Fixed March 12, 2016.

3. space: Use provided xisblank() function instead of ispace() for
matching [[:blank:]].

4. a-format: Add POSIX standard %a and %A to supported formats. Check
at runtime that this format is available.

5. decr-NF: Decrementing NF did not change $0. This is a decades-old
bug. There are interactions with the old and new value of OFS as well.
Most of the fix came from the NetBSD awk.

6. string-conv: String conversions of scalars were sticky.  Once a
conversion to string happened, even with OFMT, that value was used until
a new numeric value was assigned, even if OFMT differed from CONVFMT,
and also if CONVFMT changed.

7. unary-plus: Unary plus on a string constant returned the string.
Instead, it should convert the value to numeric and give that value.

8. concat-assign-same: Concatenation previously evaluated both sides of the
expression before doing its work, which, since assign() evaluates to the cell
being assigned to, meant that expressions like "print (a = 1) (a = 2)" would
print "22" rather than "12".

9. missing-precision: When using the format string "%*s", the precision
argument was used without checking if it was present first.

10. missing-precision: When using the format string "%*s", the precision
argument was used without checking if it was present first.

11. fmt-overflow: The buffer used for OFMT/CONVFMT conversions was written
to with sprintf(), which meant that some conversions could write past the
end.

12. numeric-subsep, numeric-fs, numeric-output-seps, numerics-rs: If SUBSEP,
FS, RS, OFS, or ORS were set to a numeric value, then their string values
wouldn't always be generated before being needed.

13. subsep-overflow: The length of SUBSEP needs to be rechecked after
calling execute(), in case SUBSEP itself has been changed.

14. split-fs-from-array: If the third argument to split() comes from the
array passed as the second argument, then split() would previously read
from the freed memory and possibly produce incorrect results (depending
on the system's malloc()/free() behaviour.)

15. getline-numeric: The `getline xx < file' syntax did not check if
values were numeric, in discordance from POSIX. Test case adapted from
one posted by Ben Bacarisse <ben.usenet@bsb.me.uk> in comp.lang.awk,
January 2019.