2e406c584f
This is a rollup of a lot of changes. In summary, lots of bug fixes, Unicode support and CSV support to match the 2nd Edition of the Awk Book. In detail, from AWK's FIXES and FIXES.1e: From FIXES: Oct 30, 2023: multiple fixes and a minor code cleanup. disabled utf-8 for non-multibyte locales, such as C or POSIX. fixed a bad char * cast that causes incorrect results on big-endian systems. also fixed an out-of-bounds read for empty CCL. fixed a buffer overflow in substr with utf-8 strings. many thanks to Todd C Miller. Sep 24, 2023: fnematch and getrune have been overhauled to solve issues around unicode FS and RS. also fixed gsub null match issue with unicode. big thanks to Arnold Robbins. Sep 12, 2023: Fixed a length error in u8_byte2char that set RSTART to incorrect (cannot happen) value for EOL match(str, /$/). ----------------------------------------------------------------- [This entry is a summary, not a precise list of changes.] Added --csv option to enable processing of comma-separated values inputs. When --csv is enabled, fields are separated by commas, fields may be quoted with " double quotes, fields may contain embedded newlines. If no explicit separator argument is provided, split() uses the setting of --csv to determine how fields are split. Strings may now contain UTF-8 code points (not necessarily characters). Functions that operate on characters, like length, substr, index, match, etc., use UTF-8, so the length of a string of 3 emojis is 3, not 12 as it would be if bytes were counted. Regular expressions are processes as UTF-8. Unicode literals can be written as \u followed by one to eight hexadecimal digits. These may appear in strings and regular expressions. From FIXES.1e: Sep 06, 2023: Fix edge case where FS is changed on commandline. Thanks to Gordon Shephard and Miguel Pineiro Jr. Fix regular expression clobbering in the lexer, where lexer does not make a copy of regexp literals. also makedfa memory leaks have been plugged. Thanks to Miguel Pineiro Jr. Dec 15, 2022: Force hex escapes in strings to be no more than two characters, as they already are in regular expressions. This brings internal consistency, as well as consistency with gawk. Thanks to Arnold Robbins. Sep 12, 2022: adjbuf minlen error (cannot be 0) in cat, resulting in NULL pbuf. discovered by todd miller. also use-after-free issue with tempfree in cat, thanks to Miguel Pineiro Jr and valgrind. Aug 30, 2022: Various leaks and use-after-free issues plugged/fixed. Thanks to Miguel Pineiro Jr. <mpj@pineiro.cc>. May 23, 2022: Memory leak when assigning a string to some of the built-in variables. allocated string erroneously marked DONTFREE. Thanks to Miguel Pineiro Jr. <mpj@pineiro.cc>. Mar 14, 2022: Historic bug: command-line "name=value" assignment had been truncating its entry in ARGV. (circa 1989) Thanks to Miguel Pineiro Jr. <mpj@pineiro.cc>. Mar 3, 2022: Fixed file management memory leak that appears to have been there since the files array was first initialized with stdin, stdout, and stderr (circa 1992). Thanks to Miguel Pineiro Jr. <mpj@pineiro.cc>. December 8, 2021: The error handling in closefile and closeall was mangled. Long standing warnings had been made fatal and some fatal errors went undetected. Thanks to Miguel Pineiro Jr. <mpj@pineiro.cc>. Nov 03, 2021: getline accesses uninitialized data after getrec() returns 0 on EOF and leaves the contents of buf unchanged. Thanks to Volodymyr Gubarkov, and Todd C Miller. Oct 12, 2021: The fix for #83 changed the code to insert 2 chars, but the call to adjbuf just above it only allows for 1 char. This can cause a heap buffer overflow. Sponsored by: Netflix |
||
---|---|---|
bugs-fixed | ||
testdir | ||
awk.1 | ||
awk.h | ||
awkgram.y | ||
b.c | ||
ChangeLog | ||
FIXES | ||
FIXES.1e | ||
lex.c | ||
lib.c | ||
LICENSE | ||
main.c | ||
makefile | ||
maketab.c | ||
parse.c | ||
proto.h | ||
README.md | ||
REGRESS | ||
run.c | ||
TODO | ||
tran.c |
The One True Awk
This is the version of awk
described in The AWK Programming Language,
Second Edition, by Al Aho, Brian Kernighan, and Peter Weinberger
(Addison-Wesley, 2024, ISBN-13 978-0138269722, ISBN-10 0138269726).
What's New?
This version of Awk handles UTF-8 and comma-separated values (CSV) input.
Strings
Functions that process strings now count Unicode code points, not bytes;
this affects length
, substr
, index
, match
, split
,
sub
, gsub
, and others. Note that code
points are not necessarily characters.
UTF-8 sequences may appear in literal strings and regular expressions.
Aribtrary characters may be included with \u
followed by 1 to 8 hexadecimal digits.
Regular expressions
Regular expressions may include UTF-8 code points, including \u
.
Character classes are likely to be limited to about 256 characters
when expanded.
CSV
The option --csv
turns on CSV processing of input:
fields are separated by commas, fields may be quoted with
double-quote ("
) characters, quoted fields may contain embedded newlines.
In CSV mode, FS
is ignored.
If no explicit separator argument is provided,
field-splitting in split
is determined by CSV mode.
Copyright
Copyright (C) Lucent Technologies 1997
All Rights Reserved
Permission to use, copy, modify, and distribute this software and its documentation for any purpose and without fee is hereby granted, provided that the above copyright notice appear in all copies and that both that the copyright notice and this permission notice and warranty disclaimer appear in supporting documentation, and that the name Lucent Technologies or any of its entities not be used in advertising or publicity pertaining to distribution of the software without specific, written prior permission.
LUCENT DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL LUCENT OR ANY OF ITS ENTITIES BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
Distribution and Reporting Problems
Changes, mostly bug fixes and occasional enhancements, are listed
in FIXES
. If you distribute this code further, please please please
distribute FIXES
with it.
If you find errors, please report them to the current maintainer, ozan.yigit@gmail.com. Please also open an issue in the GitHub issue tracker, to make it easy to track issues. Thanks.
Submitting Pull Requests
Pull requests are welcome. Some guidelines:
-
Please do not use functions or facilities that are not standard (e.g.,
strlcpy()
,fpurge()
). -
Please run the test suite and make sure that your changes pass before posting the pull request. To do so:
- Save the previous version of
awk
somewhere in your path. Call itnawk
(for example). - Run
oldawk=nawk make check > check.out 2>&1
. - Search for
BAD
orerror
in the result. In general, look over it manually to make sure there are no errors.
- Save the previous version of
-
Please create the pull request with a request to merge into the
staging
branch instead of into themaster
branch. This allows us to do testing, and to make any additional edits or changes after the merge but before merging tomaster
.
Building
The program itself is created by
make
which should produce a sequence of messages roughly like this:
bison -d awkgram.y
awkgram.y: warning: 44 shift/reduce conflicts [-Wconflicts-sr]
awkgram.y: warning: 85 reduce/reduce conflicts [-Wconflicts-rr]
awkgram.y: note: rerun with option '-Wcounterexamples' to generate conflict counterexamples
gcc -g -Wall -pedantic -Wcast-qual -O2 -c -o awkgram.tab.o awkgram.tab.c
gcc -g -Wall -pedantic -Wcast-qual -O2 -c -o b.o b.c
gcc -g -Wall -pedantic -Wcast-qual -O2 -c -o main.o main.c
gcc -g -Wall -pedantic -Wcast-qual -O2 -c -o parse.o parse.c
gcc -g -Wall -pedantic -Wcast-qual -O2 maketab.c -o maketab
./maketab awkgram.tab.h >proctab.c
gcc -g -Wall -pedantic -Wcast-qual -O2 -c -o proctab.o proctab.c
gcc -g -Wall -pedantic -Wcast-qual -O2 -c -o tran.o tran.c
gcc -g -Wall -pedantic -Wcast-qual -O2 -c -o lib.o lib.c
gcc -g -Wall -pedantic -Wcast-qual -O2 -c -o run.o run.c
gcc -g -Wall -pedantic -Wcast-qual -O2 -c -o lex.o lex.c
gcc -g -Wall -pedantic -Wcast-qual -O2 awkgram.tab.o b.o main.o parse.o proctab.o tran.o lib.o run.o lex.o -lm
This produces an executable a.out
; you will eventually want to
move this to some place like /usr/bin/awk
.
If your system does not have yacc
or bison
(the GNU
equivalent), you need to install one of them first.
NOTE: This version uses ISO/IEC C99, as you should also. We have
compiled this without any changes using gcc -Wall
and/or local C
compilers on a variety of systems, but new systems or compilers
may raise some new complaint; reports of difficulties are
welcome.
This compiles without change on Macintosh OS X using gcc
and
the standard developer tools.
You can also use make CC=g++
to build with the GNU C++ compiler,
should you choose to do so.
A Note About Releases
We don't usually do releases.
A Note About Maintenance
NOTICE! Maintenance of this program is on a ''best effort'' basis. We try to get to issues and pull requests as quickly as we can. Unfortunately, however, keeping this program going is not at the top of our priority list.
Last Updated
Sun 15 Oct 2023 06:28:36 IDT