mirror of
https://github.com/freebsd/freebsd-src
synced 2024-07-21 18:27:22 +00:00
2e406c584f
This is a rollup of a lot of changes. In summary, lots of bug fixes,
Unicode support and CSV support to match the 2nd Edition of the Awk
Book.
In detail, from AWK's FIXES and FIXES.1e:
From FIXES:
Oct 30, 2023:
multiple fixes and a minor code cleanup.
disabled utf-8 for non-multibyte locales, such as C or POSIX.
fixed a bad char * cast that causes incorrect results on big-endian
systems. also fixed an out-of-bounds read for empty CCL.
fixed a buffer overflow in substr with utf-8 strings.
many thanks to Todd C Miller.
Sep 24, 2023:
fnematch and getrune have been overhauled to solve issues around
unicode FS and RS. also fixed gsub null match issue with unicode.
big thanks to Arnold Robbins.
Sep 12, 2023:
Fixed a length error in u8_byte2char that set RSTART to
incorrect (cannot happen) value for EOL match(str, /$/).
-----------------------------------------------------------------
[This entry is a summary, not a precise list of changes.]
Added --csv option to enable processing of comma-separated
values inputs. When --csv is enabled, fields are separated
by commas, fields may be quoted with " double quotes, fields
may contain embedded newlines.
If no explicit separator argument is provided, split() uses
the setting of --csv to determine how fields are split.
Strings may now contain UTF-8 code points (not necessarily
characters). Functions that operate on characters, like
length, substr, index, match, etc., use UTF-8, so the length
of a string of 3 emojis is 3, not 12 as it would be if bytes
were counted.
Regular expressions are processes as UTF-8.
Unicode literals can be written as \u followed by one
to eight hexadecimal digits. These may appear in strings and
regular expressions.
From FIXES.1e:
Sep 06, 2023:
Fix edge case where FS is changed on commandline. Thanks to
Gordon Shephard and Miguel Pineiro Jr.
Fix regular expression clobbering in the lexer, where lexer does
not make a copy of regexp literals. also makedfa memory leaks have
been plugged. Thanks to Miguel Pineiro Jr.
Dec 15, 2022:
Force hex escapes in strings to be no more than two characters,
as they already are in regular expressions. This brings internal
consistency, as well as consistency with gawk. Thanks to
Arnold Robbins.
Sep 12, 2022:
adjbuf minlen error (cannot be 0) in cat, resulting in NULL pbuf.
discovered by todd miller. also use-after-free issue with
tempfree in cat, thanks to Miguel Pineiro Jr and valgrind.
Aug 30, 2022:
Various leaks and use-after-free issues plugged/fixed.
Thanks to Miguel Pineiro Jr. <mpj@pineiro.cc>.
May 23, 2022:
Memory leak when assigning a string to some of the built-in
variables. allocated string erroneously marked DONTFREE.
Thanks to Miguel Pineiro Jr. <mpj@pineiro.cc>.
Mar 14, 2022:
Historic bug: command-line "name=value" assignment had been
truncating its entry in ARGV. (circa 1989) Thanks to
Miguel Pineiro Jr. <mpj@pineiro.cc>.
Mar 3, 2022:
Fixed file management memory leak that appears to have been
there since the files array was first initialized with stdin,
stdout, and stderr (circa 1992). Thanks to Miguel Pineiro Jr.
<mpj@pineiro.cc>.
December 8, 2021:
The error handling in closefile and closeall was mangled. Long
standing warnings had been made fatal and some fatal errors went
undetected. Thanks to Miguel Pineiro Jr. <mpj@pineiro.cc>.
Nov 03, 2021:
getline accesses uninitialized data after getrec()
returns 0 on EOF and leaves the contents of buf unchanged.
Thanks to Volodymyr Gubarkov, and Todd C Miller.
Oct 12, 2021:
The fix for #83 changed the code to insert 2 chars, but the
call to adjbuf just above it only allows for 1 char. This can
cause a heap buffer overflow.
Sponsored by: Netflix
209 lines
7.4 KiB
C
209 lines
7.4 KiB
C
/****************************************************************
|
|
Copyright (C) Lucent Technologies 1997
|
|
All Rights Reserved
|
|
|
|
Permission to use, copy, modify, and distribute this software and
|
|
its documentation for any purpose and without fee is hereby
|
|
granted, provided that the above copyright notice appear in all
|
|
copies and that both that the copyright notice and this
|
|
permission notice and warranty disclaimer appear in supporting
|
|
documentation, and that the name Lucent Technologies or any of
|
|
its entities not be used in advertising or publicity pertaining
|
|
to distribution of the software without specific, written prior
|
|
permission.
|
|
|
|
LUCENT DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE,
|
|
INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS.
|
|
IN NO EVENT SHALL LUCENT OR ANY OF ITS ENTITIES BE LIABLE FOR ANY
|
|
SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
|
|
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER
|
|
IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION,
|
|
ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF
|
|
THIS SOFTWARE.
|
|
****************************************************************/
|
|
|
|
extern int yywrap(void);
|
|
extern void setfname(Cell *);
|
|
extern int constnode(Node *);
|
|
extern char *strnode(Node *);
|
|
extern Node *notnull(Node *);
|
|
extern int yyparse(void);
|
|
|
|
extern int yylex(void);
|
|
extern void startreg(void);
|
|
extern int input(void);
|
|
extern void unput(int);
|
|
extern void unputstr(const char *);
|
|
extern int yylook(void);
|
|
extern int yyback(int *, int);
|
|
extern int yyinput(void);
|
|
|
|
extern fa *makedfa(const char *, bool);
|
|
extern fa *mkdfa(const char *, bool);
|
|
extern int makeinit(fa *, bool);
|
|
extern void penter(Node *);
|
|
extern void freetr(Node *);
|
|
extern int quoted(const uschar **);
|
|
extern int *cclenter(const char *);
|
|
extern noreturn void overflo(const char *);
|
|
extern void cfoll(fa *, Node *);
|
|
extern int first(Node *);
|
|
extern void follow(Node *);
|
|
extern int member(int, int *);
|
|
extern int match(fa *, const char *);
|
|
extern int pmatch(fa *, const char *);
|
|
extern int nematch(fa *, const char *);
|
|
extern bool fnematch(fa *, FILE *, char **, int *, int);
|
|
extern Node *reparse(const char *);
|
|
extern Node *regexp(void);
|
|
extern Node *primary(void);
|
|
extern Node *concat(Node *);
|
|
extern Node *alt(Node *);
|
|
extern Node *unary(Node *);
|
|
extern int relex(void);
|
|
extern int cgoto(fa *, int, int);
|
|
extern void freefa(fa *);
|
|
|
|
extern int pgetc(void);
|
|
extern char *cursource(void);
|
|
|
|
extern Node *nodealloc(size_t);
|
|
extern Node *exptostat(Node *);
|
|
extern Node *node1(int, Node *);
|
|
extern Node *node2(int, Node *, Node *);
|
|
extern Node *node3(int, Node *, Node *, Node *);
|
|
extern Node *node4(int, Node *, Node *, Node *, Node *);
|
|
extern Node *node5(int, Node *, Node *, Node *, Node *, Node *);
|
|
extern Node *stat3(int, Node *, Node *, Node *);
|
|
extern Node *op2(int, Node *, Node *);
|
|
extern Node *op1(int, Node *);
|
|
extern Node *stat1(int, Node *);
|
|
extern Node *op3(int, Node *, Node *, Node *);
|
|
extern Node *op4(int, Node *, Node *, Node *, Node *);
|
|
extern Node *op5(int, Node *, Node *, Node *, Node *, Node *);
|
|
extern Node *stat2(int, Node *, Node *);
|
|
extern Node *stat4(int, Node *, Node *, Node *, Node *);
|
|
extern Node *celltonode(Cell *, int);
|
|
extern Node *rectonode(void);
|
|
extern Node *makearr(Node *);
|
|
extern Node *pa2stat(Node *, Node *, Node *);
|
|
extern Node *linkum(Node *, Node *);
|
|
extern void defn(Cell *, Node *, Node *);
|
|
extern int isarg(const char *);
|
|
extern const char *tokname(int);
|
|
extern Cell *(*proctab[])(Node **, int);
|
|
extern int ptoi(void *);
|
|
extern Node *itonp(int);
|
|
|
|
extern void syminit(void);
|
|
extern void arginit(int, char **);
|
|
extern void envinit(char **);
|
|
extern Array *makesymtab(int);
|
|
extern void freesymtab(Cell *);
|
|
extern void freeelem(Cell *, const char *);
|
|
extern Cell *setsymtab(const char *, const char *, double, unsigned int, Array *);
|
|
extern int hash(const char *, int);
|
|
extern void rehash(Array *);
|
|
extern Cell *lookup(const char *, Array *);
|
|
extern double setfval(Cell *, double);
|
|
extern void funnyvar(Cell *, const char *);
|
|
extern char *setsval(Cell *, const char *);
|
|
extern double getfval(Cell *);
|
|
extern char *getsval(Cell *);
|
|
extern char *getpssval(Cell *); /* for print */
|
|
extern char *tostring(const char *);
|
|
extern char *tostringN(const char *, size_t);
|
|
extern char *qstring(const char *, int);
|
|
extern Cell *catstr(Cell *, Cell *);
|
|
|
|
extern void recinit(unsigned int);
|
|
extern void initgetrec(void);
|
|
extern void makefields(int, int);
|
|
extern void growfldtab(int n);
|
|
extern void savefs(void);
|
|
extern int getrec(char **, int *, bool);
|
|
extern void nextfile(void);
|
|
extern int readrec(char **buf, int *bufsize, FILE *inf, bool isnew);
|
|
extern char *getargv(int);
|
|
extern void setclvar(char *);
|
|
extern void fldbld(void);
|
|
extern void cleanfld(int, int);
|
|
extern void newfld(int);
|
|
extern void setlastfld(int);
|
|
extern int refldbld(const char *, const char *);
|
|
extern void recbld(void);
|
|
extern Cell *fieldadr(int);
|
|
extern void yyerror(const char *);
|
|
extern void bracecheck(void);
|
|
extern void bcheck2(int, int, int);
|
|
extern void SYNTAX(const char *, ...)
|
|
__attribute__((__format__(__printf__, 1, 2)));
|
|
extern noreturn void FATAL(const char *, ...)
|
|
__attribute__((__format__(__printf__, 1, 2)));
|
|
extern void WARNING(const char *, ...)
|
|
__attribute__((__format__(__printf__, 1, 2)));
|
|
extern void error(void);
|
|
extern void eprint(void);
|
|
extern void bclass(int);
|
|
extern double errcheck(double, const char *);
|
|
extern int isclvar(const char *);
|
|
extern bool is_valid_number(const char *s, bool trailing_stuff_ok,
|
|
bool *no_trailing, double *result);
|
|
#define is_number(s, val) is_valid_number(s, false, NULL, val)
|
|
|
|
extern int adjbuf(char **pb, int *sz, int min, int q, char **pbp, const char *what);
|
|
extern void run(Node *);
|
|
extern Cell *execute(Node *);
|
|
extern Cell *program(Node **, int);
|
|
extern Cell *call(Node **, int);
|
|
extern Cell *copycell(Cell *);
|
|
extern Cell *arg(Node **, int);
|
|
extern Cell *jump(Node **, int);
|
|
extern Cell *awkgetline(Node **, int);
|
|
extern Cell *getnf(Node **, int);
|
|
extern Cell *array(Node **, int);
|
|
extern Cell *awkdelete(Node **, int);
|
|
extern Cell *intest(Node **, int);
|
|
extern Cell *matchop(Node **, int);
|
|
extern Cell *boolop(Node **, int);
|
|
extern Cell *relop(Node **, int);
|
|
extern void tfree(Cell *);
|
|
extern Cell *gettemp(void);
|
|
extern Cell *field(Node **, int);
|
|
extern Cell *indirect(Node **, int);
|
|
extern Cell *substr(Node **, int);
|
|
extern Cell *sindex(Node **, int);
|
|
extern int format(char **, int *, const char *, Node *);
|
|
extern Cell *awksprintf(Node **, int);
|
|
extern Cell *awkprintf(Node **, int);
|
|
extern Cell *arith(Node **, int);
|
|
extern double ipow(double, int);
|
|
extern Cell *incrdecr(Node **, int);
|
|
extern Cell *assign(Node **, int);
|
|
extern Cell *cat(Node **, int);
|
|
extern Cell *pastat(Node **, int);
|
|
extern Cell *dopa2(Node **, int);
|
|
extern Cell *split(Node **, int);
|
|
extern Cell *condexpr(Node **, int);
|
|
extern Cell *ifstat(Node **, int);
|
|
extern Cell *whilestat(Node **, int);
|
|
extern Cell *dostat(Node **, int);
|
|
extern Cell *forstat(Node **, int);
|
|
extern Cell *instat(Node **, int);
|
|
extern Cell *bltin(Node **, int);
|
|
extern Cell *printstat(Node **, int);
|
|
extern Cell *nullproc(Node **, int);
|
|
extern FILE *redirect(int, Node *);
|
|
extern FILE *openfile(int, const char *, bool *);
|
|
extern const char *filename(FILE *);
|
|
extern Cell *closefile(Node **, int);
|
|
extern void closeall(void);
|
|
extern Cell *sub(Node **, int);
|
|
extern Cell *gsub(Node **, int);
|
|
extern Cell *gensub(Node **, int);
|
|
|
|
extern FILE *popen(const char *, const char *);
|
|
extern int pclose(FILE *);
|
|
|
|
extern const char *flags2str(int flags);
|