| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
This appendix contains information mainly of interest to implementors and
maintainers of gawk. Everything in it applies specifically to
gawk and not to other implementations.
C.1 Downward Compatibility and Debugging How to disable certain gawkextensions.C.2 Making Additions to gawkMaking Additions To gawk.C.3 Adding New Built-in Functions to gawkAdding new built-in functions to gawk.
C.4 Probable Future Extensions New features that may be implemented one day.
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
See section Extensions in gawk Not in POSIX awk,
for a summary of the GNU extensions to the awk language and program.
All of these features can be turned off by invoking gawk with the
`--traditional' option or with the `--posix' option.
If gawk is compiled for debugging with `-DDEBUG', then there
is one more option available on the command line:
-W parsedebug
--parsedebug
This option is intended only for serious gawk developers
and not for the casual user. It probably has not even been compiled into
your version of gawk, since it slows down execution.
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
gawk
If you find that you want to enhance gawk in a significant
fashion, you are perfectly free to do so. That is the point of having
free software; the source code is available and you are free to change
it as you want (see section GNU General Public License).
This section discusses the ways you might want to change gawk
as well as any considerations you should bear in mind.
C.2.1 Adding New Features Adding code to the main body of gawk.
C.2.2 Porting gawkto a New Operating SystemPorting gawkto a new operating system.
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
You are free to add any new features you like to gawk.
However, if you want your changes to be incorporated into the gawk
distribution, there are several steps that you need to take in order to
make it possible for me to include your changes:
gawk itself,
consider writing it as an extension module
(see section Adding New Built-in Functions to gawk).
If that's not possible, continue with the rest of the steps in this list.
gawk. If your version of
gawk is very old, I may not be able to integrate them at all.
(See section Getting the gawk Distribution,
for information on getting the latest version of gawk.)
gawk.
(The GNU Coding Standards are available from
the GNU Project's
ftp
site, at
ftp://gnudist.gnu.org/gnu/GNUInfo/standards.text.
Texinfo, Info, and DVI versions are also available.)
gawk coding style.
The C code for gawk follows the instructions in the
GNU Coding Standards, with minor exceptions. The code is formatted
using the traditional "K&R" style, particularly as regards to the placement
of braces and the use of tabs. In brief, the coding rules for gawk
are as follows:
int, on the
line above the line with the name and arguments of the function.
if, while, for, do, switch,
and return).
for loop initialization and increment parts, and in macro bodies.
NULL and '\0' in the conditions of
if, while, and for statements, as well as in the cases
of switch statements, instead of just the
plain pointer or character value.
TRUE, FALSE and NULL symbolic constants
and the character constant '\0' where appropriate, instead of 1
and 0.
ISALPHA, ISDIGIT, etc. macros, instead of the
traditional lowercase versions; these macros are better behaved for
non-ASCII character sets.
alloca function for allocating memory off the stack.
Its use causes more portability trouble than is worth the minor benefit of not having
to free the storage. Instead, use malloc and free.
Note:
If I have to reformat your code to follow the coding style used in
gawk, I may not bother to integrate your changes at all.
man page as well.
You will also have to sign paperwork for your documentation changes.
gawk source tree with your version.
(I find context diffs to be more readable but unified diffs are
more compact.)
I recommend using the GNU version of diff.
Send the output produced by either run of diff to me when you
submit your changes.
(See section Reporting Problems and Bugs, for the electronic mail
information.)
Using this format makes it easy for me to apply your changes to the
master version of the gawk source code (using patch).
If I have to apply the changes manually, using a text editor, I may
not do so, particularly if there are lots of changes.
Although this sounds like a lot of work, please remember that while you may write the new code, I have to maintain it and support it. If it isn't possible for me to do that with a minimum of extra work, then I probably will not.
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
gawk to a New Operating System
If you want to port gawk to a new operating system, there are
several steps to follow:
gawk and the other ports. Avoid gratuitous
changes to the system-independent parts of the code. If at all possible,
avoid sprinkling `#ifdef's just for your port throughout the
code.
If the changes needed for a particular system affect too much of the code, I probably will not accept them. In such a case, you can, of course, distribute your changes on your own, as long as you comply with the GPL (see section GNU General Public License).
gawk are maintained by other
people at the Free Software Foundation. Thus, you should not change them
unless it is for a very good reason; i.e., changes are not out of the
question, but changes to these files are scrutinized extra carefully.
The files are `getopt.h', `getopt.c',
`getopt1.c', `regex.h', `regex.c', `dfa.h',
`dfa.c', `install-sh', and `mkinstalldirs'.
gawk on their systems. If noone
volunteers to maintain a port, it becomes unsupported and it may
be necessary to remove it from the distribution.
Each port's `gawkmisc.???' file has a suffix reminiscent of the machine or operating system for the port--for example, `pc/gawkmisc.pc' and `vms/gawkmisc.vms'. The use of separate suffixes, instead of plain `gawkmisc.c', makes it possible to move files from a port's subdirectory into the main subdirectory, without accidentally destroying the real `gawkmisc.c' file. (Currently, this is only an issue for the PC operating system ports.)
gawk for your system.
Following these steps makes it much easier to integrate your changes
into gawk and have them co-exist happily with other
operating systems' code that is already there.
In the code that you supply and maintain, feel free to use a coding style and brace layout that suits your taste.
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
gawk Danger Will Robinson! Danger!!
Warning! Warning!
The Robot
Beginning with gawk 3.1, it is possible to add new built-in
functions to gawk using dynamically loaded libraries. This
facility is available on systems (such as GNU/Linux) that support
the dlopen and dlsym functions.
This section describes how to write and use dynamically
loaded extentions for gawk.
Experience with programming in
C or C++ is necessary when reading this section.
Caution: The facilities described in this section
are very much subject to change in the next gawk release.
Be aware that you may have to re-do everything, perhaps from scratch,
upon the next release.
C.3.1 A Minimal Introduction to gawkInternalsA brief look at some gawkinternals.C.3.2 Directory and File Operation Built-ins A example of new functions.
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
gawk Internals
The truth is that gawk was not designed for simple extensibility.
The facilities for adding functions using shared libraries work, but
are something of a "bag on the side." Thus, this tour is
brief and simplistic; would-be gawk hackers are encouraged to
spend some time reading the source code before trying to write
extensions based on the material presented here. Of particular note
are the files `awk.h', `builtin.c', and `eval.c'.
Reading `awk.y' in order to see how the parse tree is built
would also be of use.
With the disclaimers out of the way, the following types, structure members, functions, and macros are declared in `awk.h' and are of use when writing extensions. The next section shows how they are used:
AWKNUM
AWKNUM is the internal type of awk
floating-point numbers. Typically, it is a C double.
NODE
NODE.
These contain both strings and numbers, as well as variables and arrays.
AWKNUM force_number(NODE *n)
gawk function.
void force_string(NODE *n)
NODE's string value is current.
It may end up calling an internal gawk function.
It also guarantees that the string is zero-terminated.
n->param_cnt
n->stptr
n->stlen
NODE's string value, respectively.
The string is not guaranteed to be zero-terminated.
If you need to pass the string value to a C library function, save
the value in n->stptr[n->stlen], assign '\0' to it,
call the routine, and then restore the value.
n->type
NODE. This is a C enum. Values should
be either Node_var or Node_var_array for function
parameters.
n->vname
void assoc_clear(NODE *n)
n.
Make sure that `n->type == Node_var_array' first.
NODE **assoc_lookup(NODE *symbol, NODE *subs, int reference)
symbol is the array, subs is the subscript.
This is usually a value created with tmp_string (see below).
reference should be TRUE if it is an error to use the
value before it is created. Typically, FALSE is the
correct value to use from extension functions.
NODE *make_string(char *s, size_t len)
NODE that
can be stored appropriately. This is permanent storage; understanding
of gawk memory management is helpful.
NODE *make_number(AWKNUM val)
AWKNUM and turn it into a pointer to a NODE that
can be stored appropriately. This is permanent storage; understanding
of gawk memory management is helpful.
NODE *tmp_string(char *s, size_t len);
NODE that
can be stored appropriately. This is temporary storage; understanding
of gawk memory management is helpful.
NODE *tmp_number(AWKNUM val)
AWKNUM and turn it into a pointer to a NODE that
can be stored appropriately. This is temporary storage;
understanding of gawk memory management is helpful.
NODE *dupnode(NODE *n)
NODE;
understanding of gawk memory management is helpful.
void free_temp(NODE *n)
NODE
allocated with tmp_string or tmp_number.
Understanding of gawk memory management is helpful.
void make_builtin(char *name, NODE *(*func)(NODE *), int count)
func as new built-in
function name. name is a regular C string. count
is the maximum number of arguments that the function takes.
The function should be written in the following manner:
/* do_xxx --- do xxx function for gawk */
NODE *
do_xxx(NODE *tree)
{
...
}
|
NODE *get_argument(NODE *tree, int i)
i'th argument from the function call.
The first argument is argument zero.
void set_value(NODE *tree)
awk program sees as the return value from the
new awk function.
void update_ERRNO(void)
gawk's ERRNO variable, based on the current
value of the C errno variable.
It is provided as a convenience.
An argument that is supposed to be an array needs to be handled with some extra code, in case the array being passed in is actually from a function parameter. The following "boiler plate" code shows how to do this:
NODE *the_arg;
the_arg = get_argument(tree, 2); /* assume need 3rd arg, 0-based */
/* if a parameter, get it off the stack */
if (the_arg->type == Node_param_list)
the_arg = stack_ptr[the_arg->param_cnt];
/* parameter referenced an array, get it */
if (the_arg->type == Node_array_ref)
the_arg = the_arg->orig_array;
/* check type */
if (the_arg->type != Node_var && the_arg->type != Node_var_array)
fatal("newfunc: third argument is not an array");
/* force it to be an array, if necessary, clear it */
the_arg->type = Node_var_array;
assoc_clear(the_arg);
|
Again, you should spend time studying the gawk internals;
don't just blindly copy this code.
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
Two useful functions that are not in awk are chdir
(so that an awk program can change its directory) and
stat (so that an awk program can gather information about
a file).
This section implements these functions for gawk in an
external extension library.
C.3.2.1 Using chdirandstatWhat the new functions will do. C.3.2.2 C Code for chdirandstatThe code for internal file operations. C.3.2.3 Integrating the Extensions How to use an external extension.
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
chdir and stat
This section shows how to use the new functions at the awk
level once they've been integrated into the running gawk
interpreter.
Using chdir is very straightforward. It takes one argument,
the new directory to change to:
...
newdir = "/home/arnold/funstuff"
ret = chdir(newdir)
if (ret < 0) {
printf("could not change to %s: %s\n",
newdir, ERRNO) > "/dev/stderr"
exit 1
}
...
|
The return value is negative if the chdir failed,
and ERRNO
(see section 7.5 Built-in Variables)
is set to a string indicating the error.
Using stat is a bit more complicated.
The C stat function fills in a structure that has a fair
amount of information.
The right way to model this in awk is to fill in an associative
array with the appropriate information:
file = "/home/arnold/.profile"
fdata[1] = "x" # force `fdata' to be an array
ret = stat(file, fdata)
if (ret < 0) {
printf("could not stat %s: %s\n",
file, ERRNO) > "/dev/stderr"
exit 1
}
printf("size of %s is %d bytes\n", file, fdata["size"])
|
The stat function always clears the data array, even if
the stat fails. It fills in the following elements:
"name"
stat'ed.
"dev"
"ino"
"mode"
"nlink"
"uid"
"gid"
"size"
"blocks"
"atime"
"mtime"
"ctime"
strftime
(see section Built-in Functions).
"pmode"
"drwxr-xr-x".
"type"
"blockdev"
"chardev"
"directory"
"fifo"
"file"
"socket"
AF_UNIX ("Unix domain") socket in the
filesystem.
"symlink"
Several additional elements may be present depending upon the operating
system and the type of the file. You can test for them in your awk
program by using the in operator
(see section Referring to an Array Element):
"blksize"
stat structure.
"linkval"
"rdev"
"major"
"minor"
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
chdir and stat Here is the C code for these extensions. They were written for GNU/Linux. The code needs some more work for complete portability to other POSIX-compliant systems:(65)
#include "awk.h"
#include <sys/sysmacros.h>
/* do_chdir --- provide dynamically loaded
chdir() builtin for gawk */
static NODE *
do_chdir(tree)
NODE *tree;
{
NODE *newdir;
int ret = -1;
newdir = get_argument(tree, 0);
|
The file includes the "awk.h" header file for definitions
for the gawk internals. It includes <sys/sysmacros.h>
for access to the major and minor macros.
By convention, for an awk function foo, the function that
implements it is called `do_foo'. The function should take
a `NODE *' argument, usually called tree, that
represents the argument list to the function. The newdir
variable represents the new directory to change to, retrieved
with get_argument. Note that the first argument is
numbered zero.
This code actually accomplishes the chdir. It first forces
the argument to be a string and passes the string value to the
chdir system call. If the chdir fails, ERRNO
is updated.
The result of force_string has to be freed with free_temp:
if (newdir != NULL) {
(void) force_string(newdir);
ret = chdir(newdir->stptr);
if (ret < 0)
update_ERRNO();
free_temp(newdir);
}
|
Finally, the function returns the return value to the awk level,
using set_value. Then it must return a value from the call to
the new built-in (this value ignored by the interpreter):
/* Set the return value */
set_value(tmp_number((AWKNUM) ret));
/* Just to make the interpreter happy */
return tmp_number((AWKNUM) 0);
}
|
The stat built-in is more involved. First comes a function
that turns a numeric mode into a printable representation
(e.g., 644 becomes `-rw-r--r--'). This is omitted here for brevity:
/* format_mode --- turn a stat mode field
into something readable */
static char *
format_mode(fmode)
unsigned long fmode;
{
...
}
|
Next comes the actual do_stat function itself. First come the
variable declarations and argument checking:
/* do_stat --- provide a stat() function for gawk */
static NODE *
do_stat(tree)
NODE *tree;
{
NODE *file, *array;
struct stat sbuf;
int ret;
char *msg;
NODE **aptr;
char *pmode; /* printable mode */
char *type = "unknown";
/* check arg count */
if (tree->param_cnt != 2)
fatal(
"stat: called with %d arguments, should be 2",
tree->param_cnt);
|
Then comes the actual work. First, we get the arguments.
Then, we always clear the array. To get the file information,
we use lstat, in case the file is a symbolic link.
If there's an error, we set ERRNO and return:
/*
* directory is first arg,
* array to hold results is second
*/
file = get_argument(tree, 0);
array = get_argument(tree, 1);
/* empty out the array */
assoc_clear(array);
/* lstat the file, if error, set ERRNO and return */
(void) force_string(file);
ret = lstat(file->stptr, & sbuf);
if (ret < 0) {
update_ERRNO();
set_value(tmp_number((AWKNUM) ret));
free_temp(file);
return tmp_number((AWKNUM) 0);
}
|
Now comes the tedious part: filling in the array. Only a few of the calls are shown here, since they all follow the same pattern:
/* fill in the array */
aptr = assoc_lookup(array, tmp_string("name", 4), FALSE);
*aptr = dupnode(file);
aptr = assoc_lookup(array, tmp_string("mode", 4), FALSE);
*aptr = make_number((AWKNUM) sbuf.st_mode);
aptr = assoc_lookup(array, tmp_string("pmode", 5), FALSE);
pmode = format_mode(sbuf.st_mode);
*aptr = make_string(pmode, strlen(pmode));
|
When done, we free the temporary value containing the file name, set the return value, and return:
free_temp(file);
/* Set the return value */
set_value(tmp_number((AWKNUM) ret));
/* Just to make the interpreter happy */
return tmp_number((AWKNUM) 0);
}
|
Finally, it's necessary to provide the "glue" that loads the
new function(s) into gawk. By convention, each library has
a routine named dlload that does the job:
/* dlload --- load new builtins in this library */
NODE *
dlload(tree, dl)
NODE *tree;
void *dl;
{
make_builtin("chdir", do_chdir, 1);
make_builtin("stat", do_stat, 2);
return tmp_number((AWKNUM) 0);
}
|
And that's it! As an exercise, consider adding functions to
implement system calls such as chown, chmod, and umask.
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
Now that the code is written, it must be possible to add it at
runtime to the running gawk interpreter. First, the
code must be compiled. Assuming that the functions are in
a file named `filefuncs.c', and idir is the location
of the gawk include files,
the following steps create
a GNU/Linux shared library:
$ gcc -shared -DHAVE_CONFIG_H -c -O -g -Iidir filefuncs.c $ ld -o filefuncs.so -shared filefuncs.o |
Once the library exists, it is loaded by calling the extension
built-in function.
This function takes two arguments: the name of the
library to load and the name of a function to call when the library
is first loaded. This function adds the new functions to gawk.
It returns the value returned by the initialization function
within the shared library:
# file testff.awk
BEGIN {
extension("./filefuncs.so", "dlload")
chdir(".") # no-op
data[1] = 1 # force `data' to be an array
print "Info for testff.awk"
ret = stat("testff.awk", data)
print "ret =", ret
for (i in data)
printf "data[\"%s\"] = %s\n", i, data[i]
print "testff.awk modified:",
strftime("%m %d %y %H:%M:%S", data["mtime"])
}
|
Here are the results of running the program:
$ gawk -f testff.awk -| Info for testff.awk -| ret = 0 -| data["blksize"] = 4096 -| data["mtime"] = 932361936 -| data["mode"] = 33188 -| data["type"] = file -| data["dev"] = 2065 -| data["gid"] = 10 -| data["ino"] = 878597 -| data["ctime"] = 971431797 -| data["blocks"] = 2 -| data["nlink"] = 1 -| data["name"] = testff.awk -| data["atime"] = 971608519 -| data["pmode"] = -rw-r--r-- -| data["size"] = 607 -| data["uid"] = 2076 -| testff.awk modified: 07 19 99 08:25:36 |
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
AWK is a language similar to PERL, only considerably more elegant.
Arnold RobbinsHey!
Larry Wall
This section briefly lists extensions and possible improvements
that indicate the directions we are
currently considering for gawk. The file `FUTURES' in the
gawk distribution lists these extensions as well.
Following is a list of probable future changes visible at the
awk language level:
awk-level interface to the
modules facility is as good as it should be. The interface needs to be
redesigned, particularly taking namespace issues into account, as
well as possibly including issues such as library search path order
and versioning.
RECLEN variable for fixed length records
FIELDWIDTHS, this would speed up the processing of
fixed-length records.
PROCINFO["RS"] would be "RS" or "RECLEN",
depending upon which kind of record processing is in effect.
printf specifiers
printf
format specifiers. These should be evaluated for possible inclusion
in gawk.
awk array.
gawk could handle UTF-8 and other
character sets that are larger than eight bits.
lint warnings
Following is a list of probable improvements that will make gawk's
source code easier to work with:
gawk),
but is rather primitive. It requires a fair amount of manual work
to create and integrate a loadable module.
Nor is the current mechanism as portable as might be desired.
The GNU libtool package provides a number of features that
would make using loadable modules much easier.
gawk should be changed to use libtool.
gawk "exports" should be revised.
Too many things are needlessly exposed. A new API should be designed
and implemented to make module writing easier.
gawk's management of array subscript storage could use revamping,
so that using the same value to index multiple arrays only
stores one copy of the index value.
Following is a list of probable improvements that will make gawk
perform better:
dfa
dfa pattern matcher from GNU grep has some
problems. Either a new version or a fixed one will deal with some
important regexp matching issues.
awk programs
gawk uses a Bison (YACC-like)
parser to convert the script given it into a syntax tree; the syntax
tree is then executed by a simple recursive evaluator. This method incurs
a lot of overhead, since the recursive evaluator performs many procedure
calls to do even the simplest things.
It should be possible for gawk to convert the script's parse tree
into a C program which the user would then compile, using the normal
C compiler and a special gawk library to provide all the needed
functions (regexps, fields, associative arrays, type coercion, and so on).
An easier possibility might be for an intermediate phase of gawk to
convert the parse tree into a linear byte code form like the one used
in GNU Emacs Lisp. The recursive evaluator would then be replaced by
a straight line byte code interpreter that would be intermediate in speed
between running a compiled program and doing what gawk does
now.
Finally, the programs in the test suite could use documenting in this Web page.
See section Making Additions to gawk,
if you are interested in tackling any of these projects.
| [ << ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |