The GNU Awk User's Guide

This file documents awk, a program that you can use to select particular records in a file and perform operations upon them.

This is Edition 3 of GAWK: Effective AWK Programming: A User's Guide for GNU Awk, for the 3.1.0 version of the GNU implementation of AWK.

Foreword		Some nice words about this Web page.
Preface		What this Web page is about; brief history and acknowledgments.
2. Getting Started with awk		A basic introduction to using

                                   awk. How to run an awk
                                   program. Command-line syntax.

3. Regular Expressions		All about matching things using regular expressions.
4. Reading Input Files		How to read files and manipulate fields.
5. Printing Output		How to print using awk. Describes the print and printf statements. Also describes redirection of output.
6. Expressions		Expressions are the basic building blocks of statements.
7. Patterns, Actions, and Variables		Overviews of patterns and actions.
8. Arrays in awk		The description and use of arrays. Also includes array-oriented control statements.
9. Functions		Built-in and user-defined functions.
10. Internationalization with gawk		Getting gawk to speak your language.
11. Advanced Features of gawk		Stuff for advanced users, specific to

                                   gawk.

12. Running awk and gawk		How to run gawk.
13. A Library of awk Functions
14. Practical awk Programs		Many awk programs with complete explanations.
A. The Evolution of the awk Language		The evolution of the awk language.
B. Installing gawk		Installing gawk under various operating systems.
C. Implementation Notes		Notes about gawk extensions and possible future work.
D. Basic Programming Concepts		A very quick intoduction to programming concepts.
Glossary		An explanation of some unfamiliar terms.
GNU General Public License		Your right to copy and distribute

                                   gawk.

GNU Free Documentation License		The license for this Web page.
Index		Concept and Variable Index.

History of awk and gawk

The history of gawk and

                                   awk.

1.0 A Rose by Any Other Name		What name to use to find awk.
1.1 Using This Book		Using this Web page. Includes sample input files that you can use.
1.2 Typographical Conventions
The GNU Project and This Book		Brief history of the GNU project and this Web page.
How to Contribute		Helping to save the world.
Acknowledgments
2.1 How to Run awk Programs		How to run gawk programs; includes command-line syntax.
2.1.1 One-Shot Throw-Away awk Programs		Running a short throw-away awk program.
2.1.2 Running awk Without Input Files		Using no input files (input from terminal instead).
2.1.3 Running Long Programs		Putting permanent awk programs in files.
2.1.4 Executable awk Programs		Making self-contained awk programs.
2.1.5 Comments in awk Programs		Adding documentation to gawk programs.
2.1.6 Shell Quoting Issues		More discussion of shell quoting issues.
2.2 Data Files for the Examples		Sample data files for use in the

                                   awk programs illustrated in this
                                   Web page.

2.3 Some Simple Examples		A very simple example.
2.4 An Example with Two Rules		A less simple one-line example using two rules.
2.5 A More Complex Example		A more complex example.
2.6 awk Statements Versus Lines		Subdividing or combining statements into lines.
2.7 Other Features of awk
2.8 When to Use awk		When to use gawk and when to use other things.
3.1 How to Use Regular Expressions
3.2 Escape Sequences		How to write non-printing characters.
3.3 Regular Expression Operators
3.4 Using Character Lists		What can go between `[...]'.
3.5 gawk-Specific Regexp Operators		Operators specific to GNU software.
3.6 Case Sensitivity in Matching		How to do case-insensitive matching.
3.7 How Much Text Matches?		How much text matches.
3.8 Using Dynamic Regexps
4.1 How Input Is Split into Records		Controlling how data is split into records.
4.2 Examining Fields		An introduction to fields.
4.3 Non-Constant Field Numbers		Non-constant Field Numbers.
4.4 Changing the Contents of a Field
4.5 Specifying How Fields Are Separated		The field separator and how to change it.
4.5.1 Using Regular Expressions to Separate Fields		Using regexps as the field separator.
4.5.2 Making Each Character a Separate Field		Making each character a separate field.
4.5.3 Setting FS from the Command Line		Setting FS from the command-line.
4.5.4 Field Splitting Summary		Some final points and a summary table.
4.6 Reading Fixed-Width Data		Reading constant width data.
4.7 Multiple-Line Records		Reading multi-line records.
4.8 Explicit Input with getline		Reading files under explicit program control using the getline function.
4.8.1 Using getline with No Arguments		Using getline with no arguments.
4.8.2 Using getline into a Variable		Using getline into a variable.
4.8.3 Using getline from a File		Using getline from a file.
4.8.4 Using getline into a Variable from a File		Using getline into a variable from a file.
4.8.5 Using getline from a Pipe		Using getline from a pipe.
4.8.6 Using getline into a Variable from a Pipe		Using getline into a variable from a pipe.
4.8.7 Using getline from a Coprocess		Using getline from a coprocess.
4.8.8 Using getline into a Variable from a Coprocess		Using getline into a variable from a coprocess.
4.8.9 Points About getline to Remember		Important things to know about

                                   getline.

4.8.10 Summary of getline Variants
5.1 The print Statement		The print statement.
5.2 Examples of print Statements		Simple examples of print statements.
5.3 Output Separators		The output separators and how to change them.
5.4 Controlling Numeric Output with print		Controlling Numeric Output With

                                   print.

5.5 Using printf Statements for Fancier Printing		The printf statement.
5.5.1 Introduction to the printf Statement		Syntax of the printf statement.
5.5.2 Format-Control Letters		Format-control letters.
5.5.3 Modifiers for printf Formats		Format-specification modifiers.
5.5.4 Examples Using printf		Several examples.
5.6 Redirecting Output of print and printf		How to redirect output to multiple files and pipes.
5.7 Special File Names in gawk		File name interpretation in gawk.

                                   gawk allows access to inherited
                                   file descriptors.

5.7.1 Special Files for Standard Descriptors		Special files for I/O.
5.7.2 Special Files for Process-Related Information		Special files for process information.
5.7.3 Special Files for Network Communications		Special files for network communications.
5.7.4 Special File Name Caveats		Things to watch out for.
5.8 Closing Input and Output Redirections		Closing Input and Output Files and Pipes.
6.1 Constant Expressions		String, numeric and regexp constants.
6.1.1 Numeric and String Constants		Numeric and string constants.
6.1.2 Octal and Hexadecimal Numbers		What are octal and hex numbers.
6.1.3 Regular Expression Constants		Regular Expression constants.
6.2 Using Regular Expression Constants		When and how to use a regexp constant.
6.3 Variables		Variables give names to values for later use.
6.3.1 Using Variables in a Program		Using variables in your programs.
6.3.2 Assigning Variables on the Command Line		Setting variables on the command-line and a summary of command-line syntax. This is an advanced method of input.
6.4 Conversion of Strings and Numbers		The conversion of strings to numbers and vice versa.
6.5 Arithmetic Operators		Arithmetic operations (`+', `-', etc.)
6.6 String Concatenation		Concatenating strings.
6.7 Assignment Expressions		Changing the value of a variable or a field.
6.8 Increment and Decrement Operators		Incrementing the numeric value of a variable.
6.9 True and False in awk		What is "true" and what is "false".
6.10 Variable Typing and Comparison Expressions		How variables acquire types and how this affects comparison of numbers and strings with `<', etc.
6.11 Boolean Expressions		Combining comparison expressions using boolean operators `||' ("or"),

                                   `&&' ("and") and `!' ("not").

6.12 Conditional Expressions		Conditional expressions select between two subexpressions under control of a third subexpression.
6.13 Function Calls		A function call is an expression.
6.14 Operator Precedence (How Operators Nest)		How various operators nest.
7.1 Pattern Elements		What goes into a pattern.
7.1.1 Regular Expressions as Patterns		Using regexps as patterns.
7.1.2 Expressions as Patterns		Any expression can be used as a pattern.
7.1.3 Specifying Record Ranges with Patterns		Pairs of patterns specify record ranges.
7.1.4 The BEGIN and END Special Patterns		Specifying initialization and cleanup rules.
7.1.4.1 Startup and Cleanup Actions		How and why to use BEGIN/END rules.
7.1.4.2 Input/Output from BEGIN and END Rules		I/O issues in BEGIN/END rules.
7.1.5 The Empty Pattern		The empty pattern, which matches every record.
7.2 Using Shell Variables in Programs		How to use shell variables with

                                   awk.

7.3 Actions		What goes into an action.
7.4 Control Statements in Actions		Describes the various control statements in detail.
7.4.1 The if-else Statement		Conditionally execute some awk statements.
7.4.2 The while Statement		Loop until some condition is satisfied.
7.4.3 The do-while Statement		Do specified action while looping until some condition is satisfied.
7.4.4 The for Statement		Another looping statement, that provides initialization and increment clauses.
7.4.5 The break Statement		Immediately exit the innermost enclosing loop.
7.4.6 The continue Statement		Skip to the end of the innermost enclosing loop.
7.4.7 The next Statement		Stop processing the current input record.
7.4.8 Using gawk's nextfile Statement		Stop processing the current file.
7.4.9 The exit Statement		Stop execution of awk.
7.5 Built-in Variables		Summarizes the built-in variables.
7.5.1 Built-in Variables That Control awk		Built-in variables that you change to control awk.
7.5.2 Built-in Variables That Convey Information		Built-in variables where awk gives you information.
7.5.3 Using ARGC and ARGV		Ways to use ARGC and ARGV.
8.1 Introduction to Arrays
8.2 Referring to an Array Element		How to examine one element of an array.
8.3 Assigning Array Elements		How to change an element of an array.
8.4 Basic Array Example		Basic Example of an Array
8.5 Scanning All Elements of an Array		A variation of the for statement. It loops through the indices of an array's existing elements.
8.6 The delete Statement		The delete statement removes an element from an array.
8.7 Using Numbers to Subscript Arrays		How to use numbers as subscripts in

                                   awk.

8.8 Using Uninitialized Variables as Subscripts		Using Uninitialized variables as subscripts.
8.9 Multidimensional Arrays		Emulating multidimensional arrays in

                                   awk.

8.10 Scanning Multidimensional Arrays		Scanning multidimensional arrays.
8.11 Sorting Array Values and Indices with gawk		Sorting array values and indices.
9.1 Built-in Functions		Summarizes the built-in functions.
9.1.1 Calling Built-in Functions		How to call built-in functions.
9.1.2 Numeric Functions		Functions that work with numbers, including

                                   int, sin and rand.

9.1.3 String Manipulation Functions

Functions for string manipulation, such as

                                   split, match and
                                   sprintf.

9.1.3.1 More About `\' and `&' with sub, gsub, and gensub		More than you want to know about `\' and `&' with sub, gsub, and gensub.
9.1.4 Input/Output Functions		Functions for files and shell commands.
9.1.5 Using gawk's Timestamp Functions		Functions for dealing with timestamps.
9.1.6 Using gawk's Bit Manipulation Functions		Functions for bitwise operations.
9.1.7 Using gawk's String Translation Functions		Functions for string translation.
9.2 User-Defined Functions		Describes User-defined functions in detail.
9.2.1 Function Definition Syntax		How to write definitions and what they mean.
9.2.2 Function Definition Examples		An example function definition and what it does.
9.2.3 Calling User-Defined Functions		Things to watch out for.
9.2.4 The return Statement		Specifying the value a function returns.
9.2.5 Functions and Their Effect on Variable Typing		How variable types can change at runtime.
10.1 Internationalization and Localization
10.2 GNU gettext		How GNU gettext works.
10.3 Internationalizing awk Programs		Features for the programmer.
10.4 Translating awk Programs		Features for the translator.
10.4.1 Extracting Marked Strings		Extracting marked strings.
10.4.2 Rearranging printf Arguments		Rearranging printf arguments.
10.4.3 awk Portability Issues		awk-level portability issues.
10.5 A Simple Internationalization Example		A simple i18n example.
10.6 gawk Can Speak Your Language		gawk is also internationalized.
11.1 Allowing Non-Decimal Input Data		Allowing non-decimal input data.
11.2 Two-Way Communications with Another Process		Two-way communications with another process.
11.3 Using gawk for Network Programming		Using gawk for network programming.
11.4 Using gawk with BSD Portals		Using gawk with BSD portals.
11.5 Profiling Your awk Programs		Profiling your awk programs.
12.1 Invoking awk		How to run awk.
12.2 Command-Line Options		Command-line options and their meanings.
12.3 Other Command-Line Arguments		Input file names and variable assignments.
12.4 The AWKPATH Environment Variable		Searching directories for awk programs.
12.5 Obsolete Options and/or Features		Obsolete Options and/or features.
12.6 Undocumented Options and Features
12.7 Known Bugs in gawk
13.1 Naming Library Function Global Variables		How to best name private global variables in library functions.
13.2 General Programming		Functions that are of general use.
13.2.1 Implementing nextfile as a Function		Two implementations of a nextfile function.
13.2.2 Assertions		A function for assertions in awk programs.
13.2.3 Rounding Numbers		A function for rounding if sprintf does not do it correctly.
13.2.4 The Cliff Random Number Generator
13.2.5 Translating Between Characters and Numbers		Functions for using characters as numbers and vice versa.
13.2.6 Merging an Array into a String		A function to join an array into a string.
13.2.7 Managing the Time of Day		A function to get formatted times.
13.3 Data File Management		Functions for managing command-line data files.
13.3.1 Noting Data File Boundaries		A function for handling data file transitions.
13.3.2 Rereading the Current File		A function for rereading the current file.
13.3.3 Checking for Readable Data Files		Checking that data files are readable.
13.3.4 Treating Assignments as File Names		Treating assignments as file names.
13.4 Processing Command-Line Options		A function for processing command-line arguments.
13.5 Reading the User Database		Functions for getting user information.
13.6 Reading the Group Database		Functions for getting group information.
14.1 Running the Example Programs		How to run these examples.
14.2 Reinventing Wheels for Fun and Profit		Clones of common utilities.
14.2.1 Cutting out Fields and Columns		The cut utility.
14.2.2 Searching for Regular Expressions in Files		The egrep utility.
14.2.3 Printing out User Information		The id utility.
14.2.4 Splitting a Large File into Pieces		The split utility.
14.2.5 Duplicating Output into Multiple Files		The tee utility.
14.2.6 Printing Non-Duplicated Lines of Text		The uniq utility.
14.2.7 Counting Things		The wc utility.
14.3 A Grab Bag of awk Programs		Some interesting awk programs.
14.3.1 Finding Duplicated Words in a Document		Finding duplicated words in a document.
14.3.2 An Alarm Clock Program		An alarm clock.
14.3.3 Transliterating Characters		A program similar to the tr utility.
14.3.4 Printing Mailing Labels		Printing mailing labels.
14.3.5 Generating Word Usage Counts		A program to produce a word usage count.
14.3.6 Removing Duplicates from Unsorted Text		Eliminating duplicate entries from a history file.
14.3.7 Extracting Programs from Texinfo Source Files		Pulling out programs from Texinfo source files.
14.3.8 A Simple Stream Editor
14.3.9 An Easy Way to Use Library Functions		A wrapper for awk that includes files.
A.1 Major Changes Between V7 and SVR3.1		The major changes between V7 and System V Release 3.1.
A.2 Changes Between SVR3.1 and SVR4		Minor changes between System V Releases 3.1 and 4.
A.3 Changes Between SVR4 and POSIX awk		New features from the POSIX standard.
A.4 Extensions in the Bell Laboratories awk		New features from the Bell Laboratories version of awk.
A.5 Extensions in gawk Not in POSIX awk		The extensions in gawk not in POSIX awk.
A.6 Major Contributors to gawk		The major contributors to gawk.
B.1 The gawk Distribution		What is in the gawk distribution.
B.1.1 Getting the gawk Distribution		How to get the distribution.
B.1.2 Extracting the Distribution		How to extract the distribution.
B.1.3 Contents of the gawk Distribution		What is in the distribution.
B.2 Compiling and Installing gawk on Unix		Installing gawk under various versions of Unix.
B.2.1 Compiling gawk for Unix		Compiling gawk under Unix.
B.2.2 Additional Configuration Options		Other compile-time options.
B.2.3 The Configuration Process		How it's all supposed to work.
B.3 Installation on Other Operating Systems
B.3.1 Installing gawk on an Amiga
B.3.2 Installing gawk on BeOS
B.3.3 Installation on PC Operating Systems		Installing and Compiling gawk on MS-DOS and OS/2.
B.3.3.1 Installing a Prepared Distribution for PC Systems		Installing a prepared distribution.
B.3.3.2 Compiling gawk for PC Operating Systems		Compiling gawk for MS-DOS, Win32, and OS/2.
B.3.3.3 Using gawk on PC Operating Systems		Running gawk on MS-DOS, Win32 and OS/2.
B.3.4 How to Compile and Install gawk on VMS		Installing gawk on VMS.
B.3.4.1 Compiling gawk on VMS		How to compile gawk under VMS.
B.3.4.2 Installing gawk on VMS		How to install gawk under VMS.
B.3.4.3 Running gawk on VMS		How to run gawk under VMS.
B.3.4.4 Building and Using gawk on VMS POSIX		Alternate instructions for VMS POSIX.
B.4 Unsupported Operating System Ports		Systems whose ports are no longer supported.
B.4.1 Installing gawk on the Atari ST
B.4.1.1 Compiling gawk on the Atari ST		Compiling gawk on Atari.
B.4.1.2 Running gawk on the Atari ST		Running gawk on Atari.
B.4.2 Installing gawk on a Tandem
B.5 Reporting Problems and Bugs
B.6 Other Freely Available awk Implementations		Other freely available awk implementations.
C.1 Downward Compatibility and Debugging		How to disable certain gawk extensions.
C.2 Making Additions to gawk		Making Additions To gawk.
C.2.1 Adding New Features		Adding code to the main body of

                                   gawk.

C.2.2 Porting gawk to a New Operating System		Porting gawk to a new operating system.
C.3 Adding New Built-in Functions to gawk		Adding new built-in functions to

                                   gawk.

C.3.1 A Minimal Introduction to gawk Internals		A brief look at some gawk internals.
C.3.2 Directory and File Operation Built-ins		A example of new functions.
C.3.2.1 Using chdir and stat		What the new functions will do.
C.3.2.2 C Code for chdir and stat		The code for internal file operations.
C.3.2.3 Integrating the Extensions		How to use an external extension.
C.4 Probable Future Extensions		New features that may be implemented one day.
D.1 What a Program Does		The high level view.
D.2 Data Values in a Computer		A very quick intro to data types.
D.3 Floating-Point Number Caveats		Stuff to know about floating-point numbers.

Foreword

Arnold Robbins and I are good friends. We were introduced 11 years ago by circumstances--and our favorite programming language, AWK. The circumstances started a couple of years earlier. I was working at a new job and noticed an unplugged Unix computer sitting in the corner. No one knew how to use it, and neither did I. However, a couple of days later it was running, and I was root and the one-and-only user. That day, I began the transition from statistician to Unix programmer.

On one of many trips to the library or bookstore in search of books on Unix, I found the gray AWK book, a.k.a. Aho, Kernighan and Weinberger, The AWK Programming Language, Addison-Wesley, 1988. AWK's simple programming paradigm--find a pattern in the input and then perform an action--often reduced complex or tedious data manipulations to few lines of code. I was excited to try my hand at programming in AWK.

Alas, the awk on my computer was a limited version of the language described in the AWK book. I discovered that my computer had "old awk" and the AWK book described "new awk." I learned that this was typical; the old version refused to step aside or relinquish its name. If a system had a new awk, it was invariably called nawk, and few systems had it. The best way to get a new awk was to ftp the source code for gawk from prep.ai.mit.edu. gawk was a version of new awk written by David Trueman and Arnold, and available under the GNU General Public License.

(Incidentally, it's no longer difficult to find a new awk. gawk ships with Linux, and you can download binaries or source code for almost any system; my wife uses gawk on her VMS box.)

My Unix system started out unplugged from the wall; it certainly was not plugged into a network. So, oblivious to the existence of gawk and the Unix community in general, and desiring a new awk, I wrote my own, called mawk. Before I was finished I knew about gawk, but it was too late to stop, so I eventually posted to a comp.sources newsgroup.

A few days after my posting, I got a friendly email from Arnold introducing himself. He suggested we share design and algorithms and attached a draft of the POSIX standard so that I could update mawk to support language extensions added after publication of the AWK book.

Frankly, if our roles had been reversed, I would not have been so open and we probably would have never met. I'm glad we did meet. He is an AWK expert's AWK expert and a genuinely nice person. Arnold contributes significant amounts of his expertise and time to the Free Software Foundation.

This book is the gawk reference manual, but at its core it is a book about AWK programming that will appeal to a wide audience. It is a definitive reference to the AWK language as defined by the 1987 Bell Labs release and codified in the 1992 POSIX Utilities standard.

On the other hand, the novice AWK programmer can study a wealth of practical programs that emphasize the power of AWK's basic idioms: data driven control-flow, pattern matching with regular expressions, and associative arrays. Those looking for something new can try out gawk's interface to network protocols via special `/inet' files.

The programs in this book make clear that an AWK program is typically much smaller and faster to develop than a counterpart written in C. Consequently, there is often a payoff to prototype an algorithm or design in AWK to get it running quickly and expose problems early. Often, the interpreted performance is adequate and the AWK prototype becomes the product.

The new pgawk (profiling gawk), produces program execution counts. I recently experimented with an algorithm that for n lines of input, exhibited ~ C n^2 performance, while theory predicted ~ C n log n behavior. A few minutes poring over the `awkprof.out' profile pinpointed the problem to a single line of code. pgawk is a welcome addition to my programmer's toolbox.

Arnold has distilled over a decade of experience writing and using AWK programs, and developing gawk, into this book. If you use AWK or want to learn how, then read this book.

Michael Brennan Author of mawk

Preface

Several kinds of tasks occur repeatedly when working with text files. You might want to extract certain lines and discard the rest. Or you may need to make changes wherever certain patterns appear, but leave the rest of the file alone. Writing single-use programs for these tasks in languages such as C, C++ or Pascal is time-consuming and inconvenient. Such jobs are often easier with awk. The awk utility interprets a special-purpose programming language that makes it easy to handle simple data-reformatting jobs.

The GNU implementation of awk is called gawk; it is fully compatible with the System V Release 4 version of awk. gawk is also compatible with the POSIX specification of the awk language. This means that all properly written awk programs should work with gawk. Thus, we usually don't distinguish between gawk and other awk implementations.

Using awk allows you to:

• Manage small, personal databases

• Generate reports

• Validate data

• Produce indexes and perform other document preparation tasks

• Experiment with algorithms that you can adapt later to other computer languages.

In addition, gawk provides facilities that make it easy to:

• Extract bits and pieces of data for processing

• Sort data

• Perform simple network communications.

This Web page teaches you about the awk language and how you can use it effectively. You should already be familiar with basic system commands, such as cat and ls,(1) as well as basic shell facilities, such as Input/Output (I/O) redirection and pipes.

Implementations of the awk language are available for many different computing environments. This Web page, while describing the awk language in general, also describes the particular implementation of awk called gawk (which stands for "GNU awk"). gawk runs on a broad range of Unix systems, ranging from 80386 PC-based computers, up through large-scale systems, such as Crays. gawk has also been ported to Mac OS X, MS-DOS, Microsoft Windows (all versions) and OS/2 PC's, Atari and Amiga micro-computers, BeOS, Tandem D20, and VMS.

History of awk and gawk

The history of gawk and

                                awk.

1.0 A Rose by Any Other Name		What name to use to find awk.
1.1 Using This Book		Using this Web page. Includes sample input files that you can use.
1.2 Typographical Conventions
The GNU Project and This Book		Brief history of the GNU project and this Web page.
How to Contribute		Helping to save the world.
Acknowledgments

History of awk and gawk

	1 part egrep	1 part snobol
	2 parts ed	3 parts C

Blend all parts well using lex and yacc. Document minimally and release.

After eight years, add another part egrep and two more parts C. Document very well and release.

The name awk comes from the initials of its designers: Alfred V. Aho, Peter J. Weinberger and Brian W. Kernighan. The original version of awk was written in 1977 at AT&T Bell Laboratories. In 1985, a new version made the programming language more powerful, introducing user-defined functions, multiple input streams, and computed regular expressions. This new version became widely available with Unix System V Release 3.1 (SVR3.1). The version in SVR4 added some new features and cleaned up the behavior in some of the "dark corners" of the language. The specification for awk in the POSIX Command Language and Utilities standard further clarified the language. Both the gawk designers and the original Bell Laboratories awk designers provided feedback for the POSIX specification.

Paul Rubin wrote the GNU implementation, gawk, in 1986. Jay Fenlason completed it, with advice from Richard Stallman. John Woods contributed parts of the code as well. In 1988 and 1989, David Trueman, with help from me, thoroughly reworked gawk for compatibility with the newer awk. Circa 1995, I became the primary maintainer. Current development focuses on bug fixes, performance improvements, standards compliance, and occasionally, new features.

In May of 1997, Jürgen Kahrs felt the need for network access from awk, and with a little help from me, set about adding features to do this for gawk. At that time, he also wrote the bulk of TCP/IP Internetworking with gawk (a separate document, available as part of the gawk distribution). His code finally became part of the main gawk distribution with gawk version 3.1.

See section Major Contributors to gawk, for a complete list of those who made important contributions to gawk.

1.0 A Rose by Any Other Name

The awk language has evolved over the years. Full details are provided in The Evolution of the awk Language. The language described in this Web page is often referred to as "new awk" (nawk).

Because of this, many systems have multiple versions of awk. Some systems have an awk utility that implements the original version of the awk language and a nawk utility for the new version. Others have an oawk for the "old awk" language and plain awk for the new one. Still others only have one version, which is usually the new one.(2)

All in all, this makes it difficult for you to know which version of awk you should run when writing your programs. The best advice I can give here is to check your local documentation. Look for awk, oawk, and nawk, as well as for gawk. It is likely that you already have some version of new awk on your system, which is what you should use when running your programs. (Of course, if you're reading this Web page, chances are good that you have gawk!)

Throughout this Web page, whenever we refer to a language feature that should be available in any complete implementation of POSIX awk, we simply use the term awk. When referring to a feature that is specific to the GNU implementation, we use the term gawk.

1.1 Using This Book

Documentation is like sex: when it is good, it is very, very good; and when it is bad, it is better than nothing.
Dick Brandon

The term awk refers to a particular program as well as to the language you use to tell this program what to do. When we need to be careful, we call the program "the awk utility" and the language "the awk language." This Web page explains both the awk language and how to run the awk utility. The term awk program refers to a program written by you in the awk programming language.

Primarily, this Web page explains the features of awk, as defined in the POSIX standard. It does so in the context of the gawk implementation. While doing so, it also attempts to describe important differences between gawk and other awk implementations.(3) Finally, any gawk features that are not in the POSIX standard for awk are noted.

This Web page has the difficult task of being both a tutorial and a reference. If you are a novice, feel free to skip over details that seem too complex. You should also ignore the many cross references; they are for the expert user and for the online Info version of the document.

There are subsections labelled as Advanced Notes scattered throughout the Web page. They add a more complete explanation of points that are relevant, but not likely to be of interest on first reading. All appear in the index, under the heading "advanced notes."

Most of the time, the examples use complete awk programs. In some of the more advanced sections, only the part of the awk program that illustrates the concept currently being described is shown.

While this Web page is aimed principally at people who have not been exposed to awk, there is a lot of information here that even the awk expert should find useful. In particular, the description of POSIX awk and the example programs in A Library of awk Functions, and in Practical awk Programs, should be of interest.

Getting Started with awk, provides the essentials you need to know to begin using awk.

Regular Expressions, introduces regular expressions in general, and in particular the flavors supported by POSIX awk and gawk.

Reading Input Files, describes how awk reads your data. It introduces the concepts of records and fields, as well as the getline command. I/O redirection is first described here.

Printing Output, describes how awk programs can produce output with print and printf.

6. Expressions, describes expressions, which are the basic building blocks for getting most things done in a program.

Patterns Actions and Variables, describes how to write patterns for matching records, actions for doing something when a record is matched, and the built-in variables awk and gawk use.

Arrays in awk, covers awk's one-and-only data structure: associative arrays. Deleting array elements and whole arrays is also described, as well as sorting arrays in gawk.

9. Functions, describes the built-in functions awk and gawk provide for you, as well as how to define your own functions.

Internationalization with gawk, describes special features in gawk for translating program messages into different languages at runtime.

Advanced Features of gawk, describes a number of gawk-specific advanced features. Of particular note are the abilities to have two-way communications with another process, perform TCP/IP networking, and profile your awk programs.

Running awk and gawk, describes how to run gawk, the meaning of its command-line options, and how it finds awk program source files.

A Library of awk Functions, and Practical awk Programs, provide many sample awk programs. Reading them allows you to see awk being used for solving real problems.

The Evolution of the awk Language, describes how the awk language has evolved since it was first released to present. It also describes how gawk has acquired features over time.

Installing gawk, describes how to get gawk, how to compile it under Unix, and how to compile and use it on different non-Unix systems. It also describes how to report bugs in gawk and where to get three other freely available implementations of awk.

Implementation Notes, describes how to disable gawk's extensions, as well as how to contribute new code to gawk, how to write extension libraries, and some possible future directions for gawk development.

Basic Programming Concepts, provides some very cursory background material for those who are completely unfamiliar with computer programming. Also centralized there is a discussion of some of the issues involved in using floating-point numbers.

The Glossary, defines most, if not all, the significant terms used throughout the book. If you find terms that you aren't familiar with, try looking them up.

GNU General Public License, and GNU Free Documentation License, present the licenses that cover the gawk source code, and this Web page, respectively.

1.2 Typographical Conventions

This Web page is written using Texinfo, the GNU documentation formatting language. A single Texinfo source file is used to produce both the printed and online versions of the documentation. This section briefly documents the typographical conventions used in Texinfo.

Examples you would type at the command-line are preceded by the common shell primary and secondary prompts, `$' and `>'. Output from the command is preceded by the glyph "-|". This typically represents the command's standard output. Error messages, and other output on the command's standard error, are preceded by the glyph "error-->". For example:

$ echo hi on stdout -| hi on stdout $ echo hello on stderr 1>&2 error--> hello on stderr

Characters that you type at the keyboard look like this. In particular, there are special characters called "control characters." These are characters that you type by holding down both the CONTROL key and another key, at the same time. For example, a Ctrl-d is typed by first pressing and holding the CONTROL key, next pressing the d key and finally releasing both keys.

Dark Corners

Dark corners are basically fractal -- no matter how much you illuminate, there's always a smaller but darker one.
Brian Kernighan

Until the POSIX standard (and The Gawk Manual), many features of awk were either poorly documented or not documented at all. Descriptions of such features (often called "dark corners") are noted in this Web page with "(d.c.)". They also appear in the index under the heading "dark corner."

As noted by the opening quote, though, any coverage of dark corners is, by definition, something that is incomplete.

The GNU Project and This Book

Software is like sex: it's better when it's free.
Linus Torvalds

The Free Software Foundation (FSF) is a non-profit organization dedicated to the production and distribution of freely distributable software. It was founded by Richard M. Stallman, the author of the original Emacs editor. GNU Emacs is the most widely used version of Emacs today.

The GNU(4) Project is an ongoing effort on the part of the Free Software Foundation to create a complete, freely distributable, POSIX-compliant computing environment. The FSF uses the "GNU General Public License" (GPL) to ensure that their software's source code is always available to the end user. A copy of the GPL is included in this Web page for your reference (see section GNU General Public License). The GPL applies to the C language source code for gawk. To find out more about the FSF and the GNU Project online, see the GNU Project's home page. This Web page may also be read from their web site.

A shell, an editor (Emacs), highly portable optimizing C, C++, and Objective-C compilers, a symbolic debugger and dozens of large and small utilities (such as gawk), have all been completed and are freely available. The GNU operating system kernel (the HURD), has been released but is still in an early stage of development.

Until the GNU operating system is more fully developed, you should consider using GNU/Linux, a freely distributable, Unix-like operating system for Intel 80386, DEC Alpha, Sun SPARC, IBM S/390, and other systems.(5) There are many books on GNU/Linux. One that is freely available is Linux Installation and Getting Started, by Matt Welsh. Many GNU/Linux distributions are often available in computer stores or bundled on CD-ROMs with books about Linux. (There are three other freely available, Unix-like operating systems for 80386 and other systems: NetBSD, FreeBSD, and OpenBSD. All are based on the 4.4-Lite Berkeley Software Distribution, and they use recent versions of gawk for their versions of awk.)

The Web page you are reading now is actually free--at least, the information in it is free to anyone. The machine readable source code for the Web page comes with gawk; anyone may take this Web page to a copying machine and make as many copies of it as they like. (Take a moment to check the Free Documentation License; see GNU Free Documentation License.)

Although you could just print it out yourself, bound books are much easier to read and use. Furthermore, the proceeds from sales of this book go back to the FSF to help fund development of more free software.

The Web page itself has gone through a number of previous editions. Paul Rubin wrote the very first draft of The GAWK Manual; it was around 40 pages in size. Diane Close and Richard Stallman improved it, yielding a version that was around 90 pages long and barely described the original, "old" version of awk.

I started working with that version in the fall of 1988. As work on it progressed, the FSF published several preliminary versions (numbered 0.x). In 1996, Edition 1.0 was released with gawk 3.0.0. The FSF published the first two editions under the title The GNU Awk User's Guide.

This edition maintains the basic structure of Edition 1.0, but with significant additional material, reflecting the host of new features in gawk version 3.1. Of particular note is Sorting Array Values and Indices with gawk, as well as Using gawk's Bit Manipulation Functions, Internationalization with gawk, and also Advanced Features of gawk, and Adding New Built-in Functions to gawk.

GAWK: Effective AWK Programming will undoubtedly continue to evolve. An electronic version comes with the gawk distribution from the FSF. If you find an error in this Web page, please report it! See section Reporting Problems and Bugs, for information on submitting problem reports electronically, or write to me in care of the publisher.

How to Contribute

As the maintainer of GNU awk, I am starting a collection of publicly available awk programs. For more information, see ftp://ftp.freefriends.org/arnold/Awkstuff. If you have written an interesting awk program, or have written a gawk extension that you would like to share with the rest of the world, please contact me ([email protected]). Making things available on the Internet helps keep the gawk distribution down to manageable size.

Acknowledgments

The initial draft of The GAWK Manual had the following acknowledgments:

Many people need to be thanked for their assistance in producing this manual. Jay Fenlason contributed many ideas and sample programs. Richard Mlynarik and Robert Chassell gave helpful comments on drafts of this manual. The paper A Supplemental Document for awk by John W. Pierce of the Chemistry Department at UC San Diego, pinpointed several issues relevant both to awk implementation and to this manual, that would otherwise have escaped us.

I would like to acknowledge Richard M. Stallman, for his vision of a better world and for his courage in founding the FSF and starting the GNU project.

The following people (in alphabetical order) provided helpful comments on various versions of this book, up to and including this edition. Rick Adams, Nelson H.F. Beebe, Karl Berry, Dr. Michael Brennan, Rich Burridge, Claire Coutier, Diane Close, Scott Deifik, Christopher ("Topher") Eliot, Jeffrey Friedl, Dr. Darrel Hankerson, Michal Jaegermann, Dr. Richard J. LeBlanc, Michael Lijewski, Pat Rankin, Miriam Robbins, Mary Sheehan, and Chuck Toporek.

Robert J. Chassell provided much valuable advice on the use of Texinfo. He also deserves special thanks for convincing me not to title this Web page How To Gawk Politely. Karl Berry helped significantly with the TeX part of Texinfo.

I would like to thank Marshall and Elaine Hartholz of Seattle and Dr. Bert and Rita Schreiber of Detroit for large amounts of quiet vacation time in their homes, which allowed me to make significant progress on this Web page and on gawk itself.

Phil Hughes of SSC contributed in a very important way by loaning me his laptop GNU/Linux system, not once, but twice, which allowed me to do a lot of work while away from home.

David Trueman deserves special credit; he has done a yeoman job of evolving gawk so that it performs well and without bugs. Although he is no longer involved with gawk, working with him on this project was a significant pleasure.

The intrepid members of the GNITS mailing list, and most notably Ulrich Drepper, provided invaluable help and feedback for the design of the internationalization features.

Nelson Beebe, Martin Brown, Scott Deifik, Darrel Hankerson, Michal Jaegermann, Jürgen Kahrs, Pat Rankin, Kai Uwe Rommel, and Eli Zaretskii (in alphabetical order) are long-time members of the gawk "crack portability team." Without their hard work and help, gawk would not be nearly the fine program it is today. It has been and continues to be a pleasure working with this team of fine people.

David and I would like to thank Brian Kernighan of Bell Laboratories for invaluable assistance during the testing and debugging of gawk, and for help in clarifying numerous points about the language. We could not have done nearly as good a job on either gawk or its documentation without his help.

Chuck Toporek, Mary Sheehan, and Claire Coutier of O'Reilly & Associates contributed significant editorial help for this Web page for the 3.1 release of gawk.

I must thank my wonderful wife, Miriam, for her patience through the many versions of this project, for her proof-reading, and for sharing me with the computer. I would like to thank my parents for their love, and for the grace with which they raised and educated me. Finally, I also must acknowledge my gratitude to G-d, for the many opportunities He has sent my way, as well as for the gifts He has given me with which to take advantage of those opportunities. Arnold Robbins
Nof Ayalon
ISRAEL
March, 2001

2. Getting Started with awk

The basic function of awk is to search files for lines (or other units of text) that contain certain patterns. When a line matches one of the patterns, awk performs specified actions on that line. awk keeps processing input lines in this way until it reaches the end of the input files.

Programs in awk are different from programs in most other languages, because awk programs are data-driven; that is, you describe the data you want to work with and then what to do when you find it. Most other languages are procedural; you have to describe, in great detail, every step the program is to take. When working with procedural languages, it is usually much harder to clearly describe the data your program will process. For this reason, awk programs are often refreshingly easy to write and read.

When you run awk, you specify an awk program that tells awk what to do. The program consists of a series of rules. (It may also contain function definitions, an advanced feature that we will ignore for now. See section User-Defined Functions.) Each rule specifies one pattern to search for and one action to perform upon finding the pattern.

Syntactically, a rule consists of a pattern followed by an action. The action is enclosed in curly braces to separate it from the pattern. Newlines usually separate rules. Therefore, an awk program looks like this:

pattern { action } pattern { action } ...

2.1 How to Run awk Programs		How to run gawk programs; includes command-line syntax.
2.2 Data Files for the Examples		Sample data files for use in the awk programs illustrated in this Web page.
2.3 Some Simple Examples		A very simple example.
2.4 An Example with Two Rules		A less simple one-line example using two rules.
2.5 A More Complex Example		A more complex example.
2.6 awk Statements Versus Lines		Subdividing or combining statements into lines.
2.7 Other Features of awk
2.8 When to Use awk		When to use gawk and when to use other things.

2.1 How to Run awk Programs

There are several ways to run an awk program. If the program is short, it is easiest to include it in the command that runs awk, like this:

awk 'program' input-file1 input-file2 ...

When the program is long, it is usually more convenient to put it in a file and run it with a command like this:

awk -f program-file input-file1 input-file2 ...

This section discusses both mechanisms, along with several variations of each.

2.1.1 One-Shot Throw-Away awk Programs		Running a short throw-away awk program.
2.1.2 Running awk Without Input Files		Using no input files (input from terminal instead).
2.1.3 Running Long Programs		Putting permanent awk programs in files.
2.1.4 Executable awk Programs		Making self-contained awk programs.
2.1.5 Comments in awk Programs		Adding documentation to gawk programs.
2.1.6 Shell Quoting Issues		More discussion of shell quoting issues.

2.1.1 One-Shot Throw-Away awk Programs

Once you are familiar with awk, you will often type in simple programs the moment you want to use them. Then you can write the program as the first argument of the awk command, like this:

awk 'program' input-file1 input-file2 ...

where program consists of a series of patterns and actions, as described earlier.

This command format instructs the shell, or command interpreter, to start awk and use the program to process records in the input file(s). There are single quotes around program so the shell won't interpret any awk characters as special shell characters. The quotes also cause the shell to treat all of program as a single argument for awk, and allow program to be more than one line long.

This format is also useful for running short or medium-sized awk programs from shell scripts, because it avoids the need for a separate file for the awk program. A self-contained shell script is more reliable because there are no other files to misplace.

Some Simple Examples, later in this chapter, presents several short, self-contained programs.

2.1.2 Running awk Without Input Files

You can also run awk without any input files. If you type the following command line:

awk 'program'

awk applies the program to the standard input, which usually means whatever you type on the terminal. This continues until you indicate end-of-file by typing Ctrl-d. (On other operating systems, the end-of-file character may be different. For example, on OS/2 and MS-DOS, it is Ctrl-z.)

As an example, the following program prints a friendly piece of advice (from Douglas Adams's The Hitchhiker's Guide to the Galaxy), to keep you from worrying about the complexities of computer programming. (BEGIN is a feature we haven't discussed yet.):

$ awk "BEGIN { print \"Don't Panic!\" }" -| Don't Panic!

This program does not read any input. The `\' before each of the inner double quotes is necessary because of the shell's quoting rules--in particular because it mixes both single quotes and double quotes.(6)

This next simple awk program emulates the cat utility; it copies whatever you type at the keyboard to its standard output. (Why this works is explained shortly.)

$ awk '{ print }' Now is the time for all good men -| Now is the time for all good men to come to the aid of their country. -| to come to the aid of their country. Four score and seven years ago, ... -| Four score and seven years ago, ... What, me worry? -| What, me worry? Ctrl-d

2.1.3 Running Long Programs

Sometimes your awk programs can be very long. In this case, it is more convenient to put the program into a separate file. In order to tell awk to use that file for its program, you type:

awk -f source-file input-file1 input-file2 ...

The `-f' instructs the awk utility to get the awk program from the file source-file. Any file name can be used for source-file. For example, you could put the program:

BEGIN { print "Don't Panic!" }

into the file `advice'. Then this command:

awk -f advice

does the same thing as this one:

awk "BEGIN { print \"Don't Panic!\" }"

This was explained earlier (see section Running awk Without Input Files). Note that you don't usually need single quotes around the file name that you specify with `-f', because most file names don't contain any of the shell's special characters. Notice that in `advice', the awk program did not have single quotes around it. The quotes are only needed for programs that are provided on the awk command line.

If you want to identify your awk program files clearly as such, you can add the extension `.awk' to the file name. This doesn't affect the execution of the awk program but it does make "housekeeping" easier.

2.1.4 Executable awk Programs

Once you have learned awk, you may want to write self-contained awk scripts, using the `#!' script mechanism. You can do this on many Unix systems(7) as well as on the GNU system. For example, you could update the file `advice' to look like this:

#! /bin/awk -f BEGIN { print "Don't Panic!" }

After making this file executable (with the chmod utility), simply type `advice' at the shell and the system arranges to run awk(8) as if you had typed `awk -f advice':

$ chmod +x advice $ advice -| Don't Panic!

Self-contained awk scripts are useful when you want to write a program that users can invoke without their having to know that the program is written in awk.

Advanced Notes: Portability Issues with `#!'

Some systems limit the length of the interpreter name to 32 characters. Often, this can be dealt with by using a symbolic link.

You should not put more than one argument on the `#!' line after the path to awk. It does not work. The operating system treats the rest of the line as a single argument and passes it to awk. Doing this leads to confusing behavior--most likely a usage diagnostic of some sort from awk.

Finally, the value of ARGV[0] (see section 7.5 Built-in Variables) varies depending upon your operating system. Some systems put `awk' there, some put the full pathname of awk (such as `/bin/awk'), and some put the name of your script (`advice'). Don't rely on the value of ARGV[0] to provide your script name.

2.1.5 Comments in awk Programs

A comment is some text that is included in a program for the sake of human readers; it is not really an executable part of the program. Comments can explain what the program does and how it works. Nearly all programming languages have provisions for comments, as programs are typically hard to understand without them.

In the awk language, a comment starts with the sharp sign character (`#') and continues to the end of the line. The `#' does not have to be the first character on the line. The awk language ignores the rest of a line following a sharp sign. For example, we could have put the following into `advice':

# This program prints a nice friendly message. It helps # keep novice users from being afraid of the computer. BEGIN { print "Don't Panic!" }

You can put comment lines into keyboard-composed throw-away awk programs, but this usually isn't very useful; the purpose of a comment is to help you or another person understand the program when reading it at a later time.

Caution: As mentioned in One-Shot Throw-Away awk Programs, you can enclose small to medium programs in single quotes, in order to keep your shell scripts self-contained. When doing so, don't put an apostrophe (i.e., a single quote) into a comment (or anywhere else in your program). The shell interprets the quote as the closing quote for the entire program. As a result, usually the shell prints a message about mismatched quotes, and if awk actually runs, it will probably print strange messages about syntax errors. For example, look at the following:

$ awk '{ print "hello" } # let's be cute' >

The shell sees that the first two quotes match, and that a new quoted object begins at the end of the command-line. It therefore prompts with the secondary prompt, waiting for more input. With Unix awk, closing the quoted string produces this result:

$ awk '{ print "hello" } # let's be cute' > ' error--> awk: can't open file be error--> source line number 1

Putting a backslash before the single quote in `let's' wouldn't help, since backslashes are not special inside single quotes. The next subsection describes the shell's quoting rules.

2.1.6 Shell Quoting Issues

For short to medium length awk programs, it is most convenient to enter the program on the awk command line. This is best done by enclosing the entire program in single quotes. This is true whether you are entering the program interactively at the shell prompt, or writing it as part of a larger shell script:

awk 'program text' input-file1 input-file2 ...

Once you are working with the shell, it is helpful to have a basic knowledge of shell quoting rules. The following rules apply only to POSIX-compliant, Bourne-style shells (such as bash, the GNU Bourne-Again Shell). If you use csh, you're on your own.

• Quoted items can be concatenated with nonquoted items as well as with other quoted items. The shell turns everything into one argument for the command.

• Preceding any single character with a backslash (`\') quotes that character. The shell removes the backslash and passes the quoted character on to the command.

• Single quotes protect everything between the opening and closing quotes. The shell does no interpretation of the quoted text, passing it on verbatim to the command. It is impossible to embed a single quote inside single-quoted text. Refer back to Comments in awk Programs, for an example showing what happens if you try.

• Double quotes protect most things between the opening and closing quotes. The shell does at least variable and command substitution on the quoted text. Different shells may do additional kinds of processing on double-quoted text.

  Since certain characters within double-quoted text are processed by the shell, they must be escaped within the text. Of note are the characters `$', ``', `\' and `"', all of which must be preceded by a backslash within double-quoted text if they are to be passed on literally to the program. (The leading backslash is stripped first.) Thus, the example seen previously in Running awk Without Input Files, is applicable:

  $ awk "BEGIN { print \"Don't Panic!\" }" -| Don't Panic!

  Note that the single quote is not special within double quotes.

• Null strings are removed when they occur as part of a non-null command-line argument, while explicit non-null objects are kept. For example, to specify that the field separator FS should be set to the null string, use:

  awk -F "" 'program' files # correct

  Don't use this:

  awk -F"" 'program' files # wrong!

  In the second case, awk will attempt to use the text of the program as the value of FS, and the first file name as the text of the program! This results in syntax errors at best, and confusing behavior at worst.

Mixing single and double quotes is difficult. You have to resort to shell quoting tricks, like this:

$ awk 'BEGIN { print "Here is a single quote <'"'"'>" }' -| Here is a single quote <'>

This program consists of three concatenated quoted strings. The first and the third are single-quoted, the second is double-quoted.

This can be "simplified" to:

$ awk 'BEGIN { print "Here is a single quote <'\''>" }' -| Here is a single quote <'>

Judge for yourself which of these two is the more readable.

Another option is to use double quotes, escaping the embedded, awk-level double quotes:

$ awk "BEGIN { print \"Here is a single quote <'>\" }" -| Here is a single quote <'>

This option is also painful, because double quotes, backslashes, and dollar signs are very common in awk programs.

If you really need both single and double quotes in your awk program, it is probably best to move it into a separate file, where the shell won't be part of the picture, and you can say what you mean.

2.2 Data Files for the Examples

Many of the examples in this Web page take their input from two sample data files. The first, called `BBS-list', represents a list of computer bulletin board systems together with information about those systems. The second data file, called `inventory-shipped', contains information about monthly shipments. In both files, each line is considered to be one record.

In the file `BBS-list', each record contains the name of a computer bulletin board, its phone number, the board's baud rate(s), and a code for the number of hours it is operational. An `A' in the last column means the board operates 24 hours a day. A `B' in the last column means the board only operates on evening and weekend hours. A `C' means the board operates only on weekends:

aardvark 555-5553 1200/300 B alpo-net 555-3412 2400/1200/300 A barfly 555-7685 1200/300 A bites 555-1675 2400/1200/300 A camelot 555-0542 300 C core 555-2912 1200/300 C fooey 555-1234 2400/1200/300 B foot 555-6699 1200/300 B macfoo 555-6480 1200/300 A sdace 555-3430 2400/1200/300 A sabafoo 555-2127 1200/300 C

The second data file, called `inventory-shipped', represents information about shipments during the year. Each record contains the month, the number of green crates shipped, the number of red boxes shipped, the number of orange bags shipped, and the number of blue packages shipped, respectively. There are 16 entries, covering the 12 months of last year and the first four months of the current year.

Jan 13 25 15 115 Feb 15 32 24 226 Mar 15 24 34 228 Apr 31 52 63 420 May 16 34 29 208 Jun 31 42 75 492 Jul 24 34 67 436 Aug 15 34 47 316 Sep 13 55 37 277 Oct 29 54 68 525 Nov 20 87 82 577 Dec 17 35 61 401 Jan 21 36 64 620 Feb 26 58 80 652 Mar 24 75 70 495 Apr 21 70 74 514

If you are reading this in GNU Emacs using Info, you can copy the regions of text showing these sample files into your own test files. This way you can try out the examples shown in the remainder of this document. You do this by using the command M-x write-region to copy text from the Info file into a file for use with awk (See section `Miscellaneous File Operations' in GNU Emacs Manual, for more information). Using this information, create your own `BBS-list' and `inventory-shipped' files and practice what you learn in this Web page.

If you are using the stand-alone version of Info, see Extracting Programs from Texinfo Source Files, for an awk program that extracts these data files from `gawk.texi', the Texinfo source file for this Info file.

2.3 Some Simple Examples

The following command runs a simple awk program that searches the input file `BBS-list' for the character string `foo'. (A string of characters is usually called a string. The term string is based on similar usage in English, such as "a string of pearls," or, "a string of cars in a train."):

awk '/foo/ { print $0 }' BBS-list

When lines containing `foo' are found, they are printed because `print $0' means print the current line. (Just `print' by itself means the same thing, so we could have written that instead.)

You will notice that slashes (`/') surround the string `foo' in the awk program. The slashes indicate that `foo' is the pattern to search for. This type of pattern is called a regular expression, which is covered in more detail later (see section Regular Expressions). The pattern is allowed to match parts of words. There are single quotes around the awk program so that the shell won't interpret any of it as special shell characters.

Here is what this program prints:

$ awk '/foo/ { print $0 }' BBS-list -| fooey 555-1234 2400/1200/300 B -| foot 555-6699 1200/300 B -| macfoo 555-6480 1200/300 A -| sabafoo 555-2127 1200/300 C

In an awk rule, either the pattern or the action can be omitted, but not both. If the pattern is omitted, then the action is performed for every input line. If the action is omitted, the default action is to print all lines that match the pattern.

Thus, we could leave out the action (the print statement and the curly braces) in the above example and the result would be the same: all lines matching the pattern `foo' are printed. By comparison, omitting the print statement but retaining the curly braces makes an empty action that does nothing (i.e., no lines are printed).

Many practical awk programs are just a line or two. Following is a collection of useful, short programs to get you started. Some of these programs contain constructs that haven't been covered yet. (The description of the program will give you a good idea of what is going on, but please read the rest of the Web page to become an awk expert!) Most of the examples use a data file named `data'. This is just a placeholder; if you use these programs yourself, substitute your own file names for `data'. For future reference, note that there is often more than one way to do things in awk. At some point, you may want to look back at these examples and see if you can come up with different ways to do the same things shown here:

• Print the length of the longest input line:

  awk '{ if (length($0) > max) max = length($0) } END { print max }' data

• Print every line that is longer than 80 characters:

  awk 'length($0) > 80' data

  The sole rule has a relational expression as its pattern and it has no action--so the default action, printing the record, is used.

• Print the length of the longest line in `data':

  expand data | awk '{ if (x < length()) x = length() } END { print "maximum line length is " x }'

  The input is processed by the expand utility to change tabs into spaces, so the widths compared are actually the right-margin columns.

• Print every line that has at least one field:

  awk 'NF > 0' data

  This is an easy way to delete blank lines from a file (or rather, to create a new file similar to the old file but from which the blank lines have been removed).

• Print seven random numbers from 0 to 100, inclusive:

  awk 'BEGIN { for (i = 1; i <= 7; i++) print int(101 * rand()) }'

• Print the total number of bytes used by files:

  ls -l files | awk '{ x += $5 } END { print "total bytes: " x }'

• Print the total number of kilobytes used by files:

  ls -l files | awk '{ x += $5 } END { print "total K-bytes: " (x + 1023)/1024 }'

• Print a sorted list of the login names of all users:

  awk -F: '{ print $1 }' /etc/passwd | sort

• Count lines in a file:

  awk 'END { print NR }' data

• Print the even-numbered lines in the data file:

  awk 'NR % 2 == 0' data

  If you use the expression `NR % 2 == 1' instead, it would print the odd-numbered lines.

2.4 An Example with Two Rules

The awk utility reads the input files one line at a time. For each line, awk tries the patterns of each of the rules. If several patterns match, then several actions are run in the order in which they appear in the awk program. If no patterns match, then no actions are run.

After processing all the rules that match the line (and perhaps there are none), awk reads the next line. (However, see section The next Statement, and also see section Using gawk's nextfile Statement). This continues until the end of the file is reached. For example, the following awk program contains two rules:

/12/ { print $0 } /21/ { print $0 }

The first rule has the string `12' as the pattern and `print $0' as the action. The second rule has the string `21' as the pattern and also has `print $0' as the action. Each rule's action is enclosed in its own pair of braces.

This program prints every line that contains the string `12' or the string `21'. If a line contains both strings, it is printed twice, once by each rule.

This is what happens if we run this program on our two sample data files, `BBS-list' and `inventory-shipped', as shown here:

$ awk '/12/ { print $0 } > /21/ { print $0 }' BBS-list inventory-shipped -| aardvark 555-5553 1200/300 B -| alpo-net 555-3412 2400/1200/300 A -| barfly 555-7685 1200/300 A -| bites 555-1675 2400/1200/300 A -| core 555-2912 1200/300 C -| fooey 555-1234 2400/1200/300 B -| foot 555-6699 1200/300 B -| macfoo 555-6480 1200/300 A -| sdace 555-3430 2400/1200/300 A -| sabafoo 555-2127 1200/300 C -| sabafoo 555-2127 1200/300 C -| Jan 21 36 64 620 -| Apr 21 70 74 514

Note how the line beginning with `sabafoo' in `BBS-list' was printed twice, once for each rule.

2.5 A More Complex Example

Now that we've mastered some simple tasks, let's look at what typical awk programs do. This example shows how awk can be used to summarize, select, and rearrange the output of another utility. It uses features that haven't been covered yet, so don't worry if you don't understand all the details:

ls -l | awk '$6 == "Nov" { sum += $5 } END { print sum }'

This command prints the total number of bytes in all the files in the current directory that were last modified in November (of any year). (9) The `ls -l' part of this example is a system command that gives you a listing of the files in a directory, including each file's size and the date the file was last modified. Its output looks like this:

-rw-r--r-- 1 arnold user 1933 Nov 7 13:05 Makefile -rw-r--r-- 1 arnold user 10809 Nov 7 13:03 awk.h -rw-r--r-- 1 arnold user 983 Apr 13 12:14 awk.tab.h -rw-r--r-- 1 arnold user 31869 Jun 15 12:20 awk.y -rw-r--r-- 1 arnold user 22414 Nov 7 13:03 awk1.c -rw-r--r-- 1 arnold user 37455 Nov 7 13:03 awk2.c -rw-r--r-- 1 arnold user 27511 Dec 9 13:07 awk3.c -rw-r--r-- 1 arnold user 7989 Nov 7 13:03 awk4.c

The first field contains read-write permissions, the second field contains the number of links to the file, and the third field identifies the owner of the file. The fourth field identifies the group of the file. The fifth field contains the size of the file in bytes. The sixth, seventh and eighth fields contain the month, day, and time, respectively, that the file was last modified. Finally, the ninth field contains the name of the file.(10)

The `$6 == "Nov"' in our awk program is an expression that tests whether the sixth field of the output from `ls -l' matches the string `Nov'. Each time a line has the string `Nov' for its sixth field, the action `sum += $5' is performed. This adds the fifth field (the file's size) to the variable sum. As a result, when awk has finished reading all the input lines, sum is the total of the sizes of the files whose lines matched the pattern. (This works because awk variables are automatically initialized to zero.)

After the last line of output from ls has been processed, the END rule executes and prints the value of sum. In this example, the value of sum is 140963.

These more advanced awk techniques are covered in later sections (see section Actions). Before you can move on to more advanced awk programming, you have to know how awk interprets your input and displays your output. By manipulating fields and using print statements, you can produce some very useful and impressive looking reports.

2.6 awk Statements Versus Lines

Most often, each line in an awk program is a separate statement or separate rule, like this:

awk '/12/ { print $0 } /21/ { print $0 }' BBS-list inventory-shipped

However, gawk ignores newlines after any of the following symbols and keywords:

, { ? : || && do else

A newline at any other point is considered the end of the statement.(11)

If you would like to split a single statement into two lines at a point where a newline would terminate it, you can continue it by ending the first line with a backslash character (`\'). The backslash must be the final character on the line in order to be recognized as a continuation character. A backslash is allowed anywhere in the statement, even in the middle of a string or regular expression. For example:

awk '/This regular expression is too long, so continue it\ on the next line/ { print $1 }'

We have generally not used backslash continuation in the sample programs in this Web page. In gawk, there is no limit on the length of a line, so backslash continuation is never strictly necessary; it just makes programs more readable. For this same reason, as well as for clarity, we have kept most statements short in the sample programs presented throughout the Web page. Backslash continuation is most useful when your awk program is in a separate source file instead of entered from the command line. You should also note that many awk implementations are more particular about where you may use backslash continuation. For example, they may not allow you to split a string constant using backslash continuation. Thus, for maximum portability of your awk programs, it is best not to split your lines in the middle of a regular expression or a string.

Caution: Backslash continuation does not work as described above with the C shell. It works for awk programs in files and for one-shot programs, provided you are using a POSIX-compliant shell, such as the Unix Bourne shell or bash. But the C shell behaves differently! There, you must use two backslashes in a row, followed by a newline. Note also that when using the C shell, every newline in your awk program must be escaped with a backslash. To illustrate:

% awk 'BEGIN { \ ? print \\ ? "hello, world" \ ? }' -| hello, world

Here, the `%' and `?' are the C shell's primary and secondary prompts, analogous to the standard shell's `$' and `>'.

Compare the previous example to how it is done with a POSIX-compliant shell:

$ awk 'BEGIN { > print \ > "hello, world" > }' -| hello, world

awk is a line-oriented language. Each rule's action has to begin on the same line as the pattern. To have the pattern and action on separate lines, you must use backslash continuation; there is no other way.

Another thing to keep in mind is that backslash continuation and comments do not mix. As soon as awk sees the `#' that starts a comment, it ignores everything on the rest of the line. For example:

$ gawk 'BEGIN { print "dont panic" # a friendly \ > BEGIN rule > }' error--> gawk: cmd. line:2: BEGIN rule error--> gawk: cmd. line:2: ^ parse error

In this case, it looks like the backslash would continue the comment onto the next line. However, the backslash-newline combination is never even noticed because it is "hidden" inside the comment. Thus, the BEGIN is noted as a syntax error.

When awk statements within one rule are short, you might want to put more than one of them on a line. This is accomplished by separating the statements with a semicolon (`;'). This also applies to the rules themselves. Thus, the program shown at the start of this section could also be written this way:

/12/ { print $0 } ; /21/ { print $0 }

Note: The requirement that states that rules on the same line must be separated with a semicolon was not in the original awk language; it was added for consistency with the treatment of statements within an action.

2.7 Other Features of awk

The awk language provides a number of predefined, or built-in, variables that your programs can use to get information from awk. There are other variables your program can set as well to control how awk processes your data.

In addition, awk provides a number of built-in functions for doing common computational and string related operations. gawk provides built-in functions for working with timestamps, performing bit manipulation, and for runtime string translation.

As we develop our presentation of the awk language, we introduce most of the variables and many of the functions. They are defined systematically in 7.5 Built-in Variables, and Built-in Functions.

2.8 When to Use awk

Now that you've seen some of what awk can do, you might wonder how awk could be useful for you. By using utility programs, advanced patterns, field separators, arithmetic statements, and other selection criteria, you can produce much more complex output. The awk language is very useful for producing reports from large amounts of raw data, such as summarizing information from the output of other utility programs like ls. (See section A More Complex Example.)

Programs written with awk are usually much smaller than they would be in other languages. This makes awk programs easy to compose and use. Often, awk programs can be quickly composed at your terminal, used once, and thrown away. Because awk programs are interpreted, you can avoid the (usually lengthy) compilation part of the typical edit-compile-test-debug cycle of software development.

Complex programs have been written in awk, including a complete retargetable assembler for eight-bit microprocessors (see section Glossary, for more information), and a microcode assembler for a special purpose Prolog computer. However, awk's capabilities are strained by tasks of such complexity.

If you find yourself writing awk scripts of more than, say, a few hundred lines, you might consider using a different programming language. Emacs Lisp is a good choice if you need sophisticated string or pattern matching capabilities. The shell is also good at string and pattern matching; in addition, it allows powerful use of the system utilities. More conventional languages, such as C, C++, and Java, offer better facilities for system programming and for managing the complexity of large programs. Programs in these languages may require more lines of source code than the equivalent awk programs, but they are easier to maintain and usually run more efficiently.

3. Regular Expressions

A regular expression, or regexp, is a way of describing a set of strings. Because regular expressions are such a fundamental part of awk programming, their format and use deserve a separate chapter.

A regular expression enclosed in slashes (`/') is an awk pattern that matches every input record whose text belongs to that set. The simplest regular expression is a sequence of letters, numbers, or both. Such a regexp matches any string that contains that sequence. Thus, the regexp `foo' matches any string containing `foo'. Therefore, the pattern /foo/ matches any input record containing the three characters `foo' anywhere in the record. Other kinds of regexps let you specify more complicated classes of strings.

Initially, the examples in this chapter are simple. As we explain more about how regular expressions work, we will present more complicated instances.

3.1 How to Use Regular Expressions
3.2 Escape Sequences		How to write non-printing characters.
3.3 Regular Expression Operators
3.4 Using Character Lists		What can go between `[...]'.
3.5 gawk-Specific Regexp Operators		Operators specific to GNU software.
3.6 Case Sensitivity in Matching		How to do case-insensitive matching.
3.7 How Much Text Matches?		How much text matches.
3.8 Using Dynamic Regexps

3.1 How to Use Regular Expressions

A regular expression can be used as a pattern by enclosing it in slashes. Then the regular expression is tested against the entire text of each record. (Normally, it only needs to match some part of the text in order to succeed.) For example, the following prints the second field of each record that contains the string `foo' anywhere in it:

$ awk '/foo/ { print $2 }' BBS-list -| 555-1234 -| 555-6699 -| 555-6480 -| 555-2127

Regular expressions can also be used in matching expressions. These expressions allow you to specify the string to match against; it need not be the entire current input record. The two operators `~' and `!~' perform regular expression comparisons. Expressions using these operators can be used as patterns, or in if, while, for, and do statements. (See section Control Statements in Actions.) For example:

exp ~ /regexp/

is true if the expression exp (taken as a string) matches regexp. The following example matches, or selects, all input records with the uppercase letter `J' somewhere in the first field:

$ awk '$1 ~ /J/' inventory-shipped -| Jan 13 25 15 115 -| Jun 31 42 75 492 -| Jul 24 34 67 436 -| Jan 21 36 64 620

So does this:

awk '{ if ($1 ~ /J/) print }' inventory-shipped

This next example is true if the expression exp (taken as a character string) does not match regexp:

exp !~ /regexp/

The following example matches, or selects, all input records whose first field does not contain the uppercase letter `J':

$ awk '$1 !~ /J/' inventory-shipped -| Feb 15 32 24 226 -| Mar 15 24 34 228 -| Apr 31 52 63 420 -| May 16 34 29 208 ...

When a regexp is enclosed in slashes, such as /foo/, we call it a regexp constant, much like 5.27 is a numeric constant and "foo" is a string constant.

3.2 Escape Sequences

Some characters cannot be included literally in string constants ("foo") or regexp constants (/foo/). Instead, they should be represented with escape sequences, which are character sequences beginning with a backslash (`\'). One use of an escape sequence is to include a double quote character in a string constant. Because a plain double quote ends the string, you must use `\"' to represent an actual double quote character as a part of the string. For example:

$ awk 'BEGIN { print "He said \"hi!\" to her." }' -| He said "hi!" to her.

The backslash character itself is another character that cannot be included normally; you must write `\\' to put one backslash in the string or regexp. Thus, the string whose contents are the two characters `"' and `\' must be written "\"\\".

Another use of backslash is to represent unprintable characters such as tab or newline. While there is nothing to stop you from entering most unprintable characters directly in a string constant or regexp constant, they may look ugly.

The following table lists all the escape sequences used in awk and what they represent. Unless noted otherwise, all these escape sequences apply to both string constants and regexp constants:

\\

A literal backslash, `\'.

\a

The "alert" character, Ctrl-g, ASCII code 7 (BEL). (This usually makes some sort of audible noise.)

\b

Backspace, Ctrl-h, ASCII code 8 (BS).

\f

Formfeed, Ctrl-l, ASCII code 12 (FF).

\n

Newline, Ctrl-j, ASCII code 10 (LF).

\r

Carriage return, Ctrl-m, ASCII code 13 (CR).

\t

Horizontal tab, Ctrl-i, ASCII code 9 (HT).

\v

Vertical tab, Ctrl-k, ASCII code 11 (VT).

\nnn

The octal value nnn, where nnn stands for 1 to 3 digits between `0' and `7'. For example, the code for the ASCII ESC (escape) character is `\033'.

\xhh...

The hexadecimal value hh, where hh stands for a sequence of hexadecimal digits (`0' through `9', and either `A' through `F' or `a' through `f'). Like the same construct in ISO C, the escape sequence continues until the first non-hexadecimal digit is seen. However, using more than two hexadecimal digits produces undefined results. (The `\x' escape sequence is not allowed in POSIX awk.)

\/

A literal slash (necessary for regexp constants only). This expression is used when you want to write a regexp constant that contains a slash. Because the regexp is delimited by slashes, you need to escape the slash that is part of the pattern, in order to tell awk to keep processing the rest of the regexp.

\"

A literal double quote (necessary for string constants only). This expression is used when you want to write a string constant that contains a double quote. Because the string is delimited by double quotes, you need to escape the quote that is part of the string, in order to tell awk to keep processing the rest of the string.

In gawk, a number of additional two-character sequences that begin with a backslash have special meaning in regexps. See section gawk-Specific Regexp Operators.

In a regexp, a backslash before any character that is not in the above table and not listed in gawk-Specific Regexp Operators, means that the next character should be taken literally, even if it would normally be a regexp operator. For example, /a\+b/ matches the three characters `a+b'.

For complete portability, do not use a backslash before any character not shown in the table above.

To summarize:

• The escape sequences in the table above are always processed first, for both string constants and regexp constants. This happens very early, as soon as awk reads your program.

• gawk processes both regexp constants and dynamic regexps (see section Using Dynamic Regexps), for the special operators listed in gawk-Specific Regexp Operators.

• A backslash before any other character means to treat that character literally.

Advanced Notes: Backslash Before Regular Characters

If you place a backslash in a string constant before something that is not one of the characters listed above, POSIX awk purposely leaves what happens as undefined. There are two choices:

Strip the backslash out

This is what Unix awk and gawk both do. For example, "a\qc" is the same as "aqc". (Because this is such an easy bug to both introduce and to miss, gawk warns you about it.) Consider `FS = "[ \t]+\|[ \t]+"' to use vertical bars surrounded by whitespace as the field separator. There should be two backslashes in the string, `FS = "[ \t]+\\|[ \t]+"'.)

Leave the backslash alone

Some other awk implementations do this. In such implementations, "a\qc" is the same as if you had typed "a\\qc".

Advanced Notes: Escape Sequences for Metacharacters

Suppose you use an octal or hexadecimal escape to represent a regexp metacharacter (see section Regular Expression Operators). Does awk treat the character as a literal character or as a regexp operator?

Historically, such characters were taken literally. (d.c.) However, the POSIX standard indicates that they should be treated as real metacharacters, which is what gawk does. In compatibility mode (see section Command-Line Options), gawk treats the characters represented by octal and hexadecimal escape sequences literally when used in regexp constants. Thus, /a\52b/ is equivalent to /a\*b/.

3.3 Regular Expression Operators

You can combine regular expressions with special characters, called regular expression operators or metacharacters, to increase the power and versatility of regular expressions.

The escape sequences described earlier in 3.2 Escape Sequences, are valid inside a regexp. They are introduced by a `\', and are recognized and converted into the corresponding real characters as the very first step in processing regexps.

Here is a list of metacharacters. All characters that are not escape sequences and that are not listed in the table stand for themselves:

\

This is used to suppress the special meaning of a character when matching. For example, `\$' matches the character `$'.

^

This matches the beginning of a string. For example, `^@chapter' matches `@chapter' at the beginning of a string, and can be used to identify chapter beginnings in Texinfo source files. The `^' is known as an anchor, because it anchors the pattern to match only at the beginning of the string.

It is important to realize that `^' does not match the beginning of a line embedded in a string. The condition is not true in the following example:

if ("line1\nLINE 2" ~ /^L/) ...

$

This is similar to `^' but it matches only at the end of a string. For example, `p$' matches a record that ends with a `p'. The `$' is an anchor and does not match the end of a line embedded in a string. The condition is not true in the following example:

if ("line1\nLINE 2" ~ /1$/) ...

.

This matches any single character, including the newline character. For example, `.P' matches any single character followed by a `P' in a string. Using concatenation, we can make a regular expression such as `U.A', that matches any three-character sequence that begins with `U' and ends with `A'.

In strict POSIX mode (see section Command-Line Options), `.' does not match the NUL character, which is a character with all bits equal to zero. Otherwise, NUL is just another character. Other versions of awk may not be able to match the NUL character.

[...]

This is called a character list.(12) It matches any one of the characters that are enclosed in the square brackets. For example, `[MVX]' matches any one of the characters `M', `V', or `X', in a string. A full discussion of what can be inside the square brackets of a character list is given in Using Character Lists.

[^ ...]

This is a complemented character list. The first character after the `[' must be a `^'. It matches any characters except those in the square brackets. For example, `[^awk]' matches any character that is not an `a', a `w', or a `k'.

|

This is the alternation operator and it is used to specify alternatives. The `|' has the lowest precedence of all the regular expression operators. For example, `^P|[[:digit:]]' matches any string that matches either `^P' or `[[:digit:]]'. This means it matches any string that starts with `P' or contains a digit.

The alternation applies to the largest possible regexps on either side.

(...)

Parentheses are used for grouping in regular expressions, similar to arithmetic. They can be used to concatenate regular expressions containing the alternation operator, `|'. For example, `@(samp|code)\{[^}]+\}' matches both `@code{foo}' and `@samp{bar}'. (These are Texinfo formatting control sequences.)

*

This symbol means that the preceding regular expression should be repeated as many times as necessary to find a match. For example, `ph*' applies the `*' symbol to the preceding `h' and looks for matches of one `p' followed by any number of `h's. This also matches just `p' if no `h's are present.

The `*' repeats the smallest possible preceding expression. (Use parentheses if you want to repeat a larger expression.) It finds as many repetitions as possible. For example, `awk '/\(c[ad][ad]*r x\)/ { print }' sample' prints every record in `sample' containing a string of the form `(car x)', `(cdr x)', `(cadr x)', and so on. Notice the escaping of the parentheses by preceding them with backslashes.

+

This symbol is similar to `*' except that the preceding expression must be matched at least once. This means that `wh+y' would match `why' and `whhy', but not `wy', whereas `wh*y' would match all three of these strings. The following is a simpler way of writing the last `*' example:

awk '/\(c[ad]+r x\)/ { print }' sample

?

This symbol is similar to `*' except that the preceding expression can be matched either once or not at all. For example, `fe?d' matches `fed' and `fd', but nothing else.

{n}

{n,}

{n,m}

One or two numbers inside braces denote an interval expression. If there is one number in the braces, the preceding regexp is repeated n times. If there are two numbers separated by a comma, the preceding regexp is repeated n to m times. If there is one number followed by a comma, then the preceding regexp is repeated at least n times:

wh{3}y

Matches `whhhy', but not `why' or `whhhhy'.

wh{3,5}y

Matches `whhhy', `whhhhy', or `whhhhhy', only.

wh{2,}y

Matches `whhy' or `whhhy', and so on.

Interval expressions were not traditionally available in awk. They were added as part of the POSIX standard to make awk and egrep consistent with each other.

However, because old programs may use `{' and `}' in regexp constants, by default gawk does not match interval expressions in regexps. If either `--posix' or `--re-interval' are specified (see section Command-Line Options), then interval expressions are allowed in regexps.

For new programs that use `{' and `}' in regexp constants, it is good practice to always escape them with a backslash. Then the regexp constants are valid and work the way you want them to, using any version of awk.(13)

In regular expressions, the `*', `+', and `?' operators, as well as the braces `{' and `}', have the highest precedence, followed by concatenation, and finally by `|'. As in arithmetic, parentheses can change how operators are grouped.

In POSIX awk and gawk, the `*', `+', and `?' operators stand for themselves when there is nothing in the regexp that precedes them. For example, `/+/' matches a literal plus sign. However, many other versions of awk treat such a usage as a syntax error.

If gawk is in compatibility mode (see section Command-Line Options), POSIX character classes and interval expressions are not available in regular expressions.

3.4 Using Character Lists

Within a character list, a range expression consists of two characters separated by a hyphen. It matches any single character that sorts between the two characters, using the locale's collating sequence and character set. For example, in the default C locale, `[a-dx-z]' is equivalent to `[abcdxyz]'. Many locales sort characters in dictionary order, and in these locales, `[a-dx-z]' is typically not equivalent to `[abcdxyz]'; instead it might be equivalent to `[aBbCcDdxXyYz]', for example. To obtain the traditional interpretation of bracket expressions, you can use the C locale by setting the LC_ALL environment variable to the value `C'.

To include one of the characters `\', `]', `-', or `^' in a character list, put a `\' in front of it. For example:

[d\]]

matches either `d' or `]'.

This treatment of `\' in character lists is compatible with other awk implementations and is also mandated by POSIX. The regular expressions in awk are a superset of the POSIX specification for Extended Regular Expressions (EREs). POSIX EREs are based on the regular expressions accepted by the traditional egrep utility.

Character classes are a new feature introduced in the POSIX standard. A character class is a special notation for describing lists of characters that have a specific attribute, but the actual characters can vary from country to country and/or from character set to character set. For example, the notion of what is an alphabetic character differs between the United States and France.

A character class is only valid in a regexp inside the brackets of a character list. Character classes consist of `[:', a keyword denoting the class, and `:]'. Here are the character classes defined by the POSIX standard:

(A space is printable but not visible, whereas an `a' is both.) control characters, or space characters).

[:alnum:]	Alphanumeric characters.
[:alpha:]	Alphabetic characters.
[:blank:]	Space and tab characters.
[:cntrl:]	Control characters.
[:digit:]	Numeric characters.
[:graph:]	Characters that are both printable and visible.
[:lower:]	Lowercase alphabetic characters.
[:print:]	Printable characters (characters that are not control characters).
[:punct:]	Punctuation characters (characters that are not letters, digits,
[:space:]	Space characters (such as space, tab, and formfeed, to name a few).
[:upper:]	Uppercase alphabetic characters.
[:xdigit:]	Characters that are hexadecimal digits.

For example, before the POSIX standard, you had to write /[A-Za-z0-9]/ to match alphanumeric characters. If your character set had other alphabetic characters in it, this would not match them, and if your character set collated differently from ASCII, this might not even match the ASCII alphanumeric characters. With the POSIX character classes, you can write /[[:alnum:]]/ to match the alphabetic and numeric characters in your character set.

Two additional special sequences can appear in character lists. These apply to non-ASCII character sets, which can have single symbols (called collating elements) that are represented with more than one character. They can also have several characters that are equivalent for collating, or sorting, purposes. (For example, in French, a plain "e" and a grave-accented "è" are equivalent.)

Collating Symbols

A collating symbol is a multicharacter collating element enclosed between `[.' and `.]'. For example, if `ch' is a collating element, then [[.ch.]] is a regexp that matches this collating element, whereas [ch] is a regexp that matches either `c' or `h'.

Equivalence Classes

An equivalence class is a locale-specific name for a list of characters that are equal. The name is enclosed between `[=' and `=]'. For example, the name `e' might be used to represent all of "e," "è," and "é." In this case, [[=e=]] is a regexp that matches any of `e', `é', or `è'.

These features are very valuable in non-English speaking locales.

Caution: The library functions that gawk uses for regular expression matching currently only recognize POSIX character classes; they do not recognize collating symbols or equivalence classes.

3.5 gawk-Specific Regexp Operators

GNU software that deals with regular expressions provides a number of additional regexp operators. These operators are described in this section and are specific to gawk; they are not available in other awk implementations. Most of the additional operators deal with word matching. For our purposes, a word is a sequence of one or more letters, digits, or underscores (`_'):

\w

Matches any word-constituent character--that is, it matches any letter, digit, or underscore. Think of it as short-hand for [[:alnum:]_].

\W

Matches any character that is not word-constituent. Think of it as short-hand for [^[:alnum:]_].

\<

Matches the empty string at the beginning of a word. For example, /\<away/ matches `away' but not `stowaway'.

\>

Matches the empty string at the end of a word. For example, /stow\>/ matches `stow' but not `stowaway'.

\y

Matches the empty string at either the beginning or the end of a word (i.e., the word boundary). For example, `\yballs?\y' matches either `ball' or `balls', as a separate word.

\B

Matches the empty string that occurs between two word-constituent characters. For example, /\Brat\B/ matches `crate' but it does not match `dirty rat'. `\B' is essentially the opposite of `\y'.

There are two other operators that work on buffers. In Emacs, a buffer is, naturally, an Emacs buffer. For other programs, gawk's regexp library routines consider the entire string to match as the buffer.

\`

Matches the empty string at the beginning of a buffer (string).

\'

Matches the empty string at the end of a buffer (string).

Because `^' and `$' always work in terms of the beginning and end of strings, these operators don't add any new capabilities for awk. They are provided for compatibility with other GNU software.

In other GNU software, the word-boundary operator is `\b'. However, that conflicts with the awk language's definition of `\b' as backspace, so gawk uses a different letter. An alternative method would have been to require two backslashes in the GNU operators, but this was deemed too confusing. The current method of using `\y' for the GNU `\b' appears to be the lesser of two evils.

The various command-line options (see section Command-Line Options) control how gawk interprets characters in regexps:

No options

In the default case, gawk provides all the facilities of POSIX regexps and the previously described GNU regexp operators. GNU regexp operators described in Regular Expression Operators. However, interval expressions are not supported.

--posix

Only POSIX regexps are supported; the GNU operators are not special (e.g., `\w' matches a literal `w'). Interval expressions are allowed.

--traditional

Traditional Unix awk regexps are matched. The GNU operators are not special, interval expressions are not available, nor are the POSIX character classes ([[:alnum:]] and so on). Characters described by octal and hexadecimal escape sequences are treated literally, even if they represent regexp metacharacters.

--re-interval

Allow interval expressions in regexps, even if `--traditional' has been provided.

3.6 Case Sensitivity in Matching

Case is normally significant in regular expressions, both when matching ordinary characters (i.e., not metacharacters) and inside character sets. Thus, a `w' in a regular expression matches only a lowercase `w' and not an uppercase `W'.

The simplest way to do a case-independent match is to use a character list--for example, `[Ww]'. However, this can be cumbersome if you need to use it often and it can make the regular expressions harder to read. There are two alternatives that you might prefer.

One way to perform a case-insensitive match at a particular point in the program is to convert the data to a single case, using the tolower or toupper built-in string functions (which we haven't discussed yet; see section String Manipulation Functions). For example:

tolower($1) ~ /foo/ { ... }

converts the first field to lowercase before matching against it. This works in any POSIX-compliant awk.

Another method, specific to gawk, is to set the variable IGNORECASE to a nonzero value (see section 7.5 Built-in Variables). When IGNORECASE is not zero, all regexp and string operations ignore case. Changing the value of IGNORECASE dynamically controls the case sensitivity of the program as it runs. Case is significant by default because IGNORECASE (like most variables) is initialized to zero:

x = "aB" if (x ~ /ab/) ... # this test will fail IGNORECASE = 1 if (x ~ /ab/) ... # now it will succeed

In general, you cannot use IGNORECASE to make certain rules case-insensitive and other rules case-sensitive, because there is no straightforward way to set IGNORECASE just for the pattern of a particular rule.(14) To do this, use either character lists or tolower. However, one thing you can do with IGNORECASE only is dynamically turn case-sensitivity on or off for all the rules at once.

IGNORECASE can be set on the command line or in a BEGIN rule (see section Other Command-Line Arguments; also see section Startup and Cleanup Actions). Setting IGNORECASE from the command line is a way to make a program case-insensitive without having to edit it.

Prior to gawk 3.0, the value of IGNORECASE affected regexp operations only. It did not affect string comparison with `==', `!=', and so on. Beginning with version 3.0, both regexp and string comparison operations are also affected by IGNORECASE.

Beginning with gawk 3.0, the equivalences between upper- and lowercase characters are based on the ISO-8859-1 (ISO Latin-1) character set. This character set is a superset of the traditional 128 ASCII characters, that also provides a number of characters suitable for use with European languages.

The value of IGNORECASE has no effect if gawk is in compatibility mode (see section Command-Line Options). Case is always significant in compatibility mode.

3.7 How Much Text Matches?

Consider the following:

echo aaaabcd | awk '{ sub(/a+/, "<A>"); print }'

This example uses the sub function (which we haven't discussed yet; see section String Manipulation Functions) to make a change to the input record. Here, the regexp /a+/ indicates "one or more `a' characters," and the replacement text is `<A>'.

The input contains four `a' characters. awk (and POSIX) regular expressions always match the leftmost, longest sequence of input characters that can match. Thus, all four `a' characters are replaced with `<A>' in this example:

$ echo aaaabcd | awk '{ sub(/a+/, "<A>"); print }' -| <A>bcd

For simple match/no-match tests, this is not so important. But when doing text matching and substitutions with the match, sub, gsub, and gensub functions, it is very important. See section String Manipulation Functions, for more information on these functions. Understanding this principle is also important for regexp-based record and field splitting (see section How Input Is Split into Records, and also see section Specifying How Fields Are Separated).

3.8 Using Dynamic Regexps

The righthand side of a `~' or `!~' operator need not be a regexp constant (i.e., a string of characters between slashes). It may be any expression. The expression is evaluated and converted to a string if necessary; the contents of the string are used as the regexp. A regexp that is computed in this way is called a dynamic regexp:

BEGIN { digits_regexp = "[[:digit:]]+" } $0 ~ digits_regexp { print }

This sets digits_regexp to a regexp that describes one or more digits, and tests whether the input record matches this regexp.

When using the `~' and `!~' Caution: When using the `~' and `!~' operators, there is a difference between a regexp constant enclosed in slashes and a string constant enclosed in double quotes. If you are going to use a string constant, you have to understand that the string is, in essence, scanned twice: the first time when awk reads your program, and the second time when it goes to match the string on the lefthand side of the operator with the pattern on the right. This is true of any string valued expression (such as digits_regexp shown previously), not just string constants.

What difference does it make if the string is scanned twice? The answer has to do with escape sequences, and particularly with backslashes. To get a backslash into a regular expression inside a string, you have to type two backslashes.

For example, /\*/ is a regexp constant for a literal `*'. Only one backslash is needed. To do the same thing with a string, you have to type "\\*". The first backslash escapes the second one so that the string actually contains the two characters `\' and `*'.

Given that you can use both regexp and string constants to describe regular expressions, which should you use? The answer is "regexp constants," for several reasons:

• String constants are more complicated to write and more difficult to read. Using regexp constants makes your programs less error-prone. Not understanding the difference between the two kinds of constants is a common source of errors.

• It is more efficient to use regexp constants. awk can note that you have supplied a regexp, and store it internally in a form that makes pattern matching more efficient. When using a string constant, awk must first convert the string into this internal form and then perform the pattern matching.

• Using regexp constants is better form; it shows clearly that you intend a regexp match.

Advanced Notes: Using \n in Character Lists of Dynamic Regexps

Some commercial versions of awk do not allow the newline character to be used inside a character list for a dynamic regexp:

$ awk '$0 ~ "[ \t\n]"' error--> awk: newline in character class [ error--> ]... error--> source line number 1 error--> context is error--> >>> <<<

But a newline in a regexp constant works with no problem:

$ awk '$0 ~ /[ \t\n]/' here is a sample line -| here is a sample line Ctrl-d

gawk does not have this problem, and it isn't likely to occur often in practice, but it's worth noting for future reference.

4. Reading Input Files

In the typical awk program, all input is read either from the standard input (by default, this is the keyboard but often it is a pipe from another command), or from files whose names you specify on the awk command line. If you specify input files, awk reads them in order, processing all the data from one before going on to the next. The name of the current input file can be found in the built-in variable FILENAME (see section 7.5 Built-in Variables).

The input is read in units called records, and is processed by the rules of your program one record at a time. By default, each record is one line. Each record is automatically split into chunks called fields. This makes it more convenient for programs to work on the parts of a record.

On rare occasions, you may need to use the getline command. The getline command is valuable, both because it can do explicit input from any number of files, and because the files used with it do not have to be named on the awk command line (see section Explicit Input with getline).

4.1 How Input Is Split into Records		Controlling how data is split into records.
4.2 Examining Fields		An introduction to fields.
4.3 Non-Constant Field Numbers		Non-constant Field Numbers.
4.4 Changing the Contents of a Field
4.5 Specifying How Fields Are Separated		The field separator and how to change it.
4.6 Reading Fixed-Width Data		Reading constant width data.
4.7 Multiple-Line Records		Reading multi-line records.
4.8 Explicit Input with getline		Reading files under explicit program control using the getline function.

4.1 How Input Is Split into Records

The awk utility divides the input for your awk program into records and fields. awk keeps track of the number of records that have been read so far from the current input file. This value is stored in a built-in variable called FNR. It is reset to zero when a new file is started. Another built-in variable, NR, is the total number of input records read so far from all data files. It starts at zero, but is never automatically reset to zero.

Records are separated by a character called the record separator. By default, the record separator is the newline character. This is why records are, by default, single lines. A different character can be used for the record separator by assigning the character to the built-in variable RS.

Like any other variable, the value of RS can be changed in the awk program with the assignment operator, `=' (see section Assignment Expressions). The new record-separator character should be enclosed in quotation marks, which indicate a string constant. Often the right time to do this is at the beginning of execution, before any input is processed, so that the very first record is read with the proper separator. To do this, use the special BEGIN pattern (see section The BEGIN and END Special Patterns). For example:

awk 'BEGIN { RS = "/" } { print $0 }' BBS-list

changes the value of RS to "/", before reading any input. This is a string whose first character is a slash; as a result, records are separated by slashes. Then the input file is read, and the second rule in the awk program (the action with no pattern) prints each record. Because each print statement adds a newline at the end of its output, the effect of this awk program is to copy the input with each slash changed to a newline. Here are the results of running the program on `BBS-list':

$ awk 'BEGIN { RS = "/" } > { print $0 }' BBS-list -| aardvark 555-5553 1200 -| 300 B -| alpo-net 555-3412 2400 -| 1200 -| 300 A -| barfly 555-7685 1200 -| 300 A -| bites 555-1675 2400 -| 1200 -| 300 A -| camelot 555-0542 300 C -| core 555-2912 1200 -| 300 C -| fooey 555-1234 2400 -| 1200 -| 300 B -| foot 555-6699 1200 -| 300 B -| macfoo 555-6480 1200 -| 300 A -| sdace 555-3430 2400 -| 1200 -| 300 A -| sabafoo 555-2127 1200 -| 300 C -|

Note that the entry for the `camelot' BBS is not split. In the original data file (see section Data Files for the Examples), the line looks like this:

camelot 555-0542 300 C

It has one baud rate only, so there are no slashes in the record, unlike the others which have two or more baud rates. In fact, this record is treated as part of the record for the `core' BBS; the newline separating them in the output is the original newline in the data file, not the one added by awk when it printed the record!

Another way to change the record separator is on the command line, using the variable-assignment feature (see section Other Command-Line Arguments):

awk '{ print $0 }' RS="/" BBS-list

This sets RS to `/' before processing `BBS-list'.

Using an unusual character such as `/' for the record separator produces correct behavior in the vast majority of cases. However, the following (extreme) pipeline prints a surprising `1':

$ echo | awk 'BEGIN { RS = "a" } ; { print NF }' -| 1

There is one field, consisting of a newline. The value of the built-in variable NF is the number of fields in the current record.

Reaching the end of an input file terminates the current input record, even if the last character in the file is not the character in RS. (d.c.)

The empty string "" (a string without any characters) has a special meaning as the value of RS. It means that records are separated by one or more blank lines and nothing else. See section Multiple-Line Records, for more details.

If you change the value of RS in the middle of an awk run, the new value is used to delimit subsequent records, but the record currently being processed, as well as records already processed, are not affected.

After the end of the record has been determined, gawk sets the variable RT to the text in the input that matched RS. When using gawk, the value of RS is not limited to a one-character string. It can be any regular expression (see section Regular Expressions). In general, each record ends at the next string that matches the regular expression; the next record starts at the end of the matching string. This general rule is actually at work in the usual case, where RS contains just a newline: a record ends at the beginning of the next matching string (the next newline in the input) and the following record starts just after the end of this string (at the first character of the following line). The newline, because it matches RS, is not part of either record.

When RS is a single character, RT contains the same single character. However, when RS is a regular expression, RT contains the actual input text that matched the regular expression.

The following example illustrates both of these features. It sets RS equal to a regular expression that matches either a newline or a series of one or more uppercase letters with optional leading and/or trailing whitespace:

$ echo record 1 AAAA record 2 BBBB record 3 | > gawk 'BEGIN { RS = "\n|( *[[:upper:]]+ *)" } > { print "Record =", $0, "and RT =", RT }' -| Record = record 1 and RT = AAAA -| Record = record 2 and RT = BBBB -| Record = record 3 and RT = -|

The final line of output has an extra blank line. This is because the value of RT is a newline, and the print statement supplies its own terminating newline. See section A Simple Stream Editor, for a more useful example of RS as a regexp and RT.

The use of RS as a regular expression and the RT variable are gawk extensions; they are not available in compatibility mode (see section Command-Line Options). In compatibility mode, only the first character of the value of RS is used to determine the end of the record.

Advanced Notes: RS = "\0" Is Not Portable

There are times when you might want to treat an entire data file as a single record. The only way to make this happen is to give RS a value that you know doesn't occur in the input file. This is hard to do in a general way, such that a program always works for arbitrary input files.

You might think that for text files, the NUL character, which consists of a character with all bits equal to zero, is a good value to use for RS in this case:

BEGIN { RS = "\0" } # whole file becomes one record?

gawk in fact accepts this, and uses the NUL character for the record separator. However, this usage is not portable to other awk implementations.

All other awk implementations(15) store strings internally as C-style strings. C strings use the NUL character as the string terminator. In effect, this means that `RS = "\0"' is the same as `RS = ""'. (d.c.)

The best way to treat a whole file as a single record is to simply read the file in, one record at a time, concatenating each record onto the end of the previous ones.

4.2 Examining Fields

When awk reads an input record, the record is automatically separated or parsed by the interpreter into chunks called fields. By default, fields are separated by whitespace, like words in a line. Whitespace in awk means any string of one or more spaces, tabs, or newlines;(16) other characters, such as formfeed, vertical tab, etc. that are considered whitespace by other languages, are not considered whitespace by awk.

The purpose of fields is to make it more convenient for you to refer to these pieces of the record. You don't have to use them--you can operate on the whole record if you want--but fields are what make simple awk programs so powerful.

A dollar-sign (`$') is used to refer to a field in an awk program, followed by the number of the field you want. Thus, $1 refers to the first field, $2 to the second, and so on. (Unlike the Unix shells, the field numbers are not limited to single digits. $127 is the one hundred and twenty-seventh field in the record.) For example, suppose the following is a line of input:

This seems like a pretty nice example.

Here the first field, or $1, is `This', the second field, or $2, is `seems', and so on. Note that the last field, $7, is `example.'. Because there is no space between the `e' and the `.', the period is considered part of the seventh field.

NF is a built-in variable whose value is the number of fields in the current record. awk automatically updates the value of NF each time it reads a record. No matter how many fields there are, the last field in a record can be represented by $NF. So, $NF is the same as $7, which is `example.'. If you try to reference a field beyond the last one (such as $8 when the record has only seven fields), you get the empty string. (If used in a numeric operation, you get zero.)

The use of $0, which looks like a reference to the "zeroth" field, is a special case: it represents the whole input record when you are not interested in specific fields. Here are some more examples:

$ awk '$1 ~ /foo/ { print $0 }' BBS-list -| fooey 555-1234 2400/1200/300 B -| foot 555-6699 1200/300 B -| macfoo 555-6480 1200/300 A -| sabafoo 555-2127 1200/300 C

This example prints each record in the file `BBS-list' whose first field contains the string `foo'. The operator `~' is called a matching operator (see section How to Use Regular Expressions); it tests whether a string (here, the field $1) matches a given regular expression.

By contrast, the following example looks for `foo' in the entire record and prints the first field and the last field for each matching input record:

$ awk '/foo/ { print $1, $NF }' BBS-list -| fooey B -| foot B -| macfoo A -| sabafoo C

4.3 Non-Constant Field Numbers

The number of a field does not need to be a constant. Any expression in the awk language can be used after a `$' to refer to a field. The value of the expression specifies the field number. If the value is a string, rather than a number, it is converted to a number. Consider this example:

awk '{ print $NR }'

Recall that NR is the number of records read so far: one in the first record, two in the second, etc. So this example prints the first field of the first record, the second field of the second record, and so on. For the twentieth record, field number 20 is printed; most likely, the record has fewer than 20 fields, so this prints a blank line. Here is another example of using expressions as field numbers:

awk '{ print $(2*2) }' BBS-list

awk evaluates the expression `(2*2)' and uses its value as the number of the field to print. The `*' sign represents multiplication, so the expression `2*2' evaluates to four. The parentheses are used so that the multiplication is done before the `$' operation; they are necessary whenever there is a binary operator in the field-number expression. This example, then, prints the hours of operation (the fourth field) for every line of the file `BBS-list'. (All of the awk operators are listed, in order of decreasing precedence, in Operator Precedence (How Operators Nest).)

If the field number you compute is zero, you get the entire record. Thus, `$(2-2)' has the same value as $0. Negative field numbers are not allowed; trying to reference one usually terminates the program. (The POSIX standard does not define what happens when you reference a negative field number. gawk notices this and terminates your program. Other awk implementations may behave differently.)

As mentioned in Examining Fields, awk stores the current record's number of fields in the built-in variable NF (also see section 7.5 Built-in Variables). The expression $NF is not a special feature--it is the direct consequence of evaluating NF and using its value as a field number.

4.4 Changing the Contents of a Field

The contents of a field, as seen by awk, can be changed within an awk program; this changes what awk perceives as the current input record. (The actual input is untouched; awk never modifies the input file.) Consider this example and its output:

$ awk '{ nboxes = $3 ; $3 = $3 - 10 > print nboxes, $3 }' inventory-shipped -| 13 3 -| 15 5 -| 15 5 ...

The program first saves the original value of field three in the variable nboxes. The `-' sign represents subtraction, so this program reassigns field three, $3, as the original value of field three minus ten: `$3 - 10'. (See section Arithmetic Operators.) Then it prints the original and new values for field three. (Someone in the warehouse made a consistent mistake while inventorying the red boxes.)

For this to work, the text in field $2 must make sense as a number; the string of characters must be converted to a number for the computer to do arithmetic on it. The number resulting from the subtraction is converted back to a string of characters that then becomes field three. See section Conversion of Strings and Numbers.

When the value of a field is changed (as perceived by awk), the text of the input record is recalculated to contain the new field where the old one was. In other words, $0 changes to reflect the altered field. Thus, this program prints a copy of the input file, with 10 subtracted from the second field of each line:

$ awk '{ $2 = $2 - 10; print $0 }' inventory-shipped -| Jan 3 25 15 115 -| Feb 5 32 24 226 -| Mar 5 24 34 228 ...

It is also possible to also assign contents to fields that are out of range. For example:

$ awk '{ $6 = ($5 + $4 + $3 + $2) > print $6 }' inventory-shipped -| 168 -| 297 -| 301 ...

We've just created $6, whose value is the sum of fields $2, $3, $4, and $5. The `+' sign represents addition. For the file `inventory-shipped', $6 represents the total number of parcels shipped for a particular month.

Creating a new field changes awk's internal copy of the current input record, which is the value of $0. Thus, if you do `print $0' after adding a field, the record printed includes the new field, with the appropriate number of field separators between it and the previously existing fields.

This recomputation affects and is affected by NF (the number of fields; see section Examining Fields). It is also affected by a feature that has not been discussed yet: the output field separator, OFS, used to separate the fields (see section 5.3 Output Separators). For example, the value of NF is set to the number of the highest field you create.

Note, however, that merely referencing an out-of-range field does not change the value of either $0 or NF. Referencing an out-of-range field only produces an empty string. For example:

if ($(NF+1) != "") print "can't happen" else print "everything is normal"

should print `everything is normal', because NF+1 is certain to be out of range. (See section The if-else Statement, for more information about awk's if-else statements. See section Variable Typing and Comparison Expressions, for more information about the `!=' operator.)

It is important to note that making an assignment to an existing field changes the value of $0 but does not change the value of NF, even when you assign the empty string to a field. For example:

$ echo a b c d | awk '{ OFS = ":"; $2 = "" > print $0; print NF }' -| a::c:d -| 4

The field is still there; it just has an empty value, denoted by the two colons between `a' and `c'. This example shows what happens if you create a new field:

$ echo a b c d | awk '{ OFS = ":"; $2 = ""; $6 = "new" > print $0; print NF }' -| a::c:d::new -| 6

The intervening field, $5, is created with an empty value (indicated by the second pair of adjacent colons), and NF is updated with the value six.

Decrementing NF throws away the values of the fields after the new value of NF and recomputes $0. (d.c.) Here is an example:

$ echo a b c d e f | awk '{ print "NF =", NF; > NF = 3; print $0 }' -| NF = 6 -| a b c

Caution: Some versions of awk don't rebuild $0 when NF is decremented. Caveat emptor.

4.5 Specifying How Fields Are Separated

4.5.1 Using Regular Expressions to Separate Fields		Using regexps as the field separator.
4.5.2 Making Each Character a Separate Field		Making each character a separate field.
4.5.3 Setting FS from the Command Line		Setting FS from the command-line.
4.5.4 Field Splitting Summary		Some final points and a summary table.

The field separator, which is either a single character or a regular expression, controls the way awk splits an input record into fields. awk scans the input record for character sequences that match the separator; the fields themselves are the text between the matches.

In the examples that follow, we use the bullet symbol (*) to represent spaces in the output. If the field separator is `oo', then the following line:

moo goo gai pan

is split into three fields: `m', `*g', and `*gai*pan'. Note the leading spaces in the values of the second and third fields.

The field separator is represented by the built-in variable FS. Shell programmers take note: awk does not use the name IFS that is used by the POSIX-compliant shells (such as the Unix Bourne shell, sh, or bash).

The value of FS can be changed in the awk program with the assignment operator, `=' (see section Assignment Expressions). Often the right time to do this is at the beginning of execution before any input has been processed, so that the very first record is read with the proper separator. To do this, use the special BEGIN pattern (see section The BEGIN and END Special Patterns). For example, here we set the value of FS to the string ",":

awk 'BEGIN { FS = "," } ; { print $2 }'

Given the input line:

John Q. Smith, 29 Oak St., Walamazoo, MI 42139

this awk program extracts and prints the string `*29*Oak*St.'.

Sometimes the input data contains separator characters that don't separate fields the way you thought they would. For instance, the person's name in the example we just used might have a title or suffix attached, such as:

John Q. Smith, LXIX, 29 Oak St., Walamazoo, MI 42139

The same program would extract `*LXIX', instead of `*29*Oak*St.'. If you were expecting the program to print the address, you would be surprised. The moral is to choose your data layout and separator characters carefully to prevent such problems. (If the data is not in a form that is easy to process, perhaps you can massage it first with a separate awk program.)

Fields are normally separated by whitespace sequences (spaces, tabs, and newlines), not by single spaces. Two spaces in a row do not delimit an empty field. The default value of the field separator FS is a string containing a single space, " ". If awk interpreted this value in the usual way, each space character would separate fields, so two spaces in a row would make an empty field between them. The reason this does not happen is that a single space as the value of FS is a special case--it is taken to specify the default manner of delimiting fields.

If FS is any other single character, such as ",", then each occurrence of that character separates two fields. Two consecutive occurrences delimit an empty field. If the character occurs at the beginning or the end of the line, that too delimits an empty field. The space character is the only single character that does not follow these rules.

4.5.1 Using Regular Expressions to Separate Fields

The previous subsection discussed the use of single characters or simple strings as the value of FS. More generally, the value of FS may be a string containing any regular expression. In this case, each match in the record for the regular expression separates fields. For example, the assignment:

FS = ", \t"

makes every area of an input line that consists of a comma followed by a space and a tab into a field separator. (`\t' is an escape sequence that stands for a tab; see section 3.2 Escape Sequences, for the complete list of similar escape sequences.)

For a less trivial example of a regular expression, try using single spaces to separate fields the way single commas are used. FS can be set to "[ ]" (left bracket, space, right bracket). This regular expression matches a single space and nothing else (see section Regular Expressions).

There is an important difference between the two cases of `FS = " "' (a single space) and `FS = "[ \t\n]+"' (a regular expression matching one or more spaces, tabs, or newlines). For both values of FS, fields are separated by runs (multiple adjacent occurrences) of spaces, tabs, and/or newlines. However, when the value of FS is " ", awk first strips leading and trailing whitespace from the record and then decides where the fields are. For example, the following pipeline prints `b':

$ echo ' a b c d ' | awk '{ print $2 }' -| b

However, this pipeline prints `a' (note the extra spaces around each letter):

$ echo ' a b c d ' | awk 'BEGIN { FS = "[ \t\n]+" } > { print $2 }' -| a

In this case, the first field is null or empty.

The stripping of leading and trailing whitespace also comes into play whenever $0 is recomputed. For instance, study this pipeline:

$ echo ' a b c d' | awk '{ print; $2 = $2; print }' -| a b c d -| a b c d

The first print statement prints the record as it was read, with leading whitespace intact. The assignment to $2 rebuilds $0 by concatenating $1 through $NF together, separated by the value of OFS. Because the leading whitespace was ignored when finding $1, it is not part of the new $0. Finally, the last print statement prints the new $0.

4.5.2 Making Each Character a Separate Field

There are times when you may want to examine each character of a record separately. This can be done in gawk by simply assigning the null string ("") to FS. In this case, each individual character in the record becomes a separate field. For example:

$ echo a b | gawk 'BEGIN { FS = "" } > { > for (i = 1; i <= NF; i = i + 1) > print "Field", i, "is", $i > }' -| Field 1 is a -| Field 2 is -| Field 3 is b

Traditionally, the behavior of FS equal to "" was not defined. In this case, most versions of Unix awk simply treat the entire record as only having one field. (d.c.) In compatibility mode (see section Command-Line Options), if FS is the null string, then gawk also behaves this way.

4.5.3 Setting FS from the Command Line

FS can be set on the command line. Use the `-F' option to do so. For example:

awk -F, 'program' input-files

sets FS to the `,' character. Notice that the option uses a capital `F' instead of a lowercase `-f', which specifies a file containing an awk program. Case is significant in command-line options: the `-F' and `-f' options have nothing to do with each other. You can use both options at the same time to set the FS variable and get an awk program from a file.

The value used for the argument to `-F' is processed in exactly the same way as assignments to the built-in variable FS. Any special characters in the field separator must be escaped appropriately. For example, to use a `\' as the field separator on the command line, you would have to type:

# same as FS = "\\" awk -F\\\\ '...' files ...

Because `\' is used for quoting in the shell, awk sees `-F\\'. Then awk processes the `\\' for escape characters (see section 3.2 Escape Sequences), finally yielding a single `\' to use for the field separator.

As a special case, in compatibility mode (see section Command-Line Options), if the argument to `-F' is `t', then FS is set to the tab character. If you type `-F\t' at the shell, without any quotes, the `\' gets deleted, so awk figures that you really want your fields to be separated with tabs and not `t's. Use `-v FS="t"' or `-F"[t]"' on the command line if you really do want to separate your fields with `t's.

For example, let's use an awk program file called `baud.awk' that contains the pattern /300/ and the action `print $1':

/300/ { print $1 }

Let's also set FS to be the `-' character and run the program on the file `BBS-list'. The following command prints a list of the names of the bulletin boards that operate at 300 baud and the first three digits of their phone numbers:

$ awk -F- -f baud.awk BBS-list -| aardvark 555 -| alpo -| barfly 555 -| bites 555 -| camelot 555 -| core 555 -| fooey 555 -| foot 555 -| macfoo 555 -| sdace 555 -| sabafoo 555

Note the second line of output. The second line in the original file looked like this:

alpo-net 555-3412 2400/1200/300 A

The `-' as part of the system's name was used as the field separator, instead of the `-' in the phone number that was originally intended. This demonstrates why you have to be careful in choosing your field and record separators.

Perhaps the most common use of a single character as the field separator occurs when processing the Unix system password file. On many Unix systems, each user has a separate entry in the system password file, one line per user. The information in these lines is separated by colons. The first field is the user's logon name and the second is the user's (encrypted or shadow) password. A password file entry might look like this:

arnold:xyzzy:2076:10:Arnold Robbins:/home/arnold:/bin/bash

The following program searches the system password file and prints the entries for users who have no password:

awk -F: '$2 == ""' /etc/passwd

4.5.4 Field Splitting Summary

The following table summarizes how fields are split, based on the value of FS. (`==' means "is equal to.")

FS == " "

Fields are separated by runs of whitespace. Leading and trailing whitespace are ignored. This is the default.

FS == any other single character

Fields are separated by each occurrence of the character. Multiple successive occurrences delimit empty fields, as do leading and trailing occurrences. The character can even be a regexp metacharacter; it does not need to be escaped.

FS == regexp

Fields are separated by occurrences of characters that match regexp. Leading and trailing matches of regexp delimit empty fields.

FS == ""

Each individual character in the record becomes a separate field. (This is a gawk extension; it is not specified by the POSIX standard.)

Advanced Notes: Changing FS Does Not Affect the Fields

According to the POSIX standard, awk is supposed to behave as if each record is split into fields at the time it is read. In particular, this means that if you change the value of FS after a record is read, the value of the fields (i.e., how they were split) should reflect the old value of FS, not the new one.

However, many implementations of awk do not work this way. Instead, they defer splitting the fields until a field is actually referenced. The fields are split using the current value of FS! (d.c.) This behavior can be difficult to diagnose. The following example illustrates the difference between the two methods. (The sed(17) command prints just the first line of `/etc/passwd'.)

sed 1q /etc/passwd | awk '{ FS = ":" ; print $1 }'

which usually prints:

root

on an incorrect implementation of awk, while gawk prints something like:

root:nSijPlPhZZwgE:0:0:Root:/:

4.6 Reading Fixed-Width Data

Note: This section discusses an advanced feature of gawk. If you are a novice awk user, you might want to skip it on the first reading.

(This section discusses an advanced feature of awk. If you are a novice awk user, you might want to skip it on the first reading.)

gawk version 2.13 introduced a facility for dealing with fixed-width fields with no distinctive field separator. For example, data of this nature arises in the input for old Fortran programs where numbers are run together, or in the output of programs that did not anticipate the use of their output as input for other programs.

An example of the latter is a table where all the columns are lined up by the use of a variable number of spaces and empty fields are just spaces. Clearly, awk's normal field splitting based on FS does not work well in this case. Although a portable awk program can use a series of substr calls on $0 (see section String Manipulation Functions), this is awkward and inefficient for a large number of fields.

The splitting of an input record into fixed-width fields is specified by assigning a string containing space-separated numbers to the built-in variable FIELDWIDTHS. Each number specifies the width of the field, including columns between fields. If you want to ignore the columns between fields, you can specify the width as a separate field that is subsequently ignored. It is a fatal error to supply a field width that is not a positive number. The following data is the output of the Unix w utility. It is useful to illustrate the use of FIELDWIDTHS:

10:06pm up 21 days, 14:04, 23 users User tty login idle JCPU PCPU what hzuo ttyV0 8:58pm 9 5 vi p24.tex hzang ttyV3 6:37pm 50 -csh eklye ttyV5 9:53pm 7 1 em thes.tex dportein ttyV6 8:17pm 1:47 -csh gierd ttyD3 10:00pm 1 elm dave ttyD4 9:47pm 4 4 w brent ttyp0 26Jun91 4:46 26:46 4:41 bash dave ttyq4 26Jun9115days 46 46 wnewmail

The following program takes the above input, converts the idle time to number of seconds, and prints out the first two fields and the calculated idle time.

Note: This program uses a number of awk features that haven't been introduced yet.

BEGIN { FIELDWIDTHS = "9 6 10 6 7 7 35" } NR > 2 { idle = $4 sub(/^ */, "", idle) # strip leading spaces if (idle == "") idle = 0 if (idle ~ /:/) { split(idle, t, ":") idle = t[1] * 60 + t[2] } if (idle ~ /days/) idle *= 24 * 60 * 60 print $1, $2, idle }

Running the program on the data produces the following results:

hzuo ttyV0 0 hzang ttyV3 50 eklye ttyV5 0 dportein ttyV6 107 gierd ttyD3 1 dave ttyD4 0 brent ttyp0 286 dave ttyq4 1296000

Another (possibly more practical) example of fixed-width input data is the input from a deck of balloting cards. In some parts of the United States, voters mark their choices by punching holes in computer cards. These cards are then processed to count the votes for any particular candidate or on any particular issue. Because a voter may choose not to vote on some issue, any column on the card may be empty. An awk program for processing such data could use the FIELDWIDTHS feature to simplify reading the data. (Of course, getting gawk to run on a system with card readers is another story!)

Assigning a value to FS causes gawk to return to using FS for field splitting. Use `FS = FS' to make this happen, without having to know the current value of FS. In order to tell which kind of field splitting is in effect, use PROCINFO["FS"] (see section Built-in Variables That Convey Information). The value is "FS" if regular field splitting is being used, or it is "FIELDWIDTHS" if fixed-width field splitting is being used:

if (PROCINFO["FS"] == "FS") regular field splitting ... else fixed-width field splitting ...

This information is useful when writing a function that needs to temporarily change FS or FIELDWIDTHS, read some records, and then restore the original settings (see section Reading the User Database, for an example of such a function).

4.7 Multiple-Line Records

In some databases, a single line cannot conveniently hold all the information in one entry. In such cases, you can use multiline records. The first step in doing this is to choose your data format.

One technique is to use an unusual character or string to separate records. For example, you could use the formfeed character (written `\f' in awk, as in C) to separate them, making each record a page of the file. To do this, just set the variable RS to "\f" (a string containing the formfeed character). Any other character could equally well be used, as long as it won't be part of the data in a record.

Another technique is to have blank lines separate records. By a special dispensation, an empty string as the value of RS indicates that records are separated by one or more blank lines. When RS is set to the empty string, each record always ends at the first blank line encountered. The next record doesn't start until the first non-blank line that follows. No matter how many blank lines appear in a row, they all act as one record separator. (Blank lines must be completely empty; lines that contain only whitespace do not count.)

You can achieve the same effect as `RS = ""' by assigning the string "\n\n+" to RS. This regexp matches the newline at the end of the record and one or more blank lines after the record. In addition, a regular expression always matches the longest possible sequence when there is a choice (see section How Much Text Matches?). So the next record doesn't start until the first non-blank line that follows--no matter how many blank lines appear in a row, they are considered one record separator.

There is an important difference between `RS = ""' and `RS = "\n\n+"'. In the first case, leading newlines in the input data file are ignored, and if a file ends without extra blank lines after the last record, the final newline is removed from the record. In the second case, this special processing is not done. (d.c.)

Now that the input is separated into records, the second step is to separate the fields in the record. One way to do this is to divide each of the lines into fields in the normal manner. This happens by default as the result of a special feature. When RS is set to the empty string, the newline character always acts as a field separator. This is in addition to whatever field separations result from FS.

The original motivation for this special exception was probably to provide useful behavior in the default case (i.e., FS is equal to " "). This feature can be a problem if you really don't want the newline character to separate fields, because there is no way to prevent it. However, you can work around this by using the split function to break up the record manually (see section String Manipulation Functions).

Another way to separate fields is to put each field on a separate line: to do this, just set the variable FS to the string "\n". (This simple regular expression matches a single newline.) A practical example of a data file organized this way might be a mailing list, where each entry is separated by blank lines. Consider a mailing list in a file named `addresses', that looks like this:

Jane Doe 123 Main Street Anywhere, SE 12345-6789 John Smith 456 Tree-lined Avenue Smallville, MW 98765-4321 ...

A simple program to process this file is as follows:

# addrs.awk --- simple mailing list program # Records are separated by blank lines. # Each line is one field. BEGIN { RS = "" ; FS = "\n" } { print "Name is:", $1 print "Address is:", $2 print "City and State are:", $3 print "" }

Running the program produces the following output:

$ awk -f addrs.awk addresses -| Name is: Jane Doe -| Address is: 123 Main Street -| City and State are: Anywhere, SE 12345-6789 -| -| Name is: John Smith -| Address is: 456 Tree-lined Avenue -| City and State are: Smallville, MW 98765-4321 -| ...

See section Printing Mailing Labels, for a more realistic program that deals with address lists. The following table summarizes how records are split, based on the value of RS. (`==' means "is equal to.") RS:

RS == "\n"

Records are separated by the newline character (`\n'). In effect, every line in the data file is a separate record, including blank lines. This is the default.

RS == any single character

Records are separated by each occurrence of the character. Multiple successive occurrences delimit empty records.

RS == ""

Records are separated by runs of blank lines. The newline character always serves as a field separator, in addition to whatever value FS may have. Leading and trailing newlines in a file are ignored.

RS == regexp

Records are separated by occurrences of characters that match regexp. Leading and trailing matches of regexp delimit empty records. (This is a gawk extension, it is not specified by the POSIX standard.)

In all cases, gawk sets RT to the input text that matched the value specified by RS.

4.8 Explicit Input with getline

So far we have been getting our input data from awk's main input stream--either the standard input (usually your terminal, sometimes the output from another program) or from the files specified on the command line. The awk language has a special built-in command called getline that can be used to read input under your explicit control.

The getline command is used in several different ways and should not be used by beginners. The examples that follow the explanation of the getline command include material that has not been covered yet. Therefore, come back and study the getline command after you have reviewed the rest of this Web page and have a good knowledge of how awk works.

The getline command returns one if it finds a record and zero if the end of the file is encountered. If there is some error in getting a record, such as a file that cannot be opened, then getline returns -1. In this case, gawk sets the variable ERRNO to a string describing the error that occurred.

In the following examples, command stands for a string value that represents a shell command.

4.8.1 Using getline with No Arguments		Using getline with no arguments.
4.8.2 Using getline into a Variable		Using getline into a variable.
4.8.3 Using getline from a File		Using getline from a file.
4.8.4 Using getline into a Variable from a File		Using getline into a variable from a file.
4.8.5 Using getline from a Pipe		Using getline from a pipe.
4.8.6 Using getline into a Variable from a Pipe		Using getline into a variable from a pipe.
4.8.7 Using getline from a Coprocess		Using getline from a coprocess.
4.8.8 Using getline into a Variable from a Coprocess		Using getline into a variable from a coprocess.
4.8.9 Points About getline to Remember		Important things to know about getline.
4.8.10 Summary of getline Variants

4.8.1 Using getline with No Arguments

The getline command can be used without arguments to read input from the current input file. All it does in this case is read the next input record and split it up into fields. This is useful if you've finished processing the current record, but want to do some special processing right now on the next record. Here's an example:

{ if ((t = index($0, "/*")) != 0) { # value of `tmp' will be "" if t is 1 tmp = substr($0, 1, t - 1) u = index(substr($0, t + 2), "*/") while (u == 0) { if (getline <= 0) { m = "unexpected EOF or error" m = (m ": " ERRNO) print m > "/dev/stderr" exit } t = -1 u = index($0, "*/") } # substr expression will be "" if */ # occurred at end of line $0 = tmp substr($0, u + 2) } print $0 }

This awk program deletes all C-style comments (`/* ... */') from the input. By replacing the `print $0' with other statements, you could perform more complicated processing on the decommented input, such as searching for matches of a regular expression. (This program has a subtle problem--it does not work if one comment ends and another begins on the same line.)

This form of the getline command sets NF, NR, FNR, and the value of $0.

Note: The new value of $0 is used to test the patterns of any subsequent rules. The original value of $0 that triggered the rule that executed getline is lost. By contrast, the next statement reads a new record but immediately begins processing it normally, starting with the first rule in the program. See section The next Statement.

4.8.2 Using getline into a Variable

You can use `getline var' to read the next record from awk's input into the variable var. No other processing is done. For example, suppose the next line is a comment or a special string, and you want to read it without triggering any rules. This form of getline allows you to read that line and store it in a variable so that the main read-a-line-and-check-each-rule loop of awk never sees it. The following example swaps every two lines of input. The program is as follows:

{ if ((getline tmp) > 0) { print tmp print $0 } else print $0 }

It takes the following list:

wan tew free phore

and produces these results:

tew wan phore free

The getline command used in this way sets only the variables NR and FNR (and of course, var). The record is not split into fields, so the values of the fields (including $0) and the value of NF do not change.

4.8.3 Using getline from a File

Use `getline < file' to read the next record from file. Here file is a string-valued expression that specifies the file name. `< file' is called a redirection because it directs input to come from a different place. For example, the following program reads its input record from the file `secondary.input' when it encounters a first field with a value equal to 10 in the current input file:

{ if ($1 == 10) { getline < "secondary.input" print } else print }

Because the main input stream is not used, the values of NR and FNR are not changed. However, the record it reads is split into fields in the normal manner, so the values of $0 and the other fields are changed, resulting in a new value of NF.

According to POSIX, `getline < expression' is ambiguous if expression contains unparenthesized operators other than `$'; for example, `getline < dir "/" file' is ambiguous because the concatenation operator is not parenthesized. You should write it as `getline < (dir "/" file)' if you want your program to be portable to other awk implementations. (It happens that gawk gets it right, but you should not rely on this. Parentheses make it easier to read.)

4.8.4 Using getline into a Variable from a File

Use `getline var < file' to read input from the file file, and put it in the variable var. As above, file is a string-valued expression that specifies the file from which to read.

In this version of getline, none of the built-in variables are changed and the record is not split into fields. The only variable changed is var. For example, the following program copies all the input files to the output, except for records that say `@include filename'. Such a record is replaced by the contents of the file filename:

{ if (NF == 2 && $1 == "@include") { while ((getline line < $2) > 0) print line close($2) } else print }

Note here how the name of the extra input file is not built into the program; it is taken directly from the data, from the second field on the `@include' line.

The close function is called to ensure that if two identical `@include' lines appear in the input, the entire specified file is included twice. See section Closing Input and Output Redirections.

One deficiency of this program is that it does not process nested `@include' statements (i.e., `@include' statements in included files) the way a true macro preprocessor would. See section An Easy Way to Use Library Functions, for a program that does handle nested `@include' statements.

4.8.5 Using getline from a Pipe

The output of a command can also be piped into getline, using `command | getline'. In this case, the string command is run as a shell command and its output is piped into awk to be used as input. This form of getline reads one record at a time from the pipe. For example, the following program copies its input to its output, except for lines that begin with `@execute', which are replaced by the output produced by running the rest of the line as a shell command:

{ if ($1 == "@execute") { tmp = substr($0, 10) while ((tmp | getline) > 0) print close(tmp) } else print }

The close function is called to ensure that if two identical `@execute' lines appear in the input, the command is run for each one. See section Closing Input and Output Redirections. Given the input:

foo bar baz @execute who bletch

the program might produce:

foo bar baz arnold ttyv0 Jul 13 14:22 miriam ttyp0 Jul 13 14:23 (murphy:0) bill ttyp1 Jul 13 14:23 (murphy:0) bletch

Notice that this program ran the command who and printed the result. (If you try this program yourself, you will of course get different results, depending upon who is logged in on your system.)

This variation of getline splits the record into fields, sets the value of NF and recomputes the value of $0. The values of NR and FNR are not changed.

According to POSIX, `expression | getline' is ambiguous if expression contains unparenthesized operators other than `$'---for example, `"echo " "date" | getline' is ambiguous because the concatenation operator is not parenthesized. You should write it as `("echo " "date") | getline' if you want your program to be portable to other awk implementations. (It happens that gawk gets it right, but you should not rely on this. Parentheses make it easier to read, anyway.)

4.8.6 Using getline into a Variable from a Pipe

When you use `command | getline var', the output of command is sent through a pipe to getline and into the variable var. For example, the following program reads the current date and time into the variable current_time, using the date utility, and then prints it:

BEGIN { "date" | getline current_time close("date") print "Report printed on " current_time }

In this version of getline, none of the built-in variables are changed and the record is not split into fields.

According to POSIX, `expression | getline var' is ambiguous if expression contains unparenthesized operators other than `$'; for example, `"echo " "date" | getline var' is ambiguous because the concatenation operator is not parenthesized. You should write it as `("echo " "date") | getline var' if you want your program to be portable to other awk implementations. (It happens that gawk gets it right, but you should not rely on this. Parentheses make it easier to read, anyway.)

4.8.7 Using getline from a Coprocess

Input into getline from a pipe is a one-way operation. The command that is started with `command | getline' only sends data to your awk program.

On occasion, you might want to send data to another program for processing and then read the results back. gawk allows you start a coprocess, with which two-way communications are possible. This is done with the `|&' operator. Typically, you write data to the coprocess first, and then read results back, as shown in the following:

print "some query" |& "db_server" "db_server" |& getline

which sends a query to db_server and then reads the results.

The values of NR and FNR are not changed, because the main input stream is not used. However, the record is split into fields in the normal manner, thus changing the values of $0, the other fields, and of NF.

Coprocesses are an advanced feature. They are discussed here only because this is the section on getline. See section Two-Way Communications with Another Process, where coprocesses are discussed in more detail.

4.8.8 Using getline into a Variable from a Coprocess

When you use `command |& getline var', the output from the coprocess command is sent through a two-way pipe to getline and into the variable var.

In this version of getline, none of the built-in variables are changed and the record is not split into fields. The only variable changed is var.

Coprocesses are an advanced feature. They are discussed here only because this is the section on getline. See section Two-Way Communications with Another Process, where coprocesses are discussed in more detail.

4.8.9 Points About getline to Remember

• When getline changes the value of $0 and NF, awk does not automatically jump to the start of the program and start testing the new record against every pattern. However, the new record is tested against any subsequent rules.

• Many awk implementations limit the number of pipelines that an awk program may have open to just one. In gawk, there is no such limit. You can open as many pipelines (and coprocesses) as the underlying operating system permits.

• An interesting side effect occurs if you use getline without a redirection inside a BEGIN rule. Because an unredirected getline reads from the command-line data files, the first getline command causes awk to set the value of FILENAME. Normally, FILENAME does not have a value inside BEGIN rules, because you have not yet started to process the command-line data files. (d.c.) (See section The BEGIN and END Special Patterns, also see section Built-in Variables That Convey Information.)

4.8.10 Summary of getline Variants

The following table summarizes the eight variants of getline, listing which built-in variables are set by each one.

(this is a gawk extension) (this is a gawk extension)

getline	Sets $0, NF, FNR and NR
getline var	Sets var, FNR and NR
getline < file	Sets $0 and NF
getline var < file	Sets var
command | getline	Sets $0 and NF
command | getline var	Sets var
command |& getline	Sets $0 and NF
command |& getline var	Sets var

5. Printing Output

One of the most common programming actions is to print or output, some or all of the input. Use the print statement for simple output, and the printf statement for fancier formatting. The print statement is not limited when computing which values to print. However, with two exceptions, you cannot specify how to print them--how many columns, whether to use exponential notation or not, and so on. (For the exceptions, see section 5.3 Output Separators, and Controlling Numeric Output with print.) For that, you need the printf statement (see section Using printf Statements for Fancier Printing).

Besides basic and formatted printing, this chapter also covers I/O redirections to files and pipes, introduces the special file names that gawk processes internally, and discusses the close built-in function.

5.1 The print Statement		The print statement.
5.2 Examples of print Statements		Simple examples of print statements.
5.3 Output Separators		The output separators and how to change them.
5.4 Controlling Numeric Output with print		Controlling Numeric Output With print.
5.5 Using printf Statements for Fancier Printing		The printf statement.
5.6 Redirecting Output of print and printf		How to redirect output to multiple files and pipes.
5.7 Special File Names in gawk		File name interpretation in gawk.

                                gawk allows access to inherited file
                                descriptors.

5.8 Closing Input and Output Redirections

Closing Input and Output Files and Pipes.

5.1 The print Statement

The print statement is used to produce output with simple, standardized formatting. Specify only the strings or numbers to print, in a list separated by commas. They are output, separated by single spaces, followed by a newline. The statement looks like this:

print item1, item2, ...

The entire list of items may be optionally enclosed in parentheses. The parentheses are necessary if any of the item expressions uses the `>' relational operator; otherwise it could be confused with a redirection (see section Redirecting Output of print and printf).

The items to print can be constant strings or numbers, fields of the current record (such as $1), variables, or any awk expression. Numeric values are converted to strings and then printed.

The simple statement `print' with no items is equivalent to `print $0': it prints the entire current record. To print a blank line, use `print ""', where "" is the empty string. To print a fixed piece of text, use a string constant, such as "Don't Panic", as one item. If you forget to use the double quote characters, your text is taken as an awk expression and you will probably get an error. Keep in mind that a space is printed between any two items.

5.2 Examples of print Statements

Each print statement makes at least one line of output. However, it isn't limited to only one line. If an item value is a string that contains a newline, the newline is output along with the rest of the string. A single print statement can make any number of lines this way.

The following is an example of printing a string that contains embedded newlines (the `\n' is an escape sequence, used to represent the newline character; see section 3.2 Escape Sequences):

$ awk 'BEGIN { print "line one\nline two\nline three" }' -| line one -| line two -| line three

The next example, which is run on the `inventory-shipped' file, prints the first two fields of each input record, with a space between them:

$ awk '{ print $1, $2 }' inventory-shipped -| Jan 13 -| Feb 15 -| Mar 15 ...

A common mistake in using the print statement is to omit the comma between two items. This often has the effect of making the items run together in the output, with no space. The reason for this is that juxtaposing two string expressions in awk means to concatenate them. Here is the same program, without the comma:

$ awk '{ print $1 $2 }' inventory-shipped -| Jan13 -| Feb15 -| Mar15 ...

To someone unfamiliar with the `inventory-shipped' file, neither example's output makes much sense. A heading line at the beginning would make it clearer. Let's add some headings to our table of months ($1) and green crates shipped ($2). We do this using the BEGIN pattern (see section The BEGIN and END Special Patterns) so that the headings are only printed once:

awk 'BEGIN { print "Month Crates" print "----- ------" } { print $1, $2 }' inventory-shipped

When run, the program prints the following:

Month Crates ----- ------ Jan 13 Feb 15 Mar 15 ...

The only problem, however, is that the headings and the table data don't line up! We can fix this by printing some spaces between the two fields:

awk 'BEGIN { print "Month Crates" print "----- ------" } { print $1, " ", $2 }' inventory-shipped

Lining up columns this way can get pretty complicated when there are many columns to fix. Counting spaces for two or three columns is simple, but any more than this can take up a lot of time. This is why the printf statement was created (see section Using printf Statements for Fancier Printing); one of its specialties is lining up columns of data.

Note: You can continue either a print or printf statement simply by putting a newline after any comma (see section awk Statements Versus Lines).

5.3 Output Separators

As mentioned previously, a print statement contains a list of items separated by commas. In the output, the items are normally separated by single spaces. However, this doesn't need to be the case; a single space is only the default. Any string of characters may be used as the output field separator by setting the built-in variable OFS. The initial value of this variable is the string " "---that is, a single space.

The output from an entire print statement is called an output record. Each print statement outputs one output record, and then outputs a string called the output record separator (or ORS). The initial value of ORS is the string "\n"; i.e., a newline character. Thus, each print statement normally makes a separate line.

In order to change how output fields and records are separated, assign new values to the variables OFS and ORS. The usual place to do this is in the BEGIN rule (see section The BEGIN and END Special Patterns), so that it happens before any input is processed. It can also be done with assignments on the command line, before the names of the input files, or using the `-v' command-line option (see section Command-Line Options). The following example prints the first and second fields of each input record, separated by a semicolon, with a blank line added after each newline:

$ awk 'BEGIN { OFS = ";"; ORS = "\n\n" } > { print $1, $2 }' BBS-list -| aardvark;555-5553 -| -| alpo-net;555-3412 -| -| barfly;555-7685 ...

If the value of ORS does not contain a newline, the program's output is run together on a single line.

5.4 Controlling Numeric Output with print

The built-in variable OFMT contains the default format specification that print uses with sprintf when it wants to convert a number to a string for printing. The default value of OFMT is "%.6g". The way print prints numbers can be changed by supplying different format specifications as the value of OFMT, as shown in the following example:

$ awk 'BEGIN { > OFMT = "%.0f" # print numbers as integers (rounds) > print 17.23, 17.54 }' -| 17 18

According to the POSIX standard, awk's behavior is undefined if OFMT contains anything but a floating-point conversion specification. (d.c.)

5.5 Using printf Statements for Fancier Printing

For more precise control over the output format than what is normally provided by print, use printf. printf can be used to specify the width to use for each item, as well as various formatting choices for numbers (such as what output base to use, whether to print an exponent, whether to print a sign, and how many digits to print after the decimal point). This is done by supplying a string, called the format string, that controls how and where to print the other arguments.

5.5.1 Introduction to the printf Statement		Syntax of the printf statement.
5.5.2 Format-Control Letters		Format-control letters.
5.5.3 Modifiers for printf Formats		Format-specification modifiers.
5.5.4 Examples Using printf		Several examples.

5.5.1 Introduction to the printf Statement

A simple printf statement looks like this:

printf format, item1, item2, ...

The entire list of arguments may optionally be enclosed in parentheses. The parentheses are necessary if any of the item expressions use the `>' relational operator; otherwise it can be confused with a redirection (see section Redirecting Output of print and printf).

The difference between printf and print is the format argument. This is an expression whose value is taken as a string; it specifies how to output each of the other arguments. It is called the format string.

The format string is very similar to that in the ISO C library function printf. Most of format is text to output verbatim. Scattered among this text are format specifiers---one per item. Each format specifier says to output the next item in the argument list at that place in the format.

The printf statement does not automatically append a newline to its output. It outputs only what the format string specifies. So if a newline is needed, you must include one in the format string. The output separator variables OFS and ORS have no effect on printf statements. For example:

$ awk 'BEGIN { > ORS = "\nOUCH!\n"; OFS = "+" > msg = "Dont Panic!" > printf "%s\n", msg > }' -| Dont Panic!

Here, neither the `+' nor the `OUCH' appear when the message is printed.

5.5.2 Format-Control Letters

A format specifier starts with the character `%' and ends with a format-control letter---it tells the printf statement how to output one item. The format-control letter specifies what kind of value to print. The rest of the format specifier is made up of optional modifiers that control how to print the value, such as the field width. Here is a list of the format-control letters:

%c

This prints a number as an ASCII character; thus, `printf "%c", 65' outputs the letter `A'. (The output for a string value is the first character of the string.)

%d, %i

These are equivalent; they both print a decimal integer. (The `%i' specification is for compatibility with ISO C.)

%e, %E

These print a number in scientific (exponential) notation; for example:

printf "%4.3e\n", 1950

prints `1.950e+03', with a total of four significant figures, three of which follow the decimal point. (The `4.3' represents two modifiers, discussed in the next subsection.) `%E' uses `E' instead of `e' in the output.

%f

This prints a number in floating-point notation. For example:

printf "%4.3f", 1950

prints `1950.000', with a total of four significant figures, three of which follow the decimal point. (The `4.3' represents two modifiers, discussed in the next subsection.)

%g, %G

These print a number in either scientific notation or in floating-point notation, whichever uses fewer characters; if the result is printed in scientific notation, `%G' uses `E' instead of `e'.

%o

This prints an unsigned octal integer.

%s

This prints a string.

%u

This prints an unsigned decimal integer. (This format is of marginal use, because all numbers in awk are floating-point; it is provided primarily for compatibility with C.)

%x, %X

These print an unsigned hexadecimal integer; `%X' uses the letters `A' through `F' instead of `a' through `f'.

%%

This isn't a format-control letter but it does have meaning--the sequence `%%' outputs one `%'; it does not consume an argument and it ignores any modifiers.

Note: When using the integer format-control letters for values that are outside the range of a C long integer, gawk switches to the `%g' format specifier. Other versions of awk may print invalid values or do something else entirely. (d.c.)

5.5.3 Modifiers for printf Formats

A format specification can also include modifiers that can control how much of the item's value is printed, as well as how much space it gets. The modifiers come between the `%' and the format-control letter. We will use the bullet symbol "*" in the following examples to represent spaces in the output. Here are the possible modifiers, in the order in which they may appear:

N$

An integer constant followed by a `$' is a positional specifier. Normally, format specifications are applied to arguments in the order given in the format string. With a positional specifier, the format specification is applied to a specific argument, instead of what would be the next argument in the list. Positional specifiers begin counting with one:

printf "%s %s\n", "don't", "panic" printf "%2$s %1$s\n", "panic", "don't"

prints the famous friendly message twice.

At first glance, this feature doesn't seem to be of much use. It is in fact a gawk extension, intended for use in translating messages at runtime. See section Rearranging printf Arguments, which describes how and why to use positional specifiers. For now, we will not use them.

-

The minus sign, used before the width modifier (see further on in this table), says to left-justify the argument within its specified width. Normally, the argument is printed right-justified in the specified width. Thus:

printf "%-4s", "foo"

prints `foo*'.

space

For numeric conversions, prefix positive values with a space and negative values with a minus sign.

+

The plus sign, used before the width modifier (see further on in this table), says to always supply a sign for numeric conversions, even if the data to format is positive. The `+' overrides the space modifier.

#

Use an "alternate form" for certain control letters. For `%o', supply a leading zero. For `%x' and `%X', supply a leading `0x' or `0X' for a nonzero result. For `%e', `%E', and `%f', the result always contains a decimal point. For `%g' and `%G', trailing zeros are not removed from the result.

0

A leading `0' (zero) acts as a flag that indicates that output should be padded with zeros instead of spaces. This applies even to non-numeric output formats. (d.c.) This flag only has an effect when the field width is wider than the value to print.

width

This is a number specifying the desired minimum width of a field. Inserting any number between the `%' sign and the format-control character forces the field to expand to this width. The default way to do this is to pad with spaces on the left. For example:

printf "%4s", "foo"

prints `*foo'.

The value of width is a minimum width, not a maximum. If the item value requires more than width characters, it can be as wide as necessary. Thus, the following:

printf "%4s", "foobar"

prints `foobar'.

Preceding the width with a minus sign causes the output to be padded with spaces on the right, instead of on the left.

.prec

A period followed by an integer constant specifies the precision to use when printing. The meaning of the precision varies by control letter:

%e, %E, %f

Number of digits to the right of the decimal point.

%g, %G

Maximum number of significant digits.

%d, %i, %o, %u, %x, %X

Minimum number of digits to print.

%s

Maximum number of characters from the string that should print.

Thus, the following:

printf "%.4s", "foobar"

prints `foob'.