Text Processing

Text processing is where the Unix philosophy proves itself: small line-oriented filters, each doing one thing, composed through pipes into arbitrarily powerful transformations. grep selects, sed edits, awk computes — and behind all three, regular expressions, the shared language of pattern matching. This chapter is a working reference: deep enough to actually use, organized by what each tool is for rather than its man-page order.

The shared substrate — regular expressions

Every filter speaks regex — learn it once, use it everywhere:

.        # any char        *    # 0+ of previous    +    # 1+ (ERE)
?        # 0-1 (ERE)       ^    # line start        $    # line end
[abc]    # char class      [^abc] # negated class   [a-z] # range
[a-z]{3} # exactly 3 (ERE) {2,4} # 2-4             |    # alternation (ERE)
()       # group (ERE)     \1   # backreference     \b   # word boundary
\<word\> # word match (GNU)

Two dialects matter: BRE (basic — grep/sed default: *, ., [], \(\), \{n,m\}) vs ERE (extended — grep -E/sed -E/awk native: +, ?, |, () unescaped). fgrep/grep -F is literal — no regex at all.

grep — the selector

grep pattern file              # matching lines
grep -v pattern file           # invert — non-matching
grep -i pattern file           # case-insensitive
grep -c pattern file           # count, don't print
grep -n pattern file           # line numbers
grep -l pattern *.c            # just filenames
grep -r pattern dir/           # recursive
grep -E 'a|b' file             # extended regex
grep -F 'literal.$*' file      # literal string — no regex
grep -w word file              # whole word only
grep -x 'exact line' file      # whole line
grep -A3 -B2 match file        # context — 3 after, 2 before
grep -o '[0-9]*' file          # just the matches
grep --color=always pat file | less -R   # colored through a pager

The idioms that pay: grep -rn 'TODO' src/, dmesg | grep -i error, ps aux | grep -v grep | grep myapp, grep -c . file (nonempty lines), grep -Po '(?<=key=)\w+' file (Perl mode for lookarounds).

sed — the stream editor

sed 's/old/new/' file          # first occurrence per line
sed 's/old/new/g' file         # all occurrences
sed 's/old/new/2' file         # the 2nd per line
sed -i 's/a/b/g' file          # EDIT IN PLACE — dangerous and essential
sed -i.bak 's/a/b/g' file      # in place with a .bak backup — the safe habit
sed '5d' file                  # delete line 5
sed '5,10d' file               # delete range
sed '/pattern/d' file          # delete matching lines
sed -n '5,10p' file            # print ONLY lines 5-10
sed 's/^/  /' file             # indent every line
sed 's/[[:space:]]*$//' file   # strip trailing whitespace
sed -E 's/([0-9]+)/[\1]/' file # groups + backreferences
sed 'y/abc/ABC/' file          # transliterate

Address,command grammar: sed '10,20s/a/b/', sed '/start/,/end/d' — select lines by number or pattern, then act. The one to memorize: sed -i.bak 's/x/y/g' file — the instant fix-everywhere with an escape hatch.

awk — the field processor

awk is a language on streams — fields auto-split, pattern-action pairs:

awk '{print $1}' file           # first field of every line
awk -F: '{print $1}' /etc/passwd # custom delimiter
awk '{print $NF}' file          # last field
awk 'NR==5' file                # line 5
awk 'NR>=10 && NR<=20' file     # range
awk '/error/' log               # matching lines
awk '$3 > 100' file             # numeric field test
awk '{sum+=$1} END{print sum}'  # accumulation — the killer idiom
awk '{count[$1]++} END{for(k in count)print k,count[k]}' file  # frequency table
awk '{print NR, $0}' file       # numbered lines
awk -F: '{print $1, $3}' /etc/passwd   # multi-field
awk 'BEGIN{FS=":"; OFS="\t"} {print $1,$3}' /etc/passwd
awk -v var=x '{print var}'      # pass in shell vars

The three blocks: BEGIN{} before input, {...} per line, END{} after — the accumulation pattern {sum+=$1} END{...} is awk's superpower; the {a[$1]++} END{for...} frequency table replaces half the quick scripts ever written.

The composition grammar

The real art is the chains:

# the histogram — count and rank anything
cat log | awk '{print $1}' | sort | uniq -c | sort -rn | head

# the extract-transform — pull, clean, report
grep 'ERROR' app.log | sed 's/.*ERROR: //' | sort | uniq -c | sort -rn

# the field-surgery — reshape structured data
awk -F: '{print $3, $1}' /etc/passwd | sort -n | head

# the find-and-fix — locate and rewrite in place
grep -rl 'oldname' src/ | xargs sed -i.bak 's/oldname/newname/g'

When to reach for which

Regex deep — the grammar table

# BRE (grep/sed default)          ERE (grep -E, awk, sed -E)
.                                .       # any char
*                                *       # 0+ of previous
\?                               ?       # 0-1
\+                               +       # 1+
\{n\} \{n,\} \{n,m\}            {n} {n,} {n,m}    # counts
\(...\)  \1                      (...) \1 $1      # groups + refs
\|                               |       # alternation
[abc] [^abc] [a-z]               same
^ $ \< \> \b                     same
[[:alpha:]] [[:digit:]] [[:space:]] [[:alnum:]] [[:lower:]] [[:upper:]] [[:xdigit:]]   # named classes

Grep's -P (Perl) adds lookahead/lookbehind (?=), (?!), (?<=), (?<!), lazy quantifiers *?, named groups (?<name>) — the full regex power when ERE runs out.

sed — the address grammar

sed's power is the address — which lines the command hits:

sed '5d'                    # line 5
sed '5,10d'                 # range
sed '$d'                    # last line
sed '/pat/d'                # matching
sed '/start/,/end/d'        # between two patterns — sections!
sed '/pat/!d'               # invert — keep only matching
sed -n '10,20p'             # print only a range
sed '1~2d'                  # every other line — GNU step syntax
sed '/pat/,+3d'             # match + 3 after — GNU
sed 's/a/b/2'               # the 2nd occurrence per line
sed 'y/abc/ABC/'            # transliterate
sed '/^$/d'                 # blank lines out
sed 's/^/#/'                # comment every line
sed '10i\new line' file     # insert before line 10
sed '10a\line after' file   # append after
sed '10c\replace this' file # change line 10 entirely
sed '/pat/ s/a/b/'          # only on matching lines
sed 's/a/b/g; s/x/y/'       # multiple commands
sed -e 'cmd1' -e 'cmd2'     # or -e-chained

awk — the language surface

awk 'NR==5'                      # line number test
awk 'NR%2==0'                    # every other line
awk '$NF>100'                    # last-field test
awk 'length($0)>80'              # line length
awk -F'[,;]' '{print $2}'        # multi-char delimiter
awk '{print $2, $1}'             # field reordering
awk '{a[$1]++} END{for(k in a)print k, a[k]}'  # frequency table
awk '{a[$1]+=$3} END{for(k in a)print k, a[k]}'  # sum by key
awk '{s+=$1; n++} END{print s/n}' # running average
awk '!seen[$0]++'                # dedupe preserving order — uniq without sort
awk '{print > "out_" $1 ".txt"}' # split file by field
awk '/start/,/end/'              # range extraction
awk -v OFS='\t' '{print $1,$2}'  # output separator
awk 'BEGIN{FS=":"}{print $1}'    # set FS in BEGIN
getline                          # read next line into $0
substr, length, index, tolower, toupper, gsub, sub, match, split, sprintf, system()   # the builtins

The functions that earn their keep: gsub(pat,rep) global-substitute in $0; split(s,a,delim) into an array; system("cmd") shell out; sprintf formatted; match with RSTART/RLENGTH; next/nextfile skip.

The composition grammar — deeper

# top-talkers: who logged in most
last | awk '{print $1}' | sort | uniq -c | sort -rn | head

# the running-total histogram
cat data | awk '{a[$1]+=$2} END{for(k in a) printf "%s %d\n", k, a[k]}' | sort -k2 -rn

# the dedupe-with-count
sort file | uniq -c | sort -rn

# the extract-between-markers
sed -n '/BEGIN/,/END/p' file

# the field-swap + reformat
awk -F: '{print $7, $1, $3}' /etc/passwd | sort

# project-wide find-and-replace
grep -rl 'oldname' src/ | xargs sed -i.bak 's/oldname/newname/g'

# the multi-criteria
awk '$3 > 100 && $2 ~ /warn/' log

# the section-average
awk '/section/{s+=$NF; n++} END{print s/n}'

The supporting cast — extended

paste -d, f1 f2                 # join columns
join -t: -1 3 -2 1 a b          # relational join on key fields
comm -12 <(sort a) <(sort b)    # intersection
comm -13                        # only-b
comm -23                        # only-a
diff -u old new > fix.patch     # the patch
patch -p1 < fix.patch           # apply
diff -r dir1 dir2               # tree diff
sdiff -w80 l r                  # side-by-side
column -t -s:                   # align columns
column -t -s, -o' | '           # csv to table
fold -w80                       # wrap
fmt -w72                        # reflow paragraphs
pr -2 -t file                   # two columns
nl -ba                          # number lines
tac                             # reverse lines — cat backwards
rev                             # reverse chars
od -c file | head               # byte-level
xxd file | head                 # hex
strings binary | grep -i version  # printable strings in binaries
expand -t4 / unexpand -a        # tabs ↔ spaces
iconv -f latin1 -t utf-8        # encoding conversion
dos2unix / unix2dos             # line endings

Beyond the trio — the supporting cast

paste f1 f2                    # join files column-wise
join -t: -1 3 -2 1 a b         # relational join on fields
comm -12 <(sort a) <(sort b)   # set ops — lines in BOTH
comm -23 <(sort a) <(sort b)   # in a not b
diff -u old new                # the difference — and the patch
patch -p1 < fix.patch          # apply the diff
column -t -s: file             # pretty-print columns
fold -w80 file                 # wrap at 80
nl file                        # number lines
rev file; tac file             # reverse chars; reverse lines

Where text processing goes wrong

Exercises

1. grep -c . file — count nonempty lines; then without -c.
2. history | awk '{print $2}' | sort | uniq -c | sort -rn | head — your top commands.
3. sed -i.bak 's/foo/bar/g' on a scratch file; diff the .bak.
4. awk '{s+=$1} END{print s/NR}' — an average in one line.
5. grep -rl 'old' . | xargs sed -i 's/old/new/g' — project-wide rename.
6. Write an awk frequency table for /etc/passwd shells.

See also

• Core Utilities · Shell · Shell scripting · Linux