Text Processing

Text processing is where the Unix philosophy proves itself: small line-oriented filters, each doing one thing, composed through pipes into arbitrarily powerful transformations. grep selects, sed edits, awk computes — and behind all three, regular expressions, the shared language of pattern matching. This chapter is a working reference: deep enough to actually use, organized by what each tool is for rather than its man-page order.

The shared substrate — regular expressions

Every filter speaks regex — learn it once, use it everywhere:

.        # any char        *    # 0+ of previous    +    # 1+ (ERE)
?        # 0-1 (ERE)       ^    # line start        $    # line end
[abc]    # char class      [^abc] # negated class   [a-z] # range
[a-z]{3} # exactly 3 (ERE) {2,4} # 2-4             |    # alternation (ERE)
()       # group (ERE)     \1   # backreference     \b   # word boundary
\<word\> # word match (GNU)

Two dialects matter: BRE (basic — grep/sed default: *, ., [], \(\), \{n,m\}) vs ERE (extended — grep -E/sed -E/awk native: +, ?, |, () unescaped). fgrep/grep -F is literal — no regex at all.

grep — the selector

grep pattern file              # matching lines
grep -v pattern file           # invert — non-matching
grep -i pattern file           # case-insensitive
grep -c pattern file           # count, don't print
grep -n pattern file           # line numbers
grep -l pattern *.c            # just filenames
grep -r pattern dir/           # recursive
grep -E 'a|b' file             # extended regex
grep -F 'literal.$*' file      # literal string — no regex
grep -w word file              # whole word only
grep -x 'exact line' file      # whole line
grep -A3 -B2 match file        # context — 3 after, 2 before
grep -o '[0-9]*' file          # just the matches
grep --color=always pat file | less -R   # colored through a pager

The idioms that pay: grep -rn 'TODO' src/, dmesg | grep -i error, ps aux | grep -v grep | grep myapp, grep -c . file (nonempty lines), grep -Po '(?<=key=)\w+' file (Perl mode for lookarounds).

sed — the stream editor

sed 's/old/new/' file          # first occurrence per line
sed 's/old/new/g' file         # all occurrences
sed 's/old/new/2' file         # the 2nd per line
sed -i 's/a/b/g' file          # EDIT IN PLACE — dangerous and essential
sed -i.bak 's/a/b/g' file      # in place with a .bak backup — the safe habit
sed '5d' file                  # delete line 5
sed '5,10d' file               # delete range
sed '/pattern/d' file          # delete matching lines
sed -n '5,10p' file            # print ONLY lines 5-10
sed 's/^/  /' file             # indent every line
sed 's/[[:space:]]*$//' file   # strip trailing whitespace
sed -E 's/([0-9]+)/[\1]/' file # groups + backreferences
sed 'y/abc/ABC/' file          # transliterate

Address,command grammar: sed '10,20s/a/b/', sed '/start/,/end/d' — select lines by number or pattern, then act. The one to memorize: sed -i.bak 's/x/y/g' file — the instant fix-everywhere with an escape hatch.

awk — the field processor

awk is a language on streams — fields auto-split, pattern-action pairs:

awk '{print $1}' file           # first field of every line
awk -F: '{print $1}' /etc/passwd # custom delimiter
awk '{print $NF}' file          # last field
awk 'NR==5' file                # line 5
awk 'NR>=10 && NR<=20' file     # range
awk '/error/' log               # matching lines
awk '$3 > 100' file             # numeric field test
awk '{sum+=$1} END{print sum}'  # accumulation — the killer idiom
awk '{count[$1]++} END{for(k in count)print k,count[k]}' file  # frequency table
awk '{print NR, $0}' file       # numbered lines
awk -F: '{print $1, $3}' /etc/passwd   # multi-field
awk 'BEGIN{FS=":"; OFS="\t"} {print $1,$3}' /etc/passwd
awk -v var=x '{print var}'      # pass in shell vars

The three blocks: BEGIN{} before input, {...} per line, END{} after — the accumulation pattern {sum+=$1} END{...} is awk's superpower; the {a[$1]++} END{for...} frequency table replaces half the quick scripts ever written.

The composition grammar

The real art is the chains:

# the histogram — count and rank anything
cat log | awk '{print $1}' | sort | uniq -c | sort -rn | head

# the extract-transform — pull, clean, report
grep 'ERROR' app.log | sed 's/.*ERROR: //' | sort | uniq -c | sort -rn

# the field-surgery — reshape structured data
awk -F: '{print $3, $1}' /etc/passwd | sort -n | head

# the find-and-fix — locate and rewrite in place
grep -rl 'oldname' src/ | xargs sed -i.bak 's/oldname/newname/g'

When to reach for which

Beyond the trio — the supporting cast

paste f1 f2                    # join files column-wise
join -t: -1 3 -2 1 a b         # relational join on fields
comm -12 <(sort a) <(sort b)   # set ops — lines in BOTH
comm -23 <(sort a) <(sort b)   # in a not b
diff -u old new                # the difference — and the patch
patch -p1 < fix.patch          # apply the diff
column -t -s: file             # pretty-print columns
fold -w80 file                 # wrap at 80
nl file                        # number lines
rev file; tac file             # reverse chars; reverse lines

Where text processing goes wrong

Exercises

1. grep -c . file — count nonempty lines; then without -c.
2. history | awk '{print $2}' | sort | uniq -c | sort -rn | head — your top commands.
3. sed -i.bak 's/foo/bar/g' on a scratch file; diff the .bak.
4. awk '{s+=$1} END{print s/NR}' — an average in one line.
5. grep -rl 'old' . | xargs sed -i 's/old/new/g' — project-wide rename.
6. Write an awk frequency table for /etc/passwd shells.

See also

• Core Utilities · Shell · Shell scripting · Linux