Diff — Text Processing

{{infobox

| title = Text Processing

| chapter = Linux Handbook, chapter 11 — grep, sed, awk: the deep filters

| core = Line-oriented streams · regular expressions · the Unix composability ideal

| tools | grep · sed · awk · plus sort/uniq/cut/tr/wc from chapter 10

| skill_level | The chapter that turns users into practitioners

| next | Users & permissions — the security layer

}}

**Text processing is where the Unix philosophy proves itself**: small line-oriented filters, each doing one thing, composed through pipes into arbitrarily powerful transformations. `grep` selects, `sed` edits, `awk` computes — and behind all three, **regular expressions**, the shared language of pattern matching. This chapter is a working reference: deep enough to actually use, organized by what each tool is *for* rather than its man-page order.

## The shared substrate — regular expressions

Every filter speaks regex — learn it once, use it everywhere:

```sh

.        # any char        *    # 0+ of previous    +    # 1+ (ERE)

?        # 0-1 (ERE)       ^    # line start        $    # line end

[abc]    # char class      [^abc] # negated class   [a-z] # range

[a-z]{3} # exactly 3 (ERE) {2,4} # 2-4             |    # alternation (ERE)

()       # group (ERE)     \1   # backreference     \b   # word boundary

\<word\> # word match (GNU)

```

Two dialects matter: **BRE** (basic — grep/sed default: `*`, `.`, `[]`, `\(\)`, `\{n,m\}`) vs **ERE** (extended — `grep -E`/`sed -E`/awk native: `+`, `?`, `|`, `()` unescaped). `fgrep`/`grep -F` is literal — no regex at all.

## grep — the selector

```sh

grep pattern file              # matching lines

grep -v pattern file           # invert — non-matching

grep -i pattern file           # case-insensitive

grep -c pattern file           # count, don't print

grep -n pattern file           # line numbers

grep -l pattern *.c            # just filenames

grep -r pattern dir/           # recursive

grep -E 'a|b' file             # extended regex

grep -F 'literal.$*' file      # literal string — no regex

grep -w word file              # whole word only

grep -x 'exact line' file      # whole line

grep -A3 -B2 match file        # context — 3 after, 2 before

grep -o '[0-9]*' file          # just the matches

grep --color=always pat file | less -R   # colored through a pager

```

The idioms that pay: `grep -rn 'TODO' src/`, `dmesg | grep -i error`, `ps aux | grep -v grep | grep myapp`, `grep -c . file` (nonempty lines), `grep -Po '(?<=key=)\w+' file` (Perl mode for lookarounds).

## sed — the stream editor

```sh

sed 's/old/new/' file          # first occurrence per line

sed 's/old/new/g' file         # all occurrences

sed 's/old/new/2' file         # the 2nd per line

sed -i 's/a/b/g' file          # EDIT IN PLACE — dangerous and essential

sed -i.bak 's/a/b/g' file      # in place with a .bak backup — the safe habit

sed '5d' file                  # delete line 5

sed '5,10d' file               # delete range

sed '/pattern/d' file          # delete matching lines

sed -n '5,10p' file            # print ONLY lines 5-10

sed 's/^/  /' file             # indent every line

sed 's/[[:space:]]*$//' file   # strip trailing whitespace

sed -E 's/([0-9]+)/[\1]/' file # groups + backreferences

sed 'y/abc/ABC/' file          # transliterate

```

`Address,command` grammar: `sed '10,20s/a/b/'`, `sed '/start/,/end/d'` — select lines by number or pattern, then act. The one to memorize: **`sed -i.bak 's/x/y/g' file`** — the instant fix-everywhere with an escape hatch.

## awk — the field processor

`awk` is a *language* on streams — fields auto-split, pattern-action pairs:

```sh

awk '{print $1}' file           # first field of every line

awk -F: '{print $1}' /etc/passwd # custom delimiter

awk '{print $NF}' file          # last field

awk 'NR==5' file                # line 5

awk 'NR>=10 && NR<=20' file     # range

awk '/error/' log               # matching lines

awk '$3 > 100' file             # numeric field test

awk '{sum+=$1} END{print sum}'  # accumulation — the killer idiom

awk '{count[$1]++} END{for(k in count)print k,count[k]}' file  # frequency table

awk '{print NR, $0}' file       # numbered lines

awk -F: '{print $1, $3}' /etc/passwd   # multi-field

awk 'BEGIN{FS=":"; OFS="\t"} {print $1,$3}' /etc/passwd

awk -v var=x '{print var}'      # pass in shell vars

```

The three blocks: `BEGIN{}` before input, `{...}` per line, `END{}` after — the accumulation pattern `{sum+=$1} END{...}` is awk's superpower; the `{a[$1]++} END{for...}` frequency table replaces half the quick scripts ever written.

## The composition grammar

The real art is the chains:

```sh

# the histogram — count and rank anything

cat log | awk '{print $1}' | sort | uniq -c | sort -rn | head

# the extract-transform — pull, clean, report

grep 'ERROR' app.log | sed 's/.*ERROR: //' | sort | uniq -c | sort -rn

# the field-surgery — reshape structured data

awk -F: '{print $3, $1}' /etc/passwd | sort -n | head

# the find-and-fix — locate and rewrite in place

grep -rl 'oldname' src/ | xargs sed -i.bak 's/oldname/newname/g'

```

## When to reach for which

| Task | Tool |

| --- | --- |

| "show me matching lines" | grep |

| "change this string" | sed |

| "do math/group/aggregate on fields" | awk |

| "count distinct X" | sort \| uniq -c |

| "take column N" | cut (simple) or awk |

| "swap chars / delete chars" | tr |

| "big multi-field report" | awk |

| "extract one regex bit per line" | grep -o |

## Regex deep — the grammar table

```sh

# BRE (grep/sed default)          ERE (grep -E, awk, sed -E)

.                                .       # any char

*                                *       # 0+ of previous

\?                               ?       # 0-1

\+                               +       # 1+

\{n\} \{n,\} \{n,m\}            {n} {n,} {n,m}    # counts

\(...\)  \1                      (...) \1 $1      # groups + refs

\|                               |       # alternation

[abc] [^abc] [a-z]               same

^ $ \< \> \b                     same

[[:alpha:]] [[:digit:]] [[:space:]] [[:alnum:]] [[:lower:]] [[:upper:]] [[:xdigit:]]   # named classes

```

Grep's `-P` (Perl) adds lookahead/lookbehind `(?=)`, `(?!)`, `(?<=)`, `(?<!)`, lazy quantifiers `*?`, named groups `(?<name>)` — the full regex power when ERE runs out.

## sed — the address grammar

`sed`'s power is the *address* — which lines the command hits:

```sh

sed '5d'                    # line 5

sed '5,10d'                 # range

sed '$d'                    # last line

sed '/pat/d'                # matching

sed '/start/,/end/d'        # between two patterns — sections!

sed '/pat/!d'               # invert — keep only matching

sed -n '10,20p'             # print only a range

sed '1~2d'                  # every other line — GNU step syntax

sed '/pat/,+3d'             # match + 3 after — GNU

sed 's/a/b/2'               # the 2nd occurrence per line

sed 'y/abc/ABC/'            # transliterate

sed '/^$/d'                 # blank lines out

sed 's/^/#/'                # comment every line

sed '10i\new line' file     # insert before line 10

sed '10a\line after' file   # append after

sed '10c\replace this' file # change line 10 entirely

sed '/pat/ s/a/b/'          # only on matching lines

sed 's/a/b/g; s/x/y/'       # multiple commands

sed -e 'cmd1' -e 'cmd2'     # or -e-chained

```

## awk — the language surface

```sh

awk 'NR==5'                      # line number test

awk 'NR%2==0'                    # every other line

awk '$NF>100'                    # last-field test

awk 'length($0)>80'              # line length

awk -F'[,;]' '{print $2}'        # multi-char delimiter

awk '{print $2, $1}'             # field reordering

awk '{a[$1]++} END{for(k in a)print k, a[k]}'  # frequency table

awk '{a[$1]+=$3} END{for(k in a)print k, a[k]}'  # sum by key

awk '{s+=$1; n++} END{print s/n}' # running average

awk '!seen[$0]++'                # dedupe preserving order — uniq without sort

awk '{print > "out_" $1 ".txt"}' # split file by field

awk '/start/,/end/'              # range extraction

awk -v OFS='\t' '{print $1,$2}'  # output separator

awk 'BEGIN{FS=":"}{print $1}'    # set FS in BEGIN

getline                          # read next line into $0

substr, length, index, tolower, toupper, gsub, sub, match, split, sprintf, system()   # the builtins

```

The functions that earn their keep: `gsub(pat,rep)` global-substitute in $0; `split(s,a,delim)` into an array; `system("cmd")` shell out; `sprintf` formatted; `match` with RSTART/RLENGTH; `next`/`nextfile` skip.

## The composition grammar — deeper

```sh

# top-talkers: who logged in most

last | awk '{print $1}' | sort | uniq -c | sort -rn | head

# the running-total histogram

cat data | awk '{a[$1]+=$2} END{for(k in a) printf "%s %d\n", k, a[k]}' | sort -k2 -rn

# the dedupe-with-count

sort file | uniq -c | sort -rn

# the extract-between-markers

sed -n '/BEGIN/,/END/p' file

# the field-swap + reformat

awk -F: '{print $7, $1, $3}' /etc/passwd | sort

# project-wide find-and-replace

grep -rl 'oldname' src/ | xargs sed -i.bak 's/oldname/newname/g'

# the multi-criteria

awk '$3 > 100 && $2 ~ /warn/' log

# the section-average

awk '/section/{s+=$NF; n++} END{print s/n}'

```

## The supporting cast — extended

```sh

paste -d, f1 f2                 # join columns

join -t: -1 3 -2 1 a b          # relational join on key fields

comm -12 <(sort a) <(sort b)    # intersection

comm -13                        # only-b

comm -23                        # only-a

diff -u old new > fix.patch     # the patch

patch -p1 < fix.patch           # apply

diff -r dir1 dir2               # tree diff

sdiff -w80 l r                  # side-by-side

column -t -s:                   # align columns

column -t -s, -o' | '           # csv to table

fold -w80                       # wrap

fmt -w72                        # reflow paragraphs

pr -2 -t file                   # two columns

nl -ba                          # number lines

tac                             # reverse lines — cat backwards

rev                             # reverse chars

od -c file | head               # byte-level

xxd file | head                 # hex

strings binary | grep -i version  # printable strings in binaries

expand -t4 / unexpand -a        # tabs ↔ spaces

iconv -f latin1 -t utf-8        # encoding conversion

dos2unix / unix2dos             # line endings

```

## Where text processing goes wrong

| Symptom | Cause |

| --- | --- |

| sed changed nothing | `-i` forgotten — sed prints to stdout by default |

| sed clobbered the file | `-i` without `.bak` — always `-i.bak` |

| regex matched everything | `.*` is greedy — use `[^x]*` boundaries |

| `+` didn't work in grep | BRE vs ERE — `grep -E` |

| awk fields empty | Wrong delimiter — `-F:` |

| `sort` ranked "10" before "2" | Needed `-n` |

| grep found binary garbage | `-a` to force text, or `-I` to skip binary |

| Backreference `\1` literal | Needed `-E` and `()` not `\(\)` |

## Exercises

1. `grep -c . file` — count nonempty lines; then without -c.

2. `history | awk '{print $2}' | sort | uniq -c | sort -rn | head` — your top commands.

3. `sed -i.bak 's/foo/bar/g'` on a scratch file; diff the .bak.

4. `awk '{s+=$1} END{print s/NR}'` — an average in one line.

5. `grep -rl 'old' . | xargs sed -i 's/old/new/g'` — project-wide rename.

6. Write an awk frequency table for `/etc/passwd` shells.

## See also

- [[Core Utilities]] · [[Shell]] · [[Shell scripting]] · [[Linux]]

[[Category:Linux]]

[[Category:Handbook]]

[[Category:Reference]]