| | {{infobox |
| | | title = Core Utilities |
| | | chapter = Linux Handbook, chapter 10 — the vocabulary: ls, cp, mv, find, tar, dd, and the rest |
| | | core | The ~150 commands every Linux shares — GNU coreutils + friends |
| | | skills = file ops · navigation · search · archive · measuring · the toolkit map |
| | | usage | This is a reference chapter — scan it, then keep it at hand |
| | | next | Text processing — grep, sed, awk: the deep filters |
| | }} |
| | |
| | **The core utilities are the vocabulary** — roughly 150 commands (GNU coreutils plus tar, findutils, and friends) that every Linux carries and every real task composes. This chapter is a working reference: not every flag, but the *shape* of each tool — what it's for, the flags that earn their keep, and the idioms that combine them. Skim it once, keep it forever. |
| | |
| | ## The file trio |
| | |
| | ```sh |
| | cp -r src/ dest/ # recursive; -a = archive (perms+times) |
| | cp -i file dest # ask before clobbering |
| | mv old new # rename/move — free within a filesystem |
| | mv -i file dir/ # safe move |
| | rm -r dir/ # recursive delete — NO UNDO |
| | rm -rf dir/ # force — the honest footgun |
| | rm -i file # ask each time — the habit worth keeping |
| | mkdir -p a/b/c # -p makes parents |
| | rmdir emptydir # only empty dirs — rm -r for real |
| | ``` |
| | |
| - | `cp -a` preserves everything; `mv` between filesystems silently becomes copy+delete; `rm -rf` has no recovery — the three truths nobody forgets after learning them the hard way. |
| + | `cp -a` preserves everything (GNU — non-POSIX; portable spell is `cp -RpP`); `mv` between filesystems silently becomes copy+delete; `rm -rf` has no recovery — the three truths nobody forgets after learning them the hard way. |
| | |
| | ## Navigation + inspection |
| | |
| | ```sh |
| | ls -la # long + hidden — the working form |
| | ls -lhS # human sizes, biggest first |
| | ls -lt # newest first |
| | ls -li # inode numbers |
| | pwd; cd -; cd ~ # where am I; back; home |
| | stat file # the full inode record |
| | file mystery # WHAT is this — magic bytes, not extension |
| - | du -sh dir # total size; du -h --max-depth=1 for the map |
| + | du -sh dir # total size; du -h --max-depth=1 for the map (GNU flag — BSD du lacks it) |
| | du -h | sort -rh | head # where space went |
| | df -h # filesystem free space |
| | which cmd; type -a cmd # where it lives; all resolutions |
| | ``` |
| | |
| | ## Find — the real search engine |
| | |
| | ```sh |
| | find . -name '*.c' # by name |
| | find . -iname 'readme*' # case-insensitive |
| | find / -type f -size +100M 2>/dev/null # space hogs |
| | find . -mtime -7 # changed this week |
| | find . -perm 777 # by mode |
| | find . -user alice -group devs # by owner |
| | find . -empty # empty files/dirs |
| | find . -name '*.log' -delete # act on results — careful! |
| | find . -name '*.tmp' -exec rm {} + # exec — {} is each path, + batches |
| | find . -print0 | xargs -0 cmd # the space-safe pipeline |
| | ``` |
| | |
| | `find ... -exec` vs `| xargs`: `-exec {} +` is the modern safe form; `xargs` without `-0` breaks on spaces — always pair `find -print0 | xargs -0` when filenames might be weird. |
| | |
| | ## Archives — tar, plus the compressors |
| | |
| | ```sh |
| | tar -cf archive.tar dir/ # create |
| | tar -xf archive.tar # extract |
| | tar -tf archive.tar # list contents |
| | tar -czf a.tar.gz dir/ # gzip |
| | tar -cjf a.tar.bz2 dir/ # bzip2 |
| | tar -cJf a.tar.xz dir/ # xz — best ratio |
| | gzip file; gunzip file.gz # standalone compressors |
| | xz file; xz -d file.xz |
| | zstd -19 file # modern fast+strong |
| | zip -r a.zip dir/; unzip a.zip |
| | ``` |
| | |
| | The flags decompose: `c`/`x`/`t` mode, `f` file, `z`/`j`/`J` compressor. `tar -tf` before `-xf` — always look before you unpack. |
| | |
| | ## dd — the block-level tool |
| | |
| | ```sh |
| | dd if=/dev/sda of=disk.img bs=4M status=progress # image a disk |
| | dd if=linux.iso of=/dev/sdX bs=4M status=progress && sync # bootable USB |
| | dd if=/dev/zero of=file bs=1M count=100 # a 100MB file of zeros |
| | dd if=/dev/urandom of=file bs=1M count=10 # random bytes |
| | dd if=/dev/sda of=/dev/sdb bs=64K conv=noerror,sync # rescue a dying disk |
| | ``` |
| | |
| | `if`/`of` are raw devices as easily as files — `dd` writes boot sectors, wipes disks, images drives. Check `of=` twice; it overwrites without asking. |
| | |
| | ## Text plumbing — the everyday filters |
| | |
| | ```sh |
| | cat file; cat a b > combined # concatenate (useless-cat is a meme for a reason) |
| | head -20 file; tail -20 file # ends |
| | tail -f /var/log/syslog # follow — the log-watcher |
| | less file # the pager — /to-search, q to quit |
| | wc -l file # lines/words/bytes |
| | sort file; sort -n; sort -k2 # sort; numeric; by field |
| | sort | uniq -c | sort -nr # the histogram idiom |
| | cut -d: -f1 /etc/passwd # field extraction |
| | cut -c1-80 file # by column |
| | tr 'a-z' 'A-Z' < file # translate chars |
| | tr -d '\n' < file # delete chars |
| | tee out.log # stdout → both screen AND file |
| | xargs -n1 cmd < list # build argv from stdin |
| | ``` |
| | |
| | ## The process watchers |
| | |
| | ```sh |
| | ps aux # everything running |
| | ps -ef | grep name # find a process |
| | pgrep -l name # PIDs by name |
| | top; htop # live views |
| | free -h # memory |
| | uptime # load averages |
| | kill PID; kill -9 PID; killall name # signals |
| | df -h; mount; umount # filesystem plumbing |
| | ``` |
| | |
| | ## The toolkit map — the rest worth knowing |
| | |
| | | Tool | Job | |
| | | --- | --- | |
| - | | `chmod/chown/chgrp` | permissions/ownership — deep dive in [[Users and permissions]] | |
| + | | `chmod/chown/chgrp` | permissions/ownership — deep dive in [[Users and Permissions]] | |
| | | `ln -s` / `ln` | symlinks / hard links | |
| | | `touch` | create/timestamp files | |
| | | `readlink -f` | resolve a path fully | |
| | | `basename`/`dirname` | path surgery | |
| | | `realpath` | canonicalize | |
| | | `date` | timestamps — `date -u`, `date +%s` | |
| | | `seq` | number ranges — `seq 1 10` | |
| | | `yes` | infinite stream — `yes | cmd` | |
| | | `sleep` | pause — `sleep 5` | |
| | | `timeout 10 cmd` | kill after N seconds | |
| | | `watch -n2 cmd` | rerun + refresh | |
| | | `man cmd` · `info` | the manuals | |
| | | `apropos kw` | search man pages | |
| | | `alias`/`unalias` | shell shortcuts | |
| | | `history` | the command log — `!n`, `!!`, `Ctrl-R` | |
| | | `env`/`printenv` | the environment | |
| | | `uname -a` | kernel/arch | |
| | | `hostname` | the machine's name | |
| | | `id`, `whoami`, `who`, `w` | who/what | |
| | | `cal`, `bc` | calendar, calculator | |
| | | `od`, `xxd`, `hexdump` | binary viewing | |
| | | `strings file` | printable strings in binaries | |
| | | `nm`, `objdump`, `readelf`, `ldd` | binary anatomy | |
| | |
| | ## The complete toolkit — the reference table |
| | |
| | | Tool | The job | The form that matters | |
| | | --- | --- | --- | |
| | | `ls` | list | `-la` `-lhS` `-lt` `-li` `-R` `-d dir/` | |
| | | `cp` | copy | `-r` `-a` `-i` `-u` `--preserve=all` | |
| | | `mv` | move/rename | `-i` `-u` `-n` | |
| | | `rm` | remove | `-r` `-f` `-i` `-I` `-v` | |
| | | `mkdir` | dirs | `-p` `-m mode` `-v` | |
| | | `rmdir` | empty dirs | `-p` | |
| | | `ln` | links | `-s` symlink; bare = hard link | |
| | | `touch` | create/timestamp | `-d "date"` `-t YYYYMMDDhhmm` | |
| | | `cat` | concatenate | `-n` `-A` (show nonprinting) `-s` | |
| | | `head`/`tail` | ends | `-n20` `-c100` `-f` (follow) `+5` | |
| | | `less` | pager | `+/pat` `-N` `-S` `-R` | |
| | | `wc` | count | `-l` `-w` `-c` `-L` | |
| | | `sort` | sort | `-n` `-r` `-u` `-k2` `-t:` `-h` `-R` | |
| | | `uniq` | dedupe | `-c` `-d` `-u` `-i` (needs sorted input) | |
| | | `cut` | fields | `-d: -f1` `-c1-10` `-f2-` | |
| | | `tr` | translate | `'a-z' 'A-Z'` `-d '\n'` `-s ' '` | |
| | | `paste` | join cols | `-d,` `-s` serial | |
| | | `join` | relational join | `-t:` `-1 n -2 m` `-a1` | |
| | | `comm` | set ops | `-12` `-13` `-23` on sorted | |
| | | `split` | break files | `-l 1000` `-b 1M` `-n 4` | |
| | | `csplit` | split on patterns | `/pat/` `{*}` | |
| | | `tee` | fork output | `cmd \| tee file` `-a` | |
| | | `xargs` | argv from stdin | `-0` `-n1` `-P4` `-I{}` `-t` | |
| | | `find` | the searcher | the full grammar — ch 10 | |
| | | `locate` | indexed search | `locate file` (updatedb builds db) | |
| | | `which`/`type`/`command -v` | resolution | `type -a` all hits | |
| | | `whereis` | locate binaries | `whereis -b ls` | |
| | | `stat` | inode record | `-c %a %U %s` formats | |
| | | `file` | magic bytes | `-i` mime `-b` bare | |
| - | | `du` | usage | `-sh` `--max-depth=1` `-x` `-ah` | |
| + | | `du` | usage | `-sh` `--max-depth=1`(GNU) `-x` `-ah` | |
| | | `df` | fs space | `-h` `-i` `-T` `-x tmpfs` | |
| | | `tar` | archive | `c/x/t` + `f` + `z/j/J` + `v` + `-C` | |
| | | `gzip`/`bzip2`/`xz`/`zstd` | compress | `-d` `-k` `-9` `-T0` | |
| | | `zip`/`unzip` | zip | `-r` `-l` `-d` | |
| | | `dd` | block copy | `if of bs count status=progress conv=noerror` | |
| | | `shred` | secure delete | `-u` `-n3` `-z` | |
| | | `sync` | flush write cache | after dd to removable | |
| | | `md5sum`/`sha256sum` | checksums | `-c file.sums` `-b` | |
| | | `base64`/`base32` | encode | `-d` `-w0` | |
| | | `od`/`xxd`/`hexdump` | hex views | `-c` `-C` `-An` | |
| | | `strings` | printable text | `-n4` `-t x` `-a` | |
| | | `nm`/`objdump`/`readelf`/`ldd`/`ar` | binary anatomy | `nm -C` `objdump -d` `readelf -h` `ldd` `ar rcs` | |
| | | `date` | the clock | `+%s` `-d "date"` `-u` `-R` | |
| | | `cal`/`ncal` | calendar | `-3` `-y` `-j` | |
| | | `seq` | sequences | `1 10` `-w` `-s,` `-f %02g` | |
| | | `yes` | infinite stream | `yes \| cmd` `yes n` | |
| | | `sleep`/`usleep` | pause | `5` `0.5` `1m` | |
| | | `timeout` | time cap | `30 cmd` `-k 5` `--signal=KILL` | |
| | | `watch` | refresh | `-n1` `-d` `-t` `-c` | |
| | | `time` | measure | `-v` `-p` — the builtin vs /usr/bin/time | |
| | | `env`/`printenv` | environment | `-i` `-u VAR` `env VAR=x cmd` | |
| | | `printenv`/`set`/`export` | vars | `printenv VAR` | |
| | | `uname` | system id | `-a` `-r` `-m` `-s` `-n` `-v` | |
| | | `hostname` | the name | `-f` `-i` `-I` | |
| | | `id`/`whoami`/`who`/`w`/`users`/`last` | who | `id -u` `w` `last -20` `lastlog` | |
| | | `man`/`info`/`apropos`/`whatis` | manuals | `man 5 passwd` `-k kw` | |
| | | `alias`/`unalias`/`history`/`fc` | shell memory | `!!` `!n` `!str` `Ctrl-R` | |
| | | `true`/`false`/`test`/`[` | predicates | `test -f` `[ -d dir ]` | |
| | | `expr`/`bc`/`dc` | math | `expr 1 + 2` `bc -l` `echo '2+2' \| bc` | |
| | | `printf` | formatted output | `'%s\n'` `%-10s` `%05d` | |
| | | `read` | input | `-r` `-p "prompt"` `-s` `-a arr` | |
| | | `getopts` | arg parsing | `while getopts "a:b:" o` | |
| | | `basename`/`dirname`/`realpath`/`readlink` | path surgery | `realpath -m` `readlink -f` | |
| | | `mktemp` | temp files | `-d` `-u` `-t` `--suffix=.x` | |
| | | `install` | copy+mode | `-D` `-m755` `-d` | |
| | | `truncate`/`fallocate` | file sizing | `-s 1G` `-l 1G` | |
| | | `iconv`/`dos2unix`/`unix2dos` | encoding/CRLF | `-f latin1 -t utf8` | |
| | | `locale`/`localectl` | locale | `-a` `locale -k LC_TIME` | |
| | | `stty`/`tput`/`reset`/`clear` | terminal | `stty -a` `cols 80` `tput cols` | |
| | | `script`/`scriptreplay` | session record | `-a log` `-c cmd` | |
| | | `expect`/`unbuffer` | automation | pty-driving | |
| | | `cron`/`at`/`batch`/`anacron` | scheduling | `at now +5min` `atq` `atrm` | |
| - | | `logger`/`systemd-cat` | write to syslog | `logger -t tag msg` | |
| + | | `logger`/`[[systemd]]-cat` | write to syslog | `logger -t tag msg` | |
| | | `dmesg`/`journalctl`/`logrotate` | the logs | `-T` `-l err` `-b` | |
| | | `mount`/`umount`/`findmnt`/`lsblk`/`blkid` | storage plumbing | `-o opts` `-R` `-l` `--bind` | |
| | | `fdisk`/`gdisk`/`parted`/`cfdisk`/`sfdisk` | partition | `-l` `print` `mkpart` | |
| | | `mkfs.*`/`mkswap`/`swapon`/`swapoff` | filesystems | `mkfs.ext4 -L` `swapon -a` | |
| | | `fsck.*`/`e2fsck`/`tune2fs`/`debugfs`/`dumpe2fs` | fs tools | `-n` `-y` `-b 32768` | |
| | | `badblocks`/`smartctl`/`hdparm` | disk health | `-a` `-I` `-s` | |
| | | `ddrescue`/`testdisk`/`photorec` | rescue | image-dying-disks, partition/files recovery | |
| | | `losetup`/`kpartx` | loop devices | `-f` `--show` `-a` | |
| | | `lsof`/`fuser` | open files | `-i :80` `-u user` `+D dir` | |
| | | `strace`/`ltrace`/`perf` | syscall tracing | `-c` `-e open` `-f` `-o log` | |
| | | `nm`/`ldd`/`objdump`/`readelf` | binary internals | the .so dependencies | |
| | | `ldconfig`/`ldd`/`LD_*` | the .so registry | `-p` `-v` `LD_LIBRARY_PATH` | |
| | | `getent`/`nsswitch` | name services | `hosts` `passwd` `services` | |
| | | `chroot`/`unshare`/`nsenter`/`setpriv` | containers | `chroot dir cmd` `unshare -n` | |
| | | `ip`/`ss`/`ping`/`traceroute`/`mtr`/`dig`/`host`/`nslookup`/`nc`/`socat`/`curl`/`wget`/`rsync`/`ssh`/`scp`/`sftp`/`tcpdump`/`ethtool`/`iftop`/`nethogs`/`iptraf`/`nmap`/`ncat`/`openssl s_client`/`nload`/`arp`/`arping`/`route`/`netstat`/`brctl`/`bridge`/`tc`/`iptables`/`nft`/`firewall-cmd`/`hostnamectl`/`resolvectl` | networking | the ch-18 toolkit | |
| | | `ssh`/`ssh-keygen`/`ssh-agent`/`ssh-add`/`ssh-copy-id`/`scp`/`sftp`/`rsync`/`mosh`/`autossh`/`sshfs`/`expect` | remote | `-L/-R/-D/-i/-p/-N/-f` `rsync -avz -P` | |
| | | `tmux`/`screen`/`dtach`/`abduco`/`byobu`/`nohup`/`disown`/`setsid`/`at` | sessions | `tmux new -s x` `attach -t` | |
| | | `git`/`svn`/`cvs`/`hg`/`patch`/`diff`/`quilt`/`diff3`/`wdiff`/`colordiff`/`delta`/`tig`/`lazygit`/`gh`/`glab` | vcs + diffs | `git log --oneline` `-p` `--graph` | |
| | | `make`/`cmake`/`meson`/`ninja`/`autoconf`/`automake`/`autoreconf`/`libtool`/`pkg-config`/`ccache`/`distcc`/`bear`/`compdb`/`mold`/`lld` | builds | `-j` `-C` `--dry-run` `-B` `setup` `ninja -C` | |
| | | `gcc`/`clang`/`cc`/`c++`/`g++`/`cpp`/`as`/`ld`/`ar`/`ranlib`/`strip`/`addr2line`/`c++filt`/`size`/`elfedit`/`eu-*`/`gdb`/`lldb`/`valgrind`/`strace`/`ltrace`/`perf`/`gcov`/`clang-format`/`clang-tidy`/`scan-build`/`flawfinder`/`cppcheck` | compilers+debug | `-save-temps` `-E -S -c` `-Wall` `-g` `-O2` `-fsanitize=address` | |
| | | `python3`/`perl`/`jq`/`yq`/`xmlstarlet`/`csvkit`/`miller`/`dasel`/`jc` | data-format tools | `jq .x` `miller --csv` `dasel -i json` | |
| | | `rg`/`fd`/`ag`/`ack`/`ripgrep` | the modern searchers | `rg pat` `fd name` — faster grep/find | |
| | | `bat`/`eza`/`lsd`/`tree`/`ncdu`/`ranger`/`vifm` | the modern views | `bat file` `eza -la` `tree` `ncdu` | |
| | | `fzf`/`zoxide`/`direnv`/`starship`/`atuin`/`hyperfine`/`just`/`task`/`entr`/`watchexec`/`parallel`/`moreutils`/`sponge`/`ts`/`vipe`/`vidir`/`flock`/`run-one`/`retry` | the power-user layer | `fzf` fuzzy-find, `entr` re-run-on-change, `parallel` | |
| | | `procps`/`psmisc`/`fuser`/`killall`/`pstree`/`pmap`/`pwdx`/`tload` | process toolkit | `pstree -p` `fuser -k /mnt` | |
| | | `vmstat`/`iostat`/`mpstat`/`sar`/`nmon`/`dstat`/`glances`/`btop`/`htop`/`atop`/`iotop`/`iftop`/`nethogs`/`powertop`/`s-tui`/`netdata` | monitors | `sar -u 1` `glances` `iotop -o` | |
| | | `stress`/`stress-ng`/`sysbench`/`bonnie`/`fio`/`ioping`/`lmbench` | benchmarks + load | `fio` io benchmarks, `stress-ng` load | |
| | | `strace`/`ltrace`/`perf`/`bpftrace`/`bcc`/`systemtap`/`ftrace`/`dwarves`/`pahole`/`crash`/`drgn`/`kdump`/`kexec` | tracing + kernel debug | `perf top` `bpftrace -e` `pahole` | |
| | |
| | The long tail is the point — the toolchain's ~150 core commands are chapter-10's vocabulary; the thousands beyond are the ecosystem the package manager reaches. |
| | |
| | ## The composition idioms |
| | |
| | The power isn't any tool — it's the pipes: |
| | |
| | ```sh |
| | history | awk '{print $2}' | sort | uniq -c | sort -rn | head # your top commands |
| | find . -name '*.log' -exec wc -l {} + | sort -n | tail # biggest logs |
| | cat access.log | cut -d' ' -f1 | sort | uniq -c | sort -rn | head # top IPs |
| | ps aux --sort=-%mem | head -11 # RAM hogs |
| | ``` |
| | |
| | `sort | uniq -c | sort -rn | head` — the frequency-histogram idiom — solves a third of all quick analysis questions by itself. |
| | |
| | ## Where utilities go wrong |
| | |
| | | Symptom | Cause | |
| | | --- | --- | |
| | | xargs broke on a filename | Spaces — `-print0 | xargs -0` | |
| | | rm deleted the wrong tree | Unquoted var with spaces — `rm -rf "$dir"` | |
| | | find found nothing on other mounts | `-xdev` restricting, or the dir is a mount | |
| | | tar extracted to wrong place | `-C dir` needed, or `cd` first | |
| | | dd hung | Missing `bs=` — defaults to 512-byte blocks | |
| | | `command not found` but it exists | `which` shows path — is it in `$PATH`? | |
| | | sort ordered "1 10 2" | Needed `-n` for numeric | |
| | |
| | ## Exercises |
| | |
| | 1. `find /usr/bin -name '*zip*'` — inventory the compressors. |
| | 2. `history | awk '{print $2}' | sort | uniq -c | sort -rn | head` — your actual most-used commands. |
| | 3. `dd` a 10MB file, `tar` it, `xz` it, unroll the whole chain. |
| | 4. `find . -empty -delete` on a scratch tree. |
| | 5. `file` five mystery files — extension vs magic bytes. |
| | |
| | ## See also |
| | |
| - | - [[Linux]] · [[Filesystem]] · [[Shell]] · [[Text processing]] |
| + | - [[Linux]] · [[Filesystem]] · [[Shell]] · [[Text Processing]] |
| | |
| | [[Category:Linux]] |
| | [[Category:Handbook]] |
| | [[Category:Reference]] |
| | [[Category:Linux Handbook]] |
| | |
