| | {{infobox |
| | | title = Logs, Monitoring & Time |
| | | chapter = Linux Handbook, chapter 27 — syslog, journald, the watchers, and the clock |
| | | core | The system narrates itself — logs are how you hear it |
| | | tools | journalctl · /var/log · dmesg · top/htop · vmstat · date/timedatectl |
| | | skills | reading a system · watching it breathe · keeping its clock honest |
| | | next | The distro landscape — the same machine, dressed differently |
| | }} |
| | |
| | **A Linux system narrates itself constantly** — every daemon's complaints, every login, every kernel warning, every service's lifecycle lands in logs. The discipline is knowing where they live and how to read them. This chapter covers the two logging worlds (syslog files and journald), the monitoring toolkit, and the clock machinery — NTP, timezones, locales — the quiet infrastructure every other chapter leans on. |
| | |
| | ## The two logging worlds |
| | |
| | **Syslog files** — the classic: `/var/log/` holds text files, rotated, greppable: |
| | |
| | ```sh |
| | ls -lt /var/log/ # freshest first |
| | tail -f /var/log/syslog # Debian's main |
| | tail -f /var/log/messages # Slackware/RH's main |
| | less /var/log/auth.log # logins, sudo, sshd |
| | less /var/log/kern.log # the kernel |
| | grep 'error' /var/log/syslog | tail |
| | logrotate # the rotator — /etc/logrotate.conf + .d |
| | ``` |
| | |
| | **journald** — systemd's structured binary log: |
| | |
| | ```sh |
| | journalctl # everything, this boot |
| | journalctl -b # this boot only |
| | journalctl -b -1 # LAST boot — the crash forensics |
| | journalctl -u sshd # one unit |
| | journalctl -p err # by priority — emerg..debug |
| | journalctl -f # follow live |
| | journalctl --since "1 hour ago" --until "10 min ago" |
| | journalctl -k # kernel messages |
| | journalctl -x # + explanations |
| | journalctl --disk-usage # how much it stores |
| | journalctl --vacuum-size=100M # trim it |
| | ``` |
| | |
| | The `-b -1` trick — the *previous* boot's log — is the single most useful forensic command on systemd systems: the log of the boot that failed. |
| | |
| | ## dmesg — the kernel's own voice |
| | |
| | ```sh |
| | dmesg # the ring buffer — hardware, drivers, oopses |
| | dmesg -T # human timestamps |
| | dmesg -l err,warn # filtered by level |
| | dmesg -w # follow live |
| | dmesg | grep -i usb # device arrivals |
| | ``` |
| | |
| | `dmesg` is where hardware announces itself — boot's transcript, USB plug-ins, disk I/O errors, kernel oopses — the first place to look when hardware misbehaves. |
| | |
| + | ## The journal's fields — the queryable layer |
| + | |
| + | ```sh |
| + | journalctl _SYSTEMD_UNIT=sshd.service # by unit |
| + | journalctl _PID=1234 # by process |
| + | journalctl _UID=1000 # by user |
| + | journalctl _COMM=sshd # by command name |
| + | journalctl SYSLOG_IDENTIFIER=crond # by syslog id |
| + | journalctl -g 'Failed.*password' # regex the messages |
| + | journalctl -o json # structured — jq-able |
| + | journalctl -o short-iso-precise # precise timestamps |
| + | journalctl --since '2024-01-01' --until '2024-01-02' |
| + | journalctl --since 'yesterday' --until 'now' |
| + | journalctl -u sshd --since today -f # unit + time + follow |
| + | journalctl --field _TRANSPORT # list a field's values |
| + | journalctl --vacuum-time=7d # trim by age |
| + | journalctl --vacuum-files=20 # keep N files |
| + | ``` |
| + | |
| + | ## syslog — the classic file grammar |
| + | |
| + | ```sh |
| + | /var/log/messages # RH/Slackware's main |
| + | /var/log/syslog # Debian's main |
| + | /var/log/auth.log # Debian auth · /var/log/secure on RH |
| + | /var/log/kern.log # kernel |
| + | /var/log/daemon.log # services |
| + | /var/log/mail.log # mail |
| + | /var/log/cron # cron |
| + | /var/log/dmesg # boot's kernel transcript |
| + | /var/log/wtmp btmp # logins (binary — `last`/`lastb` read) |
| + | /var/log/faillog lastlog # failed/last logins |
| + | /var/log/Xorg.0.log # the display |
| + | ``` |
| + | |
| + | `logrotate` rotates them — `/etc/logrotate.conf` + `.d/`, daily/weekly/size-triggered, `.1 .2 .gz` archives. `zcat`/`zless`/`zgrep` read the compressed ones without unrolling. |
| + | |
| | ## The watchers — monitoring |
| | |
| | ```sh |
| | top # the classic live view |
| | htop # the better one — per-core bars, trees, search |
| | ps aux --sort=-%cpu | head # the static snapshot |
| | free -h # memory — used/free/cached |
| | vmstat 1 # the pulse — cpu, io, memory per second |
| | iostat -x 1 # per-disk io |
| | mpstat -P ALL 1 # per-CPU |
| | dstat -cdn # the everything-monitor |
| | uptime # load averages — 1/5/15 min |
| | watch -n2 'command' # any command as a live monitor |
| | ``` |
| | |
| | **Load average interpretation**: `4.0` on 4 cores = fully loaded; on 8 cores = half. `free`'s `available` column is real free memory — `buff/cache` is reclaimable, not "used". |
| | |
| | ## Time — the quiet machinery |
| | |
| | ```sh |
| | date # now |
| | date -u # UTC |
| | date +%s # epoch seconds — the real format |
| | date -d @1700000000 # epoch → human |
| | timedatectl # the status — tz, NTP, sync state |
| | timedatectl set-timezone America/Chicago |
| | timedatectl set-ntp true # enable NTP |
| | hwclock --show # the hardware clock — the other clock |
| | hwclock --systohc # system → hardware — save it |
| | cal # the calendar |
| | TZ=UTC date # override the zone for one command |
| | ``` |
| | |
| | Two clocks: the **hardware clock** (RTC — survives power-off) and the **system clock** (the kernel's, set at boot). `hwclock` bridges them; NTP keeps system time true. Timezone lives at `/etc/localtime` → `/usr/share/zoneinfo/Region/City` — a symlink, changeable per-process via `TZ`. |
| | |
| | ## Locale — the language layer |
| | |
| | ```sh |
| | locale # current settings |
| | locale -a # available |
| | localectl list-locales # systemd's view |
| | export LC_ALL=C # the predictable POSIX mode — LFS's friend |
| | export LANG=en_US.UTF-8 # the normal one |
| | update-locale LANG=... # Debian's setter |
| | ``` |
| | |
| | `LC_ALL=C`/`POSIX` is the deterministic mode — sorting, collation, error messages all canonical — the LFS builds' reproducibility hack and a debugging tool (error messages in English match Google results). |
| | |
| | ## Reading a sick system — the triage |
| | |
| | 1. `journalctl -p err -b` or `grep -i 'error\|fail' /var/log/messages` — the complaints |
| | 2. `dmesg -l err,warn` — the kernel's |
| | 3. `top`/`free`/`df -h`/`df -i` — the resources |
| | 4. `ss -tlnp` — the surfaces |
| | 5. `journalctl -u <failing>` — the suspect's story |
| | |
| | ## Where logging goes wrong |
| | |
| | | Symptom | Cause | |
| | | --- | --- | |
| | | journalctl eats gigabytes | `SystemMaxUse`/`--vacuum-size` — trim it | |
| | | A log is empty | The daemon never started — `systemctl status` | |
| | | Timestamps wrong | tz vs RTC — `timedatectl` shows the divergence | |
| | | Logs stop mid-file | Rotation — `zcat`/`zless` the `.gz` archives | |
| | | "no space left" but disk fine | inode exhaustion — `df -i` | |
| | | Error in another language | `LC_ALL=C` canonicalizes | |
| | |
| | ## Exercises |
| | |
| | 1. `journalctl -b -1 -p err` — read a previous boot's failures. |
| | 2. `dmesg -T | tail` — the last hardware events. |
| | 3. `watch -n1 free -h` — watch memory breathe. |
| | 4. `timedatectl` — see both clocks' sync state. |
| | 5. `LC_ALL=C ls /nonexistent` vs `ls /nonexistent` — the locale difference. |
| | |
| | ## Timeline |
| | |
| | | Layer | The mechanism | |
| | | --- | --- | |
| | | syslog | /var/log text files | |
| | | journald | The binary indexed log | |
| | | dmesg | The kernel's ring buffer | |
| | | monitoring | top/vmstat/iostat | |
| | | time | RTC + NTP + tz | |
| | | locale | LC_ALL + LANG | |
| | |
| | ## See also |
| | |
| | - [[Inside a Linux System]] · [[Troubleshooting and Rescue]] · [[Linux]] |
| | |
| | [[Category:Linux]] |
| | [[Category:Handbook]] |
| | |
