Logs Monitoring and Time

A Linux system narrates itself constantly — every daemon's complaints, every login, every kernel warning, every service's lifecycle lands in logs. The discipline is knowing where they live and how to read them. This chapter covers the two logging worlds (syslog files and journald), the monitoring toolkit, and the clock machinery — NTP, timezones, locales — the quiet infrastructure every other chapter leans on.

The two logging worlds

Syslog files — the classic: /var/log/ holds text files, rotated, greppable:

ls -lt /var/log/           # freshest first
tail -f /var/log/syslog    # Debian's main
tail -f /var/log/messages  # Slackware/RH's main
less /var/log/auth.log     # logins, sudo, sshd
less /var/log/kern.log     # the kernel
grep 'error' /var/log/syslog | tail
logrotate                  # the rotator — /etc/logrotate.conf + .d

journald — systemd's structured binary log:

journalctl                 # everything, this boot
journalctl -b              # this boot only
journalctl -b -1           # LAST boot — the crash forensics
journalctl -u sshd         # one unit
journalctl -p err          # by priority — emerg..debug
journalctl -f              # follow live
journalctl --since "1 hour ago" --until "10 min ago"
journalctl -k              # kernel messages
journalctl -x              # + explanations
journalctl --disk-usage    # how much it stores
journalctl --vacuum-size=100M   # trim it

The -b -1 trick — the previous boot's log — is the single most useful forensic command on systemd systems: the log of the boot that failed.

dmesg — the kernel's own voice

dmesg                      # the ring buffer — hardware, drivers, oopses
dmesg -T                   # human timestamps
dmesg -l err,warn          # filtered by level
dmesg -w                   # follow live
dmesg | grep -i usb        # device arrivals

dmesg is where hardware announces itself — boot's transcript, USB plug-ins, disk I/O errors, kernel oopses — the first place to look when hardware misbehaves.

The journal's fields — the queryable layer

journalctl _SYSTEMD_UNIT=sshd.service   # by unit
journalctl _PID=1234                    # by process
journalctl _UID=1000                    # by user
journalctl _COMM=sshd                   # by command name
journalctl SYSLOG_IDENTIFIER=crond      # by syslog id
journalctl -g 'Failed.*password'        # regex the messages
journalctl -o json                      # structured — jq-able
journalctl -o short-iso-precise         # precise timestamps
journalctl --since '2024-01-01' --until '2024-01-02'
journalctl --since 'yesterday' --until 'now'
journalctl -u sshd --since today -f     # unit + time + follow
journalctl --field _TRANSPORT           # list a field's values
journalctl --vacuum-time=7d             # trim by age
journalctl --vacuum-files=20            # keep N files

syslog — the classic file grammar

/var/log/messages      # RH/Slackware's main
/var/log/syslog        # Debian's main
/var/log/auth.log      # Debian auth · /var/log/secure on RH
/var/log/kern.log      # kernel
/var/log/daemon.log    # services
/var/log/mail.log      # mail
/var/log/cron          # cron
/var/log/dmesg         # boot's kernel transcript
/var/log/wtmp btmp     # logins (binary — `last`/`lastb` read)
/var/log/faillog lastlog   # failed/last logins
/var/log/Xorg.0.log    # the display

logrotate rotates them — /etc/logrotate.conf + .d/, daily/weekly/size-triggered, .1 .2 .gz archives. zcat/zless/zgrep read the compressed ones without unrolling.

The watchers — monitoring

top                        # the classic live view
htop                       # the better one — per-core bars, trees, search
ps aux --sort=-%cpu | head # the static snapshot
free -h                    # memory — used/free/cached
vmstat 1                   # the pulse — cpu, io, memory per second
iostat -x 1                # per-disk io
mpstat -P ALL 1            # per-CPU
dstat -cdn                 # the everything-monitor
uptime                     # load averages — 1/5/15 min
watch -n2 'command'        # any command as a live monitor

Load average interpretation: 4.0 on 4 cores = fully loaded; on 8 cores = half. free's available column is real free memory — buff/cache is reclaimable, not "used".

The monitoring grammar — reading a system live

top                    # M=mem-sort P=cpu-sort 1=per-core k=kill c=full-cmd
htop                   # the better one — trees, meters, search
free -h                # memory — MemFree vs available vs buff/cache
vmstat 1 5             # 5 samples at 1s — cpu, io, memory per line
iostat -x 1            # per-disk utilization — %util, await
mpstat -P ALL 1        # per-CPU
pidstat 1              # per-process
dstat -cdn --top-cpu --top-mem   # the everything
sar -u 1               # the sysstat historian
sar -n DEV 1           # per-interface net
iostat -x | sort -k10  # busiest disks
watch -n1 'ps aux --sort=-%mem | head'  # your own monitor

free -h's available is the real free; buff/cache is reclaimable. Load 4.0 on 4 CPUs = saturated. vmstat's si/so = swap-in/out — nonzero is memory pressure.

The proc-metrics — where the numbers come from

cat /proc/loadavg      # 1.23 0.98 0.76 2/456 12345 — load + runnable + lastpid
cat /proc/meminfo      # the ledger — every kind of memory
cat /proc/vmstat       # the counters behind vmstat
cat /proc/stat         # cpu jiffies — what top computes from
cat /proc/net/dev      # per-interface bytes — what iftop reads

top reads /proc/stat + /proc/*/stat; free reads /proc/meminfo; iostat reads /proc/diskstats — the monitoring tools are readers, not sensors.

Time — the quiet machinery

date                       # now
date -u                    # UTC
date +%s                   # epoch seconds — the real format
date -d @1700000000        # epoch → human
timedatectl                # the status — tz, NTP, sync state
timedatectl set-timezone America/Chicago
timedatectl set-ntp true   # enable NTP
hwclock --show             # the hardware clock — the other clock
hwclock --systohc          # system → hardware — save it
cal                        # the calendar
TZ=UTC date                # override the zone for one command

Two clocks: the hardware clock (RTC — survives power-off) and the system clock (the kernel's, set at boot). hwclock bridges them; NTP keeps system time true. Timezone lives at /etc/localtime → /usr/share/zoneinfo/Region/City — a symlink, changeable per-process via TZ.

Locale — the language layer

locale                     # current settings
locale -a                  # available
localectl list-locales     # systemd's view
export LC_ALL=C            # the predictable POSIX mode — LFS's friend
export LANG=en_US.UTF-8    # the normal one
update-locale LANG=...     # Debian's setter

LC_ALL=C/POSIX is the deterministic mode — sorting, collation, error messages all canonical — the LFS builds' reproducibility hack and a debugging tool (error messages in English match Google results).

Reading a sick system — the triage

1. journalctl -p err -b or grep -i 'error\|fail' /var/log/messages — the complaints
2. dmesg -l err,warn — the kernel's
3. top/free/df -h/df -i — the resources
4. ss -tlnp — the surfaces
5. journalctl -u <failing> — the suspect's story

Where logging goes wrong

Exercises

1. journalctl -b -1 -p err — read a previous boot's failures.
2. dmesg -T | tail — the last hardware events.
3. watch -n1 free -h — watch memory breathe.
4. timedatectl — see both clocks' sync state.
5. LC_ALL=C ls /nonexistent vs ls /nonexistent — the locale difference.

Timeline

See also

• Inside a Linux System · Troubleshooting and Rescue · Linux