Logs Monitoring and Time

A Linux system narrates itself constantly — every daemon's complaints, every login, every kernel warning, every service's lifecycle lands in logs. The discipline is knowing where they live and how to read them. This chapter covers the two logging worlds (syslog files and journald), the monitoring toolkit, and the clock machinery — NTP, timezones, locales — the quiet infrastructure every other chapter leans on.

The two logging worlds

Syslog files — the classic: /var/log/ holds text files, rotated, greppable:

ls -lt /var/log/           # freshest first
tail -f /var/log/syslog    # Debian's main
tail -f /var/log/messages  # Slackware/RH's main
less /var/log/auth.log     # logins, sudo, sshd
less /var/log/kern.log     # the kernel
grep 'error' /var/log/syslog | tail
logrotate                  # the rotator — /etc/logrotate.conf + .d

journald — systemd's structured binary log:

journalctl                 # everything, this boot
journalctl -b              # this boot only
journalctl -b -1           # LAST boot — the crash forensics
journalctl -u sshd         # one unit
journalctl -p err          # by priority — emerg..debug
journalctl -f              # follow live
journalctl --since "1 hour ago" --until "10 min ago"
journalctl -k              # kernel messages
journalctl -x              # + explanations
journalctl --disk-usage    # how much it stores
journalctl --vacuum-size=100M   # trim it

The -b -1 trick — the previous boot's log — is the single most useful forensic command on systemd systems: the log of the boot that failed.

dmesg — the kernel's own voice

dmesg                      # the ring buffer — hardware, drivers, oopses
dmesg -T                   # human timestamps
dmesg -l err,warn          # filtered by level
dmesg -w                   # follow live
dmesg | grep -i usb        # device arrivals

dmesg is where hardware announces itself — boot's transcript, USB plug-ins, disk I/O errors, kernel oopses — the first place to look when hardware misbehaves.

The watchers — monitoring

top                        # the classic live view
htop                       # the better one — per-core bars, trees, search
ps aux --sort=-%cpu | head # the static snapshot
free -h                    # memory — used/free/cached
vmstat 1                   # the pulse — cpu, io, memory per second
iostat -x 1                # per-disk io
mpstat -P ALL 1            # per-CPU
dstat -cdn                 # the everything-monitor
uptime                     # load averages — 1/5/15 min
watch -n2 'command'        # any command as a live monitor

Load average interpretation: 4.0 on 4 cores = fully loaded; on 8 cores = half. free's available column is real free memory — buff/cache is reclaimable, not "used".

Time — the quiet machinery

date                       # now
date -u                    # UTC
date +%s                   # epoch seconds — the real format
date -d @1700000000        # epoch → human
timedatectl                # the status — tz, NTP, sync state
timedatectl set-timezone America/Chicago
timedatectl set-ntp true   # enable NTP
hwclock --show             # the hardware clock — the other clock
hwclock --systohc          # system → hardware — save it
cal                        # the calendar
TZ=UTC date                # override the zone for one command

Two clocks: the hardware clock (RTC — survives power-off) and the system clock (the kernel's, set at boot). hwclock bridges them; NTP keeps system time true. Timezone lives at /etc/localtime → /usr/share/zoneinfo/Region/City — a symlink, changeable per-process via TZ.

Locale — the language layer

locale                     # current settings
locale -a                  # available
localectl list-locales     # systemd's view
export LC_ALL=C            # the predictable POSIX mode — LFS's friend
export LANG=en_US.UTF-8    # the normal one
update-locale LANG=...     # Debian's setter

LC_ALL=C/POSIX is the deterministic mode — sorting, collation, error messages all canonical — the LFS builds' reproducibility hack and a debugging tool (error messages in English match Google results).

Reading a sick system — the triage

1. journalctl -p err -b or grep -i 'error\|fail' /var/log/messages — the complaints
2. dmesg -l err,warn — the kernel's
3. top/free/df -h/df -i — the resources
4. ss -tlnp — the surfaces
5. journalctl -u <failing> — the suspect's story

Where logging goes wrong

Exercises

1. journalctl -b -1 -p err — read a previous boot's failures.
2. dmesg -T | tail — the last hardware events.
3. watch -n1 free -h — watch memory breathe.
4. timedatectl — see both clocks' sync state.
5. LC_ALL=C ls /nonexistent vs ls /nonexistent — the locale difference.

Timeline

See also

• Inside a Linux System · Troubleshooting and Rescue · Linux