Diff — Kernel Deep Dive

{{infobox

| title = The Kernel Deep Dive

| chapter = Linux Handbook, chapter 26 — source, config, compile, modules, the anatomy

| core | The kernel is the machine's mind — one C codebase, configured to your hardware

| tree | kernel.org tarball · git tree · distro patches

| skills | menuconfig · make · modules · .config literacy · sysctl

| next | The distro landscape — the kernel dressed a hundred ways

}}

**The Linux kernel is the largest single C project on Earth** — ~30 million lines, ~80,000 files, contributed to by tens of thousands. Yet it's a program like any other: you can download it, configure it, compile it, and run the result. This chapter walks the kernel as *code* — the source tree's map, the `.config` that shapes it, the build, the modules, and the live interfaces (`/proc`, `/sys`, `sysctl`) it exposes.

## The source tree's map

```sh

git clone --depth 1 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

# or: wget https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.x.tar.xz

ls linux/

# arch/     — per-architecture code — x86, arm, riscv...

# kernel/   — the core: scheduler, signals, timers, printk

# mm/       — memory management: paging, slab, swap

# fs/       — filesystems: VFS layer + ext4, btrfs, proc, sysfs...

# drivers/  — the bulk: every device driver on Earth

# net/      — the network stack

# include/  — headers incl. the UAPI boundary (include/uapi)

# init/     — main.c — where the kernel begins

# sound/, security/, ipc/, block/, crypto/, lib/, virt/, Documentation/

```

`init/main.c`'s `start_kernel()` is where it all begins; `kernel/sched/` holds the scheduler; `fs/` the VFS; `drivers/` is ~half the tree by size. Reading the tree once makes the kernel less mythical.

## The .config — the kernel's personality

```sh

make defconfig        # sane defaults for your arch

make menuconfig       # THE interface — the ncurses option tree

make oldconfig        # upgrade a .config across versions

make olddefconfig     # same, auto-defaults

make localmodconfig   # only the modules your hardware uses

grep CONFIG_EXT4 .config

scripts/config -e CONFIG_EXT4_FS    # script edits

```

Every `CONFIG_*` option is a C macro — `[*]` built-in, `[M]` module, `[ ]` absent. The critical divides: **built-in** = in bzImage, available instantly, needed for boot-critical things (rootfs driver, disk controller); **module** = a `.ko` file, loadable after boot, fine for everything else. The boot-critical rule — *can't load a module from a filesystem you can't yet mount* — is the chapter 6 panic's explanation.

## The build

```sh

make -j$(nproc)              # bzImage + all [M] modules — hours → minutes

make modules                 # just modules

make modules_install         # → /lib/modules/<ver>/

make install                 # → /boot + bootloader hooks (distro-dependent)

# the manual form (chapter 6):

cp arch/x86_64/boot/bzImage /boot/vmlinuz-<ver>

cp System.map /boot/; cp .config /boot/config-<ver>

make clean / mrproper        # the clean levels — mrproper deletes .config too

```

Artifacts: `bzImage` (the compressed kernel), `System.map` (symbol→address table — debugging's map), `.config` (the recipe — keep forever; every later kernel starts from it).

## Modules — the pluggable kernel

```sh

lsmod                        # loaded modules

modinfo ext4                 # a module's metadata

modprobe e1000e              # load with deps

modprobe -r e1000e           # unload

insmod foo.ko                # load by file (no deps)

rmmod foo                    # raw unload

depmod -a                    # rebuild the dep map

ls /lib/modules/$(uname -r)/ # the module tree

modprobe.d/*.conf            # blacklists + options

```

Modules load on demand — `udev`/`modprobe` hotplug them as hardware appears. `modprobe blacklists` keep problem modules out.

## The live interfaces — reading the running kernel

```sh

uname -a                     # version + build metadata

cat /proc/version            # who compiled it, when

cat /proc/cmdline            # the boot parameters

sysctl -a                    # every tunable

sysctl -w net.ipv4.ip_forward=1   # live-tune — write a file, change a kernel

cat /proc/sys/kernel/hostname

dmesg                        # the kernel's own log

ls /sys/devices/             # the device object tree

```

`sysctl`/proc-sys is the kernel's control panel — networking, memory, security knobs, all writable files.

## The source tree's deep map — what each subsystem owns

```text

init/         — main.c, start_kernel() — where the kernel begins

kernel/       — sched/ the scheduler · time/ timers · printk · fork/exit/signal.c

mm/           — memory: page_alloc, slab, vmalloc, swap, mmap

fs/           — VFS layer + ext4/, btrfs/, proc/, sysfs/, devpts/, nfs/, xfs/

                + open.c read_write.c inode.c — the VFS heart

drivers/      — the bulk: ata/, net/, usb/, pci/, tty/, input/, gpu/, block/,

                acpi/, base/ — every device on Earth

net/          — the network stack: ipv4/, ipv6/, core/, sched/, netfilter/

include/      — headers incl. include/uapi/ — the stable ABI boundary

sound/        — ALSA

security/     — selinux/, apparmor/, capability.c, lsm audit

ipc/          — SysV ipc + posix mq

block/        — the block layer — io scheduling

crypto/       — the in-kernel crypto

virt/         — kvm — virtualization

lib/          — shared utilities — crc, zlib, bitmaps

tools/        — perf, objtool, the userspace tools shipped with the kernel

scripts/      — the build's helpers — kconfig, checkpatch

Documentation/— the kernel documents itself

arch/         — per-CPU: x86/, arm/, arm64/, riscv/, powerpc/, mips/

```

The sizes tell the truth — `drivers/` is ~half the tree; `fs/` and `net/` the deep machinery; `kernel/` the core. `init/main.c`'s `start_kernel()` calls `rest_init()` which spawns PID 1 — the fork everything descends from.

## The syscall surface — the kernel's API

```sh

strace -c cmd                # the syscall census

man 2 intro                  # the syscall manpages

grep -r '__x64_sys'          # the implementation table

# the core set:

open read write close stat fstat lseek mmap munmap brk ioctl

readv writev pread pwrite access dup dup2 pipe socket bind listen

accept connect sendto recvfrom socketpair sendmsg recvmsg shutdown

setsockopt getsockname getpeername

clone fork vfork execve exit wait4 waitpid kill getpid getppid

gettid futex set_tid_address sched_yield getpriority setpriority

sched_setparam sched_getparam sched_setscheduler sched_getscheduler

sched_setaffinity sched_getaffinity setrlimit getrlimit getrusage

sysinfo times vhangup modify_ldt arch_prctl

rt_sigaction rt_sigprocmask rt_sigpending rt_sigtimedwait

rt_sigqueueinfo rt_sigsuspend sigaltstack rt_sigreturn

ioperm iopl personality set_tid_address restart_syscall

nanosleep clock_nanosleep getitimer setitimer timer_create

timer_settime timer_gettime timer_getoverrun timer_delete

clock_settime clock_gettime clock_getres clock_adjtime

getpid getppid gettid getpgid setpgid getsid setsid

getgroups setgroups setregid setgid getuid geteuid getresuid

setuid setreuid setresuid getresgid setfsgid setfsuid

capget capset ptrace userfaultfd process_vm_readv process_vm_writev

```

~350 syscalls — `man 2` is their documentation; `strace` shows them firing.

## Versioning — the numbers

`6.9.3` = major.minor.patch; `.x` releases are the stable series; `LTS` versions (6.1, 6.6, 6.12…) get years of backports; `-rc` are pre-releases. `uname -r` reports yours; distro kernels carry suffixes (`-amd64`, `-generic`, `-lts`) and distro patches atop the upstream version.

## menuconfig — the search and the structure

Inside `make menuconfig`:

- `/` — **search** — type a CONFIG_ name, jump to it

- `?` — help on the highlighted option — the real documentation

- `z` — toggle between `[*]`, `[M]`, `[ ]`

- `space` — toggle; `h` help; arrows navigate

The tree's shape: `Processor type and features` (CPU), `Device Drivers` (half the kernel — storage, net, USB, input, graphics), `File systems`, `Networking support`, `Security options`, `Kernel hacking` (debug), `Power management`, `Virtualization`, `Executable file formats`. The three things that matter for a first build: your **storage controller** (`Device Drivers → Serial ATA/SCSI/NVMe`), your **root filesystem** (`File systems → ext4`), **`Devtmpfs automount`** (`Device Drivers → Generic Driver Options`) — plus the NIC.

## The build products — what each artifact is

```sh

arch/x86_64/boot/bzImage    # the kernel — bzipped + self-decompressing

System.map                  # symbol→address — crash logs decode through it

.config                     # the recipe — the single most valuable file

vmlinux                     # the uncompressed ELF — debugging format

/lib/modules/<ver>/*.ko     # the modules — loadable drivers

Module.symvers              # module symbol versions — external builds need it

```

`make modules_install` installs the .ko tree to `/lib/modules/<ver>/`; `depmod` builds `modules.dep` so modprobe resolves chains. `make install` runs the distro's install hooks (or you copy bzImage manually).

## The module lifecycle

```sh

modinfo -F version e1000e    # a module's version

modinfo -p e1000e            # its parameters

modprobe e1000e param=val    # load with options

echo 'options e1000e param=val' >> /etc/modprobe.d/e1000e.conf

echo 'blacklist badmod' >> /etc/modprobe.d/blacklist.conf

dkms status                  # out-of-tree modules' rebuilder

```

Modules live as `.ko` (or `.ko.xz`/`.zst` compressed) under `/lib/modules/`; `modules.dep` is the dependency map `depmod` builds; `modprobe.d/` holds options + blacklists.

## The UAPI boundary — the kernel/userspace contract

`include/uapi/` is the kernel's public interface — the headers userspace compiles against (the LFS chapter-3 headers install). Everything there is the stable ABI — the syscall numbers, ioctl codes, structure layouts — the boundary a kernel *cannot* break.

## Building a custom kernel — the whole move

```sh

wget https://cdn.kernel.org/.../linux-6.9.3.tar.xz && tar -xf ...

cd linux-6.9.3

zcat /proc/config.gz > .config   # steal the running kernel's config (if enabled)

# or: cp /boot/config-$(uname -r) .config

make olddefconfig

make menuconfig                  # trim to your hardware — the speedup

make -j$(nproc)

sudo make modules_install

sudo cp arch/x86_64/boot/bzImage /boot/vmlinuz-custom

# add a GRUB entry → reboot → test → keep the old kernel as fallback

```

**Always keep the old kernel** — GRUB's menu lets you boot the known-good one; the fallback is the safety net.

## Where kernel work goes wrong

| Symptom | Cause |

| --- | --- |

| Panic: can't mount root | rootfs/disk driver as `M`, no initramfs |

| Booted, no hardware | Drivers missing — `localmodconfig` trimmed too far |

| `make` takes forever | Full defconfig — menuconfig-trim it |

| Module won't load | Built for a different kernel — `uname -r` vs modules dir |

| `.config` lost | `mrproper` — it's in /boot/config-* as backup |

| Custom kernel, old system still boots | The GRUB fallback — the correct design |

## Exercises

1. `zcat /proc/config.gz | grep -c '=y'` — your kernel's built-ins count.

2. `modinfo` three loaded modules — read their parameters.

3. `sysctl -a | wc -l` — the knob count.

4. Browse `/sys/devices` — find your network card's object.

5. `make menuconfig` a downloaded tree — navigate, don't change; read the structure.

## Timeline

| Layer | The mechanism |

| --- | --- |

| Source | init/main.c → the subsystems |

| .config | Which pieces compile |

| Build | bzImage + .ko's |

| Modules | The pluggable parts |

| Interfaces | /proc, /sys, sysctl, dmesg |

## References

- kernel.org — the tree, the docs

- `Documentation/` inside the source — the kernel documents itself

## See also

- [[LFS Kernel and Boot]] · [[Inside a Linux System]] · [[Processes]] · [[Linux]]

[[Category:Linux]]

[[Category:Handbook]]

[[Category:Kernel]]