Diff — Kernel Deep Dive

{{infobox

| title = The Kernel Deep Dive

| chapter = Linux Handbook, chapter 26 — source, config, compile, modules, the anatomy

| core | The kernel is the machine's mind — one C codebase, configured to your hardware

| tree | kernel.org tarball · git tree · distro patches

| skills | menuconfig · make · modules · .config literacy · sysctl

| next | The distro landscape — the kernel dressed a hundred ways

}}

**The Linux kernel is the largest single C project on Earth** — ~30 million lines, ~80,000 files, contributed to by tens of thousands. Yet it's a program like any other: you can download it, configure it, compile it, and run the result. This chapter walks the kernel as *code* — the source tree's map, the `.config` that shapes it, the build, the modules, and the live interfaces (`/proc`, `/sys`, `sysctl`) it exposes.

## The source tree's map

```sh

git clone --depth 1 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

# or: wget https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.x.tar.xz

ls linux/

# arch/     — per-architecture code — x86, arm, riscv...

# kernel/   — the core: scheduler, signals, timers, printk

# mm/       — memory management: paging, slab, swap

# fs/       — filesystems: VFS layer + ext4, btrfs, proc, sysfs...

# drivers/  — the bulk: every device driver on Earth

# net/      — the network stack

# include/  — headers incl. the UAPI boundary (include/uapi)

# init/     — main.c — where the kernel begins

# sound/, security/, ipc/, block/, crypto/, lib/, virt/, Documentation/

```

`init/main.c`'s `start_kernel()` is where it all begins; `kernel/sched/` holds the scheduler; `fs/` the VFS; `drivers/` is ~half the tree by size. Reading the tree once makes the kernel less mythical.

## The .config — the kernel's personality

```sh

make defconfig        # sane defaults for your arch

make menuconfig       # THE interface — the ncurses option tree

make oldconfig        # upgrade a .config across versions

make olddefconfig     # same, auto-defaults

make localmodconfig   # only the modules your hardware uses

grep CONFIG_EXT4 .config

scripts/config -e CONFIG_EXT4_FS    # script edits

```

Every `CONFIG_*` option is a C macro — `[*]` built-in, `[M]` module, `[ ]` absent. The critical divides: **built-in** = in bzImage, available instantly, needed for boot-critical things (rootfs driver, disk controller); **module** = a `.ko` file, loadable after boot, fine for everything else. The boot-critical rule — *can't load a module from a filesystem you can't yet mount* — is the chapter 6 panic's explanation.

## The build

```sh

make -j$(nproc)              # bzImage + all [M] modules — hours → minutes

make modules                 # just modules

make modules_install         # → /lib/modules/<ver>/

make install                 # → /boot + bootloader hooks (distro-dependent)

# the manual form (chapter 6):

cp arch/x86_64/boot/bzImage /boot/vmlinuz-<ver>

cp System.map /boot/; cp .config /boot/config-<ver>

make clean / mrproper        # the clean levels — mrproper deletes .config too

```

Artifacts: `bzImage` (the compressed kernel), `System.map` (symbol→address table — debugging's map), `.config` (the recipe — keep forever; every later kernel starts from it).

## Modules — the pluggable kernel

```sh

lsmod                        # loaded modules

modinfo ext4                 # a module's metadata

modprobe e1000e              # load with deps

modprobe -r e1000e           # unload

insmod foo.ko                # load by file (no deps)

rmmod foo                    # raw unload

depmod -a                    # rebuild the dep map

ls /lib/modules/$(uname -r)/ # the module tree

modprobe.d/*.conf            # blacklists + options

```

Modules load on demand — `udev`/`modprobe` hotplug them as hardware appears. `modprobe blacklists` keep problem modules out.

## The live interfaces — reading the running kernel

```sh

uname -a                     # version + build metadata

cat /proc/version            # who compiled it, when

cat /proc/cmdline            # the boot parameters

sysctl -a                    # every tunable

sysctl -w net.ipv4.ip_forward=1   # live-tune — write a file, change a kernel

cat /proc/sys/kernel/hostname

dmesg                        # the kernel's own log

ls /sys/devices/             # the device object tree

```

`sysctl`/proc-sys is the kernel's control panel — networking, memory, security knobs, all writable files.

## Versioning — the numbers

`6.9.3` = major.minor.patch; `.x` releases are the stable series; `LTS` versions (6.1, 6.6, 6.12…) get years of backports; `-rc` are pre-releases. `uname -r` reports yours; distro kernels carry suffixes (`-amd64`, `-generic`, `-lts`) and distro patches atop the upstream version.

## menuconfig — the search and the structure

Inside `make menuconfig`:

- `/` — **search** — type a CONFIG_ name, jump to it

- `?` — help on the highlighted option — the real documentation

- `z` — toggle between `[*]`, `[M]`, `[ ]`

- `space` — toggle; `h` help; arrows navigate

The tree's shape: `Processor type and features` (CPU), `Device Drivers` (half the kernel — storage, net, USB, input, graphics), `File systems`, `Networking support`, `Security options`, `Kernel hacking` (debug), `Power management`, `Virtualization`, `Executable file formats`. The three things that matter for a first build: your **storage controller** (`Device Drivers → Serial ATA/SCSI/NVMe`), your **root filesystem** (`File systems → ext4`), **`Devtmpfs automount`** (`Device Drivers → Generic Driver Options`) — plus the NIC.

## The build products — what each artifact is

```sh

arch/x86_64/boot/bzImage    # the kernel — bzipped + self-decompressing

System.map                  # symbol→address — crash logs decode through it

.config                     # the recipe — the single most valuable file

vmlinux                     # the uncompressed ELF — debugging format

/lib/modules/<ver>/*.ko     # the modules — loadable drivers

Module.symvers              # module symbol versions — external builds need it

```

`make modules_install` installs the .ko tree to `/lib/modules/<ver>/`; `depmod` builds `modules.dep` so modprobe resolves chains. `make install` runs the distro's install hooks (or you copy bzImage manually).

## The module lifecycle

```sh

modinfo -F version e1000e    # a module's version

modinfo -p e1000e            # its parameters

modprobe e1000e param=val    # load with options

echo 'options e1000e param=val' >> /etc/modprobe.d/e1000e.conf

echo 'blacklist badmod' >> /etc/modprobe.d/blacklist.conf

dkms status                  # out-of-tree modules' rebuilder

```

Modules live as `.ko` (or `.ko.xz`/`.zst` compressed) under `/lib/modules/`; `modules.dep` is the dependency map `depmod` builds; `modprobe.d/` holds options + blacklists.

## The UAPI boundary — the kernel/userspace contract

`include/uapi/` is the kernel's public interface — the headers userspace compiles against (the LFS chapter-3 headers install). Everything there is the stable ABI — the syscall numbers, ioctl codes, structure layouts — the boundary a kernel *cannot* break.

## Building a custom kernel — the whole move

```sh

wget https://cdn.kernel.org/.../linux-6.9.3.tar.xz && tar -xf ...

cd linux-6.9.3

zcat /proc/config.gz > .config   # steal the running kernel's config (if enabled)

# or: cp /boot/config-$(uname -r) .config

make olddefconfig

make menuconfig                  # trim to your hardware — the speedup

make -j$(nproc)

sudo make modules_install

sudo cp arch/x86_64/boot/bzImage /boot/vmlinuz-custom

# add a GRUB entry → reboot → test → keep the old kernel as fallback

```

**Always keep the old kernel** — GRUB's menu lets you boot the known-good one; the fallback is the safety net.

## Where kernel work goes wrong

| Symptom | Cause |

| --- | --- |

| Panic: can't mount root | rootfs/disk driver as `M`, no initramfs |

| Booted, no hardware | Drivers missing — `localmodconfig` trimmed too far |

| `make` takes forever | Full defconfig — menuconfig-trim it |

| Module won't load | Built for a different kernel — `uname -r` vs modules dir |

| `.config` lost | `mrproper` — it's in /boot/config-* as backup |

| Custom kernel, old system still boots | The GRUB fallback — the correct design |

## Exercises

1. `zcat /proc/config.gz | grep -c '=y'` — your kernel's built-ins count.

2. `modinfo` three loaded modules — read their parameters.

3. `sysctl -a | wc -l` — the knob count.

4. Browse `/sys/devices` — find your network card's object.

5. `make menuconfig` a downloaded tree — navigate, don't change; read the structure.

## Timeline

| Layer | The mechanism |

| --- | --- |

| Source | init/main.c → the subsystems |

| .config | Which pieces compile |

| Build | bzImage + .ko's |

| Modules | The pluggable parts |

| Interfaces | /proc, /sys, sysctl, dmesg |

## References

- kernel.org — the tree, the docs

- `Documentation/` inside the source — the kernel documents itself

## See also

- [[LFS Kernel and Boot]] · [[Inside a Linux System]] · [[Processes]] · [[Linux]]

[[Category:Linux]]

[[Category:Handbook]]

[[Category:Kernel]]