Kernel Deep Dive

The Linux kernel is the largest single C project on Earth — ~30 million lines, ~80,000 files, contributed to by tens of thousands. Yet it's a program like any other: you can download it, configure it, compile it, and run the result. This chapter walks the kernel as code — the source tree's map, the .config that shapes it, the build, the modules, and the live interfaces (/proc, /sys, sysctl) it exposes.

The source tree's map

git clone --depth 1 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
# or: wget https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.x.tar.xz

ls linux/
# arch/     — per-architecture code — x86, arm, riscv...
# kernel/   — the core: scheduler, signals, timers, printk
# mm/       — memory management: paging, slab, swap
# fs/       — filesystems: VFS layer + ext4, btrfs, proc, sysfs...
# drivers/  — the bulk: every device driver on Earth
# net/      — the network stack
# include/  — headers incl. the UAPI boundary (include/uapi)
# init/     — main.c — where the kernel begins
# sound/, security/, ipc/, block/, crypto/, lib/, virt/, Documentation/

init/main.c's start_kernel() is where it all begins; kernel/sched/ holds the scheduler; fs/ the VFS; drivers/ is ~half the tree by size. Reading the tree once makes the kernel less mythical.

The .config — the kernel's personality

make defconfig        # sane defaults for your arch
make menuconfig       # THE interface — the ncurses option tree
make oldconfig        # upgrade a .config across versions
make olddefconfig     # same, auto-defaults
make localmodconfig   # only the modules your hardware uses
grep CONFIG_EXT4 .config
scripts/config -e CONFIG_EXT4_FS    # script edits

Every CONFIG_* option is a C macro — [*] built-in, [M] module, [ ] absent. The critical divides: built-in = in bzImage, available instantly, needed for boot-critical things (rootfs driver, disk controller); module = a .ko file, loadable after boot, fine for everything else. The boot-critical rule — can't load a module from a filesystem you can't yet mount — is the chapter 6 panic's explanation.

The build

make -j$(nproc)              # bzImage + all [M] modules — hours → minutes
make modules                 # just modules
make modules_install         # → /lib/modules/<ver>/
make install                 # → /boot + bootloader hooks (distro-dependent)
# the manual form (chapter 6):
cp arch/x86_64/boot/bzImage /boot/vmlinuz-<ver>
cp System.map /boot/; cp .config /boot/config-<ver>
make clean / mrproper        # the clean levels — mrproper deletes .config too

Artifacts: bzImage (the compressed kernel), System.map (symbol→address table — debugging's map), .config (the recipe — keep forever; every later kernel starts from it).

Modules — the pluggable kernel

lsmod                        # loaded modules
modinfo ext4                 # a module's metadata
modprobe e1000e              # load with deps
modprobe -r e1000e           # unload
insmod foo.ko                # load by file (no deps)
rmmod foo                    # raw unload
depmod -a                    # rebuild the dep map
ls /lib/modules/$(uname -r)/ # the module tree
modprobe.d/*.conf            # blacklists + options

Modules load on demand — udev/modprobe hotplug them as hardware appears. modprobe blacklists keep problem modules out.

The live interfaces — reading the running kernel

uname -a                     # version + build metadata
cat /proc/version            # who compiled it, when
cat /proc/cmdline            # the boot parameters
sysctl -a                    # every tunable
sysctl -w net.ipv4.ip_forward=1   # live-tune — write a file, change a kernel
cat /proc/sys/kernel/hostname
dmesg                        # the kernel's own log
ls /sys/devices/             # the device object tree

sysctl/proc-sys is the kernel's control panel — networking, memory, security knobs, all writable files.

The source tree's deep map — what each subsystem owns

init/         — main.c, start_kernel() — where the kernel begins
kernel/       — sched/ the scheduler · time/ timers · printk · fork/exit/signal.c
mm/           — memory: page_alloc, slab, vmalloc, swap, mmap
fs/           — VFS layer + ext4/, btrfs/, proc/, sysfs/, devpts/, nfs/, xfs/
                + open.c read_write.c inode.c — the VFS heart
drivers/      — the bulk: ata/, net/, usb/, pci/, tty/, input/, gpu/, block/,
                acpi/, base/ — every device on Earth
net/          — the network stack: ipv4/, ipv6/, core/, sched/, netfilter/
include/      — headers incl. include/uapi/ — the stable ABI boundary
sound/        — ALSA
security/     — selinux/, apparmor/, capability.c, lsm audit
ipc/          — SysV ipc + posix mq
block/        — the block layer — io scheduling
crypto/       — the in-kernel crypto
virt/         — kvm — virtualization
lib/          — shared utilities — crc, zlib, bitmaps
tools/        — perf, objtool, the userspace tools shipped with the kernel
scripts/      — the build's helpers — kconfig, checkpatch
Documentation/— the kernel documents itself
arch/         — per-CPU: x86/, arm/, arm64/, riscv/, powerpc/, mips/

The sizes tell the truth — drivers/ is ~half the tree; fs/ and net/ the deep machinery; kernel/ the core. init/main.c's start_kernel() calls rest_init() which spawns PID 1 — the fork everything descends from.

The syscall surface — the kernel's API

strace -c cmd                # the syscall census
man 2 intro                  # the syscall manpages
grep -r '__x64_sys'          # the implementation table
# the core set:
open read write close stat fstat lseek mmap munmap brk ioctl
readv writev pread pwrite access dup dup2 pipe socket bind listen
accept connect sendto recvfrom socketpair sendmsg recvmsg shutdown
setsockopt getsockname getpeername socketcall
clone fork vfork execve exit wait4 waitpid kill getpid getppid
gettid futex set_tid_address sched_yield getpriority setpriority
sched_setparam sched_getparam sched_setscheduler sched_getscheduler
sched_setaffinity sched_getaffinity setrlimit getrlimit getrusage
sysinfo times vhangup modify_ldt arch_prctl
rt_sigaction rt_sigprocmask rt_sigpending rt_sigtimedwait
rt_sigqueueinfo rt_sigsuspend sigaltstack rt_sigreturn
ioperm iopl personality set_tid_address restart_syscall
nanosleep clock_nanosleep getitimer setitimer timer_create
timer_settime timer_gettime timer_getoverrun timer_delete
clock_settime clock_gettime clock_getres clock_adjtime
getpid getppid gettid getpgid setpgid getsid setsid
getgroups setgroups setregid setgid getuid geteuid getresuid
setuid setreuid setresuid getresgid setfsgid setfsuid
capget capset ptrace userfaultfd process_vm_readv process_vm_writev

~350 syscalls — man 2 is their documentation; strace shows them firing.

Versioning — the numbers

6.9.3 = major.minor.patch; .x releases are the stable series; LTS versions (6.1, 6.6, 6.12…) get years of backports; -rc are pre-releases. uname -r reports yours; distro kernels carry suffixes (-amd64, -generic, -lts) and distro patches atop the upstream version.

menuconfig — the search and the structure

Inside make menuconfig:

• / — search — type a CONFIG_ name, jump to it
• ? — help on the highlighted option — the real documentation
• z — toggle between [*], [M], [ ]
• space — toggle; h help; arrows navigate

The tree's shape: Processor type and features (CPU), Device Drivers (half the kernel — storage, net, USB, input, graphics), File systems, Networking support, Security options, Kernel hacking (debug), Power management, Virtualization, Executable file formats. The three things that matter for a first build: your storage controller (Device Drivers → Serial ATA/SCSI/NVMe), your root filesystem (File systems → ext4), Devtmpfs automount (Device Drivers → Generic Driver Options) — plus the NIC.

The build products — what each artifact is

arch/x86_64/boot/bzImage    # the kernel — bzipped + self-decompressing
System.map                  # symbol→address — crash logs decode through it
.config                     # the recipe — the single most valuable file
vmlinux                     # the uncompressed ELF — debugging format
/lib/modules/<ver>/*.ko     # the modules — loadable drivers
Module.symvers              # module symbol versions — external builds need it

make modules_install installs the .ko tree to /lib/modules/<ver>/; depmod builds modules.dep so modprobe resolves chains. make install runs the distro's install hooks (or you copy bzImage manually).

The module lifecycle

modinfo -F version e1000e    # a module's version
modinfo -p e1000e            # its parameters
modprobe e1000e param=val    # load with options
echo 'options e1000e param=val' >> /etc/modprobe.d/e1000e.conf
echo 'blacklist badmod' >> /etc/modprobe.d/blacklist.conf
dkms status                  # out-of-tree modules' rebuilder

Modules live as .ko (or .ko.xz/.zst compressed) under /lib/modules/; modules.dep is the dependency map depmod builds; modprobe.d/ holds options + blacklists.

The UAPI boundary — the kernel/userspace contract

include/uapi/ is the kernel's public interface — the headers userspace compiles against (the LFS chapter-3 headers install). Everything there is the stable ABI — the syscall numbers, ioctl codes, structure layouts — the boundary a kernel cannot break.

Building a custom kernel — the whole move

wget https://cdn.kernel.org/.../linux-6.9.3.tar.xz && tar -xf ...
cd linux-6.9.3
zcat /proc/config.gz > .config   # steal the running kernel's config (if enabled)
# or: cp /boot/config-$(uname -r) .config
make olddefconfig
make menuconfig                  # trim to your hardware — the speedup
make -j$(nproc)
sudo make modules_install
sudo cp arch/x86_64/boot/bzImage /boot/vmlinuz-custom
# add a GRUB entry → reboot → test → keep the old kernel as fallback

Always keep the old kernel — GRUB's menu lets you boot the known-good one; the fallback is the safety net.

Where kernel work goes wrong

Exercises

1. zcat /proc/config.gz | grep -c '=y' — your kernel's built-ins count.
2. modinfo three loaded modules — read their parameters.
3. sysctl -a | wc -l — the knob count.
4. Browse /sys/devices — find your network card's object.
5. make menuconfig a downloaded tree — navigate, don't change; read the structure.

Timeline

References

• kernel.org — the tree, the docs
• Documentation/ inside the source — the kernel documents itself

See also

• LFS Kernel and Boot · Inside a Linux System · Processes · Linux