DBG-120 · Debugging & Tracing
kgdb & Live Kernel Debugging
Interactive kernel debugging over serial and network, plus dynamic debug for cases where stopping is not an option.
Who this course is for
Kernel and driver developers facing bugs that need the machine stopped mid-thought — state inspection, stepping and watchpoints — and who also need to know when stopping makes the bug vanish.
Prerequisites
Course outline
Day 1 — Getting attached
- kgdb architecture: what the stub can and cannot do
- kgdb over serial with kgdboc: cables, parameters and timing
- kgdboe over Ethernet and its constraints
- The QEMU gdb stub as a practice target: full control without extra hardware
- gdb client setup with vmlinux: symbols, source paths and the first breakpoint
Day 2 — Working a stopped kernel
- Breakpoints, stepping and backtraces in kernel context
- Inspecting kernel state: the kernel's gdb helper scripts (lx-symbols, lx-dmesg, task and module walkers)
- Debugging loadable modules: resolving module symbols at their load address
- Hardware watchpoints: catching the exact write that corrupts a variable
- Debugging early boot code before the console exists
Day 3 — When stopping is not an option
- dynamic_debug and pr_debug at runtime: enabling sites by file, function, line and format match
- Conditional debug output without recompiling
- What stopping an SMP machine does to timing-sensitive bugs: heisenbugs and kgdb's blind spots
- Choosing between kgdb, tracing and dump analysis per bug class
- Workshop: an injected timing bug pursued with both stopped and running techniques
Hands-on labs
Labs follow the academy model — 35% principles, 20% guided investigation, 45% engineering studio. Every claim you make in a lab is backed by a trace, a counter or a measurement you captured yourself. How we teach
- Lab: bring up kgdb over serial (or the QEMU gdb stub) and single-step through a system call path
- Lab: load the kernel's gdb helper scripts and walk tasks and modules on a live target
- Lab: set a hardware watchpoint on a corrupted variable and catch the instruction that clobbers it
- Lab: debug a deliberately broken module interactively: resolve its symbols and find the faulting line
- Lab: enable individual pr_debug sites at runtime with dynamic_debug and capture only the traffic you need
Capstone project
Debug an injected, timing-sensitive kernel fault end to end. You choose per phase between kgdb and dynamic_debug and must justify each choice: catch the corruption with a watchpoint or conditional breakpoint, then reproduce the surrounding timing with non-stopping debug. Deliverables: the gdb and dmesg transcripts, plus a written record of where interactive debugging helped and where it perturbed the bug away.
What you leave with
- A working kgdb/kgdboe (or QEMU stub) setup you can rebuild
- Fluency with the kernel's lx-* gdb helper scripts
- Hardware watchpoint technique for corruption bugs
- dynamic_debug as the non-stopping alternative
- Defensible judgment about when interactive debugging is the wrong tool
How it runs
Every course follows the same model: 35% principles, 20% guided investigation, 45% engineering studio. You leave with working code, raw measurements and an evidence-based report — not a certificate of attendance. Read the methodology or see a full sample lesson.
Material is adapted to your kernel version, hardware and workload before a private delivery. For public cohorts, the environment is provided and configured.
Questions
Who is this course for?
Kernel and driver developers facing bugs that need the machine stopped mid-thought — state inspection, stepping and watchpoints — and who also need to know when stopping makes the bug vanish. It sits at advanced level within the Debugging & Tracing track.
What do I need to know already?
Specific prerequisites for this course: Solid C and kernel module experience; Ability to build and boot a custom kernel; gdb fundamentals (breakpoints, stepping, backtrace). We confirm levels before the cohort starts and adapt if a group is stronger or weaker than expected.
Can this run privately for my team?
Yes. Any course runs on-site at your offices anywhere, or live online for a distributed team, with labs adapted to your hardware and codebase.
What is the difference between in-person and online?
In person is 3 full days with hardware on your desk, capped at 14. Online is 6 half-day sessions across about two weeks so you can keep working, capped at 20, with remote lab access.
Do you invoice companies?
Yes. Purchase orders are accepted and invoicing is available in USD, EUR, GBP, SAR and CAD.
Upcoming dates
| Dates | Where | Seats | Early bird | Regular | |
|---|---|---|---|---|---|
| 25 Oct – 27 Oct 20263 full days | RiyadhIn person · KAFD Conference Centre | 6 of 14 | — | SAR 9,000 | |
| 1 Nov – 3 Nov 20263 full days | Kuwait CityIn person · Al Hamra Tower | 11 of 14 | — | KWD 740 | |
| 1 Nov – 3 Nov 20263 full days | MuscatIn person · Knowledge Oasis Muscat | 6 of 14 | — | OMR 920 | |
| 8 Nov – 15 Nov 20266 half-days | Gulf bandLive online · 09:00–13:00 GMT+3 | 14 of 20 | US$1,580until 9 Oct | ||
| 9 Nov – 11 Nov 20263 full days | OttawaIn person · Kanata North Tech Park | 11 of 14 | CAD 2,930until 10 Oct | ||
| 9 Nov – 16 Nov 20266 half-days | Europe bandLive online · 09:00–13:00 CET | 3 of 20 | US$1,580until 10 Oct | ||
| 16 Nov – 18 Nov 20263 full days | TorontoIn person · MaRS Discovery District | 6 of 14 | CAD 2,930until 17 Oct | ||
| 16 Nov – 18 Nov 20263 full days | LondonIn person · Shoreditch Works | 11 of 14 | GBP 1,680until 17 Oct | ||
| 16 Nov – 23 Nov 20266 half-days | Americas bandLive online · 13:00–17:00 ET | 8 of 20 | US$1,580until 17 Oct | ||
| 23 Nov – 25 Nov 20263 full days | BerlinIn person · Factory Görlitzer Park | 6 of 14 | EUR 1,990until 24 Oct |
Dates shown for the next few months. If nothing fits, tell us where and when — cohorts are added on demand, and private delivery can be scheduled any week.
More in Debugging & Tracing
DBG-1012 days
Reading an Oops & Panic Analysis
Turning a kernel splat into a precise location in the source, and knowing what the register dump is telling you.
Practitioner-taught
SAR 5,250Next 11 Oct
DBG-1103 days
kdump & the crash Utility
Capturing a crash dump in production and doing a real post-mortem on it.
Practitioner-taught
SAR 9,000Next 8 Nov
DBG-2013 days
ftrace & trace-cmd
The kernel's built-in tracer, used properly: function graphs, events and latency tracers.
Practitioner-taught
SAR 7,880Next 1 Nov
DBG-2103 days
perf: Sampling to Flame Graphs
CPU and off-CPU analysis with perf, from first sample to a flame graph that tells you something actionable.
Practitioner-taught
SAR 7,880Next 11 Oct
DBG-2204 days
eBPF & bpftrace
Programmable observability: one-liners for immediate answers, custom programs for the questions nothing else answers.
Practitioner-taught
SAR 12,000Next 15 Nov
DBG-3013 days
Race Conditions & Lock Contention
The bugs that only appear under load on someone else's machine, and a method for actually finding them.
Practitioner-taught
SAR 10,120Next 22 Nov
DBG-3103 days
Memory Corruption: KASAN & KFENCE
Finding use-after-free, out-of-bounds and uninitialised memory before they become a security advisory.
Practitioner-taught
SAR 9,000Next 1 Nov