DBG-301 · Debugging & Tracing · Expert
Race Conditions & Lock Contention — full syllabus
The bugs that only appear under load on someone else's machine, and a method for actually finding them.
Who this course is for
Experienced kernel and systems engineers chasing the bugs that only appear under load on someone else's machine — data races, lock convoys and priority inversions — and who want a method, not luck.
Prerequisites
- Strong kernel internals knowledge (locking primitives, memory model basics)
- Solid C and concurrent-programming experience
- Comfort building and booting instrumented kernels
Course outline
Day 1 — Making the race reproducible
- Why races hide: timing windows, cache effects and load dependence
- Stress and timing pressure as engineering: making a once-a-month bug fail on demand
- Fault injection and deliberate delay insertion to widen the window
- KCSAN: setup, coverage expectations and how it watches memory accesses
- Reading a KCSAN report down to the two racing accesses and their stacks
Day 2 — Contention and inversion
- Lock contention analysis with lockstat: hold time vs wait time
- perf lock for contention on production-like kernels
- Priority inversion demonstrated and measured; priority inheritance as remedy
- Convoy effects and lock granularity: when the fix is restructuring, not tuning
- Choosing the primitive: spinlock vs mutex vs rwlock vs RCU per access pattern
Day 3 — Reviewing concurrent code
- The Linux kernel memory model at the level a reviewer needs
- Common pattern failures: check-then-act, lockless read of torn state, missing barriers
- Reviewing for correctness, not plausibility: what 'looks fine' hides
- Validating a fix: proving the race is gone with statistics, not hope
- Building a regression run that would have caught the original bug
Hands-on labs
- Lab: reproduce an injected race from rare to routine using stress, timing pressure and fault injection
- Lab: run KCSAN against a racy module and interpret the report down to the two racing accesses
- Lab: quantify contention with lockstat and perf lock, separating hold-time from wait-time problems
- Lab: demonstrate priority inversion on a test setup and measure the remedy's effect on tail latency
- Lab: review a concurrent code sample, find the correctness bug every test passes, and write the fix
Capstone project
Given a workload with an intermittent corruption symptom, take it to root cause and closure: a reproduction recipe that fails on demand, KCSAN or lockstat evidence naming the racing accesses or the contended lock, a fix, and before/after run statistics demonstrating the failure rate went to zero — plus the regression run configuration that keeps it there.
What you leave with
- A reproduction method for heisenbugs: stress, injection, timing pressure
- KCSAN deployment and report interpretation
- lockstat and perf lock contention analysis
- Priority-inversion diagnosis and remedy measurement
- A correctness-first review checklist for concurrent kernel code
Upcoming dates
| Dates | Where | Seats | Early bird | Regular | |
|---|---|---|---|---|---|
| 22 Nov – 24 Nov 20263 full days | RiyadhIn person · KAFD Conference Centre | 7 of 14 | SAR 9,110until 23 Oct | ||
| 29 Nov – 1 Dec 20263 full days | Kuwait CityIn person · Al Hamra Tower | 12 of 14 | KWD 760until 30 Oct | ||
| 29 Nov – 1 Dec 20263 full days | MuscatIn person · Knowledge Oasis Muscat | 7 of 14 | OMR 940until 30 Oct | ||
| 6 Dec – 13 Dec 20266 half-days | Gulf bandLive online · 09:00–13:00 GMT+3 | 3 of 20 | US$1,760until 6 Nov | ||
| 7 Dec – 9 Dec 20263 full days | OttawaIn person · Kanata North Tech Park | 12 of 14 | CAD 3,300until 7 Nov | ||
| 7 Dec – 14 Dec 20266 half-days | Europe bandLive online · 09:00–13:00 CET | 8 of 20 | US$1,760until 7 Nov | ||
| 14 Dec – 16 Dec 20263 full days | TorontoIn person · MaRS Discovery District | 7 of 14 | CAD 3,300until 14 Nov | ||
| 14 Dec – 16 Dec 20263 full days | LondonIn person · Shoreditch Works | 12 of 14 | GBP 1,900until 14 Nov | ||
| 14 Dec – 21 Dec 20266 half-days | Americas bandLive online · 13:00–17:00 ET | 13 of 20 | US$1,760until 14 Nov | ||
| 21 Dec – 23 Dec 20263 full days | BerlinIn person · Factory Görlitzer Park | 7 of 14 | EUR 2,230until 21 Nov |
Book a seat, or bring this course to your team
Seats can be reserved online; private delivery runs on-site or live online, adapted to your stack.
Questions about fit or prerequisites? Email hello@kernelsystems.academy. To save this syllabus, print this page to PDF from your browser.