Reading an Oops & Panic Analysis

Turning a kernel splat into a precise location in the source, and knowing what the register dump is telling you.

Practitioner 2 days in person4 half-days online Max 14 in person

Who this course is for

Driver, embedded and platform engineers — and the support engineers who back them — who get handed kernel splats and need to turn registers and a call trace into a source line and a next step.

Prerequisites

Ability to read CCommand-line LinuxBasic familiarity with kernel build artifacts (vmlinux, System.map) helpful

Course outline

Day 1 — Anatomy of an oops

What the kernel prints and why: the oops message section by section
Registers that matter: the faulting instruction pointer, stack pointer and the address in the fault
Reading the call trace: what each frame is and when to distrust it
Tainted flags: every flag, what set it, and what it does to your bug report
Decoding addresses to source with addr2line, scripts/decode_stacktrace and scripts/faddr2line
Matching the splat to the exact vmlinux it came from

Day 2 — Fault classes and evidence preservation

Common fault classes and their signatures: NULL dereference, page fault in atomic context, bad RIP, general protection fault, stack corruption
Panic vs oops vs warning, and the configuration that controls each: panic_on_oops, panic_on_warn, hung-task and softlockup/hardlockup detectors
Preserving evidence when the machine dies: pstore, ramoops and serial console capture; netconsole for headless systems
Building a triage habit: from splat to fault class to hypothesis to the right next tool
Workshop: classifying a bundle of real splats under time pressure

Hands-on labs

Labs follow the academy model — 35% principles, 20% guided investigation, 45% engineering studio. Every claim you make in a lab is backed by a trace, a counter or a measurement you captured yourself. How we teach

Lab: decode a captured oops with addr2line and scripts/decode_stacktrace down to the exact faulting source line
Lab: classify a set of raw splats by fault class using register and call-trace signatures alone, then check your answers
Lab: configure pstore/ramoops on a test kernel, trigger a panic, and recover the preserved log after reboot
Lab: set up serial console capture to a second machine and prove no line of a panic is lost
Lab: compare panic, oops and warning behaviour under different panic_on_* settings and recommend a production configuration

Capstone project

You receive a bundle of raw kernel logs from several failing machines, some incomplete. For each incident you produce a triage report: the decoded faulting line, the fault class with the register/trace evidence supporting it, what evidence is missing, and the capture configuration (pstore, ramoops, serial) that would have preserved it — ending with a written first-response checklist you take back to your own fleet.

What you leave with

The ability to take any oops to a precise source line
Working fluency with addr2line, decode_stacktrace and faddr2line
A fault-class signature reference you built from real cases
An evidence-preservation configuration (pstore, ramoops, serial) ready to deploy

How it runs

Every course follows the same model: 35% principles, 20% guided investigation, 45% engineering studio. You leave with working code, raw measurements and an evidence-based report — not a certificate of attendance. Read the methodology or see a full sample lesson.

Material is adapted to your kernel version, hardware and workload before a private delivery. For public cohorts, the environment is provided and configured.

Questions

Who is this course for?

Driver, embedded and platform engineers — and the support engineers who back them — who get handed kernel splats and need to turn registers and a call trace into a source line and a next step. It sits at practitioner level within the Debugging & Tracing track.

What do I need to know already?

Specific prerequisites for this course: Ability to read C; Command-line Linux; Basic familiarity with kernel build artifacts (vmlinux, System.map) helpful. We confirm levels before the cohort starts and adapt if a group is stronger or weaker than expected.

Can this run privately for my team?

Yes. Any course runs on-site at your offices anywhere, or live online for a distributed team, with labs adapted to your hardware and codebase.

What is the difference between in-person and online?

In person is 2 full days with hardware on your desk, capped at 14. Online is 4 half-day sessions across about two weeks so you can keep working, capped at 20, with remote lab access.

Do you invoice companies?

Yes. Purchase orders are accepted and invoicing is available in USD, EUR, GBP, SAR and CAD.

Upcoming dates

Dates	Where	Seats	Early bird	Regular
11 Oct – 12 Oct 20262 full days	RiyadhIn person · KAFD Conference Centre	5 of 14	—	SAR 5,250
18 Oct – 19 Oct 20262 full days	Kuwait CityIn person · Al Hamra Tower	10 of 14	—	KWD 430
18 Oct – 19 Oct 20262 full days	MuscatIn person · Knowledge Oasis Muscat	5 of 14	—	OMR 540
25 Oct – 28 Oct 20264 half-days	Gulf bandLive online · 09:00–13:00 GMT+3	17 of 20	—	US$1,000
26 Oct – 27 Oct 20262 full days	OttawaIn person · Kanata North Tech Park	10 of 14	—	CAD 1,900
26 Oct – 29 Oct 20264 half-days	Europe bandLive online · 09:00–13:00 CET	6 of 20	—	US$1,000
2 Nov – 3 Nov 20262 full days	TorontoIn person · MaRS Discovery District	5 of 14	—	CAD 1,900
2 Nov – 3 Nov 20262 full days	LondonIn person · Shoreditch Works	10 of 14	—	GBP 1,090
2 Nov – 5 Nov 20264 half-days	Americas bandLive online · 13:00–17:00 ET	11 of 20	—	US$1,000
9 Nov – 10 Nov 20262 full days	BerlinIn person · Factory Görlitzer Park	5 of 14	EUR 1,160until 10 Oct	~~EUR 1,290~~

Dates shown for the next few months. If nothing fits, tell us where and when — cohorts are added on demand, and private delivery can be scheduled any week.

More in Debugging & Tracing

DBG-1103 days kdump & the crash Utility Capturing a crash dump in production and doing a real post-mortem on it. Advanced Practitioner-taught SAR 9,000Next 8 Nov

DBG-1203 days kgdb & Live Kernel Debugging Interactive kernel debugging over serial and network, plus dynamic debug for cases where stopping is not an option. Advanced Practitioner-taught SAR 9,000Next 25 Oct

DBG-2013 days ftrace & trace-cmd The kernel's built-in tracer, used properly: function graphs, events and latency tracers. Practitioner Practitioner-taught SAR 7,880Next 1 Nov

DBG-2103 days perf: Sampling to Flame Graphs CPU and off-CPU analysis with perf, from first sample to a flame graph that tells you something actionable. Practitioner Practitioner-taught SAR 7,880Next 11 Oct

DBG-2204 days eBPF & bpftrace Programmable observability: one-liners for immediate answers, custom programs for the questions nothing else answers. Advanced Practitioner-taught SAR 12,000Next 15 Nov

DBG-3013 days Race Conditions & Lock Contention The bugs that only appear under load on someone else's machine, and a method for actually finding them. Expert Practitioner-taught SAR 10,120Next 22 Nov

DBG-3103 days Memory Corruption: KASAN & KFENCE Finding use-after-free, out-of-bounds and uninitialised memory before they become a security advisory. Advanced Practitioner-taught SAR 9,000Next 1 Nov