Train your team

DBG-201 · Debugging & Tracing

ftrace & trace-cmd

The kernel's built-in tracer, used properly: function graphs, events and latency tracers.

Practitioner 3 days in person6 half-days online Max 14 in person

Who this course is for

Kernel, driver and performance engineers who have touched ftrace through folklore and one-liners and want to use the kernel's built-in tracer deliberately — filters, events and latency tracers included.

Prerequisites

Command-line LinuxBasic kernel internals (what a syscall and an interrupt are)Ability to read C helpful for interpreting traces

Course outline

Day 1 — The tracer interface

  • tracefs layout and the tracing files that actually matter
  • Available tracers and what each one records
  • function and function_graph tracing with set_ftrace_filter and set_graph_function
  • Buffer sizing, trace clock and the cost of tracing itself
  • Reading a function graph: nesting, duration and where the time went

Day 2 — Events, filters and triggers

  • Tracepoints vs function tracing: stability and cost
  • Enabling events individually, by subsystem and system-wide
  • Per-event filtering on fields: capturing only the case you care about
  • Triggers and histograms: building answers inside the kernel instead of post-processing
  • Synthetic events for correlating two points in time

Day 3 — Latency tracers and capture tooling

  • irqsoff, preemptoff and wakeup: what each measures and when to use it
  • Reading a latency tracer report: the stack it hands you and the one it hides
  • trace-cmd record and report: scripting captures instead of clicking tracefs
  • KernelShark for timeline analysis of a recorded session
  • Production capture practice: overhead, wrap-around and what to leave running

Hands-on labs

Labs follow the academy model — 35% principles, 20% guided investigation, 45% engineering studio. Every claim you make in a lab is backed by a trace, a counter or a measurement you captured yourself. How we teach

  1. Lab: trace a specific kernel path with function and function_graph tracing, narrowing filters until the output answers one precise question
  2. Lab: enable tracepoint events with per-event field filters and build an in-kernel histogram that isolates one slow case
  3. Lab: capture worst-case latency with the irqsoff/preemptoff/wakeup tracers and explain the stacks they produce
  4. Lab: record a workload with trace-cmd and analyse it in KernelShark, producing an annotated timeline of one incident

Capstone project

Investigate an injected latency regression on a loaded system. You build the trace-cmd capture, use latency tracers and event filters to isolate the bad window, and deliver a KernelShark-annotated timeline plus a written causal chain from the first anomalous event to the culprit code path — with buffer sizing and measured tracing overhead stated so the evidence is reproducible.

What you leave with

  • tracefs fluency without folklore
  • function_graph tracing with disciplined filters
  • Event filters, triggers and in-kernel histograms
  • Latency tracer interpretation (irqsoff, preemptoff, wakeup)
  • A trace-cmd plus KernelShark workflow for captures you can share

How it runs

Every course follows the same model: 35% principles, 20% guided investigation, 45% engineering studio. You leave with working code, raw measurements and an evidence-based report — not a certificate of attendance. Read the methodology or see a full sample lesson.

Material is adapted to your kernel version, hardware and workload before a private delivery. For public cohorts, the environment is provided and configured.

Questions

Who is this course for?

Kernel, driver and performance engineers who have touched ftrace through folklore and one-liners and want to use the kernel's built-in tracer deliberately — filters, events and latency tracers included. It sits at practitioner level within the Debugging & Tracing track.

What do I need to know already?

Specific prerequisites for this course: Command-line Linux; Basic kernel internals (what a syscall and an interrupt are); Ability to read C helpful for interpreting traces. We confirm levels before the cohort starts and adapt if a group is stronger or weaker than expected.

Can this run privately for my team?

Yes. Any course runs on-site at your offices anywhere, or live online for a distributed team, with labs adapted to your hardware and codebase.

What is the difference between in-person and online?

In person is 3 full days with hardware on your desk, capped at 14. Online is 6 half-day sessions across about two weeks so you can keep working, capped at 20, with remote lab access.

Do you invoice companies?

Yes. Purchase orders are accepted and invoicing is available in USD, EUR, GBP, SAR and CAD.

Upcoming dates

DatesWhereSeatsEarly birdRegular
1 Nov – 3 Nov 20263 full days RiyadhIn person · KAFD Conference Centre 6 of 14 —SAR 7,880
8 Nov – 10 Nov 20263 full days Kuwait CityIn person · Al Hamra Tower 11 of 14 KWD 580until 9 OctKWD 650
8 Nov – 10 Nov 20263 full days MuscatIn person · Knowledge Oasis Muscat 6 of 14 OMR 730until 9 OctOMR 810
15 Nov – 22 Nov 20266 half-days Gulf bandLive online · 09:00–13:00 GMT+3 18 of 20 US$1,350until 16 OctUS$1,500
16 Nov – 18 Nov 20263 full days OttawaIn person · Kanata North Tech Park 11 of 14 CAD 2,570until 17 OctCAD 2,860
16 Nov – 23 Nov 20266 half-days Europe bandLive online · 09:00–13:00 CET 7 of 20 US$1,350until 17 OctUS$1,500
23 Nov – 25 Nov 20263 full days TorontoIn person · MaRS Discovery District 6 of 14 CAD 2,570until 24 OctCAD 2,860
23 Nov – 25 Nov 20263 full days LondonIn person · Shoreditch Works 11 of 14 GBP 1,480until 24 OctGBP 1,640
23 Nov – 30 Nov 20266 half-days Americas bandLive online · 13:00–17:00 ET 12 of 20 US$1,350until 24 OctUS$1,500
30 Nov – 2 Dec 20263 full days BerlinIn person · Factory Görlitzer Park 6 of 14 EUR 1,740until 31 OctEUR 1,930

Dates shown for the next few months. If nothing fits, tell us where and when — cohorts are added on demand, and private delivery can be scheduled any week.

More in Debugging & Tracing

DBG-1012 days Reading an Oops & Panic Analysis Turning a kernel splat into a precise location in the source, and knowing what the register dump is telling you. Practitioner Practitioner-taught SAR 5,250Next 11 Oct DBG-1103 days kdump & the crash Utility Capturing a crash dump in production and doing a real post-mortem on it. Advanced Practitioner-taught SAR 9,000Next 8 Nov DBG-1203 days kgdb & Live Kernel Debugging Interactive kernel debugging over serial and network, plus dynamic debug for cases where stopping is not an option. Advanced Practitioner-taught SAR 9,000Next 25 Oct DBG-2103 days perf: Sampling to Flame Graphs CPU and off-CPU analysis with perf, from first sample to a flame graph that tells you something actionable. Practitioner Practitioner-taught SAR 7,880Next 11 Oct DBG-2204 days eBPF & bpftrace Programmable observability: one-liners for immediate answers, custom programs for the questions nothing else answers. Advanced Practitioner-taught SAR 12,000Next 15 Nov DBG-3013 days Race Conditions & Lock Contention The bugs that only appear under load on someone else's machine, and a method for actually finding them. Expert Practitioner-taught SAR 10,120Next 22 Nov DBG-3103 days Memory Corruption: KASAN & KFENCE Finding use-after-free, out-of-bounds and uninitialised memory before they become a security advisory. Advanced Practitioner-taught SAR 9,000Next 1 Nov