RTL-110 · Real-Time Systems · Practitioner
Latency Analysis & cyclictest — full syllabus
Measuring latency honestly: the tools, the methodology, and the mistakes that produce meaningless numbers.
Who this course is for
Engineers who must prove a system meets a latency bound — and who suspect their current benchmark numbers would not survive review.
Prerequisites
- Comfortable with the Linux command line and shell scripting
- Basic understanding of scheduling and interrupts
- No prior PREEMPT_RT experience required
Course outline
Day 1 — What latency is and how cyclictest measures it
- Decomposing wakeup latency: timer, IRQ, scheduler, context switch
- How cyclictest works inside: clock_nanosleep, hrtimers and what it actually samples
- Correct invocation: priorities, intervals, thread count, affinity and duration
- Reading the output: min/avg/max and why only some of those numbers matter
- Invocation mistakes that silently produce meaningless results
Day 2 — The tracer arsenal
- ftrace latency tracers: wakeup, wakeup_rt, irqsoff, preemptoff
- rtla: the osnoise and timerlat tracers and the noise sources they separate
- hwlatdetect and the SMIs it exposes
- Tracing one worst-case event from detection to root cause
- Latency histograms and reading distribution shape, not single maxima
Day 3 — Methodology that survives scrutiny
- Building a representative load: CPU, memory, I/O and network interference
- Why an idle benchmark proves nothing
- Chasing the tail: percentiles, maxima and how long a run is long enough
- Controlling the environment: frequency, topology, thermal state
- Documenting a result — platform, config, load, raw data — so others can reproduce it
Hands-on labs
- Lab: Run cyclictest with a correct and a deliberately broken invocation; explain why the broken numbers look better
- Lab: Build a layered interference load with stress-ng and measure its effect on each latency component separately
- Lab: Capture a worst-case event with rtla timerlat and osnoise, then chase it to its source with ftrace
- Lab: Produce latency histograms under three load profiles and compare the tails rather than the averages
- Lab: Write a one-page latency report for a measured system that a colleague could reproduce exactly
Capstone project
Characterize a platform you have not seen before: define and justify a representative load, run a long cyclictest campaign with histograms, chase the two worst outliers to root cause with rtla and ftrace, and deliver a latency report — platform description, kernel configuration, load definition, raw data, analysis scripts and a stated worst case with the evidence behind it — that another engineer could rerun and confirm.
What you leave with
- Correct cyclictest and rtla invocation habits
- A root-cause workflow from histogram outlier to named kernel path
- Load-design skills that make a measurement mean something
- A latency report template with reproducibility built in
Upcoming dates
| Dates | Where | Seats | Early bird | Regular | |
|---|---|---|---|---|---|
| 18 Oct – 20 Oct 20263 full days | RiyadhIn person · KAFD Conference Centre | 4 of 14 | — | SAR 7,880 | |
| 25 Oct – 27 Oct 20263 full days | Kuwait CityIn person · Al Hamra Tower | 9 of 14 | — | KWD 650 | |
| 25 Oct – 27 Oct 20263 full days | MuscatIn person · Knowledge Oasis Muscat | 4 of 14 | — | OMR 810 | |
| 1 Nov – 8 Nov 20266 half-days | Gulf bandLive online · 09:00–13:00 GMT+3 | 16 of 20 | — | US$1,500 | |
| 2 Nov – 4 Nov 20263 full days | OttawaIn person · Kanata North Tech Park | 9 of 14 | — | CAD 2,860 | |
| 9 Nov – 11 Nov 20263 full days | TorontoIn person · MaRS Discovery District | 4 of 14 | CAD 2,570until 10 Oct | ||
| 9 Nov – 11 Nov 20263 full days | LondonIn person · Shoreditch Works | 9 of 14 | GBP 1,480until 10 Oct | ||
| 9 Nov – 16 Nov 20266 half-days | Europe bandLive online · 09:00–13:00 CET | 5 of 20 | US$1,350until 10 Oct | ||
| 9 Nov – 16 Nov 20266 half-days | Americas bandLive online · 13:00–17:00 ET | 10 of 20 | US$1,350until 10 Oct | ||
| 16 Nov – 18 Nov 20263 full days | BerlinIn person · Factory Görlitzer Park | 4 of 14 | EUR 1,740until 17 Oct |
Book a seat, or bring this course to your team
Seats can be reserved online; private delivery runs on-site or live online, adapted to your stack.
Questions about fit or prerequisites? Email hello@kernelsystems.academy. To save this syllabus, print this page to PDF from your browser.