RTL-110 · Real-Time Systems
Latency Analysis & cyclictest
Measuring latency honestly: the tools, the methodology, and the mistakes that produce meaningless numbers.
Who this course is for
Engineers who must prove a system meets a latency bound — and who suspect their current benchmark numbers would not survive review.
Prerequisites
Course outline
Day 1 — What latency is and how cyclictest measures it
- Decomposing wakeup latency: timer, IRQ, scheduler, context switch
- How cyclictest works inside: clock_nanosleep, hrtimers and what it actually samples
- Correct invocation: priorities, intervals, thread count, affinity and duration
- Reading the output: min/avg/max and why only some of those numbers matter
- Invocation mistakes that silently produce meaningless results
Day 2 — The tracer arsenal
- ftrace latency tracers: wakeup, wakeup_rt, irqsoff, preemptoff
- rtla: the osnoise and timerlat tracers and the noise sources they separate
- hwlatdetect and the SMIs it exposes
- Tracing one worst-case event from detection to root cause
- Latency histograms and reading distribution shape, not single maxima
Day 3 — Methodology that survives scrutiny
- Building a representative load: CPU, memory, I/O and network interference
- Why an idle benchmark proves nothing
- Chasing the tail: percentiles, maxima and how long a run is long enough
- Controlling the environment: frequency, topology, thermal state
- Documenting a result — platform, config, load, raw data — so others can reproduce it
Hands-on labs
Labs follow the academy model — 35% principles, 20% guided investigation, 45% engineering studio. Every claim you make in a lab is backed by a trace, a counter or a measurement you captured yourself. How we teach
- Lab: Run cyclictest with a correct and a deliberately broken invocation; explain why the broken numbers look better
- Lab: Build a layered interference load with stress-ng and measure its effect on each latency component separately
- Lab: Capture a worst-case event with rtla timerlat and osnoise, then chase it to its source with ftrace
- Lab: Produce latency histograms under three load profiles and compare the tails rather than the averages
- Lab: Write a one-page latency report for a measured system that a colleague could reproduce exactly
Capstone project
Characterize a platform you have not seen before: define and justify a representative load, run a long cyclictest campaign with histograms, chase the two worst outliers to root cause with rtla and ftrace, and deliver a latency report — platform description, kernel configuration, load definition, raw data, analysis scripts and a stated worst case with the evidence behind it — that another engineer could rerun and confirm.
What you leave with
- Correct cyclictest and rtla invocation habits
- A root-cause workflow from histogram outlier to named kernel path
- Load-design skills that make a measurement mean something
- A latency report template with reproducibility built in
How it runs
Every course follows the same model: 35% principles, 20% guided investigation, 45% engineering studio. You leave with working code, raw measurements and an evidence-based report — not a certificate of attendance. Read the methodology or see a full sample lesson.
Material is adapted to your kernel version, hardware and workload before a private delivery. For public cohorts, the environment is provided and configured.
Questions
Who is this course for?
Engineers who must prove a system meets a latency bound — and who suspect their current benchmark numbers would not survive review. It sits at practitioner level within the Real-Time Systems track.
What do I need to know already?
Specific prerequisites for this course: Comfortable with the Linux command line and shell scripting; Basic understanding of scheduling and interrupts; No prior PREEMPT_RT experience required. We confirm levels before the cohort starts and adapt if a group is stronger or weaker than expected.
Can this run privately for my team?
Yes. Any course runs on-site at your offices anywhere, or live online for a distributed team, with labs adapted to your hardware and codebase.
What is the difference between in-person and online?
In person is 3 full days with hardware on your desk, capped at 14. Online is 6 half-day sessions across about two weeks so you can keep working, capped at 20, with remote lab access.
Do you invoice companies?
Yes. Purchase orders are accepted and invoicing is available in USD, EUR, GBP, SAR and CAD.
Upcoming dates
| Dates | Where | Seats | Early bird | Regular | |
|---|---|---|---|---|---|
| 18 Oct – 20 Oct 20263 full days | RiyadhIn person · KAFD Conference Centre | 4 of 14 | — | SAR 7,880 | |
| 25 Oct – 27 Oct 20263 full days | Kuwait CityIn person · Al Hamra Tower | 9 of 14 | — | KWD 650 | |
| 25 Oct – 27 Oct 20263 full days | MuscatIn person · Knowledge Oasis Muscat | 4 of 14 | — | OMR 810 | |
| 1 Nov – 8 Nov 20266 half-days | Gulf bandLive online · 09:00–13:00 GMT+3 | 16 of 20 | — | US$1,500 | |
| 2 Nov – 4 Nov 20263 full days | OttawaIn person · Kanata North Tech Park | 9 of 14 | — | CAD 2,860 | |
| 9 Nov – 11 Nov 20263 full days | TorontoIn person · MaRS Discovery District | 4 of 14 | CAD 2,570until 10 Oct | ||
| 9 Nov – 11 Nov 20263 full days | LondonIn person · Shoreditch Works | 9 of 14 | GBP 1,480until 10 Oct | ||
| 9 Nov – 16 Nov 20266 half-days | Europe bandLive online · 09:00–13:00 CET | 5 of 20 | US$1,350until 10 Oct | ||
| 9 Nov – 16 Nov 20266 half-days | Americas bandLive online · 13:00–17:00 ET | 10 of 20 | US$1,350until 10 Oct | ||
| 16 Nov – 18 Nov 20263 full days | BerlinIn person · Factory Görlitzer Park | 4 of 14 | EUR 1,740until 17 Oct |
Dates shown for the next few months. If nothing fits, tell us where and when — cohorts are added on demand, and private delivery can be scheduled any week.
More in Real-Time Systems
RTL-1013 days
PREEMPT_RT Internals
What the real-time patchset actually changes in the kernel, now that most of it has landed mainline.
Practitioner-taught
SAR 9,000Next 1 Nov
RTL-1203 days
Tuning for Determinism
Turning a general-purpose platform into one that holds a deadline: isolation, affinity, firmware and BIOS.
Practitioner-taught
SAR 9,000Next 15 Nov
RTL-2012 days
Motion Control Systems
Applying real-time Linux to closed-loop control, with the architecture patterns that work in production.
Practitioner-taught
SAR 6,000Next 11 Oct
RTL-2103 days
Industrial Fieldbus & TSN
Deterministic networking for industrial systems: EtherCAT, PROFINET and Time-Sensitive Networking.
Practitioner-taught
SAR 9,000Next 8 Nov