STO-101 · Storage & Filesystems · Practitioner
Block Layer & I/O Schedulers — full syllabus
How an I/O request travels from the filesystem to the device, and what the scheduler does to it on the way.
Who this course is for
SREs, platform and performance engineers who debug storage latency on Linux and want to reason from the block layer's actual mechanics rather than from iostat folklore.
Prerequisites
- Solid Linux administration and command line
- A basic kernel mental model (system calls, VFS, page cache)
- No kernel programming required
Course outline
Day 1 — From syscall to device: the request path
- read/write from the VFS through the page cache to the block layer
- struct bio and struct request: what each carries and who transforms it
- The multiqueue (blk-mq) model: per-CPU software queues and hardware dispatch queues
- Plugging, flushing and the submission path
- Where iostat numbers come from: reading /proc/diskstats correctly
Day 2 — The I/O schedulers
- Why scheduling still matters on fast devices: reads vs writes, latency vs throughput
- mq-deadline: read/write FIFOs, expiry and write-starvation avoidance
- BFQ: proportional-share budgeting for interactive and mixed workloads
- Kyber: latency-target throttling for NVMe
- Merging, plugging and queue depth: how batching shapes throughput
- Choosing none, mq-deadline, BFQ or Kyber by workload
Day 3 — Writeback, throttling and measurement
- Buffered writeback: dirty pages, flusher threads, dirty_ratio and dirty_background_ratio
- Writeback throttling (wbt) and the cgroup v2 io controller
- blktrace/blkparse/btt: capturing and replaying the request stream
- Latency histograms with biolatency (eBPF/BCC)
- fio job design that produces evidence instead of numbers
Hands-on labs
- Lab: benchmark raw-device vs filesystem I/O with fio and explain the difference the page cache makes
- Lab: capture a mixed workload with blktrace and reconstruct who waited on whom with blkparse and btt
- Lab: compare none, mq-deadline, BFQ and Kyber under a read/write mix and pick a winner from latency distributions, not averages
- Lab: push a writer past dirty_background_ratio, observe the throttle, then bound it with the cgroup v2 io controller
- Lab: build biolatency histograms of a slow device and correlate them with iostat queue metrics
Capstone project
Diagnose a misbehaving storage service: given a host with an injected I/O problem (a mis-chosen scheduler, a writeback storm, a queue misconfiguration), you produce an incident report that traces the complaint from application latency down through blktrace and biolatency evidence to the exact layer and knob at fault — then verify the fix with before/after fio latency distributions.
What you leave with
- A working model of the blk-mq request path you can trace on any host
- Scheduler selection backed by measured latency distributions
- blktrace/btt and biolatency as everyday diagnostic tools
- Writeback and cgroup io tuning that survives mixed workloads
Upcoming dates
| Dates | Where | Seats | Early bird | Regular | |
|---|---|---|---|---|---|
| 15 Nov – 17 Nov 20263 full days | RiyadhIn person · KAFD Conference Centre | 8 of 14 | SAR 7,090until 16 Oct | ||
| 15 Nov – 17 Nov 20263 full days | Kuwait CityIn person · Al Hamra Tower | 3 of 14 | KWD 580until 16 Oct | ||
| 22 Nov – 24 Nov 20263 full days | MuscatIn person · Knowledge Oasis Muscat | 8 of 14 | OMR 730until 23 Oct | ||
| 29 Nov – 6 Dec 20266 half-days | Gulf bandLive online · 09:00–13:00 GMT+3 | 6 of 20 | US$1,350until 30 Oct | ||
| 30 Nov – 2 Dec 20263 full days | OttawaIn person · Kanata North Tech Park | 3 of 14 | CAD 2,570until 31 Oct | ||
| 30 Nov – 2 Dec 20263 full days | TorontoIn person · MaRS Discovery District | 8 of 14 | CAD 2,570until 31 Oct | ||
| 30 Nov – 7 Dec 20266 half-days | Europe bandLive online · 09:00–13:00 CET | 11 of 20 | US$1,350until 31 Oct | ||
| 7 Dec – 9 Dec 20263 full days | LondonIn person · Shoreditch Works | 3 of 14 | GBP 1,480until 7 Nov | ||
| 7 Dec – 14 Dec 20266 half-days | Americas bandLive online · 13:00–17:00 ET | 16 of 20 | US$1,350until 7 Nov | ||
| 14 Dec – 16 Dec 20263 full days | BerlinIn person · Factory Görlitzer Park | 8 of 14 | EUR 1,740until 14 Nov |
Book a seat, or bring this course to your team
Seats can be reserved online; private delivery runs on-site or live online, adapted to your stack.
Questions about fit or prerequisites? Email hello@kernelsystems.academy. To save this syllabus, print this page to PDF from your browser.