STO-101 · Storage & Filesystems
Block Layer & I/O Schedulers
How an I/O request travels from the filesystem to the device, and what the scheduler does to it on the way.
Who this course is for
SREs, platform and performance engineers who debug storage latency on Linux and want to reason from the block layer's actual mechanics rather than from iostat folklore.
Prerequisites
Course outline
Day 1 — From syscall to device: the request path
- read/write from the VFS through the page cache to the block layer
- struct bio and struct request: what each carries and who transforms it
- The multiqueue (blk-mq) model: per-CPU software queues and hardware dispatch queues
- Plugging, flushing and the submission path
- Where iostat numbers come from: reading /proc/diskstats correctly
Day 2 — The I/O schedulers
- Why scheduling still matters on fast devices: reads vs writes, latency vs throughput
- mq-deadline: read/write FIFOs, expiry and write-starvation avoidance
- BFQ: proportional-share budgeting for interactive and mixed workloads
- Kyber: latency-target throttling for NVMe
- Merging, plugging and queue depth: how batching shapes throughput
- Choosing none, mq-deadline, BFQ or Kyber by workload
Day 3 — Writeback, throttling and measurement
- Buffered writeback: dirty pages, flusher threads, dirty_ratio and dirty_background_ratio
- Writeback throttling (wbt) and the cgroup v2 io controller
- blktrace/blkparse/btt: capturing and replaying the request stream
- Latency histograms with biolatency (eBPF/BCC)
- fio job design that produces evidence instead of numbers
Hands-on labs
Labs follow the academy model — 35% principles, 20% guided investigation, 45% engineering studio. Every claim you make in a lab is backed by a trace, a counter or a measurement you captured yourself. How we teach
- Lab: benchmark raw-device vs filesystem I/O with fio and explain the difference the page cache makes
- Lab: capture a mixed workload with blktrace and reconstruct who waited on whom with blkparse and btt
- Lab: compare none, mq-deadline, BFQ and Kyber under a read/write mix and pick a winner from latency distributions, not averages
- Lab: push a writer past dirty_background_ratio, observe the throttle, then bound it with the cgroup v2 io controller
- Lab: build biolatency histograms of a slow device and correlate them with iostat queue metrics
Capstone project
Diagnose a misbehaving storage service: given a host with an injected I/O problem (a mis-chosen scheduler, a writeback storm, a queue misconfiguration), you produce an incident report that traces the complaint from application latency down through blktrace and biolatency evidence to the exact layer and knob at fault — then verify the fix with before/after fio latency distributions.
What you leave with
- A working model of the blk-mq request path you can trace on any host
- Scheduler selection backed by measured latency distributions
- blktrace/btt and biolatency as everyday diagnostic tools
- Writeback and cgroup io tuning that survives mixed workloads
How it runs
Every course follows the same model: 35% principles, 20% guided investigation, 45% engineering studio. You leave with working code, raw measurements and an evidence-based report — not a certificate of attendance. Read the methodology or see a full sample lesson.
Material is adapted to your kernel version, hardware and workload before a private delivery. For public cohorts, the environment is provided and configured.
Questions
Who is this course for?
SREs, platform and performance engineers who debug storage latency on Linux and want to reason from the block layer's actual mechanics rather than from iostat folklore. It sits at practitioner level within the Storage & Filesystems track.
What do I need to know already?
Specific prerequisites for this course: Solid Linux administration and command line; A basic kernel mental model (system calls, VFS, page cache); No kernel programming required. We confirm levels before the cohort starts and adapt if a group is stronger or weaker than expected.
Can this run privately for my team?
Yes. Any course runs on-site at your offices anywhere, or live online for a distributed team, with labs adapted to your hardware and codebase.
What is the difference between in-person and online?
In person is 3 full days with hardware on your desk, capped at 14. Online is 6 half-day sessions across about two weeks so you can keep working, capped at 20, with remote lab access.
Do you invoice companies?
Yes. Purchase orders are accepted and invoicing is available in USD, EUR, GBP, SAR and CAD.
Upcoming dates
| Dates | Where | Seats | Early bird | Regular | |
|---|---|---|---|---|---|
| 15 Nov – 17 Nov 20263 full days | RiyadhIn person · KAFD Conference Centre | 8 of 14 | SAR 7,090until 16 Oct | ||
| 15 Nov – 17 Nov 20263 full days | Kuwait CityIn person · Al Hamra Tower | 3 of 14 | KWD 580until 16 Oct | ||
| 22 Nov – 24 Nov 20263 full days | MuscatIn person · Knowledge Oasis Muscat | 8 of 14 | OMR 730until 23 Oct | ||
| 29 Nov – 6 Dec 20266 half-days | Gulf bandLive online · 09:00–13:00 GMT+3 | 6 of 20 | US$1,350until 30 Oct | ||
| 30 Nov – 2 Dec 20263 full days | OttawaIn person · Kanata North Tech Park | 3 of 14 | CAD 2,570until 31 Oct | ||
| 30 Nov – 2 Dec 20263 full days | TorontoIn person · MaRS Discovery District | 8 of 14 | CAD 2,570until 31 Oct | ||
| 30 Nov – 7 Dec 20266 half-days | Europe bandLive online · 09:00–13:00 CET | 11 of 20 | US$1,350until 31 Oct | ||
| 7 Dec – 9 Dec 20263 full days | LondonIn person · Shoreditch Works | 3 of 14 | GBP 1,480until 7 Nov | ||
| 7 Dec – 14 Dec 20266 half-days | Americas bandLive online · 13:00–17:00 ET | 16 of 20 | US$1,350until 7 Nov | ||
| 14 Dec – 16 Dec 20263 full days | BerlinIn person · Factory Görlitzer Park | 8 of 14 | EUR 1,740until 14 Nov |
Dates shown for the next few months. If nothing fits, tell us where and when — cohorts are added on demand, and private delivery can be scheduled any week.
More in Storage & Filesystems
STO-1103 days
NVMe & NVMe-oF
NVMe as a protocol and as a driver, including fabrics for disaggregated storage.
Practitioner-taught
SAR 9,000Next 25 Oct
STO-1203 days
Device Mapper & LVM
Composing block devices: linear, striped, snapshot, thin provisioning, crypt and cache targets.
Practitioner-taught
SAR 7,880Next 11 Oct
STO-2013 days
VFS Internals
The abstraction every filesystem implements: inodes, dentries, the page cache and the locking around them.
Practitioner-taught
SAR 9,000Next 18 Oct
STO-2103 days
ext4 & XFS Internals
The on-disk layout and operational behaviour of the two filesystems most production Linux runs on.
Practitioner-taught
SAR 9,000Next 15 Nov
STO-2204 days
Writing a Filesystem from Scratch
Implement a small but real filesystem, which is the fastest way to genuinely understand the VFS.
Practitioner-taught
SAR 13,500Next 1 Nov