Stop the kernel from lying about your disk latency

Technical Briefing | 9/6/2026

You check your monitoring dashboard and everything looks green. IOPS are within limits, throughput is fine, and yet your application is hanging on disk writes like it is stuck in molasses. This has bitten me in prod more times than I care to admit. The problem isn’t the monitoring tool, it is the kernel buffering. When you look at iostat, you are often seeing what the kernel queued, not what the physical platter or flash actually committed to permanent storage.

Why iostat is only telling half the story

Standard tools show you request submission times, but they often mask the wait time inside the block layer queue or, worse, the controller cache. If your storage subsystem is misconfigured or if you have a massive write-back cache, the kernel reports the IO as finished the moment it hits the memory buffer. The actual latency hit happens later when the kernel flushes those dirty pages. If you want to see the real truth, you need to look at the block layer trace points.

biolatency -D 1

• Use biolatency to break down the actual disk response time in a histogram format
• Check /proc/meminfo for Dirty and Writeback metrics to see if your kernel is backlogged
• Compare await from iostat against the hardware commit times to find the delta

If you see high outliers in your histogram that don’t show up in your standard metrics, you are likely hitting controller-level cache saturation. Stop trusting the averages that your dashboard averages out over five minutes. If your P99 is spiking but your average is fine, your users are feeling the pain, and you need to look at the raw block event traces before you start tuning filesystem parameters that won’t actually help.