Stop the kernel from lying to you about disk latency with biolatency

Technical Briefing | 11/12/2026

You check iostat, and the disk utilization is hovering at a nice, calm 30 percent. Your application developers are screaming that the database is hanging, and the logs show request queues backing up like it is Friday afternoon at the DMV. We have all been there. The reality is that average latency numbers are a trap, and iostat hides the spikes that actually kill your throughput. When you see high average wait times, you are looking at the result of a problem, not the cause.

Why averages kill your visibility

The problem with standard tools is they aggregate data over time intervals, smoothing out the outliers that cause stalls. If your disk serves a hundred fast operations and one massive, multi-second latency spike in a single window, the average looks perfectly fine. You need a histogram of block I/O latency to actually see the tail latency that causes your app to hit timeouts. Stop relying on moving averages and start looking at the distribution.

/usr/sbin/biolatency -D 5

• The -D flag lets you isolate latency by device, which is essential if you are running multi-tenant storage.
• Watch for bimodal distributions, which usually indicate contention between two different IO profiles fighting for the same spindle or controller.
• If you see latency spikes that correlate with kernel readahead or page cache pressure, you know it is a memory issue disguised as storage lag.

What to do when you see the tail

Once you have the histogram, ignore the middle of the pack and focus entirely on the right tail. If you see latency extending into the hundreds of milliseconds, you are likely looking at queue depth saturation or a failing controller cache. I have seen this catch intermittent hardware stalls that took a week of uptime before showing up as a full-blown error. If the hardware checks out, look for cgroup limits that might be artificially throttling your block device throughput. Keep this one in your back pocket for the next time you get blamed for slow disks when the dashboard says everything is fine.