Stop the kernel from lying about your CPU wait times
Technical Briefing | 11/20/2026
You see a load average of 5.0 on a 4-core machine and immediately assume your app is hitting a wall. You check top, look at the CPU utilization, and it says you have idle time. It is maddening. This happens because iowait is lumped into your idle percentages in some older utilities, but even in modern tools, the kernel metric is often misunderstood. I spent three days chasing a phantom process once, only to realize the disk controller was saturating its queue.
Why iowait is a misleading metric
The kernel tracks iowait as the time a processor is idle while there is at least one outstanding disk I/O request. It implies your CPU is sitting on its hands waiting for storage, but it does not tell you which process is holding the baton. If you have multiple threads hitting a single slow NVMe or a shared SAN, you might see massive iowait, but top won’t show the process consuming actual user time. You need to look deeper into the block layer.
iostat -xz 1
- Check the await column to see if your disk latency is actually climbing above 10ms
- Watch the %util column to see if your block device is hitting saturation
- Use pidstat -d to identify which specific PID is generating the most I/O throughput
Stop guessing and track the blockage
Most sysadmins stop at iostat, but if you really need to know what’s blocking, you have to trace the kernel functions. If you’re on a modern kernel, reach for bpftrace. It bypasses the aggregate reporting and shows you exactly which process is submitting I/O requests that result in long wait times. This is the only way to prove to your storage team that the SAN is the problem and not your code.
If you see your iowait spikes correlating with spikes in disk latency, don’t bother trying to tune your application logic. You are just fighting physics. Next time the load spikes for no apparent reason, verify the disk latency first before you go burning cycles profiling your web server.
