You spend all morning staring at high latency spikes on your ingress nodes, but the load averages look fine and the CPU usage seems low. You start blaming the application code or some latent lock contention in your database driver. I have been there, and I have wasted days chasing ghosts because the standard tools ignore the hardware interrupt handling latency. If you only look at user-space stats, you are missing half the story.

Why your metrics are blind to hardware interrupts

When a network packet hits your NIC, it triggers an interrupt. The CPU stops what it is doing, executes an interrupt handler, and then resumes its previous task. This context switch is invisible to most monitoring agents because it happens outside the scope of your processes. If your CPU affinity is misaligned or you have a single core getting hammered with softirqs, your throughput will tank while your process-level metrics show plenty of idle time.

watch -n 1 'cat /proc/interrupts | awk "{print \$1, \$2, \$3, \$4, \$NF}"'

• Check /proc/interrupts to see if one core is shouldering all the network load
• Look for NMI watchdogs firing if you see weird stalls during peak traffic
• Review your IRQ balance configuration to see if it is actually respecting your topology

Getting the kernel to stop hiding the truth

The standard approach is often just letting irqbalance handle the distribution, but that daemon is rarely optimal for high-speed throughput. If you really care about performance, pin your IRQs manually to cores that are not running your latency-sensitive user-space apps. It is a pain to manage, but it stops the CPU from thrashing context switches and keeps your packet processing path clear. Don’t let the default settings dictate your latency budget when you can control the hardware mapping yourself.

If you are running on modern hardware with a decent NIC, check the interrupt coalescing settings as well. Sometimes the kernel is just trying to be too helpful by grouping interrupts, which adds exactly the delay you are trying to avoid. Toggle it off during a maintenance window and watch how the latency distribution shifts. You might find that the jitter you blamed on your application was just the kernel trying to be efficient.