You see high CPU usage in top, but your load average is weirdly low and the application isn’t doing anything heavy. I’ve been here before. You go chasing memory leaks or locking contention, but the culprit is usually the network stack just trying to keep up with incoming packets. When your kernel spends more time processing interrupts than actually running your threads, you have a softirq imbalance.

Why only one core is doing all the work

Modern NICs try to be smart with Receive Side Scaling, but if your interrupt affinity is pinned or the kernel decides to bind the softirq processing to a single core, you end up with one core hitting 100 percent usage while the rest are idle. This bit me hard on a 10Gbps load balancer once. The interface was processing packets fast enough, but the CPU could not handle the softirq queue without dropping frames.

watch -n1 'cat /proc/softirqs | grep NET_RX'

• Check /proc/interrupts to see if your NIC queues are spread across multiple cores
• Verify your smp_affinity settings match the physical CPU topology
• Increase your ring buffer size if you see drops in ethtool -S output

Finding the silent interrupt wall

If you are pinned to a single core, you can try spreading the load by manually updating the hex mask in the smp_affinity file for your IRQ. But be careful. If you map it to a core that is already handling your application threads, you might actually kill your latency performance due to cache thrashing. Keep an eye on your context switches when you make the shift; if those skyrocket, you went too far.

Next time you see that phantom CPU usage, don’t restart the app. Pull up the softirq stats first. A bit of tuning on the IRQ balance usually fixes issues that people spend hours debugging in the application logs for no reason.