You check your IPAM dashboard and see hundreds of pods that don’t exist anymore holding onto perfectly good IP addresses. It’s a classic sign that your CNI plugin is failing to clean up after itself when the kubelet stops a container. This bit me in prod during a node scale-down event where the stale IPs caused address exhaustion in a smaller VPC subnet.

Why the IPAM database gets out of sync

Most CNI plugins rely on a local host state file or a backend like etcd to track IP allocation. When a pod is deleted, the CNI expects a DEL command from the kubelet. If the network interface is detached forcefully or the node crashes hard, the CNI plugin never gets that signal. The IP stays reserved in the IPAM store, even if the underlying pod sandbox is long gone.

grep -r 'Allocated' /var/lib/cni/networks/ | awk -F: '{print $1}' | xargs -n 1 basename

• Check the CNI logs specifically for the DEL operations that return non-zero exit codes
• Verify your IPAM state file directory for entries that lack a corresponding running container ID
• Use kubectl get pods -A to confirm the IP really is orphaned and not just assigned to a pending pod

Cleaning up the mess

If you are stuck with ghost IPs, you usually have to manually prune the state files if your plugin isn’t smart enough to self-heal on reboot. Be careful though, deleting entries while the kubelet is actively scheduling can cause collisions. It’s safer to taint the node, drain the workloads, and verify the IPAM directory is truly empty before allowing new pods to land on that node again.

Next time you scale your cluster and see the pod count drop while the subnet utilization flatlines, check the local CNI state. You might just save yourself an emergency migration if you find the leak before the next rolling deployment.