You see a Pod in CrashLoopBackOff. You run kubectl describe and there it is: OOMKilled. It’s the classic Kubernetes rite of passage. Most teams just bump the memory limits and call it a day. But if your application doesn’t actually have a memory leak, bumping the limit just masks the fact that your kernel memory management is fighting your container configuration.

The cgroup v1 vs v2 discrepancy that ruins your day

When you set a memory limit in Kubernetes, you’re interacting with cgroups. Here is the problem: the memory limit in the spec isn’t the only metric that causes an eviction. If you are on cgroup v1, the kernel often accounts for filesystem cache and buffers differently than you expect. I have seen pods getting evicted because the page cache grew too quickly, even if the application heap was perfectly healthy. If you are troubleshooting this, you need to look at the memory.stat file inside the container cgroup on the host, not just the dashboard metrics.

cat /sys/fs/cgroup/memory/kubepods.slice/kubepods-podUID.slice/docker-CONTAINERID.scope/memory.stat

• Check rss vs cache in memory.stat to see what is actually pinning your RAM
• Compare the host kernel metrics against your Prometheus memory_usage_bytes metric
• Verify if your application is spawning subprocesses that escape the container cgroup limits

Before you throw more hardware at the problem, look at the hierarchy. If you see high cache usage, your application might be performing massive I/O operations that force the kernel to keep pages in memory longer than necessary. Adjusting your application’s write-back behavior or simply tuning the swappiness of the node can keep your pods alive. Next time you see that OOMKilled flag, ignore the urge to just increase the limit and pull the stats from the node directly.