You set a 512MB memory limit on your container, Kubernetes tells the kubelet to enforce it, and yet your Java app gets an OOMKilled signal while the metrics show it only used 300MB. It’s a classic trap that I’ve seen frustrate countless engineers. The kernel doesn’t just look at what your app thinks it’s using; it looks at the page cache, the kernel structures, and how much of that memory is actually reclaimable. If your app is hitting a wall, it is usually because the cgroup pressure is being interpreted differently by the runtime than by your monitoring stack.

Why the kernel sees a different number

When you are trying to debug memory starvation, standard metrics tools often aggregate memory in ways that hide the truth. They show you RSS, but the cgroup memory.current includes everything the container owns that isn’t easily reclaimable by the kernel’s active shrinker. If you’re running on a newer kernel with cgroup v2 enabled, the interface changed. You aren’t just looking at memory.usage_in_bytes anymore; you’re dealing with memory.stat which gives a much clearer view of what’s pinned versus what’s active.

cat /sys/fs/cgroup/memory.stat

• Check anon memory to see the actual application heap usage
• Look at file_mapped to see if you are cache-heavy
• Investigate slab_unreclaimable if your kernel objects are ballooning
• Compare memory.high vs memory.max to see if the kernel is throttling instead of killing

Most of the time, the fix isn’t just increasing the limit. If you see high inactive_file usage, you might just need to tune your JVM or your application’s disk interaction patterns to release cache faster. Stop throwing memory at the problem and start looking at what the kernel considers non-negotiable memory. Next time you see a pod exit with a 137 code, bypass the dashboard metrics and go straight to the sysfs files on the node; it is the only way to know if your app is actually failing or if the kernel is just being protective.