Stop trusting cgroup v1 memory limits when your child processes are leaking

Container & Kubernetes Internals

Stop trusting cgroup v1 memory limits when your child processes are leaking

Technical Briefing | 11/7/2026

You set your container limits, deploy your pods, and everything looks stable in Grafana. Then you wake up at 3 AM because your node is thrashing and the OOM killer is nuking half your services. The problem isn’t always the main process. If your application spawns short-lived subprocesses that fork memory-heavy tasks before dying, you are likely hitting a silent wall where cgroup accounting falls behind.

Why the kernel hides memory from you

When a process forks, the kernel uses copy-on-write to keep things snappy. If your app is spawning a shell or a utility that hits memory hard, the cgroup hierarchy under /sys/fs/cgroup might not update the usage_in_bytes counter until the memory page is actually faulted in. I have seen this bite people where the container stays under its hard limit on paper, but the physical node memory is starving because the cgroup controller is lagging behind the actual kernel page allocations.

cat /sys/fs/cgroup/memory/memory.stat | grep -E 'rss|cache|hierarchical_memsw_usage'

  • Check memory.stat for major vs minor page faults to see if you are constantly thrashing memory
  • Verify that memory.kmem.usage_in_bytes isn’t holding onto kernel memory leaks that user-space tools ignore
  • Watch for oom_control in your cgroup to see if the kernel is holding processes in a suspended state instead of killing them

Dealing with the silent bloat

Stop relying on basic monitoring dashboards that just read the high-level metrics. You need to inspect the kernel-level accounting directly. If you suspect your app’s child processes are the culprits, stick an strace on the parent and look for fork and exec syscalls that precede the memory spikes. If you keep finding your containers dying with no clear reason, check if you are hitting the memory soft limit trigger that quietly throttles CPU cycles to reclaim memory, which usually causes a performance death spiral long before the process actually crashes.

Linux Admin Automation  |  © www.ngelinux.com  |  11/7/2026

0 0 votes
Article Rating
Subscribe
Notify of
guest

0 Comments
Newest
Oldest Most Voted