You spend weeks tuning your heap size and optimizing your queries, but sometimes the kernel just decides your application has to die. It’s not a memory leak, and it’s not a bad actor. It’s the OOM killer doing its job when the node is starving for RAM. If you aren’t setting your OOM scores explicitly, you’re leaving your mission-critical services at the mercy of a kernel heuristic that doesn’t care about your uptime.

How the kernel decides who gets the axe

The kernel assigns an oom_score_adj to every process, which acts as a modifier for the baseline oom_score. Most folks don’t realize that Kubernetes manages this score for you based on the Quality of Service class you assign to your pod. BestEffort pods are the first to get gutted, while Guaranteed pods usually escape the reaper. But if you have a bunch of Burstable pods, the kernel is playing favorites based on who’s using the most memory relative to their limits. This bit me in prod once when a logging sidecar was pinned by the scheduler while the actual application process got nuked.

grep oom_score_adj /proc/$(pgrep -u your-app-user)/oom_score_adj

• Guaranteed pods get -998, which effectively makes them immune unless you really hose the system
• Burstable pods get a dynamic score that scales based on memory requests versus actual usage
• BestEffort pods start at a default of 1000, basically screaming take me first
• You can manually nudge these values in your pod spec using the resources field

Take control of the process priority

If you really need to keep a specific container alive, you might be tempted to mess with the cgroup files directly from inside the container, but don’t bother. The container runtime will reset your changes or throw an error. The only sane way is to define your resource limits strictly and use PriorityClass objects. If you find your primary app is still being picked off, double-check that your memory requests aren’t set too low. The scheduler uses requests to make placement decisions, but the OOM killer looks at actual consumption versus memory limits. Stop assuming the defaults are doing what you want, and start looking at the /proc filesystem for your own running processes to see how the kernel currently views them.