Configure OOM Killer in Linux.
1. OOM Killer Introduction
OOM(Out of Memory) killer is a process which is called by our system kernel when linux system memory is critically low to recover some memory/RAM.
OOM is designed to kill some processes and reduce the memory usage on the server.
When the system memory is highly used by the processes and there is not enough memory available on the system, this critical situation should be handled asap to keep the system running.
Linux kernel invokes the OOM Killer to review all running processes and kill one or more of these processes in order to free up system memory or RAM and keep the system running.
2. Why system runs out of memory ?
Linux kernel was designed to work on over-commit memory model to make memory usage more efficient. In this over-commit model,the kernel allocates more memory to the system processes than the actual physically available memory. For example:- If a system has 4GB RAM, it used to allocate 4.5GB of memory to all processes so that the processes can work more efficiently.
Suppose if a process starts utilizing the memory it was allocated, and when too many processes start utilizing the memory they were allocated, this over-commit memory model sometimes can generate problems and the kernel must kill processes in order to stay operational.
This mechanism of kernel which is used to recover memory on the system is known as the out-of-memory killer or OOM killer.
3. How process is selected by OOM Killer ?
OOM killer assigns a score to each running process often called as badness score.
A process with high badness score is likely to be killed first in case of high memory consumption.
System, kernel and root processes are given much lower badness score compared to other user processes.
OOM killer tries to kill minimum number of processes possible to free up memory space ideally is possible a single high memory consuming process.
4. Which process is likely to be killed by OOM killer ?
Non system processes using high amount of memory and having a lot of child processes is likely to be killed first.
Due to this reason, mysql/apache processes are killed first as they are largest in-memory, and non-system processes.
5. Identify which process is killed by OOM Killer.
To identify the processes killed or affected by oom killer, we can look in the system logs like below.
a. dmesg | grep -i “killed process”; b. cat /var/log/messages | grep -i "killed process"; c. journalctl -a | grep -i "killed process"
Output
### The result or output will look like below: host kernel: Out of Memory: Killed process 3498 (apache).
6. Some Situations causing OOM condition
a. Kernel data structures can occupy too much memory space exhausting the system and can cause an OOM situation, even when swap space is available.
b. System running out of RAM & Swap.
c. NUMA architecture–based systems can encounter an OOM condition if one node is out of memory while plenty of memory is available in the remaining nodes.
d. Sometimes kernel is also not able to utilize swap space in ideal way due to the type of workload or address space required by different applications on the system. For example:- some applications need physical ram address space to run and can’t utilize swap space.
7. Enable/Disable OOM killer on our system
There is no way to disable OOM killer, we can turn off the overcommit memory to avoid OOM situation most of the times.
a. Enable Overcommit memory
Put vm.overcommit_memory = 0 in /etc/sysctl.conf
0 = “get an estimate if we have enough RAM”, 1 = “Always yes”, 2 = “no if we the memory is not enough”
vm.overcommit_memory = 2
Note:- There is no system variable with name vm.oom-kill. If you mention this, you will encounter an error below.
error: "vm.oom-kill" is an unknown key
8. Change OOM Behavior: Kill the task taking maximum memory Vs kill the task because of which OOM condition occurred.
OOM can kill a process in two ways to come out of the high memory situations:
a. Kill the process taking up most of the memory vm.oom_kill_allocating_task = 0 b. Kill the task which triggered OOM situation. vm.oom_kill_allocating_task = 1
Default value is 0.
If vm.panic_on_oom is defined, it takes precedence over the value used in vm.oom_kill_allocating_task.
9. Enable kernel panic on OOM to collect crash dump
vm.panic_on_oom = 0 (Default value)
Values that can be defined:
0 :- Kill the process, no panic
1 :- Kernel Panic on OOM only if system memory is exhausted. If cgroup/mempolicy limit, then it will kill that process.
2 :- Kernel panic even if a process defined memory is exhausted.
1 and 2 are generally used for failover in case of clustering. Please select either according to your policy of failover.
vm.panic_on_oom = 2+kdump gives us enough data to investigate the system issue.
10. Change OOM Killer Badness Score
a. Check Defauit badness Score [root@nglinux ~]# ps PID TTY TIME CMD 6565 pts/0 00:00:00 bash 6792 pts/0 00:00:00 ps [root@nglinux ~]# cat /proc/6565/oom_adj 0 [root@nglinux ~]# b. Increase the value to kill it first. [root@nglinux ~]# echo 15 > /proc/6565/oom_adj [root@nglinux ~]# cat /proc/6565/oom_adj 15 [root@nglinux ~]# c. Decrease the value to make it less likely to be killed. [root@nglinux ~]# echo -15 > /proc/6565/oom_adj [root@nglinux ~]# cat /proc/6565/oom_adj -15