What Is The OOM Killer?
The OOM Killer or Out Of Memory Killer is a process that the linux kernel employs when the system is critically low on memory. This situation occurs because the linux kernel has over allocated memory to its processes. When a process starts it requests a block of memory from the kernel. This initial request is usually a large request that the process will not immediately or indeed ever use all of. The kernel, aware of this tendency for processes to request redundant memory, over allocates the system memory. This means that when the system has, for example, 2GB of RAM the kernel may allocate 2.5GB to processes. This maximises the use of system memory by ensuring that the memory that is allocated to processes is being actively used.
Normally, this situation does not cause a problem. However, if enough processes begin to use all of their requested memory blocks then there will not be enough physical memory to support them all. This means that the running processes require more memory than is physically available. This situation is critical and must be resolved immediately.
The solution that the linux kernel employs is to invoke the OOM Killer to review all running processes and kill one or more of them in order to free up system memory and keep the system running.
How Does It Select A Process To Kill?
The OOM Killer works by reviewing all running processes and assigning them a badness score. The process that has the highest score is the one that is killed. The OOM Killer assigns a badness score based on a number of criteria. The principle of which are as follows are as follows:
- The process and its all of its child processes are using a lot of memory
- The minimum number of processes are killed (ideally one) in order to free up enough memory to resolve the situation
- Root, kernel and important system processes are given much lower scores
So Why Is Apache / MySQL Always Killed?
The above listed criteria mean that when selecting a process to kill the OOM Killer will choose a process using lots of memory and has lots of child processes and which are not system processes. An application such as apache, mysql, ftp server or a mail server will make an good candidate. However, as this situation usually occurs on a busy web server Apache or MySQL will be the largest in-memory, non-system processes and consequently get killed.
It must be remembered that although the webserver or db server are very important to you when the kernel calls the OOM Killer the situation is critical. If memory is not freed by killing a process the server will crash very shortly afterwards. Continuing normal operations at this juncture is impossible.
How To Find Out If The OOM Killer Was Running
The easiest way to find if the OOM Killer was the reason that a website went offline is to check the system log. Whenever the OOM Killer is invoked it will write a great deal of information to the system log including which process was killed and why. If the following command is run:
dmesg | egrep -i “killed process”
The output will look like the following
host kernel: Out of Memory: Killed process 2592 (mysql).
In this instance the process that was killed was mysql which has the PID (Process Identification Number) of 2592.
How To Avoid The OOM Killer
The OOM Killer will only get invoked when the system is critically low on memory. Consequently the solution to avoiding it is to either reduce the memory requirements of the server or increase the available memory.