OOMKilled

[ianmiell]

No description provided.

[StevenACoffman]

How do you detect OOMKilled events in your Kubernetes cluster?

We have been using log based metrics, looking for messages from kubelet like:

Memory cgroup out of memory: Kill process 1557026 (stress) score 1957 or sacrifice child
Killed process 1557026 (stress) total-vm:256780kB, anon-rss:100056kB, file-rss:264kB, shmem-rss:0kB

Although the OOMKilled event isn't present in the workload's (job or deployment) pod's logs, if you can detect that a pod was killed you can then use kubectl get pod -o go-template=... <pod-id> to determine the last termination reason.
As an example straight from the docs:

$ ./cluster/kubectl.sh  get pod -o go-template='{{range.status.containerStatuses}}{{"Container Name: "}}{{.name}}{{"\r\nLastState: "}}{{.lastState}}{{end}}'  simmemleak-60xbc
Container Name: simmemleak
LastState: map[terminated:map[exitCode:137 reason:OOM Killed startedAt:2015-07-07T20:58:43Z finishedAt:2015-07-07T20:58:43Z containerID:docker://0e4095bba1feccdfe7ef9fb6ebffe972b4b14285d5acdec6f0d3ae8a22fad8b2]]

[carvido1]

An interesting go throw article about OOMKilled WTH! Who killed my pod - Whodunit?

[carvido1]

There is a reported bug in java JDK-8230305 : Cgroups v2: Container awareness that can be fixed using JDK >= 11.0.16 because it uses cgroups v2, this was implemented in K8s 1.25 as well.