Skip to main content
Version: latest

trace_oomkill

The trace_oomkill gadget is used to trace OOM kill events.

Getting started​

Running the gadget:

$ kubectl gadget run ghcr.io/inspektor-gadget/gadget/trace_oomkill:latest [flags]

Guide​

Start by creating a namespace and a deployment:

$ kubectl create namespace oomkill-demo
namespace/oomkill-demo created
$ kubectl create deployment oomkill-demo --image=busybox --namespace oomkill-demo -- sleep inf
deployment.apps/oomkill-demo created

Set the memory limit of the pod to a low value:

$ kubectl set resources deployment oomkill-demo --namespace oomkill-demo --limits=memory=128Mi
deployment.apps/oomkill-demo resource requirements updated
$ kubectl wait --for=condition=Ready pod -l app=oomkill-demo --namespace oomkill-demo
pod/oomkill-demo-<...> condition met

Run the gadget in a terminal:

$ kubectl gadget run trace_oomkill:latest --namespace oomkill-demo
K8S.NODE           K8S.NAMESPACE             K8S.PODNAME               K8S.CONTAINERNAME         MNTNS_ID      FPID          FUID          FGID          TPID          PAGES        FCOMM        TCOMM

The gadget is waiting for the OOM killer to get triggered and kill a process in oomkill-demo namespace (alternatively, we could use -A and get out-of-memory killer events in all namespaces). To trigger the OOM killer, in another terminal, exec a container and run this command to exhaust the memory:

$ kubectl exec -n oomkill-demo -ti deployments/oomkill-demo -- tail /dev/zero
command terminated with exit code 137

Go back to the first terminal and see:

K8S.NODE           K8S.NAMESPACE             K8S.PODNAME               K8S.CONTAINERNAME         MNTNS_ID      FPID          FUID          FGID          TPID          PAGES        FCOMM        TCOMM
minikube-docker    oomkill-demo              oomkill-demo…dbf85d-r9tls busybox                   4026533320    728870        0             0             728870        4227071      tail         tail

The printed lined corresponds to the killing of the tail process by the OOM killer. Note that, in this case, the command which was killed by the OOM killer is the same which triggered it, this is not always the case.

Congratulations! You reached the end of this guide! You can now delete the resource we created:

$ kubectl delete namespace oomkill-demo