`pod_container_status_terminated_reason_oom_killed` metric isn't being pushed to Cloudwatch console

[ashrielbrian]

Describe the bug
The OOM metric pod_container_status_terminated_reason_oom_killed from container insights enhanced observability can't be found on the Cloudwatch console even when there's a confirmed OOMKilled error for pods in our EKS cluster. It is on the list of observable metrics found here.

Steps to reproduce

• Use the default EKS add-on for cloudwatch observability with no changes to the config - version number below.
• Simulate a pod OOM error
• Observe that the metric pod_container_status_terminated_reason_oom_killed cannot be found in Cloudwatch container insights

What did you expect to see?
pod_container_status_terminated_reason_oom_killed can be viewed and graphed on Cloudwatch container insights console. Pods with OOMKilled errors can be monitored.

What did you see instead?
The metric can't be found even when there's an OOMKilled status occuring for pods.

What version did you use?

• AmazonCloudWatchAgent CWAgent/1.300054.0b1074 (go1.23.7; linux; amd64)
• EKS v1.30
• EKS cloudwatch add-on version: v3.6.0-eksbuild.2

What config did you use?
config.txt

Environment
AL2

Additional context
In the cloudwatch agent logs, there is the line:

manager.go:306] Could not configure a source for OOM detection, disabling OOM events: open /dev/kmsg: no such file or directory

[zhihonl]

If you do

securityContext:
          privileged: true

Does that fix the issue?

[NeodymiumFerBore]

Thanks @zhihonl , it fixed it for us. Is there a way to achieve this without running in privileged mode? AFAIK the addon configuration values do not allow to set securityContext for the deployed agent DaemonSet, so this is quite disabling...

For those who have the same issue, here is how we fixed the issue and tested it:

Click to expand

We added .spec.securityContext.privileged=true to the amazoncloudwatchagent Custom Resource. DaemonSet was restarted and now running in privileged mode. The /dev/kmsg error disappeared. We are now receiving the pod_container_status_terminated_reason_oom_killed metric in CloudWatch.

Test Pod to trigger OOMKill events:

apiVersion: v1
kind: Pod
metadata:
  name: test-oom
  labels:
    run: test-oom
spec:
  containers:
  - name: test-oom
    image: busybox:latest
    args:
    - /bin/sh
    - -c
    # Fill a variable with 10Mi of random characters
    - "sleep 5 && DUMMY=$(tr -dc A-Za-z0-9 </dev/urandom | head -c $(( 1024**2 * 10 )) ) && sleep infinity"
    resources:
      requests:
        cpu: 50m
        memory: 1Mi
      limits:
        cpu: 500m
        memory: 10Mi

After applying this manifest, it goes OOMKilled 5s after creation:

$ kubectl get pod -w
NAME         READY    STATUS             RESTARTS      AGE
test-oom     1/1      Running            0             8s
test-oom     0/1      OOMKilled          0             14s
test-oom     1/1      Running            1 (7s ago)    20s
test-oom     0/1      OOMKilled          1 (13s ago)   26s
test-oom     0/1      CrashLoopBackOff   1 (15s ago)   41s
test-oom     1/1      Running            2 (17s ago)   43s
test-oom     0/1      OOMKilled          2 (23s ago)   49s

Then it goes CrashLoopBackOff -> Running -> OOMKilled indefinitely. The pod_container_status_terminated_reason_oom_killed should appear in CloudWatch.