Understanding CPU Pressure and the selector

[rimbesat]

Hi DataDog Chaos Team!

I just wanted to reach out to see if you better understand how to troubleshoot the issue we are having. I currently don't have the time to fully deep dive into this issue but if there are any steps I can take let me know. We are running the controller on version 7.13.1

Our application pod fault-injection-showcase, consists of two containers, the application container fault-injection-showcase and an istio-proxy container
When running a cpu-pressure experiment against my application, with this manifest

apiVersion: chaos.datadoghq.com/v1beta1
kind: Disruption
metadata:
  name: cpu-pressure-pod
  namespace: chaos-engineering-framework
spec:
  level: pod
  duration: 5m
  selector:
    app.kubernetes.io/name: "fault-injection-showcase"
    app: "fault-injection-showcase--plugin-master"
  count: 100%
  staticTargeting: true
  cpuPressure: 
    count: 100%

I find no evidence to suggest that the fault-injection-showcase container has increased cpu-pressure, only the istio-proxy container is showing 100% cpu pressure
The injector pod also reaches a Failed state after the duration. This only happens with the cpu-pressure experiments (not with state or network experiments)

status:
  phase: Failed
  conditions:
    - type: Initialized
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2023-03-23T11:09:06Z'
    - type: Ready
      status: 'False'
      lastProbeTime: null
      lastTransitionTime: '2023-03-23T11:12:54Z'
      reason: PodFailed
    - type: ContainersReady
      status: 'False'
      lastProbeTime: null
      lastTransitionTime: '2023-03-23T11:12:54Z'
      reason: PodFailed
    - type: PodScheduled
      status: 'True'
      lastProbeTime: null
      lastTransitionTime: '2023-03-23T11:09:06Z'
  message: Pod was active on the node longer than the specified deadline
  reason: DeadlineExceeded
  hostIP: 
  startTime: '2023-03-23T11:09:06Z'
  containerStatuses:
    - name: injector
      state:
        terminated:
          exitCode: 137
          reason: Error
          startedAt: '2023-03-23T11:0

However, if I specify the container in the manifest below

apiVersion: chaos.datadoghq.com/v1beta1
kind: Disruption
metadata:
  name: cpu-pressure-pod
  namespace: chaos-engineering-framework
spec:
  level: pod
  duration: 5m
  selector:
    app.kubernetes.io/name: "fault-injection-showcase"
    app: "fault-injection-showcase--plugin-master"
  containers: 
    - "fault-injection-showcase"
  count: 100%
  staticTargeting: true
  cpuPressure: 
    count: 100%

It correctly injects into the fault-injection-showcase container 100% cpu-pressure. Though the pod still fails.

I also ran one experiment where I specify both containers in the manifest istio-proxy and fault-injection-showcase
And I find that it targets all the necessary pods, but in one pod it injects the fault-injection-showcase containers (and not istio-proxy) and in the other pod it injects the istio-proxy (and not the fault-injection-showcase)

If you have any ideas on where to go about solving this, it would be greatly appreciated!

[expFlower]

Tagging @Devatoria for visibility on this, also thinking if we should make this an issue rather than a discussion?