`KubePodNotReady` rule is not querying whether the pod is actually ready

[FrederikNJS]

What happened?
I set up the kube-prometheus-stack, and has a pod that is not ready, and hasn't been for a long time, but the KubePodNotReady rule is not firing.

Did you expect to see some different?
Yes, when the pod is not ready I expect the KubePodNotReady rule to fire.

How to reproduce it (as minimally and precisely as possible):
Add a pod with a failing readinessProbe, and notice that KubePodNotReady is not firing.

You can deploy the basic example deployment from the kubernetes documentation with a misconfigured readinessProbe. (I have added an example in the Manifest section.)

Then notice that KubePodNotReady isn't getting triggered.

Environment

• Prometheus Operator version:

  Not relevant

• Kubernetes version information:

  Not relevant

• Kubernetes cluster kind:

  Not relevant

• Manifests:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080 # Port is intentionally wrong

• Prometheus Operator Logs:
  Not relevant

• Prometheus Logs:
  Not relevant

Anything else we need to know?:
I noticed that the expression for KubePodNotReady is watching for the kube_pod_status_phase{phase=~"Pending|Unknown|Failed"}, so this would start firing in case the pod is not in a Running state, but it doesn't actually test whether the pod is Ready as the name of the rule would imply.

It seems it should instead be looking at kube_pod_status_ready{condition="false"}, which actually detects whether the pod is Ready.

[duc00]

Thanks @FrederikNJS, I am wondering the same thing, and regret this has not been answered yet.