Support reporting on multiple instances of a uniquely named pod (such as via a statefulset)

[mwestphall]

Currently, Kuantifier's prometheus queries assume each pilot will run in a distinctly named pod (such as those scheduled via a Deployment or CronJob). UNL is currently contributing capacity via JupyterHub, which repeatedly schedules pods of the same name in serial (such that two distinct instances of a pod named eg. jupyter-mwestphall-0 might be scheduled on two different days). Kuantifier's queries currently don't handle this well, with some taking the value from a single instance of the pod name and some aggregating data from across all instances of the pod name. Queries that come from kube-state-metrics also support a uid field that can be used to differentiate between distinct instances of a shared pod name, but queries directly to the kubernetes control plane typically don't. We'll likely need a pretty substantial set of code changes to support reporting on this use case.

[rptaylor]

@mwestphall Interesting, that could require some surgical changes indeed.

Unless, perhaps, it just comes down to this one line in rearrange, which shuffles the data into a list of tuples of metrics indexed by pod name:
https://github.com/rptaylor/kuantifier/blob/master/python/KAPEL.py#L187

However, there are some comments and also #11 which note that the behaviour of the rearrange function - deduplicating results returned by the prom queries - is important and required to avoid double counting.

Conceivably, if we were to have it index by uuid instead of pod name, the rearrange function might still do the required deduplication to avoid double counting, while still reporting different unique instances of statefulset pods.
Would need careful testing to confirm and validate in any case.

Also for the individual pod accounting I think you need the individual pod names?
https://github.com/rptaylor/kuantifier/blob/master/python/KAPEL.py#L187

[mwestphall]

@rptaylor it looks like the UID field can be dug out of the response values for non-kube-state-metrics metrics via the container cgroup listed in the id field. It'll be a bit of work in-code to line these values up but it should be possible. Eg.
max_over_time(container_cpu_usage_seconds_total{namespace="osgdev", pod="access-amie-6cdb74dcb7-66nn8"}[1d])

{
	"metric": {
		...
		// uid embedded in /kubepod.slice/kubepods-pod<uid>/...
		"id": "/kubepods.slice/kubepods-pod59bf08c5_d01a_4838_b435_6ccf502523b0.slice/cri-containerd-01e1c2c9a70106a4aff19fae59062d34206a7acfdfbd5b4549162cef16fbe0fe.scope",
		"namespace": "osgdev",
		"pod": "access-amie-6cdb74dcb7-66nn8",
	},
	"value": [
		1754943241.143,
		"0.023286"
	]
}

Compared to:
max_over_time(kube_pod_start_time{namespace="osgdev", pod="access-amie-6cdb74dcb7-66nn8"}[1d])

{
	"metric": {
		...	
		"namespace": "osgdev",
		"pod": "access-amie-6cdb74dcb7-66nn8",
        // uid directly		
        "uid": "59bf08c5-d01a-4838-b435-6ccf502523b0"
	},
	"value": [
		1754943405.131,
		"1753373297"
	]
}

[mwestphall]

@rptaylor I've implemented a test version of this change and run it successfully against duplicate-named Jupyter pods scheduled in CHTC's kubernetes cluster. https://github.com/rptaylor/kuantifier/compare/master...mwestphall:feature/multi-container-quick-fix?expand=1

However, we have some concerns about the portability/robustness of differentiating between pod instances using the id field. It does not appear to be documented as "guaranteed to appear" anywhere in the kubernetes metrics docs so we are unsure that it can be used reliably across clusters/container runtimes. Do you have any thoughts on this?

[rptaylor]

Yeah, I'd also be a bit reluctant to rely on internal cgroup/cadvisor details in the string "/kubepods.slice/kubepods-pod59bf08c5_d01a_4838_b435_6ccf502523b0.slice...." for a container. On a test cluster (now using Alma 9, cgroups v2, k8s v1.28, and kube-prometheus-stack instead of bitnami's kube-prometheus), if I do max_over_time(container_cpu_usage_seconds_total{namespace="harvester",pod="grid-job-25915328-4sp67"}[2h])
I get:

{cpu="total", endpoint="https-metrics", id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e7843c_9724_43d8_a6e5_1bf25f1956f9.slice/cri-containerd-a849c730b1a00d54fd447902eaf6f928a74bc34b5dafce5f0cd26fa95edde703.scope", image="registry.k8s.io/pause:3.9", instance="10.5.7.85:10250", job="kubelet", metrics_path="/metrics/cadvisor", name="a849c730b1a00d54fd447902eaf6f928a74bc34b5dafce5f0cd26fa95edde703", namespace="harvester", node="kermes-dev-k8s-node-a02", pod="grid-job-25915328-4sp67", service="kube-prometheus-stack-kubelet"}	0.043768
{container="job-container", cpu="total", endpoint="https-metrics", id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e7843c_9724_43d8_a6e5_1bf25f1956f9.slice/cri-containerd-5bc549410f18e23ad952fe563c71927ee0fee9c60e77563c92dfd3dd90808dc1.scope", image="git.computecanada.ca:4567/rptaylor/misc/atlas-grid-almalinux9-localuser:20240830", instance="10.5.7.85:10250", job="kubelet", metrics_path="/metrics/cadvisor", name="5bc549410f18e23ad952fe563c71927ee0fee9c60e77563c92dfd3dd90808dc1", namespace="harvester", node="kermes-dev-k8s-node-a02", pod="grid-job-25915328-4sp67", service="kube-prometheus-stack-kubelet"}	421.132193
{cpu="total", endpoint="https-metrics", id="/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e7843c_9724_43d8_a6e5_1bf25f1956f9.slice", instance="10.5.7.85:10250", job="kubelet", metrics_path="/metrics/cadvisor", namespace="harvester", node="kermes-dev-k8s-node-a02", pod="grid-job-25915328-4sp67", service="kube-prometheus-stack-kubelet"}     421.14

This is bringing back memories of the double-counting discussion from #54
It looks to me like this shows:

• the pause container (0.04) with name="a849c730b1a00d54fd447902eaf6f928a74bc34b5dafce5f0cd26fa95edde703"
• the workload container (421.13) with name="5bc549410f18e23ad952fe563c71927ee0fee9c60e77563c92dfd3dd90808dc1", which is the containerID reported by k8s
• the pod total for all containers (421.14), with no name field

and the current code would double count, by including all of them. Do you get a different result on your cluster?
When I do sum by (pod) (last_over_time(container_cpu_usage_seconds_total{namespace="harvester"}[2h])) I get

{pod="grid-job-25915328-4sp67"}	842.32505

which appears to confirm double-counting of this pod. Moreover each of those 3 metrics has a different id field, so on my cluster, the code in your branch results in 3 metrics being published for each pod.

Containerd confirms those ID strings a849... and 5bc5...

$ sudo /usr/local/bin/crictl ps -a
CONTAINER           IMAGE               CREATED             STATE               NAME                                 ATTEMPT             POD ID              POD

5bc549410f18e       c955b6beb2a26       2 hours ago         Exited              job-container                        0                   a849c730b1a00       grid-job-25915328-4sp67

Originally IIRC we thought the difference between what we were seeing was cgroups v1 vs v2 but I don't think that could be the explanation after we rebuilt this cluster to Alma 9.

In any case, at least based on what I'm seeing, only the individual container metrics include a unique ID, that of the container.
So it doesn't seem like there is a good way to both combine the metrics for all containers in a pod (each with different container IDs), and have one pod ID that is shared by all containers in a pod - it would force us to parse the "id" string.
It would be really nice if we could find a better way ...

[rptaylor]

It looks like relabeling might work?
https://grafana.com/blog/2022/03/21/how-relabeling-in-prometheus-works/

This shows that __meta_kubernetes_pod_uid is available as a meta label, so it seems like it should be possible to set up some prometheus config to add that label to metrics:
https://prometheus.io/docs/prometheus/latest/configuration/configuration/#pod

[mwestphall]

@rptaylor the current code tries to just grab the "total" field by filtering on the regex [0-9a-f_]*.slice$, so it skips the two individual container level cgroups (which end in .scope) and just gets the pod level value (which ends in .slice). That being said, setting up a relabel config to explicitly include pod uid seems like a much more robust solution. I'll see if I can get that set up locally.

[mwestphall]

After taking a look I'm not 100% sure we'll be able to use the pod uid relabel config here. Per the docs, that meta label is available for role: pod scrape targets. However, the metric we're after (container_cpu_usage_seconds_total) comes from a role: endpoint scrape target pointed at the node-level cadvisor metrics. I got a sample of the raw data available from this metrics endpoint in CHTC's cluster and it doesn't look like UID is available anywhere apart from via parsing the cgroup:

$ curl localhost:8080/api/v1/nodes/tiger2003.chtc.wisc.edu/proxy/metrics/cadvisor | grep cpu_usage
# HELP container_cpu_usage_seconds_total Cumulative cpu time consumed in seconds.
# TYPE container_cpu_usage_seconds_total counter
container_cpu_usage_seconds_total{container="",cpu="total",id="/",image="",name="",namespace="",pod=""} 2.4173816833941e+07 1755199224809
container_cpu_usage_seconds_total{container="",cpu="total",id="/kubepods.slice",image="",name="",namespace="",pod=""} 1.2968997852441e+07 1755199224976
container_cpu_usage_seconds_total{container="",cpu="total",id="/kubepods.slice/kubepods-besteffort.slice",image="",name="",namespace="",pod=""} 299229.494839 1755199221309
container_cpu_usage_seconds_total{container="",cpu="total",id="/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b6e2058_f111_47ef_8f92_dae733d7b005.slice",image="",name="",namespace="chtc",pod="ldap-replica2-57565d4b77-c7rkd"} 1096.234238 1755199215736
...

[mwestphall]

@rptaylor given the expected difficulties in correlating container_cpu_usage_seconds_total to an individual pod uid, do you think it might make sense to switch the CPU time calculation back to the simpler wall time x cpu count calculation? We've confirmed that both those can successfully be correlated to a single pod uid.

[rptaylor]

@rptaylor the current code tries to just grab the "total" field by filtering on the regex [0-9a-f_]*.slice$

Ah okay, I was just referring to the PromQL part, not the regex.

A regex on the cgroup path could potentially bring into play implementation differences at the container engine layer, e.g. containerd might handle that differently than CRI-O or others (admittedly this is just speculative).

Anyway, if there are not any really good options, restate and question assumptions. Yes, the kube-state-metrics based approach used for cputime is pretty simple and robust if you're willing to go that way.
Or here's an idea, if you want the container usage metrics but can't aggregate by pod, what if you just don't aggregate by pod? One could argue that the unit of accountable work is a container.
At least based on a basic test, it looks like you should be able to do max_over_time(container_cpu_usage_seconds_total{namespace="harvester",container != ""}) to filter out the pause container and the pod-level aggregate (to avoid double counting), leaving only real unique workload containers.
At the end of the day you'll get the right total answer, just added up differently, accounting individual containers rather than individual pods.

This works for multi-container pods , and also in the simple case of pods with only one workload container it shouldn't make a difference, except perhaps in the names that are shown. If you need to map specific individual accounting records to specific jobs or pods maybe that could be considered more of a monitoring question than an accounting question? Depends on your requirements.

[mwestphall]

Switching to container-level rather than pod-level queries is an idea that's definitely worth following. At a glance, it looks like we're able to use the following cadvisor metrics as rough replacements for the existing kube-state-metrics (which appear to be aggregated at pod level).

kube_pod_start_time -> container_start_time_seconds
kube_pod_completion_time -> container_last_seen

There doesn't appear to be a good immediate drop in for the kube_pod_container_resource_requests kube-state-metrics though. It's looking like trying to align the cadvisor metrics with kube-state-metrics is going to present some degree of difficulty no matter what.

[rptaylor]

Hmm that raises a good question around how much we should diverge in the overall approach for different requirements and metrics. The different scenarios that come to mind are:

• GRACC style (individual records, CPU usage %) vs APEL style (summarized records, CPU walltime)
• single-container pods vs multi-container pods (Generalize queries to work if there is more than just 1 container in a pod #56)
• pod re-use in case of a statefulset
• include initContainers (simplified pod resource metrics (support for initContainers) #71)

Moreover each of those could intersect in different ways so it's a bit all over the place. I wonder if we can slice it in a way that solves multiple problems at once (would need some further pondering) ...

Possible variation that could avoid changing our premises about the accountable unit of work too drastically: what if we do max_over_time(container_cpu_usage_seconds_total{namespace="harvester",container != ""}) ; we should get a list of all real workload containers with at least the following labels:

container="job-container"
cpu="total"
endpoint="https-metrics"
id="/kubepods.slice/...."
image=
instance="10.5.7.85:10250"
job="kubelet"
metrics_path="/metrics/cadvisor"
name= #  UUID
namespace="harvester"
node="kermes-dev-k8s-node-a02"
pod="grid-job-25915328-4sp67"
service="kube-prometheus-stack-kubelet"

From this it should be trivially easy to aggregate together all real workload containers by pod name in python rather than in PromQL.

[rptaylor]

I guess the main question is: what are the other implications or requirements for supporting statefulsets, beyond this specific question about aggregating container_cpu_usage_seconds_total ?

I think that aggregating container-level CPU usage like this is okay, and does not have the same complications as aggregating container-level CPU requests does (#71)

However, in the case of a statefulset, the cores and memory request could change at any time, for the same re-used pod name.
Calculating duration seems like a big problem for a statefulset: you could have numerous, potentially overlapping pod start and end times and you would need to do some tricky logic and math to find the real duration, and if you get one time interval wrong for any reason (clock skew, frequent pod restarts in a short period, missing or duplicate records,...) the math would be off. Metrics for container-level start and end times, if they even exist, would also not be correct in the case of multi-container pods (especially init containers).

[rptaylor]

Statefulsets could have the same problem as deployments (kubernetes/kube-state-metrics#2118 (comment)) when deleted or scaled down.

And they could end too often (as considered above), or not often enough, aka never (at least not within the time span relevant for accounting and prometheus retention, i.e. several months).

The more I think about it, the more complicated it seems, not sure if there is a reasonable way to support statefulsets universally, but for just the cpuusage query specifically I can see how it could work.

[mwestphall]

@rptaylor the specific use-case we have for monitoring duplicate-named pods is JupyterHub. I used statefulsets as an initial example of duplicate pod names but we've looked into how JupyterHub works a bit since then and it's fairly different. Its associated operator schedules "raw" pods (with no associated deployment, statefulset, etc) based on the username of a user whenever that user logs in (for example, something like jupyter-westphall-0xabc). The username-numeric-sequence is identical across logins for a given user, which is the source of our problem. We've confirmed that these pods are "job-like" in that they'll run for a certain time and then enter the Completed state for long enough for kube-state-metrics to get the completion time metric before terminating. I don't quite know the average lifespan of a given instance but given that they correspond to interactive user logins I assume it's on the scale of hours rather than days. I guess it could be another question as to whether kuantifier should be trying to "generically" support such a specific application that doesn't follow many kubernetes norms.