Skip to content

feat: remove some resource labels from prometheus metrics #330

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat: remove some resource labels from prometheus metrics #330

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion charts/agent/Chart.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ apiVersion: v2
name: agent
description: Chart to install K8s collection stack based on Observe Agent
type: application
version: 0.46.1
version: 0.47.0
appVersion: "1.1.0"
dependencies:
- name: opentelemetry-collector
Expand Down
2 changes: 1 addition & 1 deletion charts/agent/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# agent

![Version: 0.46.1](https://img.shields.io/badge/Version-0.46.1-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 1.1.0](https://img.shields.io/badge/AppVersion-1.1.0-informational?style=flat-square)
![Version: 0.47.0](https://img.shields.io/badge/Version-0.47.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 1.1.0](https://img.shields.io/badge/AppVersion-1.1.0-informational?style=flat-square)

Chart to install K8s collection stack based on Observe Agent

Expand Down
38 changes: 23 additions & 15 deletions charts/agent/templates/_cluster-metrics-config.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,20 +117,6 @@ receivers:
target_label: __address__
################################################################

# Maps all Kubernetes pod labels to Prometheus labels with the prefix removed (e.g., __meta_kubernetes_pod_label_app becomes app).
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)

# adds new label
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace

# adds new label
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: kubernetes_pod_name

metric_relabel_configs:
- action: drop
regex: {{.Values.application.prometheusScrape.metricDropRegex}}
Expand All @@ -150,6 +136,28 @@ processors:
{{- include "config.processors.attributes.k8sattributes" . | nindent 2 }}

{{- include "config.processors.resource.observe_common" . | nindent 2 }}
resource/drop_additional_pod_metrics_labels:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we reasonably confident that this won't break any downstream datasets customers could have created?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's no way to know really. These labels existed and after this they won't so if they are relying/filtering on them then that would break things. We could also draft this change as a major version change (and ideally roll it together with the removal of TRACE_TOKEN)

attributes:
- key: http.scheme
action: delete
- key: net.host.name
action: delete
- key: net.host.port
action: delete
- key: server.address
action: delete
- key: server.port
action: delete
- key: service.instance.id
action: delete
- key: url.scheme
action: delete
- key: instance
action: delete
- key: k8s.pod.uid
action: delete
- key: job
action: delete

# attributes to append to objects
attributes/debug_source_cluster_metrics:
Expand Down Expand Up @@ -181,7 +189,7 @@ service:
{{- if .Values.application.prometheusScrape.enabled }}
metrics/pod_metrics:
receivers: [prometheus/pod_metrics]
processors: [memory_limiter, k8sattributes, batch, resource/observe_common, attributes/debug_source_pod_metrics]
processors: [memory_limiter, k8sattributes, resource/drop_additional_pod_metrics_labels, batch, resource/observe_common, attributes/debug_source_pod_metrics]
exporters: [{{ join ", " $podMetricsExporters }}]
{{ end -}}
{{- include "config.service.telemetry" . | nindent 2 }}
Expand Down
23 changes: 19 additions & 4 deletions charts/agent/templates/_config-processors.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,17 +32,32 @@ k8sattributes:
- k8s.pod.uid
- k8s.cluster.uid
- k8s.node.name
- k8s.node.uid
- k8s.container.name
- container.id
labels:
- tag_name: app.kubernetes.io/name
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WOW AWESOME, thanks for doing this!

key: app.kubernetes.io/name
from: pod
- tag_name: app.kubernetes.io/instance
key: app.kubernetes.io/instance
from: pod
- tag_name: app.kubernetes.io/version
key: app.kubernetes.io/version
from: pod
- tag_name: app.kubernetes.io/component
key: app.kubernetes.io/component
from: pod
- tag_name: app.kubernetes.io/part-of
key: app.kubernetes.io/part-of
from: pod
- tag_name: app.kubernetes.io/managed-by
key: app.kubernetes.io/managed-by
from: pod
passthrough: false
pod_association:
- sources:
- from: resource_attribute
name: k8s.pod.ip
# - sources:
# - from: resource_attribute
# name: k8s.container.restart_count
- sources:
- from: resource_attribute
name: k8s.pod.uid
Expand Down