Introduce OTel Collector mixin #1465
Conversation
Signed-off-by: Paschalis T <[email protected]>
Signed-off-by: Paschalis Tsilias <[email protected]>
Signed-off-by: Paschalis Tsilias <[email protected]>
Signed-off-by: Paschalis Tsilias <[email protected]>
Signed-off-by: Paschalis Tsilias <[email protected]>
Signed-off-by: Paschalis Tsilias <[email protected]>
Signed-off-by: Paschalis Tsilias <[email protected]>
Signed-off-by: Paschalis Tsilias <[email protected]>
| grafanaDashboards+:: { | ||
| 'collector.json': | ||
| g.dashboard.new( | ||
| 'OpenTelemetry Collector Health', |
There was a problem hiding this comment.
We normally use Sentence case for Dashboard titles and panel titles
| ) | ||
| + prometheusQuery.withIntervalFactor(2) | ||
| + prometheusQuery.withLegendFormat(||| | ||
| RSS - {{cluster}} - {{namespace}} - {{instance}} |
There was a problem hiding this comment.
what if otel collector is deployed outsude k8s and there is no namespace label?
There was a problem hiding this comment.
take a look at the concept of three variables we use across all mixins:
-
filteringSelector. Static label filter that must added to all queries in panels and alerts. This allows to deploy mixin more than once (with different filteringSlectors) or filter our unncessary metrics coming from other sources (otel-collectors you don't want to see on that dashboard). Could be empty. Example: job="integrations/otel-collector". -
groupLabels. a list of labels that represent a group of instances of SUO. Also labels you would like to keep in alerts aggregations. example:groupLabels: ['job']orgroupLabels: [environment, 'job',cluster`]. -
instanceLabels. a list of labels that represent a single instance of SUO.
example:instanceLabels: ['instance']orinstanceLabels: [instance`, 'pod'].
Moving those into config and then reusing those variables instead of static labels in queries, dashboard variables , panel legends gives maximum flexibility no matter where mixin is deployed.
| { | ||
| datasourceVariable: | ||
| variable.datasource.new('datasource', 'prometheus') | ||
| + variable.datasource.generalOptions.withLabel('Data source') | ||
| + variable.datasource.generalOptions.withCurrent(cfg._config.datasourceName) | ||
| + variable.datasource.generalOptions.showOnDashboard.withLabelAndValue(), | ||
|
|
||
| clusterVariable: | ||
| variable.query.new('cluster') | ||
| + variable.query.generalOptions.withLabel('Cluster') | ||
| + variable.query.withDatasourceFromVariable(self.datasourceVariable) | ||
| + variable.query.refresh.onTime() | ||
| + variable.query.withSort(type='alphabetical', asc=false) | ||
| + variable.query.selectionOptions.withIncludeAll(true, '.*') | ||
| + variable.query.selectionOptions.withMulti(true) | ||
| + variable.query.queryTypes.withLabelValues('cluster', metric='{__name__=~"otelcol_process_uptime.*"}'), | ||
|
|
||
| namespaceVariable: | ||
| variable.query.new('namespace') | ||
| + variable.query.generalOptions.withLabel('Namespace') | ||
| + variable.query.withDatasourceFromVariable(self.datasourceVariable) | ||
| + variable.query.refresh.onTime() | ||
| + variable.query.withSort(type='alphabetical', asc=false) | ||
| + variable.query.selectionOptions.withIncludeAll(true, '.*') | ||
| + variable.query.selectionOptions.withMulti(true) | ||
| + variable.query.queryTypes.withLabelValues('namespace', metric='{__name__=~"otelcol_process_uptime.*"}'), |
There was a problem hiding this comment.
nit: take a look at commonlib variables func from here: https://github.com/grafana/jsonnet-libs/tree/master/common-lib
you might find it handy to generated funnel-like chained variables:
https://github.com/grafana/jsonnet-libs/blob/master/windows-observ-lib/main.libsonnet#L24-L31
| cpuUsage: | ||
| prometheusQuery.new( | ||
| '$' + variables.datasourceVariable.name, | ||
| ||| | ||
| sum by (job, cluster, namespace, instance) ( | ||
| rate( | ||
| { | ||
| __name__=~"otelcol_process_cpu_seconds(_total)?", | ||
| job=~"$job", | ||
| cluster=~"$cluster", | ||
| namespace=~"$namespace", | ||
| instance=~"$instance" | ||
| } | ||
| [$__rate_interval]) | ||
| ) | ||
| ||| | ||
| ) | ||
| + prometheusQuery.withIntervalFactor(2) | ||
| + prometheusQuery.withLegendFormat(||| | ||
| {{cluster}} - {{namespace}} - {{instance}} | ||
| |||), |
There was a problem hiding this comment.
nit: You may also use 'signals' declarative format , where functions like rate and histogram_quantile are applied automatically:
https://github.com/grafana/jsonnet-libs/blob/master/common-lib/common/signal/README.md
lots of examples:
https://github.com/grafana/jsonnet-libs/tree/master?tab=readme-ov-file#observability-libraries-signal-extention
This PR introduces a new mixin for the OpenTelemetry collector that uses its internal Prometheus metrics.
It was constructed with grafonnet and tried to follow best practices. Let me know what you think!