Enrich tracing data with juju topology information

[wood-push-melon]

Enhancement Proposal

Currently, we have several Juju models. Each model has a bundle of the same charms running (same deployment topology).

Each model is "integrated" with an opentelemetry-collector-k8s-operator for metrics, logging, and tracing. And the telemetry data will be sent to a centralized COS deployed somewhere else.

However, the juju topology information is missing in the tracing data. This information is valuable in this deployment setup, so that we can tell the tracing data coming from which juju model's application.

also cc @shipperizer

[mmkay]

Hi @wood-push-melon - the first hunch is that it should be there. Do you see that only for the charm traces or for both charm and workload traces? Can you provide us with an example so that we could try to reproduce it?

[PietroPasotti]

we really need a minimal reproducer for this one; we can't guess where the labels are getting dropped, if it's at export-time, at otelcollector-reprocessing-time, or at ingestion time. @wood-push-melon can you help us here?
Also don't forget to add charm revision information and juju versions

[wood-push-melon]

Sorry for the late response. We found this potential issue in our Terraform deployment.

The involved k8s charms include the following:

• hydra-operator
• kdratos-operator
• identity-platform-login-ui

You can use the latest/edge for these charms. The Juju version, I believe, is 3.6.13. The otel charm's revision is 2/edge.

The deployment topology is sort of like the following

flowchart LR
    %% Model A
    subgraph Model_A["Juju Model: iam"]
        kratos["kratos charm"]
        hydra["hydra charm"]
        loginui["login-ui charm"]
    end

    %% Model B
    subgraph Model_B["Juju Model: cos"]
        otel["otel-collector charm"]
    end

    %% Cross-model integrations
    kratos -- "logs / traces / metrics" --> otel
    hydra -- "logs / traces / metrics" --> otel
    loginui -- "logs / traces / metrics" --> otel

Loading

[PietroPasotti]

we'll try to deploy with that terraform config and see if we can reproduce the issue, thanks!

[PietroPasotti]

on it

[PietroPasotti]

So it looks like the way to do this is to add a processor to otelcol that injects the labels into the streams that are passing through. We can help you figure out how to do that.

Or @wood-push-melon you need to go and figure out if you can configure kratos/hydra etc. to inject those labels.
After a quick look I think kratos is using opentelemetry instrumentation.
So perhaps this is as easy as adding OTEL_RESOURCE_ATTRIBUTES="juju_unit={self.unit.name},<...other labels...>" to the pebble environment definition.

see https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/

[MichaelThamm]

As Pietro is saying, the general idea is that the application instrumentation manages the telemetry labels, e.g. opentelemetry SDK, or kratos/hydra.

This sounds like the perfect use case of a (future) OTLP lib (being worked on here) since OTLP can contain logs, metrics, traces. Therefore, we can also do OTLP processing with the lib.

We can argue that aggregators like otel-collector charm can add some extra labels like which Juju or LXD hosts these telemetries were aggregated from, but the original labels should still come from the application itself

[PietroPasotti]

@MichaelThamm maybe I'm missing something but I don't think we can get away with doing the labeling in the otel collector because otelcol can't know which of the possibly many units/apps is sending the traces it's processing.
So it's still up to the individual units to ensure that the traces it's generating contain the right labels.

[PietroPasotti]

-> canonical/cos-lib#176

[PietroPasotti]

@wood-push-melon I added a short doc on how to get started with this: canonical/cos-lib#176
And created an issue on how we can get rid of some of the boilerplate for you canonical/cos-lib#176

For now that's all we can do for you I think. If you disagree, feel free to reopen the issue or reach out.