feat: send/receive relations for forwarding OTLP metrics

[MichaelThamm]

Checklist

• I have added or updated relevant documentation.
• PR title makes an appropriate release note and follows conventional commits syntax.
• Merge target is the correct branch, and relevant tandem backport PRs opened.
• Merge this PR
  • First release this feature to dev/beta and let IS test it
• Ping simme
• Finish implementing OTLP #186

Issue

Being able to send metrics cross-model is currently not possible and a feature was attempted in:

• feat: receive-remote-write #177

However, we dropped this since OTLP can achieve the same thing and is collector-native.

Tandem ingress loadbalancing issue:

• Fix loadbalancing when using Traefik's ingress #185

Solution

Tandem PR:

• feat: send-otlp relation for forwarding OTLP metrics opentelemetry-collector-operator#175

Since the workloads are handling the serving and receiving of the OTLP data, the databag only needs to communicate a few things:

• Endpoint (host and port)
• Protocol (gRPC or HTTP)
• Telemetries (logs, metrics, traces)

It is the charm’s responsibility to handle the endpoint information accordingly. For example, writing the OTLP endpoint to config file. A concrete example is the opentelemetry-collector-k8s charm needs to enable receivers, exporters, and pipelines in its config based on OTLP requirements.

Note

Not all OTLP servers can handle all telemetries and all protocols, e.g. Loki only supports logs via HTTP.

We can encapsulate all information with 2 endpoints:

• receive-otlp
• send-otlp

Otelcol-k8s charm will use the k8s service address when providing endpoints to load balance across all pipelines.

Important

The VM charm cannot do this so it must provide each unit address since it has no "service" address like in k8s, unless ingressed. This is a future TODO for when we implement the receive-otlp endpoint for the VM charm.

Databags

The provider offers in app data:

{
    "grpc": {
        "endpoint": "http://provider-0.endpoint:4317",
        "telemetries": ["logs", "metrics"],
    },
    "http": {"endpoint": "http://provider-0.endpoint:4318", "telemetries": ["logs"]},
}

and the consumer does not offer anything:

{}

The consumer aggregates all the OtlpEndpoints from all relations in this format:

{
    123: {
        "grpc": {
            "endpoint": "http://provider-0.endpoint:4317",
            "telemetries": ["logs", "metrics"],
        },
        "http": {"endpoint": "http://provider-0.endpoint:4318", "telemetries": ["logs"]},
    },
    456: {...},
}

Pydantic models

The models validate that only supported protocols and telemetries are defined in the databag.

PR Scope

This feature PR for OTLP is incomplete because it enables forwarding only metrics telemetry data between 2 otel-collector charms. Other telemetries can easily be added by updating the provider usage:

OtlpProvider(supported_telemetries=[TelemetryType.metric])

The otlp.py file is local to otelcol and is not a published lib. The plan is to eventually move this to charmlibs. This means that, for now, only two otel-collector charms can communicate OTLP.

Ingress

I added a test that we can access the OTLP HTTP receiver through ingress. Since the Traefik charm can not handle gRPC, otel-collector will currently only support OTLP over HTTP.

TLS

We have an existing config builder feature which adds TLS to all receivers. OTLP exporter offers a tls:insecure config. We toggle this config based on the state of the scheme (http vs. https) of the provider/server address.

We could support mTLS in the future:

• Add mTLS capability for Otelcol receiver/exporter communication #184
• feat: mTLS for OTLP #183

Context

• OTLP cyclic relations probe juju-doctor#73

Testing Instructions

I added integration tests for:

1. test_ingress.py::test_push_otlp_logs_through_ingress
2. test_otlp.py::test_otlp_forwarding_insecure
3. test_otlp.py::test_otlp_forwarding_secure

I added unit tests for:

1. test_config_manager.py::test_add_otlp_forwarding
2. test_ingress.py::test_otlp_url_in_databag (ingress)
3. test_otlp.py::test_provider_app_data_raises_validation_error
4. test_otlp.py::test_send_otlp_multiple_relations_and_units
5. test_otlp.py::test_send_otlp_invalid
6. test_otlp.py::test_receive_otlp
7. test_otlp.py::test_cyclic_relations

Manual

$ charmcraft pack
$ jd ./opentelemetry-collector-k8s_ubuntu@24.04-amd64.charm --resource opentelemetry-collector-image=ghcr.io/canonical/opentelemetry-collector:dev --trust otelcol-send
$ jd ./opentelemetry-collector-k8s_ubuntu@24.04-amd64.charm --resource opentelemetry-collector-image=ghcr.io/canonical/opentelemetry-collector:dev --trust otelcol-receive
$ jrel otelcol-send:send-otlp otelcol-receive:receive-otlp
$ jc otelcol-receive debug_exporter_for_metrics=true

# Check for the ingested metrics in the `otelcol-receive` metrics pipeline
$ jssh --container otelcol otelcol-receive/0 "pebble logs -f" | grep "juju_application=otelcol-send"
2026-01-30T21:37:42.566Z [otelcol] otelcol_process_runtime_heap_alloc_bytes{juju_application=otelcol-send,juju_charm=opentelemetry-collector-k8s,juju_model=demo,juju_model_uuid=d9fd7e13-ab5a-453e-873b-7a83143a7741,juju_unit=otelcol-send/0,service.instance.id=dcb7dbef-8abf-4e2f-b50e-68ead704fa9d,service.name=otelcol,service.version=0.144.0} 6.296248e+06

Testing for OTLP forwarding between VM to K8s otelcol is documented in:

• feat: send-otlp relation for forwarding OTLP metrics opentelemetry-collector-operator#175

Upgrade Notes

[MichaelThamm]

@copilot leave the functionality intact, but improve the code

[Copilot]

@MichaelThamm I've opened a new pull request, #189, to work on those changes. Once the pull request is ready, I'll request review from you.