`ca_file` not properly configured

[Abuelodelanada]

Bug Description

When otelcol is related to a juju-controller charm over the metrics-endpoint relation, TLS is not well configured.

Although juju-controller set ca_file in the MetricsEndpoint definition, and the cert content is shared over relation data:

$ juju show-unit otelcol/0 | grep "ca_file"
        {"ca_file": "-----BEGIN CERTIFICATE-----\nMIIEEjCCAnqgAwIBAgIUbxLCrdoUL88/X/tqZ6gfLbCrBdAwDQYJKoZIhvcNAQEL\nBQAwITENMAsGA1UEChMESnVqdTEQMA4GA1UEAxMHanVqdS1jYTAeFw0yNTA4MDYx\nODA4MTFaFw0zNTA4MDYxODEzMTFaMCExDTALBgNVBAoTBEp1anUxEDAOBgNVBAMT\nB2p1anUtY2EwggGiMA0GCSqGSIb3DQEBAQUAA4IBjwAwggGKAoIBgQDAsb2grN4K\nAgtpnlUxIjTz4KPB2wkTg6cSH5epE/Z4AYIiajrXjQlYmZ5TVt0C/pW25ovjfXuz\naDjvz6R/6cQSp3Ab6/SW0PGXJZGwT+meFdNj6GZr4WKdGrMfx3ykqM5nW/NClyrN\n5jd4RBE9PtHuhVHo6DKkJdhRgHO7q3X89At6UdyFKEpf7tPg1w7U9DJ4kKvyVRGI\nV8uyGzkUQRPQUKs1e4/bhC9ySjv02BmisA3yXO7FtWogjArldp/5j5iZi5lbyxXz\nMXfe5aBkqlVDvcWFEo7QLMAJBF7O2G8W2PmiSKUjCS3w8Wwtxz9KKvn0TkD17Q2+\nvK9+N9HddW52ND2rHG7wY716dXYFCCBHu+VzpznSs5QFwoGuFMks+EpWQS8gmKXD\nyPMflBcdQo8DEfK+N8p2Z22nTJnY3PTLGPtKBSCJycGXTUEazPK1W/wtehJaWXOE\nt3AqS4IUZBKMmvyzyJFcUB9lKBcv2uxN6/jXEdGz984wvjpwWPwce+MCAwEAAaNC\nMEAwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFBdU\nD5hPA/v80tve1603rDg0FdK7MA0GCSqGSIb3DQEBCwUAA4IBgQCDbRrqtsiscM5K\nNnD7eLvqZZyY8mDfordU17aOSYLx5lTzopPksmhFuFr8rlH5GkVFDVOzJOJ3wMwp\nNmOQMT1o3SIOX/gtKzPDjBnjHXb97ciruBm5LioMiUHza41KsNNJTs3ZwtoZuZJo\nWjDQnfPmeUMhLbplWKZAAai62/pQioe0wfNjvy7tpBBBQ2HPsTdouck86YyhiBoa\nh6kqeIiwzO/Z9pPX0CDdvH8xsTk0fKBoYMJIFLGoCzTrm1dOgbQl7Q4kag0x08La\n6o2tOvezclOEN3bW0rLnV2Syvh9iU6FDyVfh0BYs3rLxF+bBRnrodJWWb24CiMOu\nJULpDzNbwdinGL6srUU7kvMcY2uqT77mqzHfh2aLkMNx6TkaDKlJOWHKBkBbeyay\nz1Am3zCiec2DnzQ3hCRDVr9zHlkFzDFKCiZVUiimSPzGvKI768T95Bfnq2nkfpyC\n7A+BVLA9dpFt2N1wBupp8+BpsWmPZwfZ7U1l/cb8JL3Tn8yX6bY=\n-----END

Also the ca_file option should be a path to a file that contains the CA, not a string.

  prometheus/metrics-endpoint/otelcol/0:
    config:
      scrape_configs:
      - basic_auth:
          password: -bj7a7nTPNvhF0OukVWbHA
          username: user-juju-metrics-r5
        job_name: juju_controller_ab847f0e_controller_prometheus_scrape-0
        metrics_path: /introspection/metrics
        relabel_configs:
        - regex: (.*)
          separator: _
          source_labels:
          - juju_model
          - juju_model_uuid
          - juju_application
          - juju_unit
          target_label: instance
        scheme: https
        scrape_interval: 1m
        scrape_timeout: 10s
        static_configs:
        - labels:
            juju_application: controller
            juju_charm: juju-controller
            juju_model: controller
            juju_model_uuid: ab847f0e-aa46-4bb7-8295-416733e734af
            juju_unit: controller/0
          targets:
          - 10.1.167.112:17070
        tls_config:
          insecure_skip_verify: false

To Reproduce

Deploy COS-Lite in a cos model with the offers overlay

1. juju deploy cos-lite -m cos --channel=edge --trust --overlay ./offers-overlay.yaml

In the k8s controller model:

1. juju deploy opentelemetry-collector-k8s otelcol --channel=2/edge --trust
2. juju relate otelcol controller
3. juju consume microk8s:admin/cos.prometheus-receive-remote-write
4. Verify there is no ca_file key in prometheus/metrics-endpoint/otelcol/0 receiver: juju ssh --container otelcol otelcol/0 cat /etc/otelcol/config.yaml
5. Verify the errors in Pebble logs: juju ssh --container otelcol otelcol/0 pebble logs

Environment

juju 3.6.8

Bundle:

bundle: kubernetes
saas:
  grafana-dashboards:
    url: microk8s:admin/cos.grafana-dashboards
  prometheus-receive-remote-write:
    url: microk8s:admin/cos.prometheus-receive-remote-write
applications:
  controller:
    charm: juju-controller
    channel: 3.6/stable
    revision: 116
    scale: 1
    expose: true
    options:
      controller-url: wss://[10.152.183.58:0]:17070/api
      identity-provider-url: ""
      is-juju: true
    constraints: arch=amd64 mem=1536
    trust: true
  otelcol:
    charm: opentelemetry-collector-k8s
    channel: 2/edge
    revision: 95
    resources:
      opentelemetry-collector-image: 9
    scale: 1
    constraints: arch=amd64
    storage:
      persisted: kubernetes,1,1024M
    trust: true
relations:
- - otelcol:metrics-endpoint
  - controller:metrics-endpoint
- - otelcol:send-remote-write
  - prometheus-receive-remote-write:receive-remote-write

Relevant log output

2025-10-13T17:52:33.724Z [otelcol] 2025-10-13T17:52:33.724Z     warn    internal/transaction.go:150     Failed to scrape Prometheus endpoint    {"resource": {"service.instance.id": "d1b98f12-95fd-434e-8461-3d7c34f949fc", "service.name": "otelcol", "service.version": "0.130.1"}, "otelcol.component.id": "prometheus/metrics-endpoint/otelcol/0", "otelcol.component.kind": "receiver", "otelcol.signal": "metrics", "scrape_timestamp": 1760377953719, "target_labels": "{__name__=\"up\", instance=\"controller_ab847f0e-aa46-4bb7-8295-416733e734af_controller_controller/0\", job=\"juju_controller_ab847f0e_controller_prometheus_scrape-0\", juju_application=\"controller\", juju_charm=\"juju-controller\", juju_model=\"controller\", juju_model_uuid=\"ab847f0e-aa46-4bb7-8295-416733e734af\", juju_unit=\"controller/0\"}"}

Additional context

No response

[MichaelThamm]

We assume that all applications part of this scrape job forwarding, and eventually Prometheus scraping trust the same CA individually by having their own receive-ca-cert relations, otherwise that "that reduces to self-signed certs". I agree (with upstream) that the juju_controller adding the CA cert string to ca_file is wrong and they likely wanted this to use ca field in tls_config. Assuming we fix this, I think the next issue is, if it were to supply a path to a CA file, this could/would not be the same path in Prometheus' file system and TLS would then fail when attempting to scrape.

Thoughts

1. We can't ensure that a ca_file file path provided in job in a MetricsEndpointProvider exists in Prometheus' local FS

• Related:
  • feat: Grafana trusts Traefik's ingress observability-stack#144