invalid label in Server cause errors in Destination container

### What is the issue?

When a `Server` resource includes a `podSelector` with an invalid label value (in this case `app.kubernetes.io/instance: -elasticsearch`), the destination container logs an error and fails to process protocol detection for affected pods. This issue not only impacted the intended application but also caused unrelated meshed applications such as Emojivoto to fail.

The invalid label value (`-elasticsearch`) does not comply with the Kubernetes label validation regex, but the `Server` resource was accepted by the API server. The error then surfaces at runtime inside the Linkerd destination container, which repeatedly logs failures and degrades mesh functionality.

### How can it be reproduced?

1. Create a `Server` resource with an invalid label value, e.g.:

```yaml
apiVersion: policy.linkerd.io/v1beta3
kind: Server
metadata:
  name: backend
  namespace: server
spec:
  podSelector:
    matchLabels:
      app: backend
      app.kubernetes.io/instance: -elasticsearch  # <-- invalid
  port: 80
  proxyProtocol: HTTP/1
```

2. Deploy some meshed workloads in other namespaces (e.g. Emojivoto):

```bash
curl --proto '=https' --tlsv1.2 -sSfL https://run.linkerd.io/emojivoto.yml | kubectl apply -f -
kubectl annotate namespace emojivoto linkerd.io/inject=enabled --overwrite
kubectl rollout restart deploy -n emojivoto
```

3. Observe that Emojivoto and other meshed traffic fail or behave unpredictably.

### Logs, error output, etc

The following log lines were seen repeatedly in the destination container:

```
time="2025-06-25T14:36:13Z" level=error msg="Error computing opaque protocol for pod kube-prometheus-stack-kube-state-metrics-666565bc5c-hkvcm: \"failed to create Selector: values[0][app.kubernetes.io/instance]: Invalid value: \\\"-elasticsearch\\\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')\"" addr=":8086" component=workload-publisher ip=100.64.24.230 port=8080
time="2025-06-25T14:36:13Z" level=error msg="Error computing opaque protocol for pod linkerd-enterprise-7dd656f8df-2k6vv: \"failed to create Selector: values[0][app.kubernetes.io/instance]: Invalid value: \\\"-elasticsearch\\\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')\"" addr=":8086" component=workload-publisher ip=100.64.16.118 port=9992
time="2025-06-25T14:36:14Z" level=error msg="failed to set address OpaqueProtocol: failed to create Selector: values[0][app.kubernetes.io/instance]: Invalid value: \"-elasticsearch\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')" addr=":8086" component=service-publisher ns=emojivoto port=8080 svc=emoji-svc
time="2025-06-25T14:36:14Z" level=error msg="Error computing opaque protocol for pod linkerd-identity-6cbf59bcff-k82cq: \"failed to create Selector: values[0][app.kubernetes.io/instance]: Invalid value: \\\"-elasticsearch\\\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')\"" addr=":8086" component=workload-publisher ip=100.64.26.18 port=9990
time="2025-06-25T14:36:14Z" level=error msg="failed to set address OpaqueProtocol: failed to create Selector: values[0][app.kubernetes.io/instance]: Invalid value: \"-elasticsearch\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')" addr=":8086" component=service-publisher ns=emojivoto port=80 svc=web-svc
time="2025-06-25T14:36:14Z" level=error msg="failed to set address OpaqueProtocol: failed to create Selector: values[0][app.kubernetes.io/instance]: Invalid value: \"-elasticsearch\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')" addr=":8086" component=service-publisher ns=monitoring port=9090 svc=kube-prometheus-stack-prometheus
time="2025-06-25T14:36:14Z" level=error msg="Error computing opaque protocol for pod linkerd-identity-6cbf59bcff-k82cq: \"failed to create Selector: values[0][app.kubernetes.io/instance]: Invalid value: \\\"-elasticsearch\\\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')\"" addr=":8086" component=workload-publisher ip=100.64.26.18 port=4191
time="2025-06-25T14:36:15Z" level=error msg="failed to set address OpaqueProtocol: failed to create Selector: values[0][app.kubernetes.io/instance]: Invalid value: \"-elasticsearch\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')" addr=":8086" component=service-publisher ns=monitoring port=80 svc=loki-gateway
time="2025-06-25T14:36:16Z" level=error msg="Error computing opaque protocol for pod linkerd-destination-7449b94d89-tt7gj: \"failed to create Selector: values[0][app.kubernetes.io/instance]: Invalid value: \\\"-elasticsearch\\\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')\"" addr=":8086" component=workload-publisher ip=100.64.16.228 port=9997
time="2025-06-25T14:36:17Z" level=error msg="failed to set address OpaqueProtocol: failed to create Selector: values[0][app.kubernetes.io/instance]: Invalid value: \"-elasticsearch\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')" addr=":8086" component=service-publisher ns=monitoring port=9090 svc=kube-prometheus-stack-prometheus
time="2025-06-25T14:36:17Z" level=error msg="Error computing opaque protocol for pod buoyant-cloud-metrics-gkpnz: \"failed to create Selector: values[0][app.kubernetes.io/instance]: Invalid value: \\\"-elasticsearch\\\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')\"" addr=":8086" component=workload-publisher ip=100.64.25.1 port=4191
time="2025-06-25T14:36:18Z" level=error msg="Error computing opaque protocol for pod linkerd-proxy-injector-779775d84d-82dph: \"failed to create Selector: values[0][app.kubernetes.io/instance]: Invalid value: \\\"-elasticsearch\\\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')\"" addr=":8086" component=workload-publisher ip=100.64.26.99 port=4191
time="2025-06-25T14:36:18Z" level=error msg="failed to set address OpaqueProtocol: failed to create Selector: values[0][app.kubernetes.io/instance]: Invalid value: \"-elasticsearch\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')" addr=":8086" component=service-publisher ns=monitoring port=9000 svc=loki-minio
time="2025-06-25T14:36:21Z" level=error msg="failed to set address OpaqueProtocol: failed to create Selector: values[0][app.kubernetes.io/instance]: Invalid value: \"-elasticsearch\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')" addr=":8086" component=service-publisher ns=emojivoto port=8080 svc=voting-svc
time="2025-06-25T14:36:21Z" level=error msg="failed to set address OpaqueProtocol: failed to create Selector: values[0][app.kubernetes.io/instance]: Invalid value: \"-elasticsearch\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')" addr=":8086" component=service-publisher ns=monitoring port=80 svc=loki-gateway
time="2025-06-25T14:36:22Z" level=error msg="failed to set address OpaqueProtocol: failed to create Selector: values[0][app.kubernetes.io/instance]: Invalid value: \"-elasticsearch\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')" addr=":8086" component=service-publisher ns=monitoring port=9090 svc=kube-prometheus-stack-prometheus
time="2025-06-25T14:36:22Z" level=error msg="failed to set address OpaqueProtocol: failed to create Selector: values[0][app.kubernetes.io/instance]: Invalid value: \"-elasticsearch\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')" addr=":8086" component=service-publisher ns=monitoring port=3100 svc=loki
time="2025-06-25T14:36:24Z" level=error msg="failed to set address OpaqueProtocol: failed to create Selector: values[0][app.kubernetes.io/instance]: Invalid value: \"-elasticsearch\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')" addr=":8086" component=service-publisher ns=emojivoto port=8080 svc=voting-svc
time="2025-06-25T14:36:24Z" level=error msg="failed to set address OpaqueProtocol: failed to create Selector: values[0][app.kubernetes.io/instance]: Invalid value: \"-elasticsearch\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')" addr=":8086" component=service-publisher ns=emojivoto port=80 svc=web-svc
time="2025-06-25T14:36:25Z" level=error msg="Error computing opaque protocol for pod alloy-2pvd6: \"failed to create Selector: values[0][app.kubernetes.io/instance]: Invalid value: \\\"-elasticsearch\\\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')\"" addr=":8086" component=workload-publisher ip=100.64.16.123 port=4191
time="2025-06-25T14:36:26Z" level=error msg="Error computing opaque protocol for pod tap-injector-944f4d496-cc4h5: \"failed to create Selector: values[0][app.kubernetes.io/instance]: Invalid value: \\\"-elasticsearch\\\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')\"" addr=":8086" component=workload-publisher ip=100.64.24.153 port=4191
time="2025-06-25T14:36:27Z" level=error msg="failed to set address OpaqueProtocol: failed to create Selector: values[0][app.kubernetes.io/instance]: Invalid value: \"-elasticsearch\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')" addr=":8086" component=service-publisher ns=monitoring port=9090 svc=kube-prometheus-stack-prometheus
time="2025-06-25T14:36:28Z" level=error msg="failed to set address OpaqueProtocol: failed to create Selector: values[0][app.kubernetes.io/instance]: Invalid value: \"-elasticsearch\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')" addr=":8086" component=service-publisher ns=emojivoto port=8080 svc=emoji-svc
time="2025-06-25T14:36:32Z" level=error msg="failed to set address OpaqueProtocol: failed to create Selector: values[0][app.kubernetes.io/instance]: Invalid value: \"-elasticsearch\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')" addr=":8086" component=service-publisher ns=monitoring port=3100 svc=loki
time="2025-06-25T14:36:33Z" level=error msg="failed to set address OpaqueProtocol: failed to create Selector: values[0][app.kubernetes.io/instance]: Invalid value: \"-elasticsearch\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')" addr=":8086" component=service-publisher ns=monitoring port=80 svc=loki-gateway
```

### output of `linkerd check -o short`

```
linkerd-version
---------------
‼ cli is up-to-date
    is running version 2.18.0 but the latest enterprise version is 2.18.1
    see https://linkerd.io/2/checks/#l5d-version-cli for hints

control-plane-version
---------------------
‼ control plane is up-to-date
    is running version 2.18.0 but the latest enterprise version is 2.18.1
    see https://linkerd.io/2/checks/#l5d-version-control for hints

linkerd-control-plane-proxy
---------------------------
‼ control plane proxies are up-to-date
    some proxies are not running the current version:
        * linkerd-destination-84ff9f6c98-62g9g (enterprise-2.18.0)
        * linkerd-destination-84ff9f6c98-g6pxf (enterprise-2.18.0)
        * linkerd-destination-84ff9f6c98-q9tqs (enterprise-2.18.0)
        * linkerd-enterprise-7dd656f8df-2k6vv (enterprise-2.18.0)
        * linkerd-identity-6cbf59bcff-k82cq (enterprise-2.18.0)
        * linkerd-proxy-injector-779775d84d-82dph (enterprise-2.18.0)
    see https://linkerd.io/2/checks/#l5d-cp-proxy-version for hints

linkerd-viz
-----------
‼ viz extension proxies are up-to-date
    some proxies are not running the current version:
        * metrics-api-7ffb4866b8-25zn4 (enterprise-2.18.0)
        * tap-6fbd85db47-x7rsv (enterprise-2.18.0)
        * tap-injector-944f4d496-cc4h5 (enterprise-2.18.0)
        * web-59db46b696-5p8xt (enterprise-2.18.0)
    see https://linkerd.io/2/checks/#l5d-viz-proxy-cp-version for hints

Status check results are √
```

### Environment

Kubernetes Version: v1.33.2
Cluster Environment: on-prem
Host OS: Ubuntu 22.04
Linkerd version: enterprise-2.18.0

### Possible solution

_No response_

### Additional context

_No response_

### Would you like to work on fixing this bug?

None

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

invalid label in Server cause errors in Destination container #14176

What is the issue?

How can it be reproduced?

Logs, error output, etc

output of `linkerd check -o short`

Environment

Possible solution

Additional context

Would you like to work on fixing this bug?

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

invalid label in Server cause errors in Destination container #14176

Description

What is the issue?

How can it be reproduced?

Logs, error output, etc

output of linkerd check -o short

Environment

Possible solution

Additional context

Would you like to work on fixing this bug?

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

output of `linkerd check -o short`