GCP Provider: Add Support for System CA When Using GKE DNS Endpoint (*.gke.goog)

[nirmeshjaingothane]

Feature Request

Summary

When using Flux's GCP provider with a GKE DNS endpoint (*.gke.goog) and dnsLookup: "true", the kustomize-controller fails with a TLS certificate verification error:

tls: failed to verify certificate: x509: certificate signed by unknown authority

This is because the provider unconditionally uses the cluster's self-signed CA (MasterAuth.ClusterCaCertificate) for TLS verification — but DNS endpoints serve certificates signed by Google Trust Services (a public CA), not the cluster CA.

We would like the GCP provider to support using the system's trusted root CA bundle when a DNS endpoint is in use, either automatically or via an explicit ConfigMap field.

---

Environment

• Flux with GCP Workload Identity Federation (WIF)
• GKE cluster with DNS endpoint enabled
• Management cluster connecting to workload clusters via kubeConfig.configMapRef

---

ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-cluster-configmap
  namespace: wc-my-cluster
data:
  provider: gcp
  cluster: projects/my-project/locations/europe-west4/clusters/my-cluster
  dnsLookup: "true"
  address: gke-XXXXXXXXXXXXXXXXXXXX-XXXXXXXXXXXX.europe-west4.gke.goog

---

Current Behaviour

Could not determine release state: failed to retrieve last release from storage:
query: failed to query with labels:
Get "https://gke-XXXXXXXXXXXXXXXXXXXX-XXXXXXXXXXXX.europe-west4.gke.goog/api/v1/namespaces/kube-system/secrets?...": tls: failed to verify certificate: x509: certificate signed by unknown authority

---

Possible Root Cause

The issue may lie in auth/gcp/provider.go, in the NewRESTConfig function, which fetches CAData from the GKE API whenever it is not explicitly provided:

if host == "" || caData == "" {
    clusterResource, _ := p.impl().GetCluster(ctx, cluster, client)
    caData, _ = base64.StdEncoding.DecodeString(
        clusterResource.MasterAuth.ClusterCaCertificate)
}

When address is a DNS endpoint but caData is empty, the following may be happening:

1. host is set to the *.gke.goog DNS endpoint
2. caData is empty → provider fetches the cluster's self-signed CA
3. The DNS endpoint's certificate is signed by Google Trust Services (public CA)
4. The self-signed cluster CA cannot verify it → TLS handshake fails

This could point to a mismatch: IP-based endpoints use the cluster's self-signed CA, but DNS endpoints serve certificates signed by a public CA — and the provider may not currently distinguish between the two.

Note: fluxcd/pkg#1071 removed the address matching validation, enabling DNS endpoints to be specified — but it may not have updated CA handling, potentially leaving this gap.

---

Desired Behaviour

When dnsLookup: "true" is set (or the address is a *.gke.goog host), the GCP provider should support falling back to the system's trusted root CA bundle (which includes Google Trust Services) instead of always using the cluster's self-signed CA.

---

Suggested Implementation

Option 1 — Automatic detection: Skip cluster CA when a DNS endpoint is detected:

// Only use cluster CA for IP-based endpoints, not DNS endpoints
if !isDNSEndpoint(host) {
    caData, err = base64.StdEncoding.DecodeString(
        clusterResource.MasterAuth.ClusterCaCertificate)
}

Option 2 — Explicit user control: Expose a new ConfigMap field useSystemCA: "true" so users can opt in:

data:
  provider: gcp
  cluster: projects/my-project/locations/europe-west4/clusters/my-cluster
  dnsLookup: "true"
  address: gke-XXXXXXXXXXXXXXXXXXXX-XXXXXXXXXXXX.europe-west4.gke.goog
  useSystemCA: "true"

---

Related

• auth: remove cluster host matching requirement pkg#1071 — removed address matching validation (partial fix)
• GCP plugin for Flux WIF does not support anything other than the public IP endpoint. #5677 — original issue for DNS endpoint support

---

Note

We are also open to the possibility that this may be a misconfiguration on our end. If dnsLookup: "true" with a *.gke.goog address requires additional configuration that we may have missed, please do let us know.

[matheuscscp]

Possible Root Cause

As explained in the other issue, there's no bug here to describe a root cause for. This use case has never been contemplated in the feature design. The feature allows using a CA either set directly in the ConfigMap, or retrieved from the cluster resource in the GKE API, nothing else. Please remove or reframe this section.

---

Implementation-wise, would it suffice checking the address for the .gke.goog suffix and omitting TLSClientConfig in the *rest.Config? I don't think we need to introduce any new fields. Would there ever exist an address ending in .gke.goog where a different CA not from the system could be used?