[Bug]: SecretIAMMember stuck in Terminating when member is WIFv2 federated principal (principal://...) — external-name parser fails on '/' inside member #936

Is there an existing issue for this?

• I have searched the existing issues

Note: this rewrites #934 (originally filed 2026-05-26) to incorporate the empirical validation done on 2026-05-28. The original suggested a kubectl annotate crossplane.io/external-name=<secretId> workaround; in practice that does not unstick the delete on upbound-provider-gcp-secretmanager:v2.5.3. The validated workaround and the suggested fix direction are inlined below. #934 is being closed as superseded by this issue.

Affected Resource(s)

• secretmanager.gcp.upbound.io/v1beta1 — SecretIAMMember

The same parser pattern likely affects all *IAMMember resources from the GCP family whose import format includes the member string verbatim — BucketIAMMember, ServiceAccountIAMMember, ProjectIAMMember, FolderIAMMember, OrganizationIAMMember, ManagedFolderIAMMember.

Resource MRs required to reproduce the bug

apiVersion: secretmanager.gcp.upbound.io/v1beta1
kind: SecretIAMMember
metadata:
  name: my-binding
  namespace: crossplane
spec:
  forProvider:
    secretId: my-secret
    role: roles/secretmanager.secretAccessor
    # Workload Identity Federation v2 direct federated principal.
    # The member contains forward slashes inside its path — this is what
    # breaks the parser later. It is the documented GCP IAM principal
    # format for direct WIF (no intermediate GSA).
    member: principal://iam.googleapis.com/projects/123456789012/locations/global/workloadIdentityPools/my-pool/subject/ns/my-namespace/sa/my-ksa
  providerConfigRef:
    name: my-providerconfig

Steps to Reproduce

1. Create a SecretIAMMember whose spec.forProvider.member is a WIFv2 federated principal of the form principal://iam.googleapis.com/projects/<num>/locations/global/workloadIdentityPools/<pool>/subject/ns/<ns>/sa/<ksa>.
2. Crossplane creates the GCP binding successfully and writes crossplane.io/external-name to the auto-composite form projects/<p>/secrets/<s>/roles/<r>/principal://iam.googleapis.com/.../sa/<ksa>. The additional / characters inside the member portion are what break the parser later.
3. Delete the resource (kubectl delete, or force a recreate via an immutable-field change such as mutating the member format).
4. The provider attempts to delete the IAM binding on GCP. The Terrajet-generated importer re-parses the external-name annotation to reconstruct the Import ID.
5. The importer returns an empty Import ID ("") because the extra / characters inside principal://... collide with the path separator used by the accepted regex set; all three accepted patterns fail to match cleanly.
6. The async delete operation fails. Crossplane retries on its own schedule; each retry hits the same parser failure.
7. The finalizer finalizer.managedresource.crossplane.io is never released. The resource is stuck in Terminating indefinitely.

What happened?

The resource enters an eternal Terminating state. Cascading consequences observed in the field:

• Parent Flux Kustomization health check flips to Healthy=False (the still-existing Terminating resource is reported as unhealthy by Flux).
• Downstream resources that depend on the Kustomization (e.g. HelmReleases consuming credential secrets pushed by the same orchestration) get stuck in their own retry loops.
• In an incident on 2026-05-20 this caused 3 h+ of HelmRelease cascade failures.

Relevant Error Output

delete failed: async delete failed: failed to delete the resource:
[{0 Import id "" doesn't match any of the accepted formats: [projects/(?P<project>[^/]+)/secrets/(?P<secret_id>[^/]+) (?P<project>[^/]+)/(?P<secret_id>[^/]+) (?P<secret_id>[^/]+)]  []}]

Conditions on the stuck resource:

Synced=False reason=ReconcileError      (message as above)
Ready=False reason=Deleting
LastAsyncOperation=False reason=AsyncDeleteFailure

Empirical validation of candidate workarounds (2026-05-28)

We deliberately reproduced the stuck on 3 SecretIAMMember resources in a staging environment and walked through every candidate fix that had been considered:

Why the imperative annotate workaround does not unstick the delete

Two distinct failure modes:

1. Healthy resource (preventive variant) — annotating crossplane.io/external-name=<short> before the bascule: Crossplane reverts the annotation back to the auto-composite form within ~5 seconds (re-asserted from spec on every reconcile of the steady state). Preventive timing is essentially impossible.
2. Terminating resource (curative variant) — annotating after the stuck: the annotation does hold (controller is in delete mode, not re-asserting external-name), but the async delete op continues to fail with Import id "". The accepted-formats regex set captures only <project> and <secret_id> — neither sufficient to identify a SecretIAMMember, which is keyed by (secret_id, role, member). So even when the short external-name does parse, the resulting state has no role/member and cannot drive the removeMember call.

Also tested: declarative pin of crossplane.io/external-name in Kustomize source with the parser-safe minimal form. Crossplane rewrites the value on the next reconcile. Validated 2026-05-22.

Working workaround

Remove the offending principal://... binding directly on GCP via gcloud secrets remove-iam-policy-binding. Crossplane's next async retry observes "binding absent" → no-op success → finalizer releases → the K8s resource is GC'd. The owner (Flux in our case) then recreates the SecretIAMMember with the new spec, and Crossplane creates the new binding cleanly.

NAME=<stuck-name>
NS=crossplane

PROJECT=$(kubectl get secretiammember.secretmanager.gcp.upbound.io "$NAME" -n "$NS" -o jsonpath='{.spec.forProvider.project}')
SECRET_ID=$(kubectl get secretiammember.secretmanager.gcp.upbound.io "$NAME" -n "$NS" -o jsonpath='{.spec.forProvider.secretId}')
ROLE=$(kubectl get secretiammember.secretmanager.gcp.upbound.io "$NAME" -n "$NS" -o jsonpath='{.spec.forProvider.role}')
OLD_MEMBER=$(kubectl get secretiammember.secretmanager.gcp.upbound.io "$NAME" -n "$NS" -o jsonpath='{.status.atProvider.member}')

gcloud secrets remove-iam-policy-binding "$SECRET_ID" \
  --project="$PROJECT" --member="$OLD_MEMBER" --role="$ROLE"

The operator needs only roles/secretmanager.admin (or a tightly-scoped equivalent) — no broad IAM-admin, no permanent CronJob, no provider patch.

Suggested fix direction

Rather than redesigning the external-name composite (breaking change for existing installations), a smaller backwards-compatible fix would be to add a delete fast-path in the provider's Delete for *IAMMember resources:

Before importing/re-parsing the external-name, read the live IAM policy on the parent resource (secretId in this case, taken from spec.forProvider). If the policy does not contain a binding matching (role, member) from spec.forProvider, the external resource is effectively already absent — return success and release the finalizer.

This sidesteps the parser bug entirely on the delete path (the only place it bites in practice — Create / Update do not re-parse the external-name) and matches operator intuition ("the binding is gone, the K8s resource should clean up"). The same fast-path would apply to BucketIAMMember, ServiceAccountIAMMember, ProjectIAMMember, FolderIAMMember, OrganizationIAMMember, ManagedFolderIAMMember.

Crossplane Version

Crossplane v2 (with ops.crossplane.io/v1alpha1 CronOperation, ManagedResourceDefinition, ManagedResourceActivationPolicy enabled).

Provider Version

upbound-provider-gcp-secretmanager:v2.5.3 (family provider, installed via the Provider package).

Kubernetes Version

GKE noprod, K8s 1.31.x.

Kubernetes Distribution

GKE (Google Kubernetes Engine, Standard).

Additional Info

Happy to provide additional cluster diagnostics, full condition dumps, or test a candidate patch on a noprod cluster. See #934 for the original report and discussion thread.