v3 Provider appears to randomly remove helm_release resources from TF state - Hard to reproduce

[gpapakyriakopoulos]

Terraform, Provider, Kubernetes and Helm Versions

Terraform version: 1.5.7
Provider version: 3.0.2
Kubernetes version: 1.31 - EKS

Affected Resource(s)

• helm_release

Terraform Configuration Files

N/A

Expected Behavior

helm_release resources that have not been changed do not get their TF state files changed at all

Actual Behavior

After several successful Terraform apply runs (no changes to any helm_release resources), the terraform state for several (but not all) helm_release resources was updated and those resources completely removed from the state.

This resulted in Terraform trying to recreate those Helm Releases even though they already existed in the EKS cluster, resulting in an error :

cannot re-use a name that is still in use

Important Factoids

We cannot debug the whole thing and we are having trouble easily reproducing this as we are seeing the outcomes and comparing terraform states (v1 is ok and then v2 is empty).
We are also a bit suspicious of this method: https://github.com/hashicorp/terraform-provider-helm/blob/main/helm/resource_helm_release.go#L873 especially the resourceReleaseExists return that removes resources from the context below.

[tsigouris007]

Also probably related to this issue: #1665 (comment)

[stemid]

Also probably related to this issue: #1665 (comment)

Just in case these two issues are related, I can say I have no trouble reproducing mine. Follow the steps to reproduce and you should be able to. 1) Install a Talos cluster with no CNI, and kube-proxy disabled. Maybe it can also work with k3s or microk8s with no CNI and no kube-proxy. 2) Install Cilium helm chart with helm_release, to take over for kube-proxy. 3) It always produces this error, and a ghost install of Cilium.

[fedeostrit]

I have the same behavior, I have the resources applied with helm_realese and if I do terraform plan without any changes some ask me to update and others ask me to add directly as if they are not installed, if I do an import it adds them again

[wyardley]

Should this have the "post-v3" label too?

[gpapakyriakopoulos]

This happened to us again after a successful apply with no issues, we just lost half of our helm_release resources from the state.

Example output from Terraform (we use Terragrunt) :

Acquiring state lock. This may take a few moments...
helm_release.eck-agent: Refreshing state... [id=cf-logpush-audit]
helm_release.eck-agent: Refreshing state... [id=sast]
helm_release.eck-agent: Refreshing state... [id=zt-logpush-device-posture]
helm_release.eck-agent: Refreshing state... [id=zt-logpush-network-session]
Acquiring state lock. This may take a few moments...
helm_release.eck-agent: Refreshing state... [id=zt-logpush-gateway-network]
helm_release.eck-agent: Refreshing state... [id=cloudtrail-agent]

Example state file after data loss :

{
  "version": 4,
  "terraform_version": "1.5.7",
  "serial": 11,
  "lineage": "91be261f-bf5c-687d-b2ec-cbe09f588146",
  "outputs": {},
  "resources": [],
  "check_results": null
}

This is a MAJOR bug and it is now affecting our production cluster heavily. We will try to downgrade back to provider v2 but this needs to be addresses as soon as possible. cc @arybolovlev for bumping this

[fedeostrit]

@arybolovlev
Any updates on this or any roadmap for fixing this?

[ekersale]

@arybolovlev any ETA for a fix ? That bug is becoming really problematic.

[pguinard-public-com]

I'm not sure if this is helpful but I've been able to reproduce this around 50% of the time for the first plan during a pipeline only. Steps to reproduce, I'm more baffled than anything based on the behavior we've noticed on our side:

1. Run terraform init and restore your remote state
2. Run a terraform init and plan expecting no changes (changes occur)
3. rm -rf .terraform* (we do not use lock files but a local mirror in a docker image) to get back to the original state.
4. Run a terraform init and plan expecting no changes (changes are not in the plan)
5. Repeat the previous 2 steps and out of 8 retries we are unable to get the same issue to come back

[wyardley]

Just guessing, but I'm imagining it might be helpful for the team working on this to have a copy of the debug output? It looks like they have a gpg key available to encrypt the output, which would probably be smart in this case.

[deja001-sme]

We have a similar issue, but with alb_controller helm release. Once deployed everything is ok, but once we change something that make changes throughout the state, like tags for example, it will reappear as ghost creation, and we would get the same error as stated in this issue:

cannot re-use a name that is still in use

As this is only happening with changes that ( mostly ) affect the whole state, my guess would be that the new provider schema is not properly developed.

[tsigouris007]

Happened again now, randomly after an apply. No clear indication or reproduction path on how the provider fails that hard.
This is very frustrating as we have to restore multiple state files back to their prior state - on top of searching which states broke.

[alexmbird]

This is happening for us as well, seemingly all of the time. In our case it knocks out the ingress controller (!) which means until the bug is fixed, the provider is borderline unusable.

[wjf3121]

A somewhat related issue we are observing: when creating a new helm_release resource, if terraform apply failed because helm install leads to failed status, the second apply will throw cannot re-use a name that is still in use error.

In v2, second apply will uninstall the failed chart and recreate the resource, it seems that in v3 the resource is not written into the state if the creation is in failed status.