Build fails with "Failed to query available provider packages"

[tanya-borisova]

Describe the bug

Main build sometimes fails with an intermittent issue:

INFO     resources.resource:resource.py:74 True, deleting_failed, bf99439c-a834-438b-93aa-0b7013d58d39: Error context message = ╷ │ Error: Failed to query available provider packages │  │ Could not retrieve the list of available versions for provider │ hashicorp/random: could not query provider registry for │ registry.terraform.io/hashicorp/random: the request failed after 2 │ attempts, please try again later: Get │ "https://registry.terraform.io/v1/providers/hashicorp/random/versions": EOF ╵  Error: error running command /cnab/app ./delete_vm_extensions.sh rg-***-mgmt ***mgmt tfstate bf99439c-a834-438b-93aa-0b7013d58d39: exit status 1 Error: mixin execution failed: exit status 1 Error: 1 error occurred: 	* container exit code: 1, message: <nil>   az login --identity -u 261b930f-39a8-4266-bf69-abdb98fddebe && az acr login --name ***mgmt && porter uninstall "bf99439c-a834-438b-93aa-0b7013d58d39"  --reference ***mgmt.azurecr.io/tre-service-guacamole-windowsvm:v0.4.3  --param arm_use_msi="true" --param id="bf99439c-a834-438b-93aa-0b7013d58d39" --param os_image="Windows 10" --param parent_service_id="ecb53982-2437-4ca2-8616-d2de913119b1" --param tfstate_container_name="tfstate" --param tfstate_resource_group_name="rg-***-mgmt" --param tfstate_storage_account_name="***mgmt" --param tre_id="***" --param workspace_id="d5534ecf-7e65-45a4-9c18-4f1f49594b90" --allow-docker-host-access --force --cred ./vmss_porter/arm_auth_local_debugging.json --cred ./vmss_porter/aad_auth.json && porter show bf99439c-a834-438b-93aa-0b7013d58d39

Example runs:

https://github.com/microsoft/AzureTRE/runs/7470499092?check_suite_focus=true
https://github.com/microsoft/AzureTRE/runs/7508258200?check_suite_focus=true
https://github.com/microsoft/AzureTRE/runs/7513578222?check_suite_focus=true

[tanya-borisova]

Known mitigation is to retry

[tamirkamara]

I'm considering this for the bundle level.
Will also enable us to remove terraform endpoints from the firewall rules.
Not sure it's possible, but if it is - is it a good idea?

[tamirkamara]

This issue is a combination of several things:

1. Due to using initFile setting in porter.yaml, not all the plugins were initially downloaded when sub modules were included. This put additional pressure when installing the bundle that required more interaction with Terraform/Hashcorp endpoints.
2. There seem to be something off in the DNS. It's not reasonable to see all those errors... We see also DNS errors in the Firewall logs

We should really move to self sufficient bundles that don't require any communication with the outside world where possible.
Read here for more info (discussion as well): getporter/terraform-mixin#90

[marrobi]

Agree this sounds like a good thing to do as removes the dependency on the terraform registry at runtime.

[stuartleeks]

Nice digging @tamirkamara - this sounds like a good idea!

[marrobi]

@tamirkamara does the new TF mixin include you PR so this can now be removed from bundles?