On 1.17.0, instance metadata ECR credentials authorization on EC2 doesn't work anymore

[rdbisme]

Actual behavior
We have the following line in our CI script:

• echo "{\"credsStore\":\"ecr-login\"}" > /kaniko/.docker/config.json

This enables kaniko to authorize the push to ECR registry using the EC2 machine instance metadata it's running on. This stopped working with 1.17.0 with the following error:

panic: failed to get shared config profile, <redacted>
goroutine 1 [running]:
github.com/awslabs/amazon-ecr-credential-helper/ecr-login/api.DefaultClientFactory.NewClientFromRegion({}, {0xc00015945d?, 0x0?})
	/src/vendor/github.com/awslabs/amazon-ecr-credential-helper/ecr-login/api/factory.go:84 +0x219
github.com/awslabs/amazon-ecr-credential-helper/ecr-login.ECRHelper.Get({{0x882240?, 0xae6780?}, 0xab0a40?}, {0xc000132540, 0x2c})
	/src/vendor/github.com/awslabs/amazon-ecr-credential-helper/ecr-login/ecr.go:101 +0x113
github.com/docker/docker-credential-helpers/credentials.Get({0x881e50, 0xc00011a0d8}, {0x87d840?, 0xc000110020?}, {0x87d7c0, 0xc000110028})
	/src/vendor/github.com/docker/docker-credential-helpers/credentials/credentials.go:130 +0x1fa
github.com/docker/docker-credential-helpers/credentials.HandleCommand({0x881e50?, 0xc00011a0d8?}, {0x7ffedf9c87be?, 0xc00018bec0?}, {0x87d840?, 0xc000110020?}, {0x87d7c0?, 0xc000110028?})
	/src/vendor/github.com/docker/docker-credential-helpers/credentials/credentials.go:73 +0x85
github.com/docker/docker-credential-helpers/credentials.Serve({0x881e50?, 0xc00011a0d8?})
	/src/vendor/github.com/docker/docker-credential-helpers/credentials/credentials.go:58 +0xee
main.main()
	/src/vendor/github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login/main.go:44 +0x154
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "[MASKED]/xxxx/xxx": resolving authorization for [MASKED] failed: error getting credentials - err: exit status 2, out: ``

Expected behavior
Authorization should keep working.

To Reproduce
Steps to reproduce the behavior:

1. ...
2. ...

Additional Information

• Dockerfile
  Please provide either the Dockerfile you're trying to build or one that can reproduce this error.
• Build Context
  Please provide or clearly describe any files needed to build the Dockerfile (ADD/COPY commands)
• Kaniko Image Using docker image sha256:91ffcd7c7450560c235406479a476b632efeef9ca036ca4ce32de395a580f83a for gcr.io/kaniko-project/executor:debug with digest gcr.io/kaniko-project/executor@sha256:97c78eedb0560b8fcf64900abdb810f84f9882d033421f4aee1e6559f42b7e87 ...

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[csm-kb]

Can confirm the same! via executor:debug as of 10 minutes ago:

$ echo "{\"credsStore\":\"ecr-login\",\"credHelpers\":{\"${DOCKER_REGISTRY}\":\"ecr-login\"}}" > /kaniko/.docker/config.json
$ echo "Creating ${CI_env} build for ${DOCKER_REGISTRY}/${APP_NAME}:${IMAGE_TAG}"
Creating staging build for [MASKED].dkr.ecr.[MASKED].amazonaws.com/[MASKED]:v0.33.7
$ export AWS_PROFILE=default
$ /kaniko/executor --context ${CI_PROJECT_DIR} --dockerfile ${CI_PROJECT_DIR}/Dockerfile --build-arg "BUILD_APP_ENV=${CI_env}" --destination "${DOCKER_REGISTRY}/${APP_NAME}:${IMAGE_TAG}" --cache=true --cache-repo "${DOCKER_REGISTRY}/${APP_NAME}" --cache-ttl ${CACHE_TTL}
panic: failed to get shared config profile, default
goroutine 1 [running]:
github.com/awslabs/amazon-ecr-credential-helper/ecr-login/api.DefaultClientFactory.NewClientFromRegion({}, {0xc00002d8dd?, 0x0?})
	/src/vendor/github.com/awslabs/amazon-ecr-credential-helper/ecr-login/api/factory.go:84 +0x[21](https://gitlab.com/[MASKED]/-/jobs/5335232340#L21)9
github.com/awslabs/amazon-ecr-credential-helper/ecr-login.ECRHelper.Get({{0x88[22](https://gitlab.com/[MASKED]/-/jobs/5335232340#L22)40?, 0xae6780?}, 0xab0a40?}, {0xc000026b70, 0x2c})
	/src/vendor/github.com/awslabs/amazon-ecr-credential-helper/ecr-login/ecr.go:101 +0x113
github.com/docker/docker-credential-helpers/credentials.Get({0x881e50, 0xc0000100f0}, {0x87d840?, 0xc000068028?}, {0x87d7c0, 0xc000068030})
	/src/vendor/github.com/docker/docker-credential-helpers/credentials/credentials.go:130 +0x1fa
github.com/docker/docker-credential-helpers/credentials.HandleCommand({0x881e50?, 0xc0000100f0?}, {0x7ffc5[24](https://gitlab.com/[MASKED]/-/jobs/5335232340#L24)af69f?, 0xc000161ec0?}, {0x87d840?, 0xc0000680[28](https://gitlab.com/[MASKED]/-/jobs/5335232340#L28)?}, {0x87d7c0?, 0xc0000680[30](https://gitlab.com/[MASKED]/-/jobs/5335232340#L30)?})
	/src/vendor/github.com/docker/docker-credential-helpers/credentials/credentials.go:73 +0x85
github.com/docker/docker-credential-helpers/credentials.Serve({0x881e50?, 0xc0000100f0?})
	/src/vendor/github.com/docker/docker-credential-helpers/credentials/credentials.go:58 +0xee
main.main()
	/src/vendor/github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login/main.go:44 +0x154
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "[MASKED].dkr.ecr.[MASKED].amazonaws.com/[MASKED]:v0.[33](https://gitlab.com/[MASKED]/-/jobs/5335232340#L33).7": resolving authorization for [MASKED].dkr.ecr.[MASKED].amazonaws.com failed: error getting credentials - err: exit status 2, out: ``

[aaron-prindle]

@rdbisme and @csm-kb - thank you for flagging the issue here. I haven't had a chance to investigate this regression more deeply. Below is a list of all of the changes made from v1.16.0 - v1.17.0, likely one of these changes caused this regression IIUC. From this list it seems that this would likely be related to one of the updated deps, please add additional information/investigation if anyone in the thread here has a sense of what the root cause might be.

Docs, Test, and CI/CD Updates:

• docs: fix readme sample typo #2792
• docs: Update designdoc.md with correct link to skaffold repository #2775
• ci: add automated way of cutting releases w/ generation of CHANGELOG.md Makefile changes #2786
• ci: remove log line from listpullreqs.go and additional release.sh fixes #2790
• ci: resolve issue with integration tests where lack of disk space caused k3s issues #2804
• test: add test cases and docString for regex in COPY command #2773

Updates and Refactors:

• chore(deps): bump github.com/aws/aws-sdk-go-v2/config from 1.18.42 to 1.18.44 #2777
• chore(deps): bump github.com/aws/aws-sdk-go-v2/feature/s3/manager from 1.11.83 to 1.11.86 #2757
• chore(deps): bump github.com/aws/aws-sdk-go-v2/feature/s3/manager from 1.11.86 to 1.11.87 #2770
• chore(deps): bump github.com/aws/aws-sdk-go-v2/feature/s3/manager from 1.11.87 to 1.11.91 #2805
• chore(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.39.0 to 1.40.0 #2771
• chore(deps): bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.40.0 to 1.40.1 #2780
• chore(deps): bump github.com/containerd/containerd from 1.7.6 to 1.7.7 #2797
• chore(deps): bump github.com/google/go-cmp from 0.5.9 to 0.6.0 #2796
• chore(deps): bump github.com/otiai10/copy from 1.12.0 to 1.14.0 #2772
• chore(deps): bump github.com/spf13/afero from 1.9.5 to 1.10.0 #2758
• chore(deps): bump golang.org/x/net from 0.16.0 to 0.17.0 #2791
• chore(deps): bump golang.org/x/oauth2 from 0.12.0 to 0.13.0 #2781
• chore(deps): bump golang.org/x/sync from 0.3.0 to 0.4.0 #2798
• chore(deps): bump google.golang.org/api from 0.141.0 to 0.142.0 #2756
• chore(deps): bump google.golang.org/api from 0.142.0 to 0.143.0 #2769
• chore(deps): bump google.golang.org/api from 0.143.0 to 0.145.0 #2778
• refactor: Remove fallbackToUID bool option from Kaniko code #2767