error: 401 Must authenticate to access this API when deploying via Kustomize on GKE connected to a self-hosted Github Enterprise Server

[alan707]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

When Atlantis v0.33.0 is deployed against a GitHub Enterprise Server (GHES) instance, it repeatedly fails to post comments on pull requests with:

401 Must authenticate to access this API

Despite:

1. Generating and validating a GitHub App JWT (returns HTTP 200 from /api/v3/app).
2. Exchanging that JWT for an installation access token (manually via curl).
3. Confirming the App has Read & Write permissions on Issues and Pull Requests and is installed on the target repo.
4. Mounting the private key and webhook secret correctly via a Kubernetes Secret.
5. Setting both ATLANTIS_GH_APP_KEY_FILE and ATLANTIS_GH_WEBHOOK_SECRET in the pod environment.

Restarting the StatefulSet after every configuration change.

What does work

I’ve verified that posting an atlantis help comment on a PR in the allowed repository successfully appears in the pod’s logs.

Reproduction Steps

1. Deploy Atlantis via kubectl apply
2. Bootstraped the GitHub App via Atlantis gtihub-app/setup URL
3. Store credentials in Kubernetes secrets encoding them via base64 (including app-key file contents)
4. Confirmed Github Enterprise server can ping atlantis
5. Create a PR in the allowed repo and create a comment atlantis help

Logs

Environment details

• Atlantis version: v0.33.0
• Deployment method: Kustomization via GKE
• Atlantis flags: None
• Env Vars:

        - name: ATLANTIS_DATA_DIR
          value: /atlantis
        - name: ATLANTIS_PORT
          value: "4141" 
        - name: ATLANTIS_GH_USER
          value: fake # recommended by Atlantis docs: https://www.runatlantis.io/docs/access-credentials.html#github-app
        - name: ATLANTIS_GH_TOKEN
          value: fake # recommended by Atlantis docs: https://www.runatlantis.io/docs/access-credentials.html#github-app
        - name: ATLANTIS_GH_ORG
          value: readacted_org
        - name: ATLANTIS_ATLANTIS_URL
          value: https://<ATLANTIS_URL>
        - name: ATLANTIS_LOG_LEVEL
          value: debug
        - name: ATLANTIS_REPO_ALLOWLIST
          value: URL/REPO
        - name: ATLANTIS_GH_HOSTNAME
          value: "HOSTNAME"
        - name: ATLANTIS_GH_APP_ID
          value: "redacted"
        - name: ATLANTIS_GH_APP_KEY_FILE
          value: /etc/atlantis/gh-app-key.pem
        - name: ATLANTIS_GH_WEBHOOK_SECRET
          valueFrom:
            secretKeyRef:
              name: atlantis-vcs
              key: webhook-secret # must generate this with base64 encode

Additional Context

Decided to use an ATLANTIS_GH_APP_KEY_FILE but I also tried just passing the value for ATLANTIS_GH_APP_KEY directly from the Kubernetes secret. That failed to authenticate as well.

[dosubot]

Hello @alan707! I'm Dosu, your friendly bot here to help you tackle bugs, answer questions, and guide you on your journey to becoming a contributor. While you're waiting for a human maintainer, feel free to share your thoughts or questions! 🚀

The 401 authentication error you're encountering when deploying Atlantis on GKE with a self-hosted GitHub Enterprise Server could be due to several factors. Here are some troubleshooting steps and considerations:

1. Verify GitHub App Permissions: Ensure that the GitHub App has the necessary permissions. It should have read and write permissions for issues and pull requests, as well as other required permissions like administration (read-only), checks (read and write), commit statuses (read and write), contents (read and write), metadata (read-only), webhooks (read and write), members (read-only), and actions (read-only) ^[1].

2. JWT and Token Exchange: Double-check the process of generating the JWT and exchanging it for an installation access token. Ensure that the JWT is correctly generated and that the token exchange is successful. Any discrepancies in this process can lead to authentication failures.

3. Environment Variables and Secrets: Confirm that all environment variables are correctly set and that the Kubernetes secrets are properly mounted. Ensure that the ATLANTIS_GH_APP_KEY_FILE and ATLANTIS_GH_WEBHOOK_SECRET are correctly referenced and accessible by the Atlantis pod.

4. Webhook Configuration: Verify that the webhook is correctly configured with the necessary events (pull request reviews, pushes, issue comments, pull requests) and that the payload URL is correctly set to http://$URL/events or https://$URL/events if using SSL ^[2].

5. Logs and Debugging: Since you have set the log level to debug, check the logs for any specific error messages or indicators that might provide more context on the authentication failure. Look for any discrepancies or errors in the logs that could point to the root cause.

6. Network and Connectivity: Ensure that there are no network issues preventing Atlantis from communicating with the GitHub Enterprise Server. Verify that the server can reach Atlantis and that there are no firewall or proxy issues blocking the requests.

By following these steps, you should be able to identify and resolve the 401 authentication error. If the issue persists, consider reviewing the specific log messages for more detailed insights into the failure.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}