Updating 2.30.0 to 2.33.0 caused connection pool depletion

[gregsdennis]

How do you use Sentry?

Sentry Saas (sentry.io)

Version

2.32.0

Steps to Reproduce

I don't have explicit reproduction step for this, but we did narrow it down to this particular package update.

We had this package unpinned in our requirements.txt. When we released a build on 18 Jun, the latest package was 2.30.0. This worked fine for us.

Earlier this week, when releasing a new build, our app started reporting a lot of timeouts from the API. Looking into it, we found that the authentication library we were using was unable to establish a connection to the authority while attempting to verify the JWT.

Performing a diff of the two deployments found that this package was updated from 2.30.0 to 2.33.0. Today, we pinned the package to 2.30.0 and the issue has been resolved for us.

---

Looking at the diff between the two versions, it appears that sessions appear to be handled more aggressively. This was introduced in 2.32.0.

We believe that this change in session handling is what caused our problems. The theory is that by more quickly shutting down sessions, new sessions need to be created, which depletes the connection pool.

This is all we know. Sorry I can't give you more.

Expected Result

Don't deplete the connection pool.

Actual Result

Other activity which needs to open HTTP connections can't.

[linear]

PY-1786 Updating 2.30.0 to 2.33.0 caused connection pool depletion

[szokeasaurusrex]

Hi @gregsdennis, thank you for reporting this problem, and for providing all the information you were able to.

However, I think we will need to debug this problem a bit to confirm what exactly is going wrong and how we can fix the problem.

First, I would like to ask you to please try upgrading your SDK to version 2.33.2, which is currently the latest version, to confirm whether the problem still exists in 2.33.2.

If you are still experiencing the problem in version 2.33.2, please try pinning your SDK to commit ceb826b, which is based on 2.33.2 and reverts 5709b25, the commit which introduced the more aggressive session flushing that you suspect is causing the problem. If pinning to ceb826b solves the problem, then we at least can confirm the cause.

[sl0thentr0py]

what connection pool are we talking about here? The sessions don't have anything to do with any kind of connection pools, neither database nor http.

[gregsdennis]

@szokeasaurusrex thanks for the effort of setting that up for us, but I don't think I'm able to do that testing for you. This problem only occurred in our production environment, where we have significant traffic, and I'm not willing to risk breaking that again for a test. I can share some logs though.

These logs represent a lambda that is invoked via an HTTP request with an Authorization header containing a token from Cognito. While attempting to verify the token with Cognito, it appears that the Sentry client was also attempting to connect, and as a result the auth verification could never reach Cognito.

You can clearly see that HTTP connections to Sentry were repeatedly attempted. My assumption is that this depleted the HTTP connection pool, and the Cognito call could never be made. Maybe that assessment is wrong, but when rolling just this package back, the problem went away.

I don't any other changes in the delta from 2.30.0 to 2.33.0 that affect anything regarding HTTP calls, so I figured it was the sessions change.

To be clear, we did get a LOT of errors reported to Sentry, so it seems the connection to Sentry succeeded.

[sentrivana]

Hey @gregsdennis 👋🏻

In the absence of an example to repro it's generally not easy for us to diagnose what the issue is. I understand you don't want to test with the branch we provided -- would it be possible to at least isolate the specific SDK version between 2.30.0 and 2.33.0 where this started happening? There's basically only two candidates, if we assume it works on 2.30.0 and doesn't on 2.33.0: 2.31.0 and 2.32.0.

Thanks for the logs. Would need couple clarifications there:

• Can you post the whole message from the NewConnectionErrors? It's truncated in the screenshot. Based on what you wrote I assume it's something about failing to get a connection from the pool, but would be good to see it in full.
• I might be missing something but it looks like only the retried request to Sentry is throwing the NewConnectionErrors. I only see three log lines related to the Cognito request (Verifying Cognito JWT, Starting new HTTPS connection (1): cognito... and finally Failed to verify Cognito JWT...) and that doesn't at first glance indicate that there was a problem fetching a connection from a pool. Is there anything in the logs that indicates what the exact problem was connecting to Cognito (e.g. connection reset, failure to connect, etc.?).

Have you been able to rule out this being a temporary connection issue to Cognito, or an issue with the network in general? Have you seen this happen over a prolonged period of time or multiple times with SDK version 2.33.0?

Thanks for helping us remote-debug this.