urllib3 version 2 incompatibility

[ceesem]

We are starting to use some code in caveclient that requires urllib3 version 2, but with urllib3, e.g. version 2.2.3, import cloudvolume raises the error:

cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_

This seems to be a breaking change in urllib3 between version 1.x.x and 2.x.x, but that version req is not enforced by requirements.txt from cloudvolume. It would be nice to be compatible with recent versions of urllib3.

[william-silversmith]

It looks like this warning comes from the use of botocore:

https://urllib3.readthedocs.io/en/stable/v2-migration-guide.html

ImportError: cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_'

This likely happens because you’re using botocore which [does not support urllib3 2.0 yet](https://github.com/boto/botocore/issues/2921). The good news is that botocore explicitly declares in its dependencies that it only supports urllib3<2. Make sure to use a recent pip. That way, pip will install urllib3 1.26.x until botocore starts supporting urllib3 2.0.

If you’re deploying to an AWS environment such as Lambda or a host using Amazon Linux 2, you’ll need to explicitly pin to urllib3<2 in your project to ensure urllib3 2.0 isn’t brought into your environment. Otherwise, this may result in unintended side effects with the default boto3 installation.

botocore is required for the use of Amazon S3 or S3 emulating endpoints like the Cloudian Hyperstore (matrix://) and Tigerdata (tigerdata://).

botocore has the following pin on <2 to support Amazon Linux 2 and AWS Lambda.

https://github.com/boto/botocore/blob/develop/setup.py#L25-L33

Does the latest CaveClient need urllib>=2? What kind of resolution would you like to see? There is significant use of S3 and S3 emulating endpoints by CloudVolume and CloudFiles users.

[ceesem]

Yup, that totally checks out.

The most recent version of caveclient added some increased retry configuration that used on an import that does not work in urllib3 1.x.x. I'm honestly not sure what the right answer for us is — it feels pretty negligent of the botocore package to not support a well-telegraphed version change after well more than a year, but it's also critical functionality as you point out.

[william-silversmith]

Hmm have you considered using https://github.com/jd/tenacity for the retry function? That might get you out of the jam.

[jefferis]

@ceesem I’ve been getting test errors on Ubuntu for the last few weeks eg
https://github.com/natverse/fafbseg/actions/runs/11430315793/job/31797722275
This seems to be triggered when I try to import cloud volume on the Python side and the error is:

cannot import name 'ssl' from 'urllib3.util.ssl_' (/home/runner/.local/share/r-miniconda/envs/r-reticulate/lib/python3.10/site-packages/urllib3/util/ssl_.py)

Do you think this might be related? Do you have a workaround? Thanks!

[fcollman]

2.2.3 urllib3 is quite new (september).. https://github.com/urllib3/urllib3/tags , and is a major version bump from 1.26.20 that came before it on github.

1.26.20 had ssl in that file.. https://github.com/urllib3/urllib3/blob/6f2ad7ca0cdde53751bab29cbc10bcc965bb4387/src/urllib3/util/ssl_.py#L49

2.2.3 does not.

I don't see any direct reference to this in cloud-volume, it must be coming from one of it's dependancies that isn't properly controlling its urllib3 compatability with v2.

my workaround advice would be to add an explicit dependency on urllib3<2 on whatever your install route is, or pre-installing urllib3 with something <2 but >=1.25.7. cloud-volume installation should not cause it to be re-installed.

Longer term we need a fuller traceback to figure out who is the depenancy that is at fault and they need to fix their compatability with urllib3>=2

[jefferis]

Thanks @fcollman. I've looked into this some more and my github actions builds previously worked with urllib 2.2.3 (Sep 12 build) so unfortunately this seems to be a red herring. However, I have narrowed it down to one of the following and I'm steadily working through the options ...

Python 3.10.14 vs 3.10.15
botocore 1.35.17 vs 1.35.24
boto3 1.35.17 vs 1.35.24
google_auth 2.34.0 vs 2.35.0
cloud-volume==10.4.0 vs 10.5.0
caveclient==5.29.1 vs 5.29.11

[jefferis]

It's one of these

Python 3.10.14 vs 3.10.15
botocore 1.35.17 vs 1.35.24
boto3 1.35.17 vs 1.35.24

[jefferis]

So it seems to be:

Python 3.10.14 vs 3.10.15

just why is a bit beyond me ... but I guess it probably doesn't have anything to do with this issue. Although if anyone can enlighten me I would be v grateful.

or in more detail

	440	441	442	443	444	445	446	447
Python 3.10.14 vs 3.10.15	old	old	old	old	new	old	old	new
botocore 1.35.17 vs 1.35.24	old	old	old	new	old	old	new	old
boto3 1.35.17 vs 1.35.24	old	old	old	new	old	old	new	old
google_auth 2.34.0 vs 2.35.0	old	new	old	old	new	old	new	old
cloud-volume==10.4.0 vs 10.5.0	old	new	old	old	new	old	new	old
caveclient==5.29.1 vs 5.29.11	old	new	new	old	new	new	new	old
outcome	ok	ok	ok	ok	fail	ok	ok	fail

[fcollman]

so odd... https://www.python.org/downloads/release/python-31015/

"gh-114572: ssl.SSLContext.cert_store_stats() and ssl.SSLContext.get_ca_certs() now correctly lock access to the certificate store, when the ssl.SSLContext is shared across multiple threads."

This seems to me to be the top candidate in that release set based on the ssl related error that came up..