CDRIVER-6042 Migrate Python scripts from Poetry to Astral UV

[eramongodb]

Resolves CDRIVER-6042. Followup to mongodb/mongo-cxx-driver#1244.

Migrates our Python script configuration and runner from Poetry to Astral UV. This removes the need for manual virtual environment handling, package dependency installation steps, and Python binary compatibility and version management.

Tip

As a reminder, all uv commands should be invoked with --frozen by default to avoid updating the uv.lock file. The UV_FROZEN environment variable can also be set to 1, true, on, etc. instead.

---

Scripts which are affected by these changes include:

• .evergreen/scripts/clang-format-all.sh
• .evergreen/config_generator/*
• .evergreen/legacy_config_generator/*
• .evergreen/scripts/build-docs.sh

These scripts and related docs have been updated to require/assume invocation via uv run. All necessary dependencies (e.g. clang-format, shrub-py, sphinx, etc.) are managed by the pyproject.toml file and implicitly installed during the uv run command. As a CI optimization (not a requirement), the --only-group flag is used to limit the set of packages downloaded for the given task (e.g. format for clang-format, docs for sphinx).

Because Poetry is no longer used, the following helper scripts are removed by this PR:

• tools/poetry.sh (provided by user/system)
• tools/poetry.ps1 (provided by user/system)
• tools/python.sh (handled by uv)

Following DEVPROD-13413, most (but not quite all) EVG distros provide a system-installed uv binary. Due to the set of tasks requiring uv being relatively small compared to the C++ Driver, the uv-installer.sh script and InstallUV component function do not need to be ported over from the C++ Driver. Instead, the ubuntu2004 distro (one of the distros currently missing a system-installed uv) is removed from the list of allowed distros for the clang-format task. As a result, all tasks which invoke uv are on distros which provide a system-installed uv binary (specifically, ubuntu2204).

---

All dependencies and affected scripts were tested with:

• -p <python-version> (Python versions 3.10 through 3.13).
• --no-project (do not use an existing project or workspace).
• --isolated (do not reuse the project's virtual environment).
• --with <dependency> (to discover and minimize required dependencies).
• --resolution lowest-direct (to validate minimum version requirements, but without lowering current requirements).
• --resolution highest (the default; to validate forward compatibility with latest versions)

This PR does not update minimum required versions for existing dependencies. However, this PR proposes relaxing the max version requirements so newer package versions may be used. Specifically, >= is used instead of ~= (equivalent to ^ for Poetry). Only packages which must be pinned to a specific version use ~=, which is currently limited to clang-format.

Note

The uv tree command displays a helpful visualization of the current package dependency tree. This tree can be used to optimize the list of dependencies. For example, uv tree --outdated --resolution lowest-direct for the currently proposed dependency list produces the following (abbreviated):

mongo-c-driver v0.1.0
├── clang-format v17.0.6 (group: dev) (latest: v20.1.6)
├── furo v2023.5.20 (group: dev) (latest: v2024.8.6)
│   ├── sphinx v7.1.1 (latest: v8.1.3)
│   └── ...
├── packaging v21.0 (group: dev) (*)
├── pydantic v2.8.0 (group: dev) (latest: v2.11.6)
│   ├── pydantic-core v2.20.0 (latest: v2.35.1)
│   │   └── typing-extensions v4.14.0
│   └── ...
├── shrub-py v3.7.0 (group: dev) (latest: v3.10.0)
│   ├── pyyaml v6.0.2
│   └── ...
├── sphinx v7.1.1 (group: dev) (*)
├── sphinx-design v0.5.0 (group: dev) (latest: v0.6.1)
└── yamlloader v1.5.1 (group: dev)
(*) Package tree already displayed

Aside: I do not know to prevent uv tree with --resolution lowest-direct from updating the lock file while still evaluating the lowest-direct tree as intended. Passing --frozen returns the highest resolution tree (unmodified). 🤔

Additional notes:

• Unnecessary supplemental packages (e.g. sphinx-autobuild) are removed. Instead, the --with <package> flag can be used when needed (e.g. see updated instructions in CONTRIBUTING.md). For local development environments, uv pip install can also be used obtain additional packages without modifying the lock file.
• The yamlordereddictloader dependency, which is deprecated, has been replaced with yamlloader instead. This is used by the legacy config generator. Since no prior minimum requirement exists, I've set it to the latest release version (1.5.1 due to 1.5.0 being yanked).
• To continue supporting the mc-evg-generate entry point, uv requires defining a build system. This PR proposes using Hatchling, the default choice for uv. This also allows config_generator to be defined as a package, enabling import config_generator by other Python scripts within this project (analogous to packages = [ include = "config_generator" ] with Poetry).
• The --fresh CMake flag is added to build-docs.sh so that the CMake cache will not incorrectly continue to reuse a Sphinx binary when the script is rerun with an updated Python virtual environment (possibly an entirely new, isolated virtual environment) which provides its own distinct Sphinx binary.
• There are several RemovedInSphinx90Warning warnings. These are deliberately ignored for now in favor of applying a <9.0 dependency requirement to avoid breaking changes when Sphinx 9.0 is released. These should be addressed by CDRIVER-6031.
• Sphinx docs indicate the second parameter to doc2path() should be named base, which is neither absolute (current) nor basedir (previous) following Fix Debian package build #1470 (CDRIVER-4773). I do not know what's going on with this function, but I've opted to remove the named parameter entirely in favor of simply passing True. (@rcsanchez97 does this change break Debian doc builds again?)

[rcsanchez97]

Sphinx docs indicate the second parameter to doc2path() should be named base, which is neither absolute (current) nor basedir (previous) following #1470 (CDRIVER-4773). I do not know what's going on with this function, but I've opted to remove the named parameter entirely in favor of simply passing True. (@rcsanchez97 does this change break Debian doc builds again?)

@eramongodb TBH, I am not certain. But don't worry about that for now. Things are a bit of a mess with the packaging since the 2.0 release and I will need to rework several different things. If this happens to break building the docs, I'm sure that I'll encounter the failure as I work on updating the packaging for 2.0 and I'll deal with it then.

[kevinAlbs]

LGTM with one remaining update. Thank you for the change. uv seems simpler to use (no separate install step) and faster.

[vector-of-bool]

Sorry to delay, thought I had approved this last week. LGTM.

🎉🎉🎉🥳

I think there is potential benefit to a uv.sh shim akin to poetry.sh, but YAGNI until we actually need it.