CXX-3126 Refactor EVG config to use config_generator #1244

eramongodb · 2024-10-29T20:42:37Z

Summary

Resolves CXX-3126. Verified by this patch. Reattempt of #1242.

Imports the EVG config generator from the C Driver and converts the current EVG config into generator components. Changes to task matrices are deliberately minimized. Auditing and improving the task matrices is outside the scope of this PR.

Once this PR is merged, the .mci.yml file may be deleted in favor of the new .evergreen/config.yml file (for consistency with the C Driver). They are equivalent given the changes in this PR.

Astral uv

The projects managed by our team have seen a steady progression of Python tooling: a pip requirements.txt file -> virtual environments (venv) -> Poetry project configuration (pyproject.toml) -> Python binary management (pyenv in ./tools/python.sh). This PR takes this opportunity to take another step by adopting a new Python tool which aims to supercede all such tooling which came prior: uv.

Important

This PR does not use uv in Evergreen and does not add any scripts to assist with obtaining and using uv. Users should install uv themselves according to uv's installation instructions as best appropriate for their local development environment. Exploring the use of uv in Evergreen scripts is outside the scope of this PR.

The shockingly impressive convenience of this tool is demonstrated by running the following command, whose only prerequisite is that uv is installed and available to use:

uv run --frozen .evergreen/config_generator/generate.py

Note

The --frozen argument is needed to avoid updating the lock file when running uv commands.

Running this single command accomplishes all of the following steps:

automatically detects and selects a suitable Python binary on the system if one already exists; otherwise, automatically downloads a suitable Python binary for subsequent (re)use.
- This supercedes Python binary management, such as with pyenv, system package managers, and manual installation, which is arguably the most frustrating part of maintaining reproducibility of Python scripts.
automatically creates a project virtual environment (.venv by default) for (re)use by all subsequent Python package operations and script execution, using the Python binary selected in the earlier step.
- This supercedes virtual environment management, such as with venv or poetry, but goes even further by properly handling Python binary version compatibility through (re)generation of the virtual environment according to the requested Python version (via -p/--python).
automatically resolves and installs script dependencies within the project virtual environment according to the project configuration file.
- This supercedes project-level package requirement specifications, such as with requirements.txt or pyproject.toml (Poetry), but goes even further by supporting resolution strategies to easily validate minimum (or maximum) dependency version requirements.

The package version requirements in this PR were obtained by initially using inline script metadata and --resolution lowest-direct for requirement validation before copying the dependencies into the pyproject.toml file, demonstrating the power of built-in Python binary management and Python-version-aware virtual environment management:

# Inline script metadata -> automatic isolated virtual environment.
# Straightforward Python (minimum) version testing with `-p`/`--python`.
uv run -p 3.10 --resolution lowest-direct path/to/script.py

Note

This behavior can be reproduced despite the presence of the project configuration file using a combination of --no-project, --isolated, and --with:

run_args=(
    --no-project # Avoid detecting project dependencies.
    --isolated   # Avoid reusing existing virtual environments.
    # Validate the following package version requirements are correct.
    --resolution lowest-direct
    # Package requirements to use only for the following command.
    --with 'shrub-py>=3.3.1,pydantic>=2.0,packaging>=14.0'
)
uv run "${run_args[@]:?}" .evergreen/config_generator/generate.py

Note

Inline script metadata is not used to specify dependencies due to isolated virtual environments interacting poorly with external tooling which expect a persistent virtual environment to be present. The dependencies are instead (redundantly) listed per associated script in the project configuration file, which may be used to generate a local and persistent virtual environment (.venv by default) using uv sync --frozen (or as part of a uv run --frozen command).

Note

Although testing with lowest would be ideal, the lowest-direct resolution was used instead due to insufficiently-specified requirements in transitive dependencies leading to unresolvable failures. This PR avoids introducing constraints or overrides to keep things simple.

Note

The clang_format.py script still requires Python 2. A requires-python was added to enforce this requirement (note uv does not support Python 2):

$ uv run etc/clang_format.py
Reading inline script metadata from `etc/clang_format.py`
error: No interpreter found for Python <3.0 in virtual environments, managed installations, or system path

$ uv run -p 2 etc/clang_format.py
Reading inline script metadata from `etc/clang_format.py`
error: Invalid version request: Python <3 is not supported but 2 was requested.

Updating this script to work with Python 3 is deferred, but would make a good case for using uv tool/uvx, which supercedes pipx, e.g. uvx clang-tools -t clang-format -i <version> -d build && ./build/clang-format --version.

Despite uv not yet supporting a stable API, I believe its powerful featureset (and already high popularity) makes it an very valuable tool which we can and should adopt in our projects. This PR hopes to be a leading test case for its eventual adoption by other projects as well.

EVG Config Generator Adjustments

Some adjustments were required to support the C++ Driver EVG config. These adjustments include:

Adding .evergreen to sys.path to permit relative imports of config_generator modules regardless of the current working directory.
Adding missing distros to distros.py and support for *-latest distros.
Adding VS 2019+ support to distros.py and extending compiler helpers to help specify corresponding CMake generators and platforms (e.g. vs2019x64 -> CMAKE_GENERATOR="Visual Studio 16 2019" + CMAKE_GENERATOR_PLATFORM="x64").
Adding support for the teardown_task_can_fail_task field for task groups.
Fixing pydantic 2.0 compatibility issues with subclass field behavior with serialize_as_any=True:

In V1, we would always include all fields from the subclass instance. In V2, when we dump a model, we only include the fields that are defined on the annotated type of the field. This helps prevent some accidental security bugs.

EVG Config Migration

Tip

Reviewing this PR by-commit is recommended to more easily compare how individual functions and components are migrated from the old config into their generator component form.

Most functions are translated as-is into their component form with minimal changes. Functions commonly reused by components are given explicit parameters in their call() functions to help with consistency and reduce verbosity (e.g. mongodb_version -> TOPOLOGY, polyfill -> BSONCXX_POLYFILL, etc.).

Scripts under .evergreen which are invoked by the EVG config are relocated into the .evergreen/scripts directory, formatted, given executable permissions, and audited with shellcheck (excluding the packaging-related scripts). Scripts under etc are left in their current location.

Some lessons learned during the C Driver's migration are applied in this PR. Unlike with the C Driver's generator components, the C++ Driver's components minimize cross-component inclusion and reuse (with the exclusion of function components). Despite leading to some repetition and verbosity across components, this is done to improve separation-of-concerns of components and matrices. In particular, despite its large matrix and complicated generation routine, I believe the single integration component is nevertheless more straightforward and understandable than the layered sasl -> cse -> asan/tsan components in the C Driver. Unlike with the C Driver, sanitizer and valgrind tasks are grouped into seperate, completely independent components.

Note

Although many tasks are grouped into a display task (e.g. auth-matrix, compile-only-matrix, etc.), this is not done for the integration matrix. This is to facilitate better filtering, selection, and sorting in Spruce, as such operations appear to be limited for members of a display task. Grouping by distro, by build type, and by server version/topology were all considered but reverted in favor of the current no-grouping state. This may be reconsidered if the Spruce UI is improved to better handle filtering/selecting/sorting members of display tasks: see DEVPROD-12402.

Some additional notes:

Some ARM64 distros had batchtimes, which does not appear to be necessary per EVG distro guidelines. These batchtimes were therefore removed.
Auth tests appear to be somewhat flaky, but it is unclear why. They appear to be passing for the moment.

Extra Alignment

Important

Adding support for configuring the auto-downloaded C Driver with ENABLE_EXTRA_ALIGNMENT=ON was considered but rejected to avoid encouraging use of extra alignment; see also CDRIVER-2813 and MONGOCRYPT-725.

Validating correct migration revealed that extra alignment tasks were broken by #967. The BSON_EXTRA_ALIGNMENT has no affect on fetch_c_driver_source nor the subsequent call to the compile EVG function (despite misleadingly being set for the compile function). Extra alignment was therefore not being enabled and tested with auto-downloaded C Driver configurations, as the auto-downloaded C Driver disables extra alignment by default. All current extra alignment tasks use the auto-downloaded C Driver, therefore extra alignment current has zero test coverage.

Fixing the enabling of extra alignment (by reverting relevant tasks to using install_c_driver with BSON_EXTRA_ALIGNMENT=ON instead of fetch_c_driver_source) revealed compiler errors due to -Werror + -Waligned-new (enabled by -Wall):

src/bsoncxx/include/bsoncxx/v_noabi/bsoncxx/stdx/make_unique.hpp: In substitution of 'template<class ... Args, class> static std::unique_ptr<bsoncxx::v_noabi::builder::core::impl> bsoncxx::v_noabi::stdx::detail::make_unique_impl<bsoncxx::v_noabi::builder::core::impl>::make<Args ..., <template-parameter-1-2> >(std::true_type, Args&& ...) [with Args = {long unsigned int}; <template-parameter-1-2> = <missing>]':
src/bsoncxx/include/bsoncxx/v_noabi/bsoncxx/stdx/make_unique.hpp:134:48:   required by substitution of 'template<class T, class Impl, typename std::enable_if<std::is_array< <template-parameter-1-1> >::value, decltype ((Impl::make(std::integral_constant<bool, true>{}, declval<long unsigned int>()), void()))>::type* <anonymous> > std::unique_ptr<T> bsoncxx::v_noabi::stdx::make_unique(std::size_t) [with T = bsoncxx::v_noabi::builder::core::impl; Impl = bsoncxx::v_noabi::stdx::detail::make_unique_impl<bsoncxx::v_noabi::builder::core::impl>; typename std::enable_if<std::is_array< <template-parameter-1-1> >::value, decltype ((Impl::make(std::integral_constant<bool, true>{}, declval<long unsigned int>()), void()))>::type* <anonymous> = <missing>]'
src/bsoncxx/lib/bsoncxx/v_noabi/bsoncxx/builder/core.cpp:259:45:   required from here
src/bsoncxx/include/bsoncxx/v_noabi/bsoncxx/stdx/make_unique.hpp:52:15: error: 'new' of type 'bsoncxx::v_noabi::builder::core::impl' with extended alignment 128 [-Werror=aligned-new=]
   52 |               typename = decltype(new T(std::declval<Args>()...))>
      |               ^~~~~~~~
src/bsoncxx/include/bsoncxx/v_noabi/bsoncxx/stdx/make_unique.hpp:52:15: note: uses 'void* operator new(std::size_t)', which does not have an alignment parameter
src/bsoncxx/include/bsoncxx/v_noabi/bsoncxx/stdx/make_unique.hpp:52:15: note: use '-faligned-new' to enable C++17 over-aligned new support

This appears to be due to #1049 which removed the -Wno-aligned-new flag "due to conflicts with Clang tasks + appearing to cause no trouble for existing tasks (not entirely clear what motivated its addition or if it still applies)". This PR being dated Nov 2023 likely explains why "no trouble" was observed following #967 on Aug 2023. The -Wno-aligned-new flag is restored with exclusions for Clang using the CC/CXX env vars.

eramongodb · 2024-10-29T21:21:50Z

Accidentally dropped extra alignment fixes for valgrind tasks during a rebase. Addressed and verified by this patch. Aside: libmongocrypt is disabled for sanitizer tasks for more accurate current-config reproduction, but we probably want to "revert" this and extend sanitizer tasks to cover CSFLE code paths soon.

eramongodb · 2024-10-30T16:51:21Z

Latest changes verified by this patch.

vector-of-bool

Love it. Some minor Python cleanups recommended.

I haven't used uv yet, but intend to get into it when I have time. Poetry was a significant improvement over pipenv (which was better than requirements.txt), but Poetry's implicit virtualenv management is so finicky and getting it to play with pyenv is a chore. If this works well, I would like to port the same changes over to C and libmongoc.

vector-of-bool · 2024-11-04T18:48:47Z

.evergreen/config_generator/components/atlas_search_indexes.py

+    @classmethod
+    def call(cls, **kwargs):
+        return cls.default_call(**kwargs)


Most versions of call are like this. Can it be lifted into the Function base class?

IIRC forcing the definition of call per class was meant to encourage evaluating meaningful parameters for each function, but I do not think that played out as intended. I'm in favor of the reduced boilerplate.

vector-of-bool · 2024-11-04T18:52:49Z

.evergreen/config_generator/components/clang_tidy.py

+    )
+
+    @classmethod
+    def call(cls, compiler: str, vars: Mapping[str, str] = {}):


Python pitfall (mis-feature from the 90s?): Don't use mutable values as default arguments. The value is evaluated and attached to the function definition, meaning that mutating it persists between function calls. Instead, use a None.

Suggested change

def call(cls, compiler: str, vars: Mapping[str, str] = {}):

def call(cls, compiler: str, vars: Mapping[str, str] | None = None):

vector-of-bool · 2024-11-04T18:55:16Z

.evergreen/config_generator/components/clang_tidy.py

+        vars = vars if vars else {}
+
+        vars |= compiler_to_vars(compiler)


The dict operator |= mutates the object in-place. If the caller passes in a dict, then this function will mutate their copy. Create a clone of the dict up-front:

Suggested change

vars = vars if vars else {}

vars |= compiler_to_vars(compiler)

vars = dict(vars or {})

vars |= compiler_to_vars(compiler)

Aside: I'd be surprised if the type checker doesn't flag this, because Mapping is an immutable type and does not have a |= operator defined (as opposed to MutableMapping)

TIL the dict or {} pattern (which does not evaluate to bool).

vector-of-bool · 2024-11-04T18:57:14Z

.evergreen/config_generator/etc/utils.py

+    with open(filename.resolve(), 'w', encoding='utf-8') as file:
+        file.write(yml)


pathlib.Path has a shorthand:

Suggested change

with open(filename.resolve(), 'w', encoding='utf-8') as file:

file.write(yml)

with filename.open('w', encoding='utf-8') as file:

file.write(yml)

Or, even shorter in this case:

Suggested change

with open(filename.resolve(), 'w', encoding='utf-8') as file:

file.write(yml)

filename.write_text(yml, encoding='utf-8')

vector-of-bool · 2024-11-04T18:59:10Z

.evergreen/config_generator/etc/utils.py

+        ]
+
+        ordered = {
+            field: mapping.pop(field) for field in before if field in mapping


Recommend copying mapping at the top of this func (i.e. mapping = mapping.copy()) before calling pop, otherwise this modifies the caller's value.

vector-of-bool · 2024-11-04T19:01:31Z

.evergreen/scripts/compile-scan-build.sh

+echo "Configuring with CMake flags:"
+for flag in "${cmake_flags[@]}"; do
+  echo " - ${flag:?}"
+done
+echo


Minor shorthand with printf:

Suggested change

echo "Configuring with CMake flags:"

for flag in "${cmake_flags[@]}"; do

echo " - ${flag:?}"

done

echo

echo "Configuring with CMake flags:"

printf " - %s\n" "${cmake_flags[@]}"

echo

I think my aversion to using printf was due to discrepancies in printf behavior across distros, but I do not recall the details (I might be conflating this with some other printf behavior), and a quick scan of EVG output across all three major OS's (Ubuntu, MacOS, Windows) appears to be just fine, so I will apply the pattern as suggested.

kevinAlbs

LGTM. The config generation, use of display tasks, and fixes to extra alignment tasks are very much appreciated.

kevinAlbs · 2024-10-30T19:02:31Z

.evergreen/config_generator/components/scan_build.py

+from config_generator.etc.utils import bash_exec
+
+from shrub.v3.evg_build_variant import BuildVariant, DisplayTask
+from shrub.v3.evg_command import EvgCommandType


Suggested change

from shrub.v3.evg_command import EvgCommandType

EvgCommandType already imported above.

kevinAlbs · 2024-11-01T16:43:04Z

.evergreen/config_generator/generate.py

@@ -0,0 +1,30 @@
+#!/usr/bin/env python3


Consider adding a README file (similar to the C driver README.md) or a comment to config.yml to document this script can be run through uv.

kevinAlbs · 2024-11-01T16:47:52Z

.evergreen/config_generator/components/sanitizers.py

+from config_generator.components.funcs.compile import Compile
+from config_generator.components.funcs.fetch_det import FetchDET
+from config_generator.components.funcs.install_c_driver import InstallCDriver
+from config_generator.components.funcs.install_c_driver import InstallCDriver


Suggested change

from config_generator.components.funcs.install_c_driver import InstallCDriver

kevinAlbs · 2024-11-01T16:48:06Z

.evergreen/config_generator/components/silk.py

+from config_generator.etc.utils import bash_exec
+
+from shrub.v3.evg_build_variant import BuildVariant
+from shrub.v3.evg_command import EvgCommandType


Suggested change

from shrub.v3.evg_command import EvgCommandType

kevinAlbs · 2024-11-01T16:48:16Z

.evergreen/config_generator/components/silk.py

+from config_generator.components.funcs.setup import Setup
+
+from config_generator.etc.distros import find_small_distro
+from config_generator.etc.function import Function


Suggested change

from config_generator.etc.function import Function

kevinAlbs · 2024-11-01T16:48:33Z

.evergreen/config_generator/components/valgrind.py

@@ -0,0 +1,113 @@
+from config_generator.components.funcs.compile import Compile
+from config_generator.components.funcs.fetch_c_driver_source import FetchCDriverSource


Suggested change

from config_generator.components.funcs.fetch_c_driver_source import FetchCDriverSource

kevinAlbs · 2024-11-04T15:20:38Z

.evergreen/config_generator/components/funcs/run_kms_servers.py

+                # Python: ImportError: DLL load failed while importing _rust: The specified procedure could not be found.
+                echo "Preparing CSFLE venv environment... skipped."
+                exit 0


Suggested change

# Python: ImportError: DLL load failed while importing _rust: The specified procedure could not be found.

echo "Preparing CSFLE venv environment... skipped."

exit 0

# Python: ImportError: DLL load failed while importing _rust: The specified procedure could not be found.

echo "Preparing CSFLE venv environment... skipped."

exit 0

To indent.

kevinAlbs · 2024-11-04T15:20:53Z

.evergreen/config_generator/components/funcs/run_kms_servers.py

+                # Python: ImportError: DLL load failed while importing _rust: The specified procedure could not be found.
+                echo "Starting mock KMS servers... skipped."
+                exit 0


Suggested change

# Python: ImportError: DLL load failed while importing _rust: The specified procedure could not be found.

echo "Starting mock KMS servers... skipped."

exit 0

# Python: ImportError: DLL load failed while importing _rust: The specified procedure could not be found.

echo "Starting mock KMS servers... skipped."

exit 0

To indent.

eramongodb added 30 commits October 29, 2024 15:28

uv: convert Python requirements into pyproject.toml

c0dd140

evg: move EVG files into evergreen-old

8529646

evg: update references to old EVG files

dff8771

evg: expand all YAML anchors

f301d2d

generator: import config_generator from C Driver

2651de8

generator: fix pydantic 2.0 compatibility for dumping subclass fields

e04fb80

generator: add support for C++ Driver EVG config distros

210740a

generator: add VS compiler -> CMake generator+platform helpers

2ade344

generator: add support for teardown_task_can_fail_task

d8c1150

evg: include generated functions.yml in legacy config

a6cffdf

functions: abi-compliance-check

c0faac8

functions: abidiff

cc2c7e6

functions: abi-prohibited-symbols

c556b5d

functions: setup

3c90e39

functions: build_mongohouse

faee0c1

functions: run_mongohouse

6a62698

functions: test_mongohouse

2cc3b4c

functions: start_mongod

92ded0f

functions: stop_mongod

980000a

functions: install_c_driver

86a48dc

functions: fetch_c_driver_source

08e0c4a

functions: lint

2518429

functions: clang-tidy

3c7952a

functions: fetch-det (clone_drivers-evergreen-tools)

f5404ac

functions: run_kms_servers

51787b2

functions: compile

ca454f5

functions: compile_benchmarks

db2e904

functions: test

acfb9fe

functions: test auth

19920bd

functions: test atlas connectivity

d1a5945

eramongodb added 3 commits October 29, 2024 16:01

etc: update release instructions with uv commands

9cd187a

generator: extend extra alignment fixes to valgrind tasks

85c01b6

generator: do not install libmongocrypt for sanitizer tasks

3bf5f97

kevinAlbs requested a review from vector-of-bool October 30, 2024 14:09

evg: backport root-level config options to .mci.yml

8f9aca2

vector-of-bool approved these changes Nov 4, 2024

View reviewed changes

kevinAlbs approved these changes Nov 7, 2024

View reviewed changes

eramongodb added 10 commits November 7, 2024 16:27

Use printf to print CMake arguments

e8ca6aa

Avoid splitting -D<arg> in CMake arguments list

080ac18

Avoid unintentionally modifying arguments of Python function calls

9fde835

Provide and use default call() in Function

efb5da3

Use Path.write_text() instead of open() + write()

2f3db7d

Fix Bash script indentation

d8bf224

Remove redundant imports

be97570

Add a README documenting the Evergreen config generator

a5bc5fd

README wording tweaks

6d6deba

Update README to account for uv lockfile updates

47b184d

eramongodb merged commit 954cfbe into mongodb:master Nov 8, 2024
15 of 19 checks passed

eramongodb deleted the cxx-3126 branch November 8, 2024 16:26

eramongodb mentioned this pull request Nov 8, 2024

CXX-3126 Remove obsolete .mci.yml #1261

Merged

kevinAlbs mentioned this pull request Nov 8, 2024

CXX-2797 Remove support for external polyfill libraries #1257

Merged

This was referenced Nov 12, 2024

evg: remove workaround for missing teardown_task_can_fail_task field #1267

Merged

CXX-2797 Update Augmented SBOM with mnmlstc/core removal #1271

Merged

eramongodb mentioned this pull request Nov 20, 2024

Fix runtime errors during execution of etc/make_release.py #1280

Merged

This was referenced Dec 3, 2024

CXX-3194 Migrate from macos-1100 to macos-14-arm64 #1293

Merged

CXX-3198 upgrade to ClangFormat 19 and replace clang_format.py with clang-format-all.sh #1299

Merged

eramongodb mentioned this pull request Apr 22, 2025

Use looseversion instead of deprecated distutils #1391

Merged

eramongodb mentioned this pull request Jun 13, 2025

CDRIVER-6042 Migrate Python scripts from Poetry to Astral UV mongodb/mongo-c-driver#2039

Open

	def call(cls, compiler: str, vars: Mapping[str, str] = {}):
	def call(cls, compiler: str, vars: Mapping[str, str] \| None = None):

		vars = vars if vars else {}

		vars \|= compiler_to_vars(compiler)

		with open(filename.resolve(), 'w', encoding='utf-8') as file:
		file.write(yml)

	with open(filename.resolve(), 'w', encoding='utf-8') as file:
	file.write(yml)
	filename.write_text(yml, encoding='utf-8')

		@@ -0,0 +1,113 @@
		from config_generator.components.funcs.compile import Compile
		from config_generator.components.funcs.fetch_c_driver_source import FetchCDriverSource

CXX-3126 Refactor EVG config to use config_generator #1244

CXX-3126 Refactor EVG config to use config_generator #1244

Uh oh!

Conversation

eramongodb commented Oct 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Astral uv

EVG Config Generator Adjustments

EVG Config Migration

Extra Alignment

Uh oh!

eramongodb commented Oct 29, 2024

Uh oh!

eramongodb commented Oct 30, 2024

Uh oh!

vector-of-bool left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kevinAlbs left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

eramongodb commented Oct 29, 2024 •

edited

Loading