refactor: unified interpreter resolution pipeline#3032
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR consolidates the scattered Python interpreter discovery, validation, and filtering logic into a unified, single-pass pipeline. The refactoring reduces build_options.rs from ~1400 lines while improving maintainability, testability, and fixing several bugs.
Changes:
- Introduced
InterpreterResolverwith a singleresolve()entry point that replaces 6+ overlapping free functions - Extracted interpreter discovery logic into
python_interpreter/discovery.rsmodule - Extracted ABI flag logic into
python_interpreter/abiflags.rsmodule - Extracted bridge detection into
bridge/detection.rsmodule - Fixed multiple bugs: Windows Python 3.14+ ABIFLAGS handling, interpreter deduplication, pyenv fallback, abi3 filtering for free-threaded/PyPy interpreters, and VIRTUAL_ENV python path resolution
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
src/python_interpreter/resolver.rs |
New unified interpreter resolution pipeline with InterpreterResolver struct (987 lines, 11 unit tests) |
src/python_interpreter/discovery.rs |
Extracted interpreter discovery functions with WindowsInterpreterFinder struct replacing macro (1011 lines, 5 tests) |
src/python_interpreter/abiflags.rs |
Extracted ABI flag resolution and tag calculation logic (164 lines, 1 test) |
src/python_interpreter/config.rs |
Added soabi_from_ext_suffix() helper method |
src/python_interpreter/mod.rs |
Module reorganization with delegation methods for public API |
src/bridge/detection.rs |
Extracted bridge detection logic with find_bridge() and related functions (332 lines) |
src/bridge/mod.rs |
Added module structure for bridge detection |
src/build_options.rs |
Simplified by removing ~600 lines of interpreter/bridge logic and extracting helper functions |
src/target/mod.rs |
Enhanced VIRTUAL_ENV handling to use venv python directly |
src/main.rs |
Updated import paths for PythonInterpreter |
src/ci.rs |
Updated bridge detection function calls |
tests/common/integration.rs |
Updated PythonInterpreter import path |
messense
force-pushed
the
refactor/interpreter-resolver
branch
6 times, most recently
from
ae2a19d to
95d1b51
Compare
Consolidate the 6+ overlapping interpreter resolution functions (find_interpreters, resolve_interpreters, find_interpreter, find_interpreter_in_host, find_interpreter_in_sysconfig, find_single_python_interpreter) into a single InterpreterResolver struct in src/python_interpreter/resolver.rs. The resolver provides a unified resolve() entry point that handles all combinations of abi3/non-abi3, cross-compile/native, and Windows/Unix. This eliminates the deeply nested match arms and duplicated fake interpreter construction (previously 5 copies) from build_options.rs. The helper functions are preserved as module-private functions in the resolver module for internal reuse.
Addresses PyO3#2751, PyO3#2312: - find_all_windows: Track seen executable paths (via canonical path) to avoid processing the same interpreter twice when found via multiple discovery mechanisms (py launcher + pythonX.Y alias). Also key versions_found by (major, minor, gil_disabled) instead of just (major, minor) to properly distinguish free-threaded builds. - find_all (Unix): Add deduplication by (kind, major, minor, gil_disabled) to prevent duplicate entries. - find_all (Unix): Add fallback to `python3` and `python` names when no versioned interpreters are found, helping pyenv environments where only the generic shim is available. - Derive Hash on InterpreterKind to support using it as HashSet key.
Fixes PyO3#2740. Python 3.14+ on Windows now defines ABIFLAGS in sysconfig (upstream change). The previous code rejected any non-empty ABIFLAGS on Windows for Python < 3.14, and for 3.14+ it fell through to the Unix code path which expected ABIFLAGS to always be defined -- causing a confusing error. Restructure fun_with_abiflags() to handle Windows as a single unified branch: - Python <= 3.7: return "m" (historical) - Python 3.8-3.12: return "" (empty) - Python 3.13+: handle gil_disabled -> "t" - Python 3.14+: accept ABIFLAGS from sysconfig when defined Add test covering Python 3.14 and 3.14t on Windows.
Addresses PyO3#2772, PyO3#2607, PyO3#2852: - Add filter_for_abi3() that partitions interpreters into abi3-capable and non-abi3-capable groups. Non-abi3 interpreters (PyPy, free- threaded CPython) are only included when explicitly requested via the -i flag. This prevents: - Free-threaded CPython being chosen over regular CPython (PyO3#2772) - Unexpected PyPy wheels in abi3 cross-compile builds (PyO3#2852) - Honor user's explicit -i pypy3.X request even with abi3 (PyO3#2607): PyPy interpreters are preserved when the user specifically requests them, producing version-specific PyPy wheels alongside abi3 wheels. - In resolve_abi3_cross_compile(), only include PyPy when explicitly requested, preventing auto-discovered PyPy from generating unwanted wheels in cross-compile abi3 scenarios.
Addresses PyO3#2198. When VIRTUAL_ENV is set, Target::get_python() now returns the full path to the venv's python executable (via get_venv_python) instead of just "python" (Unix) or "python.exe" (Windows). This ensures the correct python version is used, avoiding cases where PATH resolution finds a different python version than the one the virtualenv was created with. This is particularly important in environments with multiple python versions (nix, pyenv) where the generic 'python' name may resolve to a different version than the venv expects.
- Extract user_requested_pypy() and user_requested_free_threaded() helpers to deduplicate detection logic between filter_for_abi3() and resolve_abi3_cross_compile() - Remove dead code in filter_for_abi3() (unreachable empty result debug log) - Consolidate discover_interpreters() by inlining the auto_discover, find_specified_interpreters, and find_specified_interpreters_from methods into a single method - Make struct fields pub(crate) instead of pub - Make find_interpreter_in_sysconfig and soabi_from_ext_suffix private (not used outside the resolver module)
Introduce Candidate + CandidateSource types and restructure the resolver
into a single pipeline that handles all combinations:
PYO3_CONFIG_FILE → discover_candidates → filter_for_abi3 → finalize
Key changes:
- PYO3_CONFIG_FILE is now checked first for ALL paths (previously missed
for abi3-py{version} on non-Windows)
- Discovery is organized by source: cross_lib_dir, cross_sysconfig, native
- Abi3 filtering is a distinct pipeline stage, not interleaved with discovery
- Native discovery for abi3 has graceful sysconfig fallback on error
- Collapsed resolve_pyo3_no_fixed_abi3/resolve_pyo3_abi3 into resolve_pyo3
- Collapsed resolve_cross_compile/resolve_cross_no_lib_dir into discovery
- Collapsed resolve_abi3_windows/resolve_abi3_cross_compile into finalize
Methods removed (replaced by pipeline stages):
- resolve_pyo3_no_fixed_abi3 → resolve_pyo3 pipeline
- resolve_pyo3_abi3 → resolve_pyo3 pipeline
- resolve_cross_compile → discover_from_cross_lib_dir
- resolve_cross_no_lib_dir → discover_cross_sysconfig
- resolve_abi3_windows → finalize_abi3_windows
- resolve_abi3_cross_compile → handle_abi3_cross
- try_find_host_interpreters → discover_native (with fallback)
- discover_interpreters → find_native_interpreters
The resolver now returns a ResolveResult containing both the resolved interpreters and the optional host_python path discovered during cross-compilation. The caller (resolve_interpreters in build_options.rs) is responsible for setting PYO3_PYTHON and PYTHON_SYS_EXECUTABLE. This separates concerns: the resolver discovers interpreters, while environment setup for the build is handled at the build boundary.
- Inline find_interpreter_in_host into find_host_python (sole caller) - Remove duplicated default-python logic from build_options.rs (resolver already handles PYO3_PYTHON / target.get_python() fallback) - Simplify resolve_interpreters to just pass raw user inputs to resolver - MATURIN_TEST_PYTHON override kept in build_options.rs for test compat
…iant - Remove CandidateSource::ConfigFile (PYO3_CONFIG_FILE returns before any Candidate is created, so this variant was unreachable) - Replace print_found with print_found_candidates that reads source to skip CrossCompileLib/Placeholder candidates (they already have their own messages printed during discovery/finalization) - Eliminates all allow(dead_code) suppressions
1. Remove Candidate named constructors, use struct literals directly (less indirection, clearer at call sites) 2. interpreter_from_sysconfigdata: return single PythonInterpreter instead of Vec (it always produced exactly one) 3. Inline find_interpreter() free function into find_native_interpreters() (sole caller, resolver owns all needed context) 4. Fix user_requested_free_threaded: check file_name() component only and reuse maybe_free_threaded() parser instead of fragile suffix heuristic that could match paths like /usr/local/python3.13/latest 5. Fix /// doc comment on nested fn mark_seen in mod.rs to // regular comment (doc comments on private inner functions are misleading) 6. Remove bail from handle_abi3_cross on empty: finalize_abi3 already handles the empty case (fallback to placeholder), so the bail was making that fallback unreachable for cross-compile + abi3
Address further review suggestions: 1. Inline find_single_python_interpreter into resolve_single (sole caller, resolver owns all needed context) 2. Move find_interpreter_in_sysconfig into resolver as find_in_sysconfig method. Call sites simplify from find_interpreter_in_sysconfig(self.bridge, &x, self.target, self.requires_python) to self.find_in_sysconfig(&x) 3. Extract InterpreterSpec for interpreter string parsing. The if/else chain that parsed 'pypy3.11', 'python3.14t', '3.9' etc. is now a self-contained type with parse() and try_parse_filename() methods. user_requested_pypy/user_requested_free_threaded now reuse the same parser instead of ad-hoc string matching. 4. Move soabi_from_ext_suffix to InterpreterConfig as an associated fn (pure function deriving a property of an interpreter config) 5. Move to_candidates into Candidate::from_interpreters as an associated fn All freestanding functions in resolver.rs are eliminated — the module is now fully self-contained in the InterpreterResolver impl block.
mod.rs was 1456 lines mixing struct definitions, discovery logic, platform heuristics, wheel tag generation, and tests. Split into: - abiflags.rs (164 lines): fun_with_abiflags + calculate_abi_tag Self-contained platform-specific ABI flag resolution. - discovery.rs (1046 lines): find_all_windows, find_all (Unix), check_executable, check_executables, find_by_target, run_script, from_metadata_message, InterpreterMetadataMessage. All interpreter discovery and validation logic. - mod.rs (331 lines): PythonInterpreter struct, InterpreterKind enum, Deref/Display impls, get_tag, get_library_name, from_config, and other data-oriented methods. Also replaces the maybe_add_interp! macro in find_all_windows with a WindowsInterpreterFinder struct that makes state explicit (versions_found, seen_executables, interpreters) instead of capturing 5 locals by reference. Tests moved to their respective modules.
…ld() C: Extract bridge/cargo detection from build_options.rs into bridge/detection.rs Convert bridge.rs to a bridge/ module directory and move detection functions there: - find_bridge (pub) - determines BridgeModel from cargo metadata - is_generating_import_lib (pub) - checks pyo3 generate-import-lib feature - has_abi3 - checks abi3 feature flags - find_pyo3_bindings - detects pyo3/pyo3-ffi bindings - current_crate_dependencies - walks the dependency graph - PYO3_BINDING_CRATES constant bridge/mod.rs keeps the type definitions (BridgeModel, PyO3, etc.). D: Break up BuildContextBuilder::build() (~317 lines) Extract two self-contained functions: - resolve_target() - target triple / ARCHFLAGS / universal2 / cross detection (~55 lines) - resolve_platform_tags() - platform tag resolution from CLI flags, pyproject.toml, zig, musl defaults (~50 lines) build() is now ~220 lines with clearer flow: project → bridge → target → interpreters → platform_tags → context E: Discovery statics (partial) Discovery methods (find_all, check_executable, etc.) already live in python_interpreter/discovery.rs from the previous commit. They remain as PythonInterpreter:: associated fns since changing the call syntax (used in develop.rs, main.rs, resolver.rs) would be a larger API change with diminishing returns.
Convert check_executable, check_executables, find_all, and find_by_target from PythonInterpreter associated functions to free functions in the python_interpreter::discovery module, making PythonInterpreter a pure data type with only instance methods (run_script, from_config, get_tag, etc.) and formatting. The free functions are re-exported from python_interpreter::mod.rs and the module is made pub so callers can use python_interpreter::find_all() etc. directly. from_metadata_message is kept as a private helper in discovery.rs (only used by check_executable). Call site changes: - PythonInterpreter::check_executable(...) → python_interpreter::check_executable(...) in develop.rs, main.rs, tests/ - PythonInterpreter::find_all(...) → super::find_all(...) in resolver.rs - PythonInterpreter::find_by_target(...) → super::find_by_target(...) in resolver.rs - Internal calls within discovery.rs simplified (no Self:: prefix needed)
pyo3_features_from_conditional extracts pyo3/pyo3-ffi feature names from pyproject.toml conditional features and its sole consumer is find_bridge → has_abi3. Move it into bridge/detection.rs as a private helper and have find_bridge call it internally. This changes find_bridge's third parameter from `&HashMap<&str, Vec<String>>` to `Option<&PyProjectToml>`, letting callers (build_options.rs, ci.rs) pass the pyproject directly instead of performing a two-step extract-then-pass dance.
Cover all supported formats (python3.14t, pypy3.11, graalpy-3.10, 3.9), version-less names (returns None), error cases (free-threaded PyPy/GraalPy, too-old free-threaded version, unsupported interpreter names), and try_parse_filename behavior (returns None instead of Err).
The placeholder concept belongs to the PythonInterpreter type, not the resolver. Move it to an associated function on PythonInterpreter that takes (major, minor, target) — the target is only needed to decide the ext_suffix (.pyd on Windows, empty otherwise).
DiscoveryResult had no behavior — just two fields used as a return type. Replace it with a type alias for (Vec<Candidate>, Option<PythonInterpreter>) to reduce boilerplate in discovery methods.
The bridge parameter flows through check_executable → from_metadata_message → fun_with_abiflags for a single check: skipping the platform-system mismatch validation for cffi bindings. Document this coupling in the doc comment rather than refactoring it away, since the check is a legitimate safety guard.
…section comments Move check_executable, check_executables, find_all, and lookup_target back to associated functions on PythonInterpreter (delegating to the discovery module internally). This makes python_interpreter module private again — callers use PythonInterpreter::find_all() etc. Also remove all section separator comments (dashed-line banners) throughout the refactored modules to match project style.
When the user explicitly provides interpreters (via -i or develop passing a venv python), skip the abi3 filter that excludes PyPy and free-threaded CPython. The venv python filename is just 'python', so user_requested_pypy()/user_requested_free_threaded() can't detect the interpreter kind from it, causing the filter to incorrectly remove the only available interpreter. This fixes develop_pyo3_ffi_pure failures on 3.14t and pypy3.11 in CI.
When --find-interpreter is used and find_all returns an empty list (e.g. no python3.X/python3/python on PATH), bail with a clear error instead of silently continuing with no interpreters.
Inlines find_native_interpreters into discover_native, eliminating the error-as-control-flow pattern where one function bailed specifically for the other to catch. The merged function has linear control flow: 1. Try to find real interpreters (find_all or find_specified_interpreters) 2. If found, return immediately 3. If not and abi3, try broader sysconfig fallback 4. If still nothing, bail with an appropriate error message The per-interpreter sysconfig fallback for user-specified interpreters is extracted to find_specified_interpreters, which returns Ok(vec![]) when nothing is found instead of bailing.
run_script is a general-purpose method on PythonInterpreter, not a discovery operation. Move it to mod.rs where the type is defined.
…rors InterpreterSpec::parse is used from both native and cross-compile paths (via find_in_sysconfig), so its error messages should not mention cross-compiling.
…ames parse() is used from both native and cross-compile paths. In native builds, file paths like '/usr/bin/python3' are valid -i inputs handled by check_executable — parse() should skip them (return None) not error. The cross-compile path (discover_cross_sysconfig) already validates paths before calling find_in_sysconfig, so the error in parse was redundant there too.
parse() now returns None for version strings that don't start with a digit (e.g. 'python.exe' → '.exe') or lack a minor version (e.g. 'python3' → '3'). Previously these would cause parse errors like 'Invalid python interpreter major version' when falling back to sysconfig lookup. Also removes the 'Cross-compiling is poorly supported' warning.
When the setup-python action is used, it sets the pythonLocation environment variable pointing to the Python installation directory. Use this to resolve the exact Python binary rather than relying on PATH resolution, similar to how VIRTUAL_ENV is handled.
Apply filter_for_abi3 whenever interpreters weren't explicitly provided by the user via -i. Previously the filter only ran with --find-interpreter, but the default python (no flags, no -i) could also pick up free-threaded or PyPy interpreters that shouldn't be used for abi3 builds. Skip the filter only when user_interpreters is non-empty and find_interpreter is false — that's the explicit -i case (including develop passing a venv python).
- Fix panic in calculate_abi_tag when soabi has no dashes (e.g. bare 'pyston'): return None instead of unwrapping None from nth(1) - Add warning when version-less interpreter specs (e.g. bare 'pypy') are silently skipped in find_in_sysconfig - Fix user_requested_pypy to recognize version-less paths like /usr/bin/pypy via prefix match fallback, preventing incorrect filtering during abi3 builds - Add test case for dashless soabi edge case
…fied interpreters When the user explicitly passes interpreters via -i, propagate errors from find_in_sysconfig instead of swallowing them with unwrap_or_default. This surfaces actionable diagnostics like 'Free-threaded Python is only supported on 3.13+' instead of a generic 'couldn't find' message. Auto-discovery (no -i) still swallows errors to fall through to placeholder/error handling.
fun_with_abiflags only checked for "graalvm" while from_metadata_message already accepted both "graalvm" and "graalpy". If GraalPy ever reports "graalpy" as platform.python_implementation(), abiflags resolution would incorrectly fall through to the Unix ABIFLAGS check and bail because GraalPy doesn't set ABIFLAGS.
messense
force-pushed
the
refactor/interpreter-resolver
branch
from
95d1b51 to
d65a57b
Compare
- Fix InterpreterSpec::parse doc: returns Ok(None), not Err, for paths - Use file_name() instead of display() in find_in_sysconfig to correctly parse path-like interpreter inputs and improve warning messages - Add mark_seen guard for Microsoft Store alias fallback to avoid duplicate processing - Update get_python doc comment to reflect actual precedence: VIRTUAL_ENV -> pythonLocation -> fallback
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Consolidates the scattered interpreter discovery, validation, and filtering logic into a clean, single-pass pipeline. Previously,
build_options.rsalone was ~1400 lines with interleaved interpreter resolution, bridge detection, target resolution, and build context construction spread across 6+ overlapping functions with separate native/cross-compile codepaths.What changed
New
InterpreterResolver(resolver.rs)resolve()entry point replaces 6+ overlapping free functionsresolve()→discover_candidates()→filter_for_abi3()→finalize()Candidate+CandidateSourcetypes track how each interpreter was found (executable, sysconfig, cross-lib, placeholder)InterpreterSpecparser handles"python3.14t","pypy3.11","3.9"etc. with proper validation (replaces ad-hoc string matching)Split
python_interpreter/mod.rsdiscovery.rs:check_executable,find_all,lookup_target,check_executables;WindowsInterpreterFinderstruct replacesmaybe_add_interp!macroabiflags.rs:fun_with_abiflags+calculate_abi_tagmod.rsretains:PythonInterpreterstruct,InterpreterKind, formatting,get_tag,from_config,placeholder()PythonInterpreterassociated functions delegating todiscovery.rs— module stays privateExtracted bridge detection (
bridge/detection.rs)find_bridge,is_generating_import_lib,has_abi3,find_pyo3_bindings,current_crate_dependenciesmoved frombuild_options.rspyo3_features_from_conditionalabsorbed intofind_bridge— callers passOption<&PyProjectToml>directlyCleaned up
build_options.rsresolve_target()andresolve_platform_tags()frombuild()PYO3_PYTHON/PYTHON_SYS_EXECUTABLEenv-var setting at the call siteBug fixes included
find_alldeduplicates by(kind, major, minor, gil_disabled)to avoid building duplicate wheelspython3/pythonwhen versioned binaries aren't available-iPYO3_CONFIG_FILEchecked for all paths: previously missed forabi3-py{version}on non-WindowsFixes #2740
Fixes #2772
Fixes #2852
Fixes #2607
Fixes #2312
Fixes #2751
Fixes #2198