Unexpected PyPy wheel for Linux aarch ABI3 build

[linhr]

I'm using Maturin to release ABI3 wheels. Here is the corresponding GitHub Actions workflow file:

https://github.com/lakehq/sail/blob/611daf6b92433eaedb428ab4cbf2679241bacaf0/.github/workflows/release.yml#L186

Previously (v1.9.6), for the aarch64 target on Linux, a single wheel tagged cp38-abi3-manylinux_2_24_aarch64 was generated.

This week, using the latest version (v1.10.1), besides the expected wheel, another wheel tagged pp39-pypy39_pp73-manylinux_2_24_aarch64 was generated.

In my setup, the Linux aarch64 build runs inside containers on x86_64 hosts.

Here is a comparison of the published wheel list:

• https://pypi.org/project/pysail/0.4.1/#files
• https://pypi.org/project/pysail/0.4.2/#files

Is this something that could be fixed?

(I skimmed through the changelog and this change seems most relevant to me: #2829. But I don't have a clear story of what's going on here.)

[messense]

Seems related to #2829.

[linhr]

Is this something that can be fixed, or is there any workaround for this?

(cc @freakboy3742 who might have more context for #2829)

[freakboy3742]

I'm currently on leave, so I can't do a deep dive right now. However, the motivation for the change was that if you have an interpreter that can be asked for platform details, that interpreter should be used, rather than a "faked" interpreter.

It sounds like the case being reported is a cross-platform build, so it should be falling back to the original logic. I'm not sure if thats because of a fault in the logic in the patch from #2829, or because there's something unusual about the cross-platform+pypy case.

I'll be back in my office in early January, but I don't have any experience with cross-platform manylinux builds, so I don't know what I'll be able to add there.

[linhr]

Thanks for the quick response!

Yeah this seems a cross-platform build (inside container). (I'm using Maturin in GitHub Actions. I'd imagine if we pull the Docker images used by the action, we could see the installed interpreters and the issue can hopefully be reproduced.)

[freakboy3742]

Apologies for the delay; I've taken a closer look at this, and I'm not sure I see where the logic has gone wrong for your case (or, at least, it seems like there's something else going on that I'm not really in a position to diagnose).

The biggest thing that seems odd to me is that you're saying two wheels are being produced, one of which is a pypy wheel. That suggests to me that the issue isn't with maturin, but maturin-action, and that action is picking up pypy as a valid target (and thus building a wheel).

FWIW, there does appear to be some similarity to what has been reported by PyO3/maturin-action#364.

[linhr]

Yeah I skimmed through #2829 and haven't found clues either.

I looked for the log for the previous run (before this issue occurred) and fortunately it's still there for now: https://github.com/lakehq/sail/actions/runs/19007703673/job/54284116426

🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings with abi3 support
🐍 Not using a specific python interpreter
📡 Using build options features from pyproject.toml
...
📦 Built wheel for abi3 Python ≥ 3.8 to dist/pysail-0.4.1-cp38-abi3-manylinux_2_24_aarch64.whl

And here is the log after this issue was noticed: https://github.com/lakehq/sail/actions/runs/19324040933/job/55271018623

🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings with abi3 support
📡 Using build options features from pyproject.toml
...
📦 Built wheel for abi3 Python ≥ 3.8 to dist/pysail-0.4.2-cp38-abi3-manylinux_2_24_aarch64.whl
⚠️ Warning: PyPy 3.9 does not yet support abi3 so the build artifacts will be version-specific.
...
📦 Built wheel for PyPy 3.9 to dist/pysail-0.4.2-pp39-pypy39_pp73-manylinux_2_24_aarch64.whl

[linhr]

Here are the log archives in case the GitHub Actions history gets cleaned up.

logs_48943531250.zip
logs_49800833772.zip

[linhr]

Ok I looked at the code carefully again and I feel that the behavior change was indeed caused by #2829. The PR changed the order of checking various conditions (notably self.interpreter.is_empty() and target.cross_compiling()), so the found interpreters instead of a dummy interpreter is being used for the cross-compilation case.

If #2829 is not meant to change the cross-compilation use case, would it be possible to revert to the original behavior? Or if we decide that this is not necessarily a regression, is there a workaround to accommodate this new behavior? (I guess the user may need to specify the interpreter explicitly?)

[freakboy3742]

I'm sure there are ways to re-arrange the order of evaluation, but it's not as simple as just reverting, because the old behavior was demonstrably incorrect. It was falling to the dummy interpreter construction in every case where an interpreter wasn't being explicitly specified. A dummy interpreter definition should be the last thing found, not the first.

My question for your debugging is why is the pypy interpreter being found in the environment at all? If the environment that is triggering maturin isn't a pypy environment, why is a pypy interpreter being found?

[messense]

It could be that there is a python3 executable in PATH which is actually a pypy interpreter, depending on the order of PATH

pypy3.11-v7.3.20-linux64/bin/
pypy3.11-v7.3.20-linux64/bin/libpypy3.11-c.so
pypy3.11-v7.3.20-linux64/bin/python3.11
pypy3.11-v7.3.20-linux64/bin/python
pypy3.11-v7.3.20-linux64/bin/pypy3.11
pypy3.11-v7.3.20-linux64/bin/pypy3.11.debug
pypy3.11-v7.3.20-linux64/bin/python3
pypy3.11-v7.3.20-linux64/bin/pypy3
pypy3.11-v7.3.20-linux64/bin/libpypy3.11-c.so.debug
pypy3.11-v7.3.20-linux64/bin/pypy

[freakboy3742]

Yes - that's the sort of thing that I was guessing might be happening. python3 exists on the path - so there's a valid python3 interpreter being discovered. To my mind, if the environment where you're running a Python build process isn't returning a consistent (or compatible) python3 then it's the build environment that is the problem, not maturin.

[linhr]

The Docker image in use for this case is messense/manylinux_2_24-cross:aarch64. I guess the find_interpreter_in_host logic detects that both interpreters (CPython and PyPy) are available?

If I follow the code paths correctly, it seems that PyPy will be considered when finding interpreters, even for the ABI3 case:

maturin/src/python_interpreter/mod.rs

Lines 751 to 764 in a733c9d

	// Also try to find PyPy for cffi and pyo3 bindings
	if *bridge == BridgeModel::Cffi || bridge.is_pyo3() {
	executables.extend(
	(bridge.minimal_pypy_minor_version()..=bridge.maximum_pypy_minor_version())
	.filter(|minor| {
	requires_python
	.map(|requires_python| {
	requires_python.contains(&Version::new([3, *minor as u64]))
	})
	.unwrap_or(true)
	})
	.map(|minor| format!("pypy3.{minor}")),
	);
	}

[messense]

Note that manylinux_2_24 is already deprecated so the docker image hasn't been updated for a long time, consider upgrade to manylinux_2_28 instead.

it seems that PyPy will be considered when finding interpreters, even for the ABI3 case:

I remember because PyPy does not support abi3 so we choose to build a version specific wheel for it.