Explore support for abi3.abi3t wheels (PEP 803)

[ngoldbaum]

I'm working on enabling support for a free-threaded stable ABI. See Quansight-Labs/free-threaded-compatibility#310 for more detail about this work.

Right now the most concrete path forward towards that is getting PEP 803 approved. Towards that end, I'm working on a set of end-to-end tests for PEP 803 using experimental support available for opaque PyObject builds in CPython 3.15 alphas and patched versions of packaging tools.

On the Rust end of things, I've been working on this in PyO3 and currently have a branch that can build abi3 extensions on the free-threaded build. The next step is to update maturin to support building abi3.abi3t wheels.

My plan is to open a draft pull request implementing this. For now I'm going to set it up so a free-threaded interpreter is needed to produce an abi3t.abi3 wheel, but in principle cross-compiling should also work so long as the cross-compile build config sets both Py_LIMITED_API and Py_GIL_DISABLED. PEP 803 also proposes a new Py_TARGET_ABI3T macro to control this directly without overloading Py_LIMITED_API and Py_GIL_DISABLED, but that is not yet implemented in CPython so it's not straightforward to test.

[ngoldbaum]

As a first attempt, I'm going to try starting with the following change to the Abi3Version enum:

diff --git a/src/bridge/mod.rs b/src/bridge/mod.rs
index 241f74aa..96a5ad73 100644
--- a/src/bridge/mod.rs
+++ b/src/bridge/mod.rs
@@ -114,16 +114,25 @@ impl TryFrom<PyO3MetadataRaw> for PyO3Metadata {
     }
 }

-/// Python version to use as the abi3 target.
+/// Python version to use as the abi3/abi3t target.
 #[derive(Clone, Debug, PartialEq, Eq)]
-pub enum Abi3Version {
-    /// abi3 wheels will have a minimum Python version matching the version of
-    /// the current Python interpreter
+pub enum StableAbiVersion {
+    /// Stable ABI wheels will have a minimum Python version matching the
+    /// version of the current Python interpreter
     CurrentPython,
-    /// abi3 wheels will have a fixed minimum Python version
+    /// Stable ABI wheels will have a fixed user-specified minimum Python
+    /// version
     Version(u8, u8),
 }

+#[derive(Clone, Debug, PartialEq, Eq)]
+pub enum StableAbiKind {
+    /// The original stable ABI, supporting Python 3.2 and up
+    Abi3,
+    /// The free-threaded stable ABI, supporting Python 3.15 and up
+    Abi3t,
+}
+
 /// The name and version of the pyo3 bindings crate
 #[derive(Clone, Debug, PartialEq, Eq)]
 pub struct PyO3 {
@@ -132,7 +141,7 @@ pub struct PyO3 {
     /// pyo3 bindings crate version
     pub version: semver::Version,
     /// abi3 support
-    pub abi3: Option<Abi3Version>,
+    pub abi3: Option<(StableAbiVersion, StableAbiKind)>,
     /// pyo3 metadata
     pub metadata: Option<PyO3Metadata>,
 }

We'll see if that ends up being workable.

One thing that's not clear to me: does maturin have any kind of API stability for its Rust API? Said another way: does it matter that Abi3Version is pub, or can I not do refactors like this without considering backward compatibility?

[ngoldbaum]

I think it'll probably be easier to implement this if I add an explicit abi3t feature that people can opt into in PyO3. Right now maturin doesn't know anything about the "host" interpreter when it builds, so it can't easily check ABI tags to get a default behavior.

What I can do though is make turning on the abi3 feature on the free-threaded build fail over (with a warning) to turning on the abi3t feature. Builds based on a GIL-enabled interpreter would need to opt in to an abi3t build, which is also probably the behavior we want.

[ngoldbaum]

I have an experimental branch that I'm able to build abi3t wheels using: main...ngoldbaum:maturin:abi3t

[messense]

Feel free to send a draft PR!

[ngoldbaum]

I think I'll hold off because it requires CPython and PyO3 patches to be viable.

Instead I think I'm going to extract the pure refactoring bits of my branch and try to upstream that while we wait on the steering council decision on PEP 803. I like having the enums to describe ABI options instead of bools so I think it's a worthwhile refactoring independent of upstream decisions for stable ABI support.

[ngoldbaum]

PEP 803 is in but before pyo3 can add support we'll also need to close out the discussion about adding bindings for the critical section API to the stable ABI. That means work here and in pyo3 is on hold for now.

@messense, how would you like me
to handle testing when I eventually open a PR to enable abi3t support? Just follow the existing tests and example
projects for abi3 support?

Is there any kind of testing that uses multiple Python interpreters? It might be neat to have a test that builds a wheel using a GIL-enabled interpreter and runs tests in a free-threaded interpreter and vice verse. Maybe with a test that includes a mutable pyclass that needs special handling to avoid runtime borrow checker errors on the free-threaded build.