fix: handle implementations without subinterpreter support #5732

msimacek · 2025-06-17T14:45:13Z

We're working on updating GraalPy to 3.12, but we're currently not planning to add this internal module yet, so the few tests using it should get skipped. (I'm doing this now because we run pybind11 tests from master in our CI)

Suggested changelog entry:

Support implementations without subinterpreter support

📚 Documentation preview 📚: https://pybind11--5732.org.readthedocs.build/

henryiii · 2025-06-17T17:50:06Z

Would it be better to gate this as cpython only? It's not a public API until 3.14 lands concurrent.interpreters and concurrent.futures.InterpreterPoolExecutor.

henryiii · 2025-06-17T17:52:30Z

tests/test_multiple_interpreters.py

    if sys.version_info >= (3, 15):
        import interpreters


When accepting the PEP the steering council made this concurrent.interpreters, and it's been given an exception to land in 3.14.0b3. Not sure there's any action worth taking yet, once b3 is out we can just drop support for b2, I assume. Unless we want to go all in on importorskip. Normally not a fan of importorskip for version dependent things, but maybe we should go that way instead.

b3 just got tagged, so I'm going to see if I can test this locally on my build.

henryiii · 2025-06-17T18:56:35Z

There's also sys.implementation.supports_isolated_interpreters, though that might be 3.14+, since it's in the PEP. Edit: is this missing, actually, in beta 3? Opened python/cpython#135645 to ask.

henryiii · 2025-06-17T19:51:57Z

@ericsnowcurrently, would it make sense for Interpreter to support context managers? I can wrap it in contextlib.closing, but it seems somewhat natural if it supported it directly.

henryiii · 2025-06-17T20:15:16Z

(@b-pass, perhaps?) I'm trying to use the new module. Do you know how to replace interpreter.run_string(interp, code, shared={"pipeo": pipeo}) with Interpreter methods? (Specifically, I don't know what to do with the shared= part)

Edit: I think I got it. I'm guessing the "legacy" interpreter creation option isn't available in the new API.

Signed-off-by: Henry Schreiner <[email protected]>

henryiii · 2025-06-18T02:10:39Z

2.13? Did you mean 3.12 or 3.13, perhaps?

henryiii · 2025-06-18T03:56:47Z

tests/test_multiple_interpreters.py

+    assert res1 != res2, "internals should differ between interpreters"
+
+    # do this after the two interpreters are destroyed and only one remains
+    import mod_per_interpreter_gil as m3


FYI, importing again just pulls this from sys.modules again, so not sure why this is done this way, would be clearer to just keep reusing m, but not changing this here.

Ah, yeah, this import and the next assert don't really do anything.

I think I'll go ahead and remove these, since m2 and m3 are identical to m, which means they are really just testing m.internals_at() == m.internals_at() which isn't interesting.

rwgk

I glanced through, looks good to me, but my subinterpreter know-how isn't very deep.

@b-pass, could you please help with a more expert review?

tests/test_multiple_interpreters.py

Copilot

Pull Request Overview

Updates the multiple interpreters tests to support Python 3.12–3.14+ and skip cases when the native interpreters module isn't available.

Introduce a get_interpreters helper to abstract over stdlib and private interpreter APIs by version.
Refactor existing tests to use run_string/create returned from the helper and remove manual cleanup.
Add a new test_independent_subinterpreters_modern for Python 3.14+ using the concurrent.interpreters API.

tests/test_multiple_interpreters.py

b-pass

Looks good to me. I haven't tried 14b3 yet though...

b-pass · 2025-06-18T04:11:33Z

tests/test_multiple_interpreters.py

+    assert res1 != res2, "internals should differ between interpreters"
+
+    # do this after the two interpreters are destroyed and only one remains
+    import mod_per_interpreter_gil as m3


Ah, yeah, this import and the next assert don't really do anything.

Signed-off-by: Henry Schreiner <[email protected]>

henryiii · 2025-06-18T04:51:30Z

Beta 3 should show up soon: https://github.com/actions/python-versions/releases/tag/3.14.0-beta.3-15723154677

The manifest needs updating to show up: [versions-manifest] Update for release from 06/18/2025 actions/python-versions#345

(it did pass locally for me with beta 3 on macOS)

msimacek · 2025-06-18T08:09:41Z

Would it be better to gate this as cpython only?

I wanted to leave it open to the possibility that graalpy might implement the internal module at some later point, although it's not very likely.

2.13? Did you mean 3.12 or 3.13, perhaps?

Yeah, sorry, I meant 3.12, I permuted the digits 🤦‍♂️

Thanks for extending the PR. I rechecked the changes on GraalPy and everything is correctly skipped.

Signed-off-by: Henry Schreiner <[email protected]>

henryiii · 2025-06-18T19:51:22Z

These aren't actually running concurrently, maybe we should do that too in a followup?

I've updated the check to be unconditional on 3.14.0b3, and after b3 it should just use sys.implementation.supports_isolated_interpreters. We can simplify in a few months.

henryiii · 2025-06-18T20:39:25Z

The latest build used b3. :)

rwgk · 2025-06-18T22:06:23Z

I just had a minute to look here. There was one failed job due to network issue or similar. I triggered a rerun.

ericsnowcurrently · 2025-06-19T00:08:16Z

@ericsnowcurrently, would it make sense for Interpreter to support context managers? I can wrap it in contextlib.closing, but it seems somewhat natural if it supported it directly.

Yeah, it could make sense. Feel free to open a feature request. FWIW, I purposefully kept PEP 734 as simple as possible, with the expectation that we would build from there.

Use pytest.importorskip to get _xxsubinterpreters

75f2d9c

henryiii reviewed Jun 17, 2025

View reviewed changes

tests: use modern interpreter API

1ce6ed5

Signed-off-by: Henry Schreiner <[email protected]>

henryiii marked this pull request as draft June 17, 2025 23:10

tests: try to fix beta2

31d5fa4

Signed-off-by: Henry Schreiner <[email protected]>

henryiii force-pushed the msimacek/graalpy branch from bfde4d7 to 31d5fa4 Compare June 17, 2025 23:52

henryiii marked this pull request as ready for review June 18, 2025 00:38

henryiii changed the title ~~Use pytest.importorskip to get _xxsubinterpreters~~ tests: handle 3.12 and 3.13 implementations and 3.14.0b3+ Jun 18, 2025

henryiii reviewed Jun 18, 2025

View reviewed changes

henryiii requested a review from Copilot June 18, 2025 03:57

rwgk approved these changes Jun 18, 2025

View reviewed changes

tests/test_multiple_interpreters.py Outdated Show resolved Hide resolved

Copilot AI reviewed Jun 18, 2025

View reviewed changes

tests/test_multiple_interpreters.py Outdated Show resolved Hide resolved

tests/test_multiple_interpreters.py Show resolved Hide resolved

tests/test_multiple_interpreters.py Show resolved Hide resolved

b-pass reviewed Jun 18, 2025

View reviewed changes

henryiii added 2 commits June 18, 2025 00:39

fix: remove debug printout

39a1d06

Signed-off-by: Henry Schreiner <[email protected]>

tests: drop useless checks

0faec07

Signed-off-by: Henry Schreiner <[email protected]>

tests: improve check for 3.14.0b3 and b4+

02b0c36

Signed-off-by: Henry Schreiner <[email protected]>

style: pre-commit fixes

a2c1c67

henryiii merged commit c60c149 into pybind:master Jun 18, 2025
145 of 146 checks passed

github-actions bot added the needs changelog Possibly needs a changelog entry label Jun 18, 2025

henryiii changed the title ~~tests: handle 3.12 and 3.13 implementations and 3.14.0b3+~~ fix: handle implementations without subinterpreter support Jun 18, 2025

BrewTestBot mentioned this pull request Jul 10, 2025

pybind11 3.0.0 Homebrew/homebrew-core#229675

Open

rwgk removed the needs changelog Possibly needs a changelog entry label Jul 10, 2025

fix: handle implementations without subinterpreter support #5732

fix: handle implementations without subinterpreter support #5732

Uh oh!

Conversation

msimacek commented Jun 17, 2025 • edited by henryiii Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Suggested changelog entry:

Uh oh!

henryiii commented Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

henryiii Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

henryiii Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

henryiii commented Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

henryiii commented Jun 17, 2025

Uh oh!

henryiii commented Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

henryiii commented Jun 18, 2025

Uh oh!

henryiii Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

b-pass Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

henryiii Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

rwgk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

b-pass left a comment

Choose a reason for hiding this comment

Uh oh!

b-pass Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

henryiii commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

msimacek commented Jun 18, 2025

Uh oh!

henryiii commented Jun 18, 2025

Uh oh!

henryiii commented Jun 18, 2025

Uh oh!

rwgk commented Jun 18, 2025

Uh oh!

Uh oh!

ericsnowcurrently commented Jun 19, 2025

Uh oh!

Uh oh!

msimacek commented Jun 17, 2025 •

edited by henryiii

Loading

henryiii commented Jun 17, 2025 •

edited

Loading

henryiii commented Jun 17, 2025 •

edited

Loading

henryiii commented Jun 17, 2025 •

edited

Loading

henryiii commented Jun 18, 2025 •

edited

Loading