PEP 703 -- Making the Global Interpreter Lock Optional in CPython

[colesbury]

Feature or enhancement

The steering council has accepted PEP 703. This is intended as a top-level issue to keep track of integration status.

The "up for grabs" list contains issues that no one is currently working on and are ready to be implemented. If you are interested in working on one of them, please comment on the specific issue and CC me (@colesbury) on related PRs.

### Up For Grabs (comment on specific issue to take)

### Linked issues
- [ ] https://github.com/python/cpython/issues/111506
- [ ] https://github.com/python/cpython/issues/111870
- [ ] https://github.com/python/cpython/issues/111924
- [ ] https://github.com/python/cpython/issues/112606
- [ ] https://github.com/python/cpython/issues/114203
- [ ] https://github.com/python/cpython/issues/114214
- [ ] https://github.com/python/cpython/issues/115999
- [ ] https://github.com/python/cpython/issues/116024
- [ ] https://github.com/python/cpython/issues/116738
- [ ] https://github.com/python/cpython/issues/117139
- [ ] https://github.com/python/cpython/issues/117721
- [ ] https://github.com/python/cpython/issues/117657
- [ ] https://github.com/python/cpython/issues/119053

Completed Issues

### Completed Issues
- [ ] https://github.com/python/cpython/issues/108223
- [ ] https://github.com/python/cpython/issues/108374
- [ ] https://github.com/python/cpython/issues/108337
- [ ] https://github.com/python/cpython/issues/108724
- [ ] https://github.com/python/cpython/issues/109549
- [ ] https://github.com/python/cpython/issues/109740
- [ ] https://github.com/python/cpython/issues/110119
- [ ] https://github.com/pypa/packaging/issues/727
- [ ] https://github.com/python/cpython/issues/111062
- [ ] https://github.com/python/cpython/issues/111569
- [ ] https://github.com/python/cpython/issues/112062
- [ ] https://github.com/python/cpython/issues/111903
- [ ] https://github.com/python/cpython/issues/111916
- [ ] https://github.com/python/cpython/issues/111928
- [ ] https://github.com/python/cpython/issues/112070
- [ ] https://github.com/python/cpython/issues/111956
- [ ] https://github.com/python/cpython/issues/111965
- [ ] https://github.com/python/cpython/issues/112213
- [ ] https://github.com/python/cpython/issues/112071
- [ ] https://github.com/python/cpython/issues/111972
- [ ] https://github.com/python/cpython/issues/112538
- [ ] https://github.com/python/cpython/issues/111962
- [ ] https://github.com/python/cpython/issues/112535
- [ ] https://github.com/python/cpython/issues/112205
- [ ] https://github.com/python/cpython/issues/111971
- [ ] https://github.com/python/cpython/issues/111650
- [ ] https://github.com/python/cpython/issues/113750
- [ ] https://github.com/python/cpython/issues/111964
- [ ] https://github.com/python/cpython/issues/113884
- [ ] https://github.com/python/cpython/issues/114569
- [ ] https://github.com/python/cpython/issues/114329
- [ ] https://github.com/python/cpython/issues/112066
- [ ] https://github.com/python/cpython/issues/110481
- [ ] https://github.com/python/cpython/issues/112532
- [ ] https://github.com/python/cpython/issues/112050
- [ ] https://github.com/python/cpython/issues/111968
- [ ] https://github.com/python/cpython/issues/115491
- [ ] https://github.com/python/cpython/issues/112175
- [ ] https://github.com/python/cpython/issues/113743
- [ ] https://github.com/python/cpython/issues/115103
- [ ] https://github.com/python/cpython/issues/112087
- [ ] https://github.com/python/cpython/issues/116616
- [ ] https://github.com/python/cpython/issues/116167
- [ ] https://github.com/python/cpython/issues/112536
- [ ] https://github.com/python/cpython/issues/116522
- [ ] https://github.com/python/cpython/issues/116664
- [ ] https://github.com/python/cpython/issues/117323
- [ ] https://github.com/python/cpython/issues/117300
- [ ] https://github.com/python/cpython/issues/111926
- [ ] https://github.com/python/cpython/issues/117293
- [ ] https://github.com/python/cpython/issues/117435
- [ ] https://github.com/python/cpython/issues/114271
- [ ] https://github.com/python/cpython/issues/112069
- [ ] https://github.com/python/cpython/issues/116329
- [ ] https://github.com/python/cpython/issues/116818
- [ ] https://github.com/python/cpython/issues/117514
- [ ] https://github.com/python/cpython/issues/112529
- [ ] https://github.com/python/cpython/issues/112075
- [ ] https://github.com/python/cpython/issues/116322

### Deferred tasks
- [ ] Revisit and update [`--disable-gil`](https://docs.python.org/dev/using/configure.html#cmdoption-disable-gil) configure documentation
- [x] Reenable test_cppext on `--disable-gil` builds
- [x] Audit usage of PyObject_Malloc for non-PyObject allocations
- [ ] Consider avoiding refcounting `tp_mro` during PyType_IsSubtype

Upstream functionality from nogil-3.12

This is a list of commits from the nogil-3.12 PR plan. The crossed-out entries are commits that do not need to be upstreamed, usually because the functionality is already in the main branch.

• cefe5dfee9 configure: disallow "--with-trace-refs" for "--disable-gil" builds
• dcddbe2ddb configure: add support for --with-thread-sanitizer (Add support for thread sanitizer (TSAN) via --with-thread-sanitizer #112536)
• f546dbf16a Enable/disable the GIL at runtime (Add a mechanism to disable the GIL #116167)
• f30d8d8f50 Add pyatomic.h
• 385eb1d99c pyport: add new macros
• de2be447b3 Make PyThreadState_GET thread-local
• a24dc2ecc3 pystate: keep track of attached vs. detached state
• 4584be5950 parking_lot: add mutexes and one-time notifications
• 6845b133cc critical_section: helpers for fine-grained locking (Implement Python Critical Sections from PEP 703 #111569)
• 8ed62cab6a pystate: use _PyRawMutex for internal mutexes (Avoid changing the PYMEM_DOMAIN_RAW allocator during initialization and finalization #111924)
• e15443b1f2 ceval: move eval_breaker to per-thread state (Move the eval_breaker to PyThreadState #112175)
• b6b12a9a94 Implement biased reference counting
• b6b12a9a94 Implement BRC inter-thread queue (Implement biased reference counting in --disable-gil builds #110481)
• 7b6b6f1a01 unicode: immortalize interned strings
• fc173e3711 unicode: always immortalize interned strings
• dd9b78460c Add safe memory reclamation scheme based on FreeBSD's GUS
• 901e134921 Add mimalloc v2.0.9 (DinoV)
• b6980856df Additional mimalloc changes (mimalloc: additional integration and changes for --disable-gil builds #112532)
• d13c63dee9 pymem: remove uses of _PyMem_SetDefaultAllocator during finalization (Avoid changing the PYMEM_DOMAIN_RAW allocator during initialization and finalization #111924)
• 654be8ffd6 gc: make the garbage collector non-generational (Make the garbage collector thread-safe in --disable-gil builds #112529)
• 967fe31473 gc: Traverese mimalloc heaps to find all objects. (Make the garbage collector thread-safe in --disable-gil builds #112529)
• 2864b6b36e Implement stop-the-world pauses (Implement stop-the-world functionality (for --disable-gil builds) #111964)
• 2864b6b36e gc: implement stop-the-world GC (Make the garbage collector thread-safe in --disable-gil builds #112529)
• c1befd7689 Stop the world before fork() and Python shutdown (Use stop-the-world to make fork and interpreter shutdown thread-safe #116522)
• 82800d8ec8 ceval: stop the world when enabling profiling/tracing for all threads (Make sys.settrace, sys.setprofile, and sys.monitoring thread safe in --disable-gil builds #116818)
• 7423dff344 pystate: use stop-the-world in a few places (Use stop-the-world to make sys._current_frames() and sys._current_exceptions() thread-safe in free-threaded build #117300)
• 86efa7dfe3 pystate: implement _PyRuntime.multithreaded
• d896dfc8db dict: make dict thread-safe (Make dict objects thread-safe in --disable-gil builds #112075)
• df4c51f82b list: make list thread-safe (Make list objects thread-safe in --disable-gil builds #112087)
• 9c1f7ba1b4 mro: thread-safe MRO cache (Make the MRO cache thread-safe in free-threaded builds #113743)
• 7a7aca096b getargs.c: make parser_init thread-safe (Thread-safe one-time initialization #111956)
• 0dddcb6f9d weakref: make weakrefs thread-safe without the GIL (Make weakref thread-safe without the GIL #111926)
• 410ba1036a dtoa: make dtoa thread-safe (dtoa: thread safety in --disable-gil builds #111962)
• 6540bf3e6a unicode: make unicodeobject.c thread-safe (unicode: make _PyUnicode_FromId thread-safe in --disable-gil builds #111971)
• 5d006db9fa codecs.c: fix race condition (unicode: make ucnhash_capi initialization thread-safe in --disable-gil builds #111972)
• d1b5ed128e _threadmodule: make _thread.lock thread-safe (Make _threadmodule.c thread-safe in --disable-gil builds #114271)
• cfc11bcb1a typeobject: thread safety
• cfecf6f4eb threading: remove _tstate_lock from threading.Thread (Make _threadmodule.c thread-safe in --disable-gil builds #114271)
• 74df7785f5 pyqueue: add internal queue data structure
• 4450445f51 pymem: add _PyMem_FreeQsbr (Add delayed reclamation mechanism for free-threaded build (QSBR) #115103)
• 7e60a01aee queue: make SimpleQueue thread-safe (Make queue.SimpleQueue thread-safe in --disable-gil builds #113884)
• 4ca2924f0d set: make set thread-safe (Make set thread-safe in --disable-gil builds #112069)
• 3cfbc49229 moduleobject: fix data races
• 5722416ef5 _threadmodule: thread-safety fixes (Make _threadmodule.c thread-safe in --disable-gil builds #114271)
• 31ec6f0290 pystate: refcount threads to handle race between interpreter shutdown and thread exit
• 45bdd27ee5 threading: make _thread.lock thread-safe
• 07f5f8c318 slice: move slice_cache to per-thread state (Use per-thread freelists in --disable-gil builds #111968)
• ea1160c6d7 asyncio: fix race conditions in enter_task and leave_task
• 212fef480e object.c: fix race when accessing attributes and methods (Use PyDict_GetItemRef() #111789)
• 70856f126d asdl: use _PyOnceFlag in Python-ast.c (Thread-safe one-time initialization #111956)
• 360a79cb88 socketmodule.c: use relaxed atomics for global 'defaulttimeout' (socketmodule.c: use atomics to access defaulttimeout in free-threaded build  #116616)
• 041a08e339 functools: make lru_cache thread-safe (Make functools.lru_cache thread-safe in --disable-gil builds #112070)
• 9bf62ffc4b random: add a mutex to guard random.Random (Make random.Random thread-safe in --disable-gil builds #112071)
• a8251a8d25 clinic: support '@' syntax for recursive mutexes (Argument Clinic: Add support for PEP 703's critical sections (Py_BEGIN_CRITICAL_SECTION()) #111903)
• ffade9d6f6 bufferedio: add locks to make print() thread-safe (Use critical sections to protect I/O objects (in --disable-gil builds) #111965)
• 5b83c16dcd textio: add locks to make textio thread-safe (Use critical sections to protect I/O objects (in --disable-gil builds) #111965)
• 6323ca60f9 stringio: make stringio thread-safe (Use critical sections to protect I/O objects (in --disable-gil builds) #111965)
• f1e4742eaa deque: make most functions thread-safe (Make collections.deque thread-safe in --disable-gil builds #112050)
• 78825e0508 importlib: fix data race in imports (PyImport_ImportModuleLevelObject)
• 2f5c90a284 semaphore.c: decrease count before release sem_lock (Make multiprocessing SemLock thread-safe in the free-threaded build #117435)
• 22eca6e215 sha1: make sha1module thread-safe (Make hashlib related modules thread-safe without the GIL #111916)
• ada9b73feb _struct: fix race condition in cache_struct_converter (Make _struct module thread-safe in --disable-gil builds #112062)
• 0e0b3899d1 signalmodule: fix thread-safety issue on macOS (Unclear if this is still an issue on macOS)
• 964bb33962 json: make JSON scanner thread safe (json: make "memo" dict local to scan_once call #111928)
• 86e7772c64 http: fix dependency on finalization order
• 9ab96964e7 faulthandler: don't dump all threads when running without the GIL
• cff32694a4 test_gdb: skip test_threads when running without GIL
• 2ae5ee5ed4 tests: fix and work around some race conditions in tests
• 9f9b3d085f ceval: fix some thread-safety issues
• 2a4c17e896 pystate: move freelists to per-thread state (Use per-thread freelists in --disable-gil builds #111968)
• 149ea9dc43 Deferred reference counting
• 7e7568672d specialize: make specialization thread-safe
• 90d34f0d18 specialize: optimize for single-threaded programs
• 42d3e11d8c code: make code object use deferred reference counting
• c9fc49666c test: add support for checking for TSAN
• 7507a77a98 thread: don't use sem_clockwait with TSAN
• a62d37674c _posixsubprocess: disable vfork when running with ASAN
• 398204d57b object: fix reported TSAN races
• 4526c07cae Disable the GIL by default in --disable-gil builds (Disable the GIL by default in free-threaded builds #116329)

Linked PRs

• gh-108219: Add credits to free-threading project #123802
• [3.13] gh-108219: Add credits to the free-threading entry in What's New (GH-123802) #123847