ctypes.util.find_library() does not find macOS 11+ system libraries when built on older macOS systems

[bergkvist]

BPO	44689
Nosy	@ronaldoussoren, @ned-deily, @ambv, @indygreg, @miss-islington, @bergkvist
PRs	bpo-44689: ctypes.util.find_library() now finds macOS 11+ system libraries when built on older macOS systems #27251 [3.10] bpo-44689: ctypes.util.find_library() now finds macOS 11+ system libraries when built on older macOS systems (GH-27251) #28052 [3.9] bpo-44689: ctypes.util.find_library() now finds macOS 11+ system libraries when built on older macOS systems (GH-27251) #28053 [3.8] bpo-44689: ctypes.util.find_library() now finds macOS 11+ system libraries when built on older macOS systems (GH-27251) #28054

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ned-deily'
closed_at = None
created_at = <Date 2021-07-20.21:56:46.079>
labels = ['OS-mac', 'type-bug', '3.8', '3.9', '3.10', '3.11', 'ctypes']
title = 'ctypes.util.find_library() does not find macOS 11+ system libraries when built on older macOS systems'
updated_at = <Date 2021-09-02.20:04:38.819>
user = 'https://github.com/bergkvist'

bugs.python.org fields:

activity = <Date 2021-09-02.20:04:38.819>
actor = 'ned.deily'
assignee = 'ned.deily'
closed = False
closed_date = None
closer = None
components = ['macOS', 'ctypes']
creation = <Date 2021-07-20.21:56:46.079>
creator = 'bergkvist'
dependencies = []
files = []
hgrepos = []
issue_num = 44689
keywords = ['patch']
message_count = 23.0
messages = ['397916', '397926', '397927', '397930', '397979', '397984', '397990', '398959', '399052', '399079', '400575', '400580', '400581', '400582', '400797', '400830', '400831', '400895', '400896', '400898', '400899', '400947', '400949']
nosy_count = 6.0
nosy_names = ['ronaldoussoren', 'ned.deily', 'lukasz.langa', 'indygreg', 'miss-islington', 'bergkvist']
pr_nums = ['27251', '28052', '28053', '28054']
priority = 'normal'
resolution = None
stage = 'commit review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue44689'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10', 'Python 3.11']

[bergkvist]

Python-binaries compiled on either Big Sur or Catalina - and moved to the other MacOS-version will not work as expected when code depends on ctypes.util.find_library.

Example symptom of this issue: jupyterlab/jupyterlab#9863
I have personally faced this when using Python from nixpkgs - since nixpkgs Python has been built on Catalina - and I'm using Big Sur.

Scenario 1: Compile on Catalina, copy binaries to BigSur, and call ctypes.util.find_library('c')
Python 3.11.0a0 (heads/main:635bfe8162, Jul 19 2021, 08:09:05) [Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes.util import find_library; print(find_library('c'))
None

Scenario 2: Compile on Big Sur, copy binaries to Catalina, and call ctypes.util.find_library('c'):
Python 3.11.0a0 (heads/main:635bfe8162, Jul 19 2021, 08:28:48) [Clang 12.0.5 (clang-1205.0.22.11)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes.util import find_library; print(find_library('c'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.11/ctypes/__init__.py", line 8, in <module>
    from _ctypes import Union, Structure, Array
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ImportError: dlopen(/usr/local/lib/python3.11/lib-dynload/_ctypes.cpython-311-darwin.so, 2): Symbol not found: __dyld_shared_cache_contains_path
  Referenced from: /usr/local/lib/python3.11/lib-dynload/_ctypes.cpython-311-darwin.so (which was built for Mac OS X 11.4)
  Expected in: /usr/lib/libSystem.B.dylib
 in /usr/local/lib/python3.11/lib-dynload/_ctypes.cpython-311-darwin.so