pip doesn't work from statically linked executables

[indygreg]

Environment

• pip version: 19
• Python version: 3.7.3 compiled with MUSL
• OS: Linux

Description

In the bowels of pip, the ctypes module is imported. Importing this module on statically linked executables (such as Python builds compiled with MUSL) fails with OSError because executing ctypes/__init__.py attempts to call dlopen(), which will always fail on binaries that don't support dynamic loading.

This is arguably a bug in CPython's ctypes module, as it could possibly gracefully fail on failure calling dlopen(). But that bug has shipped for years and it is everyone else's responsibility to work around it.

Expected behavior

I think pip should handle failure to import the ctypes module gracefully and not abort.

How to Reproduce

With a statically linked Python executable that doesn't have a .dynamic or other related ELF sections, attempt to run python -m ensurepip install <package>.

The zstd compressed tarball at https://github.com/indygreg/python-build-standalone/releases/download/20190505/cpython-3.7.3-linux64-musl-20190526T0219.tar.zst contains such an executable under python/install/bin/python3.7.

Output

$ python/install/bin/python3.7 -m ensurepip install pyflakes
Traceback (most recent call last):
  File "/tmp/python-distribution.Dh2VnKV1hPCw/python/install/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/tmp/python-distribution.Dh2VnKV1hPCw/python/install/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmp/python-distribution.Dh2VnKV1hPCw/python/install/lib/python3.7/ensurepip/__main__.py", line 5, in <module>
    sys.exit(ensurepip._main())
  File "/tmp/python-distribution.Dh2VnKV1hPCw/python/install/lib/python3.7/ensurepip/__init__.py", line 204, in _main
    default_pip=args.default_pip,
  File "/tmp/python-distribution.Dh2VnKV1hPCw/python/install/lib/python3.7/ensurepip/__init__.py", line 117, in _bootstrap
    return _run_pip(args + [p[0] for p in _PROJECTS], additional_paths)
  File "/tmp/python-distribution.Dh2VnKV1hPCw/python/install/lib/python3.7/ensurepip/__init__.py", line 27, in _run_pip
    import pip._internal
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/tmp/tmp6jcsm053/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/__init__.py", line 40, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/tmp/tmp6jcsm053/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/cli/autocompletion.py", line 8, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/tmp/tmp6jcsm053/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/cli/main_parser.py", line 12, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/tmp/tmp6jcsm053/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/commands/__init__.py", line 6, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/tmp/tmp6jcsm053/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/commands/completion.py", line 6, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/tmp/tmp6jcsm053/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/cli/base_command.py", line 20, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/tmp/tmp6jcsm053/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/download.py", line 37, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/tmp/tmp6jcsm053/pip-19.0.3-py2.py3-none-any.whl/pip/_internal/utils/glibc.py", line 3, in <module>
  File "/tmp/python-distribution.Dh2VnKV1hPCw/python/install/lib/python3.7/ctypes/__init__.py", line 444, in <module>
    pythonapi = PyDLL(None)
  File "/tmp/python-distribution.Dh2VnKV1hPCw/python/install/lib/python3.7/ctypes/__init__.py", line 356, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: Dynamic loading not supported

The problem with CPython specifically:

$ ldd python/install/bin/python3.7
        not a dynamic executable

$ python/install/bin/python3.7
Python 3.7.3 (default, May 26 2019, 04:33:50)
[Clang 7.0.1 (tags/RELEASE_701/final)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/gps/src/python-build-standalone.git/build/python/install/lib/python3.7/ctypes/__init__.py", line 444, in <module>
    pythonapi = PyDLL(None)
  File "/home/gps/src/python-build-standalone.git/build/python/install/lib/python3.7/ctypes/__init__.py", line 356, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: Dynamic loading not supported

[indygreg]

I filed a CPython issue at https://bugs.python.org/issue37060.

[indygreg]

setuptools is also affected. I figure I'll wait for someone from the PyPA to comment on matters before I file an issue / submit a PR to setuptools.

cpython-linux64-musl> Traceback (most recent call last):
cpython-linux64-musl>   File "/build/setuptools-41.0.1/setup.py", line 11, in <module>
cpython-linux64-musl>     import setuptools
cpython-linux64-musl>   File "/build/setuptools-41.0.1/setuptools/__init__.py", line 20, in <module>
cpython-linux64-musl>     from setuptools.dist import Distribution, Feature
cpython-linux64-musl>   File "/build/setuptools-41.0.1/setuptools/dist.py", line 35, in <module>
cpython-linux64-musl>     from setuptools import windows_support
cpython-linux64-musl>   File "/build/setuptools-41.0.1/setuptools/windows_support.py", line 2, in <module>
cpython-linux64-musl>     import ctypes
cpython-linux64-musl>   File "/build/out/python/install/lib/python3.7/ctypes/__init__.py", line 444, in <module>
cpython-linux64-musl>     pythonapi = PyDLL(None)
cpython-linux64-musl>   File "/build/out/python/install/lib/python3.7/ctypes/__init__.py", line 356, in __init__
cpython-linux64-musl>     self._handle = _dlopen(self._name, mode)
cpython-linux64-musl> OSError: Dynamic loading not supported

[cjerdonek]

I would go ahead and file an issue on the setuptools tracker as you'll get a different set of readers there.

[pfmoore]

Note - your actual issue here is in pip/_internal/utils/glibc.py and seems relatively shallow - you've posted #6543 which seems to cover it by deciding that if you don't have ctypes, you're not glibc. With that in mind, it's entirely reasonable to ignore the broader question of ctypes on statically linked systems, and just fix that one issue and stop there.

But assuming you do want to treat the broader issue as worth addressing:

I strongly believe that this is a ctypes bug - there's no way that it's reasonable to expect every import of a stdlib module to be checked for errors in every project. And it's not as if ctypes is documented as an optional stdlib module (at least not that I could see). Having said that, I see the reasoning that we have to live with the reality, and therefore we might have to deal with this. But many of our ctypes usages are in vendored libraries (colorama, urllib3) so you'll need to raise issues against those projects as well. And I'd still argue for fixing issues on a case by case basis, prompted by real world problems, rather than just making a blanket rule that we always have to protect ctypes usage (it's not like we make provision for systems where threading isn't available, for example).

I would want to watch the Python bug report closely, though - if the CPython core devs don't accept that it's a bug for "import ctypes" to fail, then at a minimum the CPython docs should make that clear - there's nothing in the docs at the moment to suggest that it's any more acceptable for "import ctypes" to fail than (say) "import subprocess".

[indygreg]

I agree that this is a ctypes bug. I think it is best for ctypes import to gracefully handle the OSError and continue with import.

Then the question becomes what should consumers of ctypes do - if anything - to work around the import time failure in existing Python releases. I highly doubt many people out there have built a non-dynamic Python binary. I think it is a reasonable stance to tell people they need a minimum version of Python - perhaps 3.7.4 - to run a fully static binary.

But there's still the run-time issue of OSError when calling ctypes.CDLL(None). Presumably pip, setuptools, and the rest of the world will need to be aware that ctypes.CDLL(None) can fail. That's of course a different patch from what I wrote. Perhaps it is best to wait for upstream to weigh in about what ctypes should do before we land any patches.

In any case, I've worked around this issue in python-build-standalone by patching cpython, setuptools, and pip.