[BUG] Python 3.11.1 breaks setuptools builds on Fedora

[stratakis]

setuptools version

65.5.1-65.6.3

Python version

Python 3.11.1

OS

Fedora Linux

Additional environment information

This was observed in Fedora Linux as soon as we updated to 3.11.1

Description

Since python/cpython@07734a4 setuptools builds have started failing on Fedora.

It seems there are two issues I've discovered so far and bisecting the problem led me to the specific Python commit.

First when trying to build setuptools through the pypi tarball after an initial bootstrap build (meaning we delete the egg-info directory on the second build) and then our pyproject script tries to generate the build requirements we get a KeyError: 'entry_points' error.

Secondly if I add the build requirements manually and then create the wheel through python3 setup.py bdist_wheel after again removing the pre-generated egg-info directory I get an error: [Errno 2] No such file or directory: 'build/bdist.linux-x86_64/wheel/setuptools-65.6.3-py3.11.egg-info/PKG-INFO' error.

Tested it from 65.5.1 and on-wards. By removing that commit or by not removing the egg-info directory, everything works like before.

Expected behavior

Expected for the errors not to happen

How to Reproduce

For the first part:

On a Fedora 37 or Rawhide system
dnf install python3.11 which will bring you the latest Python 3.11.1 or compile CPython from the v3.11.1 tag
Get the setuptools tarball from pypi and unzip it
Delete the egg-info directory
dnf install '/usr/lib/rpm/redhat/pyproject_buildrequires.py'
And inside the dir do python3 -B /usr/lib/rpm/redhat/pyproject_buildrequires.py --generate-extras --python3_pkgversion 3

By downgrading to 3.11.0 or skipping python/cpython@07734a4 the issue disappears.

For the second part:

A reproducer which should be non-fedora specific although it shows a different error.
Compile and/or install Python 3.11.1
In a system where setuptools is already installed.
Get the pypi tarball
Unzip it and delete the egg-info directory
Install wheel
python3 setup.py bdist_wheel

Same as before, if I downgrade to 3.11.0 or skip the specific commit the issue disappears.

Output

First part:

$ python3 -B /usr/lib/rpm/redhat/pyproject_buildrequires.py --generate-extras --python3_pkgversion 3
/builddir/setuptools-65.6.3/setuptools/command/egg_info.py:643: SetuptoolsDeprecationWarning: Custom 'build_py' does not implement 'get_data_files_without_manifest'.
Please extend command classes from setuptools instead of distutils.
  warnings.warn(
warning: no files found matching '*.py' under directory 'tests'
warning: no files found matching '*.txt' under directory 'docs'
warning: no files found matching '*.conf' under directory 'docs'
warning: no files found matching '*.css' under directory 'docs'
warning: no files found matching '*.css_t' under directory 'docs'
warning: no files found matching 'Makefile' under directory 'docs'
warning: no files found matching 'indexsidebar.html' under directory 'docs'
warning: no files found matching 'msvc-build-launcher.cmd'
HOOK STDOUT: running egg_info
HOOK STDOUT: creating setuptools.egg-info
HOOK STDOUT: writing manifest file 'setuptools.egg-info/SOURCES.txt'
HOOK STDOUT: reading manifest file 'setuptools.egg-info/SOURCES.txt'
HOOK STDOUT: reading manifest template 'MANIFEST.in'
HOOK STDOUT: adding license file 'LICENSE'
HOOK STDOUT: writing manifest file 'setuptools.egg-info/SOURCES.txt'
Handling wheel from get_requires_for_build_wheel
Requirement satisfied: wheel
   (installed: wheel 0.38.4)
python3dist(wheel)
Traceback (most recent call last):
  File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 508, in main
    generate_requires(
  File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 420, in generate_requires
    generate_run_requirements(backend, requirements, build_wheel=build_wheel, wheeldir=wheeldir)
  File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 330, in generate_run_requirements
    generate_run_requirements_hook(backend, requirements)
  File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 286, in generate_run_requirements_hook
    dir_basename = prepare_metadata('.')
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/setuptools-65.6.3/setuptools/build_meta.py", line 377, in prepare_metadata_for_build_wheel
    self.run_setup()
  File "/builddir/setuptools-65.6.3/setuptools/build_meta.py", line 335, in run_setup
    exec(code, locals())
  File "<string>", line 87, in <module>
  File "/builddir/setuptools-65.6.3/setuptools/__init__.py", line 87, in setup
    return distutils.core.setup(**attrs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/builddir/setuptools-65.6.3/setuptools/_distutils/core.py", line 159, in setup
    dist.parse_config_files()
  File "/builddir/setuptools-65.6.3/setuptools/dist.py", line 863, in parse_config_files
    setupcfg.parse_configuration(
  File "/builddir/setuptools-65.6.3/setuptools/config/setupcfg.py", line 162, in parse_configuration
    options.parse()
  File "/builddir/setuptools-65.6.3/setuptools/config/setupcfg.py", line 496, in parse
    section_parser_method(section_options)
  File "/builddir/setuptools-65.6.3/setuptools/config/setupcfg.py", line 724, in parse_section_entry_points
    self['entry_points'] = parsed
    ~~~~^^^^^^^^^^^^^^^^
  File "/builddir/setuptools-65.6.3/setuptools/config/setupcfg.py", line 268, in __setitem__
    raise KeyError(option_name)
KeyError: 'entry_points'

Second part:

$ python3 setup.py bdist_wheel
<snip>
running egg_info
creating setuptools.egg-info
writing manifest file 'setuptools.egg-info/SOURCES.txt'
reading manifest file 'setuptools.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*.py' under directory 'tests'
warning: no files found matching '*.txt' under directory 'docs'
warning: no files found matching '*.conf' under directory 'docs'
warning: no files found matching '*.css' under directory 'docs'
warning: no files found matching '*.css_t' under directory 'docs'
warning: no files found matching 'Makefile' under directory 'docs'
warning: no files found matching 'indexsidebar.html' under directory 'docs'
warning: no files found matching 'msvc-build-launcher.cmd'
adding license file 'LICENSE'
writing manifest file 'setuptools.egg-info/SOURCES.txt'
Copying setuptools.egg-info to build/bdist.linux-x86_64/wheel/setuptools-65.6.3-py3.11.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/distutils-precedence.pth
error: [Errno 2] No such file or directory: 'build/bdist.linux-x86_64/wheel/setuptools-65.6.3-py3.11.egg-info/PKG-INFO'

[stratakis]

cc @hroncok

[pojoba02]

Also having the same issue with ubuntu

[jaraco]

Thanks for the nudge. I'd forgotten about this issue and plan to look tomorrow.

[jaraco]

The repro instructions aren't working for me. I created this Dockerfile:

from fedora:rawhide
run dnf install -y python3.11 python-launcher
run curl https://files.pythonhosted.org/packages/fe/31/2c88d37d4f8036c6648745e74ec51f73a6ba945a06e069601900f81fe00a/setuptools-67.4.0.tar.gz -o - | tar xz
run dnf install -y '/usr/lib/rpm/redhat/pyproject_buildrequires.py'
run ln -s setuptools* setuptools-src
workdir setuptools-src
run rm -r *.egg-info
cmd py -B /usr/lib/rpm/redhat/pyproject_buildrequires.py --generate-extras --python3_pkgversion 3

And running it emits an error about packaging.

 draft $ docker run -it @$(docker build -q .)
Import error: No module named 'packaging'

Edit: Adding python-packaging to the installed packages works around that error and the KeyError reproduces.

[hroncok]

run dnf install -y '/usr/lib/rpm/redhat/pyproject_buildrequires.py' python3-packaging

[jaraco]

I tried entering a debugger where the issue occurs, but something about the pyproject_buildrequires.py script is interfering.

 draft $ docker run -it @$(docker build -q .) py -B -m pdb /usr/lib/rpm/redhat/pyproject_buildrequires.py --generate-extras --python3_pkgversion 3
> /usr/lib/rpm/redhat/pyproject_buildrequires.py(1)<module>()
-> import glob
(Pdb) c
Traceback (most recent call last):
  File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 532, in main
    generate_requires(
  File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 439, in generate_requires
    generate_build_requirements(backend, requirements)
  File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 289, in generate_build_requirements
    new_reqs = get_requires()
               ^^^^^^^^^^^^^^
  File "/setuptools-67.4.0/setuptools/build_meta.py", line 338, in get_requires_for_build_wheel
    return self._get_build_requires(config_settings, requirements=['wheel'])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/setuptools-67.4.0/setuptools/build_meta.py", line 320, in _get_build_requires
    self.run_setup()
  File "/setuptools-67.4.0/setuptools/build_meta.py", line 335, in run_setup
    exec(code, locals())
  File "<string>", line 87, in <module>
  File "/setuptools-67.4.0/setuptools/__init__.py", line 107, in setup
    _install_setup_requires(attrs)
  File "/setuptools-67.4.0/setuptools/__init__.py", line 78, in _install_setup_requires
    dist.parse_config_files(ignore_option_errors=True)
  File "/setuptools-67.4.0/setuptools/dist.py", line 881, in parse_config_files
    setupcfg.parse_configuration(
  File "/setuptools-67.4.0/setuptools/config/setupcfg.py", line 163, in parse_configuration
    options.parse()
  File "/setuptools-67.4.0/setuptools/config/setupcfg.py", line 508, in parse
    section_parser_method(section_options)
  File "/setuptools-67.4.0/setuptools/config/setupcfg.py", line 736, in parse_section_entry_points
    self['entry_points'] = parsed
    ~~~~^^^^^^^^^^^^^^^^
  File "/setuptools-67.4.0/setuptools/config/setupcfg.py", line 280, in __setitem__
    raise KeyError(option_name)
KeyError: 'entry_points'
The program exited via sys.exit(). Exit status: 1
> /usr/lib/rpm/redhat/pyproject_buildrequires.py(1)<module>()
-> import glob
(Pdb) Traceback (most recent call last):
  File "/usr/lib64/python3.11/pdb.py", line 1774, in main
    pdb._run(target)
  File "/usr/lib64/python3.11/pdb.py", line 1652, in _run
    self.run(target.code)
  File "/usr/lib64/python3.11/bdb.py", line 597, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 1, in <module>
    import glob
  File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 1, in <module>
    import glob
  File "/usr/lib64/python3.11/bdb.py", line 90, in trace_dispatch
    return self.dispatch_line(frame)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/bdb.py", line 114, in dispatch_line
    self.user_line(frame)
  File "/usr/lib64/python3.11/pdb.py", line 340, in user_line
    self.interaction(frame, None)
  File "/usr/lib64/python3.11/pdb.py", line 435, in interaction
    self._cmdloop()
  File "/usr/lib64/python3.11/pdb.py", line 400, in _cmdloop
    self.cmdloop()
  File "/usr/lib64/python3.11/cmd.py", line 126, in cmdloop
    line = input(self.prompt)
           ^^^^^^^^^^^^^^^^^^
ValueError: I/O operation on closed file.
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> /usr/lib64/python3.11/cmd.py(126)cmdloop()
-> line = input(self.prompt)
(Pdb) Traceback (most recent call last):
  File "/usr/lib64/python3.11/pdb.py", line 1774, in main
    pdb._run(target)
  File "/usr/lib64/python3.11/pdb.py", line 1652, in _run
    self.run(target.code)
  File "/usr/lib64/python3.11/bdb.py", line 597, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 1, in <module>
    import glob
  File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 1, in <module>
    import glob
  File "/usr/lib64/python3.11/bdb.py", line 90, in trace_dispatch
    return self.dispatch_line(frame)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/bdb.py", line 114, in dispatch_line
    self.user_line(frame)
  File "/usr/lib64/python3.11/pdb.py", line 340, in user_line
    self.interaction(frame, None)
  File "/usr/lib64/python3.11/pdb.py", line 435, in interaction
    self._cmdloop()
  File "/usr/lib64/python3.11/pdb.py", line 400, in _cmdloop
    self.cmdloop()
  File "/usr/lib64/python3.11/cmd.py", line 126, in cmdloop
    line = input(self.prompt)
           ^^^^^^^^^^^^^^^^^^
ValueError: I/O operation on closed file.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/lib64/python3.11/pdb.py", line 1801, in <module>
    pdb.main()
  File "/usr/lib64/python3.11/pdb.py", line 1793, in main
    pdb.interaction(None, t)
  File "/usr/lib64/python3.11/pdb.py", line 435, in interaction
    self._cmdloop()
  File "/usr/lib64/python3.11/pdb.py", line 400, in _cmdloop
    self.cmdloop()
  File "/usr/lib64/python3.11/cmd.py", line 126, in cmdloop
    line = input(self.prompt)
           ^^^^^^^^^^^^^^^^^^
ValueError: I/O operation on closed file.

[jaraco]

Looking at the traceback for the KeyError, there's nothing in the traceback that interacts with importlib.metadata. The failure occurs when entry_points doesn't exist in the ConfigOptionsHandler. I'm still trying to determine what factors affect that state.

I added this statement to the Dockerfile to enable debugging:

run sed -i -e 's/except Exception:/except Exception:\n        raise/' /usr/lib/rpm/redhat/pyproject_buildrequires.py

[jaraco]

Aha. So part of the issue is that when parsing the configuration, a MinimalDistribution is present and it's that object that has no entry_points attribute.

[hroncok]

I think we saw something similar in pypa/virtualenv#2493 (comment)

Since the problem only occurred at test-time, we solved that by deleting the prepare_metadata_for_build_wheel-produced .dist-info directory after reading the data from it. I wonder if pyproject_buildrequires.py could do that sooner, but since this fails right when the hook is invoked, I don't think so.

[jaraco]

Good news. I've been able to replicate the failure without involving pyproject_buildrequires. Simply invoking egg_info on a tarball with the metadata removed triggers the failure:

from fedora:rawhide
run dnf install -y python3.11 python-launcher python-packaging
run curl https://files.pythonhosted.org/packages/fe/31/2c88d37d4f8036c6648745e74ec51f73a6ba945a06e069601900f81fe00a/setuptools-67.4.0.tar.gz -o - | tar xz
run ln -s setuptools* setuptools-src
workdir setuptools-src
run rm -r *.egg-info
cmd py setup.py egg_info

This repro should make debugging easier because it (a) leaves . on sys.path and (b) doesn't interfere with stdio, allowing for one to set breakpoints.