test_peg_generator.test_c_parser emits DeprecationWarning due to distutils

[tirkarthi]

BPO	43425
Nosy	@terryjreedy, @vstinner, @ambv, @zooba, @corona10, @pablogsal, @miss-islington, @tirkarthi
PRs	bpo-43425: Update test_c_parser not to use TempdirManager #26693 [3.10] bpo-43425: Update test_c_parser not to use TempdirManager (GH-26693) #26713 bpo-43425: Update _osx_support not to use distutils.log #26968 bpo-43425: Update setup.py not to use distutils.log #26969 [3.10] bpo-43425: Update _osx_support not to use distutils.log (GH-26968) #26978 bpo-43425: Remove test2to3 from Tools #26981 [3.10] bpo-43425: Update setup.py not to use distutils.log (GH-26969) #26986

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-03-07.05:07:09.546>
labels = ['type-bug', 'tests', '3.10']
title = 'test_peg_generator.test_c_parser emits DeprecationWarning due to distutils'
updated_at = <Date 2021-09-15.13:15:47.651>
user = 'https://github.com/tirkarthi'

bugs.python.org fields:

activity = <Date 2021-09-15.13:15:47.651>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Tests']
creation = <Date 2021-03-07.05:07:09.546>
creator = 'xtreak'
dependencies = []
files = []
hgrepos = []
issue_num = 43425
keywords = ['patch']
message_count = 13.0
messages = ['388222', '388242', '388243', '388572', '388780', '395716', '395768', '396805', '396808', '396837', '396838', '397716', '401831']
nosy_count = 8.0
nosy_names = ['terry.reedy', 'vstinner', 'lukasz.langa', 'steve.dower', 'corona10', 'pablogsal', 'miss-islington', 'xtreak']
pr_nums = ['26693', '26713', '26968', '26969', '26978', '26981', '26986']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue43425'
versions = ['Python 3.10']

[tirkarthi]

distutils was deprecated for removal in Python 3.10. It is used in test_peg_generator.test_c_parser which emits a deprecation warning. It also seems to be used in test_support for missing_compiler_executable that will emit a deprecation warning.

./python -Wall -m test test_peg_generator test_c_parser
0:00:00 load avg: 0.02 Run tests sequentially
0:00:00 load avg: 0.02 [1/2] test_peg_generator
/root/cpython/Lib/test/test_peg_generator/test_c_parser.py:4: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP-632 for potential alternatives
from distutils.tests.support import TempdirManager

Other places on grep :

rg 'from distutils' | rg -v 'Lib/distutils|rst'
Modules/_decimal/tests/formathelper.py:from distutils.spawn import find_executable
Doc/includes/setup.py:from distutils.core import setup, Extension
Doc/includes/test.py:from distutils.util import get_platform
setup.py:from distutils import log
setup.py:from distutils.command.build_ext import build_ext
setup.py:from distutils.command.build_scripts import build_scripts
setup.py:from distutils.command.install import install
setup.py:from distutils.command.install_lib import install_lib
setup.py:from distutils.core import Extension, setup
setup.py:from distutils.errors import CCompilerError, DistutilsError
setup.py:from distutils.spawn import find_executable
Lib/_osx_support.py: from distutils import log
Lib/_osx_support.py: Currently called from distutils.sysconfig
Lib/test/support/init.py: from distutils import ccompiler, sysconfig, spawn, errors
Lib/test/test_importlib/test_windows.py:from distutils.util import get_platform
Lib/test/test_peg_generator/test_c_parser.py:from distutils.tests.support import TempdirManager
Tools/peg_generator/pegen/build.py: from distutils.core import Distribution, Extension
Tools/peg_generator/pegen/build.py: from distutils.command.clean import clean # type: ignore
Tools/peg_generator/pegen/build.py: from distutils.command.build_ext import build_ext # type: ignore
Tools/peg_generator/pegen/build.py: from distutils.tests.support import fixup_build_ext # type: ignore
Tools/test2to3/setup.py:from distutils.core import setup
Tools/test2to3/setup.py: from distutils.command.build_py import build_py_2to3 as build_py
Tools/test2to3/setup.py: from distutils.command.build_py import build_py
Tools/test2to3/setup.py: from distutils.command.build_scripts import build_scripts_2to3 as build_scripts
Tools/test2to3/setup.py: from distutils.command.build_scripts import build_scripts
Tools/test2to3/test/runtests.py: from distutils.util import copydir_run_2to3
Misc/HISTORY:- Issue bpo-5394: removed > 2.3 syntax from distutils.msvc9compiler.

[pablogsal]

We just need to suppress the warning unless I am missing somey. Distutils will still be used internally by CPython.

[tirkarthi]

Thanks for the clarification. As per bpo-41282 I thought disutils will be removed. The accepted PEP-632 states this scenario so I guess silencing the warning is good enough.

The final dependency on distutils is CPython itself, which uses it to build native extension modules in the standard library (except on Windows). Because this is a CPython build-time dependency, it is possible to continue to use distutils for this specific case without it being part of the standard library.

[terryjreedy]

Steve, PEP-632 lacks a reference to the SC acceptance thereof. Could you add one?
https://mail.python.org/archives/list/python-dev@python.org/thread/TXU6TVOMBLQU3SV57DMMOA5Y2E67AW7P/
And can you verify my interpretation below?

The warning is correct as to removal. PEP-632 says, perhaps a bit tersely, that when the 3.11 branch is established for 3.11.0b1 and master/main becomes the 3.12.0a development branch, disutils/* be no longer be installed as part of the stdlib (ie, in /Lib). At that point, 'import disutils' will cease working in installed CPython, outside of a cpython repository. Although this could be worked around by catching the import error and skipping disutils-dependent tests, the dependency should better be removed, if possible, before 3.12.0a.

The PEP gives no timetable for doing so, and anticipates separate PRs like this.

It further says that when "the CPython build process no longer depends on distutils being in the standard library, the entire Lib/distutils directory and Lib/test/test_distutils.py file will be removed from the repository." Leaving it deprecated but present was rejected.

Part of the extensive discussion at https://discuss.python.org/t/pep-632-deprecate-distutils-module/5134/128 was about possibly putting parts of disutiles under /Tools and replacing 'import disutils' with a reference to <python dir>/Tools/distutiles/whatever. If this is done for tests, the access and use has to be conditional. Our Windows installer only installs 4 or 21 Tools subdirectories thought appropriate for uses and others may include less, and maybe omit Tools altogether.

Karthikeyan posted the opening of this and another issue 5 days ago on bpo-41281, with no comment so far.

[zooba]

The link appears to be in the Resolution header of PEP-632?

But yes, your analysis is correct. If/when we move distutils elsewhere, we'll probably have to update all of these sections.

Ideally we'd have a helper function to do the initial "import distutils", handle failure (because it's not in Tools), and suppress the warning. But I don't know that there's anywhere we could put it reliably anyway, so probably just need to suppress the warnings directly and we'll handle failures if/when we move it into Tools.

[corona10]

New changeset 736ed6f by Dong-hee Na in branch 'main':
bpo-43425: Update test_c_parser not to use TempdirManager (GH-26693)
736ed6f

[miss-islington]

New changeset 2b57ad3 by Miss Islington (bot) in branch '3.10':
bpo-43425: Update test_c_parser not to use TempdirManager (GH-26693)
2b57ad3

[corona10]

New changeset c8979f7 by Dong-hee Na in branch 'main':
bpo-43425: Update _osx_support not to use distutils.log (GH-26968)
c8979f7

[miss-islington]

New changeset 94a4136 by Miss Islington (bot) in branch '3.10':
bpo-43425: Update _osx_support not to use distutils.log (GH-26968)
94a4136