3.11.0a3: under tox, sys._base_executable is wrong

[nedbat]

BPO	46028
Nosy	@vsajip, @vstinner, @tiran, @nedbat, @zooba, @pablogsal, @saaketp
PRs	bpo-46028: Calculate base_executable by resolving symlinks in a venv #30144

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-12-10.00:24:21.754>
labels = ['3.11']
title = '3.11.0a3: under tox, sys._base_executable is wrong'
updated_at = <Date 2022-01-18.15:47:49.048>
user = 'https://github.com/nedbat'

bugs.python.org fields:

activity = <Date 2022-01-18.15:47:49.048>
actor = 'steve.dower'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2021-12-10.00:24:21.754>
creator = 'nedbat'
dependencies = []
files = []
hgrepos = []
issue_num = 46028
keywords = ['patch', '3.11regression']
message_count = 30.0
messages = ['408166', '408167', '408168', '408180', '408195', '408206', '408278', '408311', '408312', '408313', '408318', '408319', '408483', '408484', '408489', '408490', '408491', '408492', '408493', '408494', '408525', '408526', '408528', '408569', '408692', '408710', '408744', '410850', '410872', '410873']
nosy_count = 7.0
nosy_names = ['vinay.sajip', 'vstinner', 'christian.heimes', 'nedbat', 'steve.dower', 'pablogsal', 'saaketp']
pr_nums = ['30144']
priority = 'normal'
resolution = None
stage = 'commit review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue46028'
versions = ['Python 3.11']

[nedbat]

Under tox, sys._base_executable is not an actual file for 3.11.0a3. It was fine in 3.11.0a2.

To reproduce:

--- 8< --------------------
# tox.ini
[tox]
envlist = py{310,311a2,311}
skipsdist = True

[testenv]
commands =
python -c "import sys,os.path; print(e := sys._base_executable); print(os.path.exists(e))"

[testenv:py311a2]
# This is the path to 3.11.0a2 if you have it.
basepython = /usr/local/pyenv/pyenv/versions/3.11.0a2/bin/python3
----------------------------

Create a new Python 3.8 virtualenv, and install latest tox (3.24.4 for me).

Then run "tox". I see:

--------------------------------------------------------------------------------
py310 create: /Users/nedbatchelder/coverage/lab/fix-3.11a3/.tox/py310
py310 run-test-pre: PYTHONHASHSEED='534434199'
py310 run-test: commands[0] | python -c 'import sys,os.path; print(e := sys._base_executable); print(os.path.exists(e))'
/Users/nedbatchelder/coverage/lab/fix-3.11a3/.tox/py310/bin/python
True
py311a2 create: /Users/nedbatchelder/coverage/lab/fix-3.11a3/.tox/py311a2
py311a2 run-test-pre: PYTHONHASHSEED='534434199'
py311a2 run-test: commands[0] | python -c 'import sys,os.path; print(e := sys._base_executable); print(os.path.exists(e))'
/Users/nedbatchelder/coverage/lab/fix-3.11a3/.tox/py311a2/bin/python
True
py311 create: /Users/nedbatchelder/coverage/lab/fix-3.11a3/.tox/py311
py311 run-test-pre: PYTHONHASHSEED='534434199'
py311 run-test: commands[0] | python -c 'import sys,os.path; print(e := sys._base_executable); print(os.path.exists(e))'
/usr/local/pyenv/pyenv/versions/3.11.0a3/python
False
_________________________________________________________ summary _________________________________________________________
py310: commands succeeded
py311a2: commands succeeded
py311: commands succeeded
congratulations :)
--------------------------------------------------------------------------------

This came to my attention because the coverage.py test suite uses "python -m venv" to create environments. They worked under 3.11.0a2, but failed under a3. I tracked it down to a difference in sys._base_executable.

I couldn't see a difference in those values without tox, but I'm not sure why it changes the results.

[pablogsal]

Steve, could this be related to the changes in getpath?

[pablogsal]

Ned, are you able to bisect this or provide a simpler reproducer that doesn't involve tox?

[tiran]

Commit 9f2f7e4 has correct _base_executable

$ venv/bin/python
Python 3.11.0a2+ (heads/bpo-45847-simple-115-g9f2f7e42269:9f2f7e42269, Dec 10 2021, 10:09:54) [GCC 11.2.1 20211203 (Red Hat 11.2.1-7)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys._base_executable
'/home/heimes/dev/python/cpython/venv/bin/python'

_base_executable in commit 99fcf15 is wrong

$ venv/bin/python
Python 3.11.0a2+ (heads/bpo-45847-simple-116-g99fcf150521:99fcf150521, Dec 10 2021, 10:12:35) [GCC 11.2.1 20211203 (Red Hat 11.2.1-7)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys._base_executable
'/home/heimes/dev/python/cpython/python'

[nedbat]

git bisect also identifies that commit as the first bad:

99fcf15 is the first bad commit
commit 99fcf15
Author: Steve Dower <steve.dower@python.org>
Date: Fri Dec 3 00:08:42 2021 +0000

bpo-45582: Port getpath[p].c to Python (GH-29041)

The getpath.py file is frozen at build time and executed as code over a namespace. It is never imported, nor is it meant to be importable or reusable. However, it should be easier to read, modify, and patch than the previous code.

This commit attempts to preserve every previously tested quirk, but these may be changed in the future to better align platforms.

[pablogsal]

Indeed, seems my original hunch is correct. Steve, could you take a look when you have some time?

[zooba]

I'm going to need a decent amount of time to learn all of these components, because I never use this OS, Tox, nor virtualenv :) I'll try and get to it, but don't hold your breath. Luckily, Modules/getpath.py is much easier to follow and modify than the old systems.

If there's a pyvenv.cfg involved, base_executable should be calculated based on the "home" key in it. Previously, I don't think we calculated it at all on Linux - it was just sys.executable before site.py changes anything.

On Windows, it was always intended to be "the executable that new venvs should be created with" so that venvs created from venvs would be based off the same install, rather than trying to chain. I have no idea what the correct path for that is in this context, so could do with some help.

[nedbat]

Tox isn't needed, just venv from the stdlib:

$ python3.11.0a2 -m venv venv_a2

$ venv_a2/bin/python -c "import sys,os.path; print(e := sys._base_executable); print(os.path.exists(e))"
/private/tmp/venv_a2/bin/python
True

$ python3.11.0a3 -m venv venv_a3

$ venv_a3/bin/python -c "import sys,os.path; print(e := sys._base_executable); print(os.path.exists(e))"
/usr/local/bin/python
False

[zooba]

What's the contents of the pyvenv.cfg in these cases?

It looks like the first case is definitely wrong, because the base
executable should not be in "venv_a2" (that's sys.executable), but I
don't know where it should be on your system.

[nedbat]

The two venvs seem analogous:

$ cat venv_a2/pyvenv.cfg
home = /usr/local/bin
include-system-site-packages = false
version = 3.11.0

$ ls -al venv_a2/bin
total 72
drwxr-xr-x  13 nedbatchelder  wheel   416 Dec 11 10:43 ./
drwxr-xr-x   6 nedbatchelder  wheel   192 Dec 11 10:43 ../
-rw-r--r--   1 nedbatchelder  wheel  9033 Dec 11 10:43 Activate.ps1
-rw-r--r--   1 nedbatchelder  wheel  1993 Dec 11 10:43 activate
-rw-r--r--   1 nedbatchelder  wheel   919 Dec 11 10:43 activate.csh
-rw-r--r--   1 nedbatchelder  wheel  2061 Dec 11 10:43 activate.fish
-rwxr-xr-x   1 nedbatchelder  wheel   244 Dec 11 10:43 pip*
-rwxr-xr-x   1 nedbatchelder  wheel   244 Dec 11 10:43 pip3*
-rwxr-xr-x   1 nedbatchelder  wheel   244 Dec 11 10:43 pip3.11*
lrwxr-xr-x   1 nedbatchelder  wheel    14 Dec 11 10:43 python@ -> python3.11.0a2
lrwxr-xr-x   1 nedbatchelder  wheel    14 Dec 11 10:43 python3@ -> python3.11.0a2
lrwxr-xr-x   1 nedbatchelder  wheel    14 Dec 11 10:43 python3.11@ -> python3.11.0a2
lrwxr-xr-x   1 nedbatchelder  wheel    29 Dec 11 10:43 python3.11.0a2@ -> /usr/local/bin/python3.11.0a2

$ cat venv_a3/pyvenv.cfg
home = /usr/local/bin
include-system-site-packages = false
version = 3.11.0

$ ls -al venv_a3/bin
total 72
drwxr-xr-x  13 nedbatchelder  wheel   416 Dec 11 10:43 ./
drwxr-xr-x   6 nedbatchelder  wheel   192 Dec 11 10:43 ../
-rw-r--r--   1 nedbatchelder  wheel  9033 Dec 11 10:43 Activate.ps1
-rw-r--r--   1 nedbatchelder  wheel  1993 Dec 11 10:43 activate
-rw-r--r--   1 nedbatchelder  wheel   919 Dec 11 10:43 activate.csh
-rw-r--r--   1 nedbatchelder  wheel  2061 Dec 11 10:43 activate.fish
-rwxr-xr-x   1 nedbatchelder  wheel   244 Dec 11 10:43 pip*
-rwxr-xr-x   1 nedbatchelder  wheel   244 Dec 11 10:43 pip3*
-rwxr-xr-x   1 nedbatchelder  wheel   244 Dec 11 10:43 pip3.11*
lrwxr-xr-x   1 nedbatchelder  wheel    14 Dec 11 10:43 python@ -> python3.11.0a3
lrwxr-xr-x   1 nedbatchelder  wheel    14 Dec 11 10:43 python3@ -> python3.11.0a3
lrwxr-xr-x   1 nedbatchelder  wheel    14 Dec 11 10:43 python3.11@ -> python3.11.0a3
lrwxr-xr-x   1 nedbatchelder  wheel    29 Dec 11 10:43 python3.11.0a3@ -> /usr/local/bin/python3.11.0a3

[saaketp]

I tried the same stuff as nedbat on WSL2, and I see similar change in the path of sys._base_executable (though I get a different "base" path on a3, so the path exists even there).

$ ~/.pyenv/versions/3.11.0a2/bin/python -m venv venv_a2
$ ~/.pyenv/versions/3.11.0a3/bin/python -m venv venv_a3
$ venv_a2/bin/python -c "import sys,os.path; print(e := sys._base_executable); print(os.path.exists(e))"
/home/ss/venv_a2/bin/python
True

$ venv_a3/bin/python -c "import sys,os.path; print(e := sys._base_executable); print(os.path.exists(e))"
/home/ss/.pyenv/versions/3.11.0a3/bin/python
True

$ cat venv_a2/pyvenv.cfg
home = /home/ss/.pyenv/versions/3.11.0a2/bin
include-system-site-packages = false
version = 3.11.0

$ cat venv_a3/pyvenv.cfg
home = /home/ss/.pyenv/versions/3.11.0a3/bin
include-system-site-packages = false
version = 3.11.0