Using venv creator creates broken environment in Ubuntu 22.04

[icezyclon]

Issue

Using venv as creator - virtualenv --creator=venv myenv - creates a broken virtual environment on Ubuntu 22.04.
It produces two different activate scripts, one in myenv/bin and one in myenv/local/bin.
Depending on which one I activate, either it uses the wrong Python Interpreter or the wrong Pip:

>>> source myenv/bin/activate 
>>> which python3 && which pip3
/home/loki/myenv/bin/python3
/home/loki/.local/bin/pip3
>>> deactivate
>>> source myenv/local/bin/activate
>>> which python3 && which pip3
/usr/bin/python3
/home/loki/myenv/local/bin/pip3

I found this post regarding a change in the sysconfig.get_default_scheme() for Ubuntu 22.04, which might be the reason why virtualenv tries to install into myenv/local/bin instead if just myenv/bin.

Environment

Provide at least:

• OS: Ubuntu 22.04 LTS x86_64
• pip list of the host python where virtualenv is installed:

Package                 Version
----------------------- --------------
apturl                  0.5.2
asn1crypto              1.4.0
bcrypt                  3.2.0
beautifulsoup4          4.10.0
blinker                 1.4
Brlapi                  0.8.3
certifi                 2020.6.20
chardet                 4.0.0
click                   8.0.3
colorama                0.4.4
command-not-found       0.3
cryptography            3.4.8
cupshelpers             1.0
dbus-python             1.2.18
defer                   1.0.6
distlib                 0.3.4
distro                  1.7.0
distro-info             1.1build1
dnspython               2.1.0
duplicity               0.8.21
evdev                   1.4.0
fasteners               0.14.1
filelock                3.6.0
future                  0.18.2
gitinspector            0.4.4
gpg                     1.16.0-unknown
html5lib                1.1
httplib2                0.20.2
idna                    3.3
importlib-metadata      4.6.4
jeepney                 0.7.1
keyring                 23.5.0
language-selector       0.1
launchpadlib            1.10.16
lazr.restfulclient      0.14.4
lazr.uri                1.0.6
lockfile                0.12.2
louis                   3.20.0
lxml                    4.8.0
macaroonbakery          1.3.1
Mako                    1.1.3
Markdown                3.3.6
MarkupSafe              2.0.1
mechanize               0.4.7
menulibre               2.2.2
monotonic               1.6
more-itertools          8.10.0
netifaces               0.11.0
numpy                   1.21.5
oauthlib                3.2.0
olefile                 0.46
paramiko                2.9.3
pbr                     5.9.0
pexpect                 4.8.0
Pillow                  9.0.1
pip                     22.0.4
platformdirs            2.5.1
powerline-status        2.8.2
protobuf                3.12.4
psutil                  5.9.0
ptyprocess              0.7.0
pycairo                 1.20.1
pycups                  2.0.1
Pygments                2.11.2
PyGObject               3.42.0
PyJWT                   2.3.0
pymacaroons             0.13.0
PyNaCl                  1.5.0
pyparsing               2.4.7
pyRFC3339               1.1
python-apt              2.3.0+ubuntu2
python-dateutil         2.8.1
python-debian           0.1.43ubuntu1
python-magic            0.4.24
pytz                    2022.1
pyxdg                   0.27
PyYAML                  5.4.1
reportlab               3.6.8
requests                2.25.1
requests-toolbelt       0.9.1
scour                   0.38.2
screen-resolution-extra 0.0.0
SecretStorage           3.3.1
setproctitle            1.2.2
setuptools              62.1.0
six                     1.16.0
soupsieve               2.3.1
sshuttle                1.0.5
stevedore               3.5.0
systemd-python          234
ubuntu-advantage-tools  27.8
ubuntu-drivers-common   0.0.0
ufw                     0.36.1
unattended-upgrades     0.1
urllib3                 1.26.5
usb-creator             0.3.7
vboxapi                 1.0
virtualenv              20.14.1
virtualenv-clone        0.5.7
virtualenvwrapper       4.8.4
wadllib                 1.3.6
webencodings            0.5.1
wheel                   0.37.1
xdg                     5
xkit                    0.0.0
zipp                    1.0.0

Output of the virtual environment creation

virtualenv --creator=venv myenv -vvv --with-traceback

287 setup logging to NOTSET [DEBUG report:39]
291 find interpreter for spec PythonSpec(path=/usr/bin/python3) [INFO builtin:62]
291 proposed PythonInfo(spec=CPython3.10.4.final.0-64, exe=/usr/bin/python3, platform=linux, version='3.10.4 (main, Apr  2 2022, 09:04:19) [GCC 11.2.0]', encoding_fs_io=utf-8-utf-8) [INFO builtin:69]
291 accepted PythonInfo(spec=CPython3.10.4.final.0-64, exe=/usr/bin/python3, platform=linux, version='3.10.4 (main, Apr  2 2022, 09:04:19) [GCC 11.2.0]', encoding_fs_io=utf-8-utf-8) [DEBUG builtin:71]
292 filesystem is case-sensitive [DEBUG info:29]
308 create virtual environment via Venv(dest=/home/loki/myenv, clear=False, no_vcs_ignore=False, global=False, describe=CPython3Posix) [INFO session:52]
309 create folder /home/loki/myenv/local/lib/python3.10/dist-packages [DEBUG _sync:24]
309 create virtualenv import hook file /home/loki/myenv/local/lib/python3.10/dist-packages/_virtualenv.pth [DEBUG api:95]
309 create /home/loki/myenv/local/lib/python3.10/dist-packages/_virtualenv.py [DEBUG api:98]
309 ============================== target debug ============================== [DEBUG session:54]
310 debug via /home/loki/myenv/local/bin/python /home/loki/.local/lib/python3.10/site-packages/virtualenv/create/debug.py [DEBUG creator:224]
309 {
  "out": "",
  "err": "No such file or directory",
  "returncode": 2,
  "exception": "SyntaxError('invalid syntax', ('<unknown>', 0, 0, '', 0, 0))"
} [DEBUG session:55]
310 add seed packages via FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/loki/.local/share/virtualenv) [INFO session:59]
311 got embed update of distribution pip from /home/loki/.local/share/virtualenv/wheel/3.10/embed/3/pip.json [DEBUG via_disk_folder:135]
312 got embed update of distribution wheel from /home/loki/.local/share/virtualenv/wheel/3.10/embed/3/wheel.json [DEBUG via_disk_folder:135]
313 got embed update of distribution setuptools from /home/loki/.local/share/virtualenv/wheel/3.10/embed/3/setuptools.json [DEBUG via_disk_folder:135]
314 got embed update of distribution setuptools from /home/loki/.local/share/virtualenv/wheel/3.10/embed/3/setuptools.json [DEBUG via_disk_folder:135]
314 got embed update of distribution pip from /home/loki/.local/share/virtualenv/wheel/3.10/embed/3/pip.json [DEBUG via_disk_folder:135]
314 got embed update of distribution wheel from /home/loki/.local/share/virtualenv/wheel/3.10/embed/3/wheel.json [DEBUG via_disk_folder:135]
315 install setuptools from wheel /home/loki/.local/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/setuptools-62.1.0-py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:49]
315 install pip from wheel /home/loki/.local/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/pip-22.0.4-py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:49]
315 Attempting to acquire lock 140460470067568 on /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-22.0.4-py3-none-any.lock [DEBUG _api:169]
315 Attempting to acquire lock 140460469877152 on /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-62.1.0-py3-none-any.lock [DEBUG _api:169]
315 Lock 140460470067568 acquired on /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-22.0.4-py3-none-any.lock [DEBUG _api:173]
315 Lock 140460469877152 acquired on /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-62.1.0-py3-none-any.lock [DEBUG _api:173]
315 install wheel from wheel /home/loki/.local/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/wheel-0.37.1-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:49]
316 Attempting to release lock 140460470067568 on /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-22.0.4-py3-none-any.lock [DEBUG _api:203]
316 Lock 140460470067568 released on /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-22.0.4-py3-none-any.lock [DEBUG _api:206]
316 Attempting to acquire lock 140460467461504 on /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.37.1-py2.py3-none-any.lock [DEBUG _api:169]
316 Attempting to release lock 140460469877152 on /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-62.1.0-py3-none-any.lock [DEBUG _api:203]
316 Lock 140460467461504 acquired on /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.37.1-py2.py3-none-any.lock [DEBUG _api:173]
316 Lock 140460469877152 released on /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-62.1.0-py3-none-any.lock [DEBUG _api:206]
316 Attempting to release lock 140460467461504 on /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.37.1-py2.py3-none-any.lock [DEBUG _api:203]
316 copy directory /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-22.0.4-py3-none-any/pip-22.0.4.dist-info to /home/loki/myenv/local/lib/python3.10/dist-packages/pip-22.0.4.dist-info [DEBUG _sync:51]
316 Lock 140460467461504 released on /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.37.1-py2.py3-none-any.lock [DEBUG _api:206]
317 copy /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-62.1.0-py3-none-any/setuptools-62.1.0.virtualenv to /home/loki/myenv/local/lib/python3.10/dist-packages/setuptools-62.1.0.virtualenv [DEBUG _sync:51]
317 copy /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.37.1-py2.py3-none-any/wheel-0.37.1.virtualenv to /home/loki/myenv/local/lib/python3.10/dist-packages/wheel-0.37.1.virtualenv [DEBUG _sync:51]
317 copy /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-62.1.0-py3-none-any/distutils-precedence.pth to /home/loki/myenv/local/lib/python3.10/dist-packages/distutils-precedence.pth [DEBUG _sync:51]
317 copy directory /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.37.1-py2.py3-none-any/wheel to /home/loki/myenv/local/lib/python3.10/dist-packages/wheel [DEBUG _sync:51]
318 copy directory /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-62.1.0-py3-none-any/_distutils_hack to /home/loki/myenv/local/lib/python3.10/dist-packages/_distutils_hack [DEBUG _sync:51]
319 copy directory /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-62.1.0-py3-none-any/setuptools to /home/loki/myenv/local/lib/python3.10/dist-packages/setuptools [DEBUG _sync:51]
319 copy directory /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-22.0.4-py3-none-any/pip to /home/loki/myenv/local/lib/python3.10/dist-packages/pip [DEBUG _sync:51]
322 copy directory /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.37.1-py2.py3-none-any/wheel-0.37.1.dist-info to /home/loki/myenv/local/lib/python3.10/dist-packages/wheel-0.37.1.dist-info [DEBUG _sync:51]
326 generated console scripts wheel3.10 wheel3 wheel-3.10 wheel [DEBUG base:45]
347 copy directory /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-62.1.0-py3-none-any/pkg_resources to /home/loki/myenv/local/lib/python3.10/dist-packages/pkg_resources [DEBUG _sync:51]
352 copy directory /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-62.1.0-py3-none-any/setuptools-62.1.0.dist-info to /home/loki/myenv/local/lib/python3.10/dist-packages/setuptools-62.1.0.dist-info [DEBUG _sync:51]
354 generated console scripts  [DEBUG base:45]
374 copy /home/loki/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-22.0.4-py3-none-any/pip-22.0.4.virtualenv to /home/loki/myenv/local/lib/python3.10/dist-packages/pip-22.0.4.virtualenv [DEBUG _sync:51]
375 generated console scripts pip3 pip pip3.10 pip-3.10 [DEBUG base:45]
375 add activators for Bash, CShell, Fish, Nushell, PowerShell, Python [INFO session:64]
376 write /home/loki/myenv/pyvenv.cfg [DEBUG pyenv_cfg:34]
376     home = /usr/bin [DEBUG pyenv_cfg:38]
376     implementation = CPython [DEBUG pyenv_cfg:38]
376     version_info = 3.10.4.final.0 [DEBUG pyenv_cfg:38]
376     virtualenv = 20.14.1 [DEBUG pyenv_cfg:38]
376     include-system-site-packages = false [DEBUG pyenv_cfg:38]
376     version = 3.10.4 [DEBUG pyenv_cfg:38]
376 created virtual environment CPython3.10.4.final.0-64 in 90ms
  creator Venv(dest=/home/loki/myenv, clear=False, no_vcs_ignore=False, global=False, describe=CPython3Posix)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/loki/.local/share/virtualenv)
    added seed packages: pip==22.0.4, setuptools==62.1.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator [WARNING __main__:19]

[icezyclon]

As a follow up question I wanted to ask:
Is there a way to make virtualenv install into myenv/bin instead of myenv/local/binagain, possibly ignoring the posix_local prefix?
Thank you! 💯

[d3al]

Can confirm, I'm having the same issue with PopOS 22.04

[gaborbernat]

Posting a docker image that reproduces this would be helpful here.

[icezyclon]

Posting a docker image that reproduces this would be helpful here.

I am not an expert with Docker, but I could not reproduce this bug in the ubuntu:22.04 image.
This kind of does mean that it is not just sysconfig.get_default_scheme() == 'posix_local' that is responsible for the bug, as virtualenv does install into myenv/bin in the Docker image...

However, the Ubuntu Docker image is much more slim and includes way fewer debs than the Ubuntu Desktop ISO and of course no Gnome or similar. There could easily be some configuration in the Desktop ISO that is not included in the Docker image.

What I tried:

FROM ubuntu:22.04

RUN	apt-get update \
    && apt-get upgrade -y \
    && apt-get install -y \
    python3 \
    python3-pip \
    python3-venv \
    python3-virtualenv \
    && apt-get clean

CMD ["/usr/bin/bash"]

putting the above into a Dockerfile in a directy, then building and running it with:

docker build --tag=virtualenvtest .
docker run -it virtualenvtest
python3 -c "import sysconfig; print(sysconfig.get_default_scheme())"
virtualenv --version
virtualenv --creator=venv myenv -vvv --with-traceback
ls myenv/bin
ls myenv/local/bin   # says there is no myenv/local directory

Again, this does not reproduce the bug.
If someone else could take a look that would be great.

We do have someone with the same problem here so it does seem to not only be on my system.

[bozhodimitrov]

Similar issue manifested for me as well, after manual upgrade from 21.04 to 22.04, so it might be related to the upgrade process.
If it is not reproducible on a clean 22.04 container, maybe try to upgrade existing container from 21.04 to 22.04.

For me, the result is that I get a complete random list of files/dirs inside my virtualenv dir.

Sometimes it's like this:

user@host:~$ ls -a venv
.  ..  .gitignore  lib  local  pyvenv.cfg

Sometimes, there is a bin folder with just the symlinks ...

user@host:~$ ls venv/bin/
python  python3  python3.10

I am using the latest available virtualenv from https://bootstrap.pypa.io/virtualenv.pyz

[icezyclon]

Similar issue manifested for me as well, after manual upgrade from 21.04 to 22.04, so it might be related to the upgrade process. [...]

I just wanted to add that my 22.04 installation was a fresh install, so it might not be directly related to the upgrade process but some settings or config from 22.04 in general.

[d3al]

I wanted to add. I initially upgraded from PopOS 21.10 to 22.04 and that's when things fell apart. My django project wouldn't run at all, couldn't update pipenv packages so I recreated my interpreter and it failed. I had other issues since the upgrade not related to virtualenv so I reinstall the OS.

I did a refresh, so the OS was reinstalled, which keeps your home directory but I lost all of my apps. Issue with virtual env was there but the other issues and errors went away. I'm getting the /bin and /lib folders inside a local folder when I run virtualenv manually. My PyCharm IDE fails to create the env as well. I can install the OS on a virtual machine and see if I get the same results.

[d3al]

I installed PopOS 22.04 on a virtual machine and did two tests, the first was with no system updates after install and the other was after installing system updates. Both test created the lib and bin in .venv/.

sudo apt-get install python3-pip
sudo pip3 install virtualenv
mkdir project
cd project
virtualenv .venv

On my machine, I ran the same commands and I get a lib and local folder. Inside the local I have lib and bin.
I also ran the commands again as sudo and got the local folder again.

I'm not sure what I can do to help.

[icezyclon]

I believe that if we knew why/when virtualenv installs into the local folder, then we can get to the bottom of this.
At the very least, disabling creating this local folder should create a normal install, right?

[gaborbernat]

PR addressing the underlying issue is welcome.

[thenewnano]

if we are to attempt a fix, we need a bit of guidance as to where should we start?

[gaborbernat]

Finding out what is happening and how to fix it is like 80% of the work needed here 😅 so what questions do you have?