Python 3.11 and above adding default module search path from the “PythonPath” registry key if the standard library was found already

[tanzim]

Bug report

Bug description:

Please see https://discuss.python.org/t/behaviour-change-in-python-3-11-compatibility-break-or-genuine-fix/54729

There's also another issue highlited in the same module by the author, but it's not important as we can workaround by setting PYTHONNOUSERSITE as seen below.

"Including import site inadvertently enables the user site packages directory because the configuration with a ._pth file doesn’t explicitly disable it. I think it should because I can’t imagine why an embedding application would ever want that. You’ll have to set PYTHONNOUSERSITE to work around this."

CPython versions tested on:

3.10, 3.11, 3.12

Operating systems tested on:

Windows

[tanzim]

The most relevant part of the discussion that relates to this bug:

It seems you uncovered a bug in “Modules/getpath.py”. It’s not supposed to add the default module search path from the “PythonPath” registry key if the standard library was found already. However, the code is only checking the variables home and stdlib_dir, which aren’t set if the standard library was found in a zip file such as “python311.zip”. A variable needs to be defined when the standard library was found as a zip file, which can be included in the check that determines whether to fetch the default search path from the registry.

[zooba]

It looks like your entire workaround is to add the path you do want into sys.path (i.e. keep the ._pth file), rather than trying to remove the ones you don't want.

Personally, I really dislike the reliance on registry keys for finding the install. Unfortunately, it broke too many people to turn it off (i.e. don't install the default keys). I seem to recall that we also broke people when we accidentally turned it off in an earlier version of 3.11.

In any case, getpath.py is much easier to change than before. Anyone is welcome to submit a PR. If the detection is clear enough, we can probably backport the fix to maintenance releases.

The more important bug (perhaps already filed separately?) is import site in a ._pth file also including user site packages. The only reasonable workaround is to add sys.path.append(site.getusersitepackages()) but it seems fairly reasonable given the context.

[tanzim]

The more important bug (perhaps already filed separately?) is import site in a ._pth file also including user site packages. The only reasonable workaround is to add sys.path.append(site.getusersitepackages()) but it seems fairly reasonable given the context.

I haven't raised a separate issue for this but sounds like I should. I'll also send a patch alongside.

Edit. Raised #120037

[zooba]

I'll also send a patch alongside.

I don't think you've contributed before? The basic idea is to send a PR against main and then we'll worry about backporting it to earlier releases. There's more info at devguide.python.org.

The changes for this one should be entirely in Modules/getpath.py. The other is probably a bit more complex - I don't remember whether the flag for no user site packages can be set from getpath.py.