Implement PEP 518 support

[brettcannon]

https://www.python.org/dev/peps/pep-0518/

This is the pyproject.toml stuff.

[takluyver]

I will attempt to implement this.

[takluyver]

I made a start (branch), but I realised I'm not sure what to do about environments. Currently, it looks like wheels are built in the Python environment that's already active. PEP 517 says:

A build frontend SHOULD, by default, create an isolated environment for each build, containing only the standard library and any explicitly requested build-dependencies.

That's not a requirement, but it is a justified recommendation. Creating a new environment would probably break installation of existing packages which rely on something else being installed first (e.g. using numpy.distutils). We can probably get round that by creating a new environment only if pyproject.toml exists.

However, how should pip create a new environment? On Python 3, it can use the stdlib venv module, but that's not present on Python 2. We could vendor virtualenv for Python 2, but that seems a bit ugly, since virtualenv in turn bundles pip. Options:

1. Install build dependencies into the current environment when running pip wheel, or autobuilding wheels for pip install.
2. Bundle virtualenv for Python 2, and use it or venv to create new environments to install build dependencies into. Put up with the circular bundling between pip and virtualenv.
3. Bundle a backport of venv for Python 2 (would this only work on Python >= 2.7.9 with ensurepip?)
4. Implement simple environment creation for the wheel building. From the environment requirements specified, I think it's sufficient to install to a temporary prefix and then modify PATH and PYTHONPATH to make the packages available to subprocesses.
5. Use venv on Python 3, but install build dependencies without creating an environment on Python 2 (this would give confusing different behaviours).

[dstufft]

I think the fourth option makes the most sense.

…

Sent from my iPhone

On Nov 25, 2016, at 10:12 AM, Thomas Kluyver ***@***.***> wrote: I made a start (branch), but I realised I'm not sure what to do about environments. Currently, it looks like wheels are built in the Python environment that's already active. PEP 517 says: A build frontend SHOULD, by default, create an isolated environment for each build, containing only the standard library and any explicitly requested build-dependencies. That's not a requirement, but it is a justified recommendation. Creating a new environment would probably break installation of existing packages which rely on something else being installed first (e.g. using numpy.distutils). We can probably get round that by creating a new environment only if pyproject.toml exists. However, how should pip create a new environment? On Python 3, it can use the stdlib venv module, but that's not present on Python 2. We could vendor virtualenv for Python 2, but that seems a bit ugly, since virtualenv in turn bundles pip. Options: Install build dependencies into the current environment when running pip wheel, or autobuilding wheels for pip install. Bundle virtualenv for Python 2, and use it or venv to create new environments to install build dependencies into. Put up with the circular bundling between pip and virtualenv. Bundle a backport of venv for Python 2 (would this only work on Python >= 2.7.9 with ensurepip?) Implement simple environment creation for the wheel building. From the environment requirements specified, I think it's sufficient to install to a temporary prefix and then modify PATH and PYTHONPATH to make the packages available to subprocesses. Use venv on Python 3, but install build dependencies without creating an environment on Python 2 (this would give confusing different behaviours). — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[pfmoore]

Agreed, option 4 seems reasonable. I don't think the additional complexities of virtualenv/venv are necessary for this use case, just manipulating paths seems like an entirely reasonable approach.

[njsmith]

There are two potential advantages to using isolated installs:

• We don't want to pollute the user's actual environment by dumping build requirements into it (esp. since different packages might have conflicting build requirements).
• To minimize the number of broken packages that get uploaded to PyPI, we want to fail fast when there are build requirements that the user happens to have installed locally when testing, but that aren't actually listed in the build-requirements -- so ideally the build should only be able to "see" packages that it has listed.

Option 4 solves the first problem, but doesn't do anything for the second.

The second is a "would be nice" thing, so even partial solutions are helpful and it's okay if we're not perfect to start. But Option 4 + as much isolation as we can conveniently manage would be nice.

I think the most we can easily do is to invoke the build process with PYTHONNOUSERSITE=1 and -S. (This removes both user site-packages and system site-packages from the build process's sys.path.) It's not quite perfect because the -S won't be inherited if the build process in turn spawns another child python -- and unfortunately there doesn't seem to be any envvar equivalent to -S. But it should still be good enough to catch a lot of silly mistakes.

And Rob will get grumpy at me if I don't point out that there should probably be an option the user can pass to pip to disable the isolation, for those cases where you know the package has a broken build-requirements and you want to override them. (Or maybe there should just be an option to override the build-requirements directly?)

[njsmith]

Oh, and yeah, agreed that we'd have to default to no-isolation mode for legacy projects that don't have pyproject.toml.

[brettcannon]

My worry with option 4 is what if Python changes what is required for a venv in some future version? One of the key reasons of creating the venv module was so that virtual environment creation could be tied to the interpreter as necessary. By going with option 4 you break the abstraction of virtual environments through venv/virtualenv.

We are talking about wheel building, which shouldn't be as common as installation. Having virtualenv installed is also very common. Would it be so bad to simply use venv if available, fall back to virtualenv, and then if neither is available to error out and ask the user to install it? Anyone with a newer version of Python 2.7 will have pip so getting it installed shouldn't be that much of a burden (even if it means a virtual environment to create a virtual environment 😄), plus if they're doing builds with pip you can assume they know how to use pip to install something, so providing instructions to install virtualenv shouldn't be a burden.

[pfmoore]

My biggest concern here is that we could significantly increase build times. As @brettcannon says, the whole build mechanism should only be invoked when there's no wheel available, and given local wheel caches plus the availability of more and more wheels on PyPI, this is not as big an issue as it once was. But the worst hit will be on simple, pure-Python packages with no build dependencies. At the moment, we just run setup.py bdist_wheel. Changing that to unpacking a setuptools wheel, setting PYTHONPATH and running setup.py bdist_wheel isn't a huge hit. Changing it to create a complete venv might be. I honestly don't have a feel for the difference.

Some other thoughts:

1. To install build dependencies into a venv, we'd need pip installed in that venv. That's yet more time to build the venv, and we'd in theory need to uninstall pip before starting the build (in practice, I can't see pip counting as "polluting" the venv, though).
2. To install using option 4, I assume we'd do pip install -t, but that doesn't install scripts, so we have an issue in that the requirement "All command-line scripts provided by the build-required packages must be present in the build environment's PATH" would need extra work to satisfy.

Given the second of these, I'm starting to think we might need to go down the venv/virtualenv route as Brett suggests. I'd also be inclined to allow for a pip wheel --non-isolated option to bypass the building of an isolated environment (as suggested in the PEP, although it talks in terms of a system-site-packages venv) to allow Python 2.7 users to build without needing to install virtualenv.

And while premature optimisation and all that, I think we should consider reusing the venv. I'd hate pip install foo bar baz spam ham another etc to potentially create 7 venvs. And if we ever get an "upgrade all" command (or users simulate their own via scripts) things could get nasty very fast.

[RonnyPfannschmidt]

I dont think virtualenv in its own is even in the direction of a solution

Buildout and easy_install silver this problem using working sets and multi version installs,
Nixos has a very similar solution where creating environments equals creating a tree of symlinks

I believe in order to support the PEPs adequately there is a need to disengage from the model of virtualenv, which was ever since its inception expensive