[2020-resolver] No longer can get list of available versions.

[vityas]

What did you want to do?

I would like to list all the available versions of a package. In the past we would use this:

pip install pip==asdf  # any gibberish here :)

Output

ERROR: Could not find a version that satisfies the requirement pip==asdf
ERROR: No matching distribution found for pip==asdf

expected output (pip==20.2.4)

ERROR: Could not find a version that satisfies the requirement pip==asdf (from versions: 0.2, 0.2.1, 0.3, 0.3.1, 0.4, 
0.5, 0.5.1, 0.6, 0.6.1, 0.6.2, 0.6.3, 0.7, 0.7.1, 0.7.2, 0.8, 0.8.1, 0.8.2, 0.8.3, 1.0, 1.0.1, 1.0.2, 1.1, 1.2, 
1.2.1, 1.3, 1.3.1, 1.4, 1.4.1, 1.5, 1.5.1, 1.5.2, 1.5.3, 1.5.4, 1.5.5, 1.5.6, 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 
6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.1.0, 6.1.1, 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.1.0, 7.1.1, 7.1.2, 8.0.0, 8.0.1, 8.0.2, 
8.0.3, 8.1.0, 8.1.1, 8.1.2, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 10.0.0b1, 10.0.0b2, 10.0.0, 10.0.1, 18.0, 18.1, 19.0, 
19.0.1, 19.0.2, 19.0.3, 19.1, 19.1.1, 19.2, 19.2.1, 19.2.2, 19.2.3, 19.3, 19.3.1, 20.0, 20.0.1, 20.0.2, 20.1b1, 
20.1, 20.1.1, 20.2b1, 20.2, 20.2.1, 20.2.2, 20.2.3, 20.2.4, 20.3b1)

Additional information

• Using verbose flag: pip install -v pip==asdf we can sorta see that versions. (300 line output, too long to post here)
  Seems like no extra requests needed. I wasn't 100% sure if it was the case.
• This is useful when debugging failed installation of a requirement.txt, pipenv or other 3rd party package managers

[brainwane]

@vityas thank you for this bug report, which I've just discussed with pip maintainer @pradyunsg.

I understand what you're concerned about. It used to be the case that if you ran that particular command, pip would tell you about all the versions of that package available in the index you were hitting (such as PyPI). Now pip doesn't print that info about versions by default, with the new resolver.

This was not a useful error message for people who UNINTENTIONALLY got it, but it was the only way to get (on the command line) a list of all the available versions of a package on PyPI or on another index.

If you're hitting PyPI, you can go the release history page for a project, such as https://pypi.org/project/pip/#history , but with the new resolver behavior, currently, there's no easy way to get this info in the CLI directly and in a concise format. pip hits an index and gets the list, but now doesn't present it by default. And if you turn up verbosity, you get the info but in a hard-to-use format, as you said.

Right now, in order to expedite the release of pip 20.3 this week. I'm marking it as post-release work.

For now, we advise you to (per the deprecation timeline) opt-out and choose the old resolver behavior for now, using the flag --use-deprecated=legacy-resolver.

But also: you're bringing further attention to two problems with pip's functionality more generally. pip has a few functions, such as pip search, which offer a little bit of discoverability to help users see what is available in package indexes. Pradyun wants to consolidate functionality and design for these features. And we know that pip's output, especially its --verbose output, needs work. So I hope you will please talk to our UX people, @nlhkabu @ei8fdb and @georgiamoon, by signing up for the user experience studies, to help them work on the larger project of making our verbose output and our package discoverability functionality better.

Thank you!

[RafalSkolasinski]

I would really love to see this back - it seemed like the most streamlined way to find out what versions one can install. See this highly voted stack answer here.

[pfmoore]

It's definitely not (and never was) a supported way of getting that information. But I can imagine the possibility that we might implement a supported way of getting the same data. Although to be honest, as long as you just want to get the information from PyPI (and don't need to support extra indexes, or local package repositories) it's not that hard to write something yourself that gets the same information, using the PyPI JSON interface, plus packaging.tags for doing the wheel compatibility checks.

Maybe someone could publish a "PyPI query" package that did things like this? It might be an interesting project for someone who wants to get into working with the packaging ecosystem.

[RafalSkolasinski]

I believe that at this point this is just a behaviour that many people may have relied on for years. Sometimes things that were not meant to be present but were just become part of what is expected.

[brainwane]

@RafalSkolasinski Thanks for sharing your experience! Indeed, we're getting a lot of demonstrations of Hyrum's Law as we engage in this work this year. :-)

As you saw in a comment above, when prioritizing what to address and what to leave for later, I decided to leave this particular issue for later. We've had to be incredibly limited in what we build and fix with the funding we got to work on pip this year. It runs out at the end of December. One of the things we did with that funding was set up a more systematic way to hear from our users about how you use pip, what features you use, and so on, to help us make better decisions -- user experience research and testing. I hope you will sign up for the user experience surveys and share your opinion in more depth that way. (For example, you could share whether you always use PyPI or whether this functionality is important to you when installing from other sources as well.)

And the pip team has no reliable future funding, and can do a lot more if companies that depend on Python packaging tools chip in (here's the packaging sponsorship program).

[NoahGorny]

Hello folks!
I had a PR #8978 which proposes a new way to look at the available versions (pip search <packages> --list-all-versions)

If you support this idea, please let folks here know so we can move forward with this offer 😄

cc @ei8fdb

[wimglenn]

This was not a useful error message for people who UNINTENTIONALLY got it

@brainwane I disagree with that, actually it's still useful even when UNINTENTIONALLY getting it. If I did pip install distlib==0.29.post0 and saw:

ERROR: Could not find a version that satisfies the requirement distlib==0.29.post0
(from versions: 0.1.0, 0.1.1, 0.1.2, 0.1.3, 0.1.4, 0.1.5, 0.1.6, 0.1.7, 0.1.8, 0.1.9, 0.2.0, 0.2.1, 
0.2.2, 0.2.3, 0.2.4, 0.2.5, 0.2.6, 0.2.7, 0.2.8, 0.2.9, 0.2.9.post0, 0.3.0, 0.3.1)

That tells me I just had a silly typo and I wanted 0.2.9.post0. But if I saw

ERROR: Could not find a version that satisfies the requirement distlib==0.29.post0 (from versions: none)

It tells me something else, probably that I have misconfigured my index urls and I should check the pip config. In 20.3.x

ERROR: Could not find a version that satisfies the requirement

This is less helpful again. Is my index misconfigured? Could it find any version at all? Do I just need to wait a moment or clear pip's cache - maybe I just published this package recently?

pip search afaik is broken and slated to be removed (#5216). At least, I could not find any way to use it reliably (and couldn't get it working at all with a devpi-server index).

Thank you for mentioning the --use-deprecated=legacy-resolver option! For the purposes of keeping the project johnnydep functioning on pip 20.3.x, this helped for now. Currently it scrapes pip's CLI output in a way that needs careful tweaking and testing each time a new version of pip is released. So here's a bit of UX feedback, it would be really nice if there was a less fragile way to get:

• download urls
• package dependencies
• versions available

It's definitely not (and never was) a supported way of getting that information.

@pfmoore So how do we get that information? A while ago pip moved all package finding code under _internal and made it explicit that a CLI subprocess was the only supported way to use pip from your program. This is arguably an important part of the CLI.

it's not that hard to write something yourself that gets the same information, using the PyPI JSON

I think that is oversimplifying the problem. It might be true if pip was only a PyPI client, but it's a lot more than that isn't it? Pip's the de facto tool for working with Python packages, whether from PyPI or not, since we don't really have one in the stdlib.

To repeat again the points in #1884: the PyPI JSON is not in general "the same information" because where pip looks is highly configurable, e.g. we may have another --index-url and then an --extra-index-url and maybe that index is just a simple repository which is supporting a JSON api at all. There are a lot of little details here and doing it in a way that accurately produces the same packages that pip would see would mean reproducing a significant amount of package finder code found within pip._internal.

[ddelange]

@brainwane thanks for your explanation on why this feature (I'm gonna call it a feature as we rely on it in johnnydep, pipgrip, probably more) was deprecated in the new resolver.

Like mentioned above, it would be quite cumbersome and most likely introducing bugs to move away from using this feature in pip, so I've dropped support for pip 21 (hoping to loosen that constraint again though!), and will keep an eye out for this issue.

Thanks for all your hard work and responses throughout these exciting times! 🙏