pip 23.0: `pip list` fails with error "ValueError: not enough values to unpack" for archives installed from URL

[chopeen]

Description

When a project contains installed archives installed by specifying URL (e.g. spaCy models), pip list does not display the list of installed packages, because it crashes.

Probable cause: 38681f3

Expected behavior

No response

pip version

pip 23.0

Python version

Python 3.8.16

OS

Ubuntu 20.04.5 LTS (Focal Fossa)

How to Reproduce

pyenv virtualenv 3.8.16 pip-issue
pyenv local pip-issue

cat << EOF > ./pyproject.toml
[tool.poetry]
name = "pip_issue"
version = "0.0.1"
authors = [ "Foo Bar <foo@example.com>" ]
description = "pip issue"

[tool.poetry.dependencies]
python = ">=3.8, <3.11"
en_core_web_sm = { url = "https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.5.0/en_core_web_sm-3.5.0.tar.gz" }

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
EOF

pip install --upgrade poetry
poetry install

# ✅ previous version works fine
pip list  

# 💥 latest version crashes
python3.8 -m pip install --upgrade pip
pip list

Output

$ pip list                              
ERROR: Exception:
Traceback (most recent call last):
  File "/home/users/foo/.pyenv/versions/pip-issue/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper
    status = run_func(*args)
  File "/home/users/foo/.pyenv/versions/pip-issue/lib/python3.8/site-packages/pip/_internal/commands/list.py", line 192, in run
    self.output_package_listing(packages, options)
  File "/home/users/foo/.pyenv/versions/pip-issue/lib/python3.8/site-packages/pip/_internal/commands/list.py", line 273, in output_package_listing
    data, header = format_for_columns(packages, options)
  File "/home/users/foo/.pyenv/versions/pip-issue/lib/python3.8/site-packages/pip/_internal/commands/list.py", line 316, in format_for_columns
    has_editables = any(x.editable for x in pkgs)
  File "/home/users/foo/.pyenv/versions/pip-issue/lib/python3.8/site-packages/pip/_internal/commands/list.py", line 316, in <genexpr>
    has_editables = any(x.editable for x in pkgs)
  File "/home/users/foo/.pyenv/versions/pip-issue/lib/python3.8/site-packages/pip/_internal/metadata/base.py", line 338, in editable
    return bool(self.editable_project_location)
  File "/home/users/foo/.pyenv/versions/pip-issue/lib/python3.8/site-packages/pip/_internal/metadata/base.py", line 176, in editable_project_location
    direct_url = self.direct_url
  File "/home/users/foo/.pyenv/versions/pip-issue/lib/python3.8/site-packages/pip/_internal/metadata/base.py", line 306, in direct_url
    return DirectUrl.from_json(content)
  File "/home/users/foo/.pyenv/versions/pip-issue/lib/python3.8/site-packages/pip/_internal/models/direct_url.py", line 217, in from_json
    return cls.from_dict(json.loads(s))
  File "/home/users/foo/.pyenv/versions/pip-issue/lib/python3.8/site-packages/pip/_internal/models/direct_url.py", line 200, in from_dict
    ArchiveInfo._from_dict(_get(d, dict, "archive_info")),
  File "/home/users/foo/.pyenv/versions/pip-issue/lib/python3.8/site-packages/pip/_internal/models/direct_url.py", line 124, in _from_dict
    return cls(hash=_get(d, str, "hash"), hashes=_get(d, dict, "hashes"))
  File "/home/users/foo/.pyenv/versions/pip-issue/lib/python3.8/site-packages/pip/_internal/models/direct_url.py", line 111, in __init__
    hash_name, hash_value = hash.split("=", 1)
ValueError: not enough values to unpack (expected 2, got 1)

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[uranusjr]

poetry install

Can this be reproduced without involving Poetry? (i.e. change this to pip install .)

[chopeen]

No, it's not reproducible without Poetry.

Only the environment created with Poetry contains file direct_url.json:

The contents is:

{"url": "https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.5.0/en_core_web_sm-3.5.0.tar.gz", "archive_info": {"hash": "sha256:63d38fecdd4290635c7af4d4f6da50902bdc6c1732ce416b55c2b76c4b0c4626"}}

src/pip/_internal/models/direct_url.py tries to split the hash value on = (i.e. hash_name, hash_value = hash.split("=", 1)) and the character is not there.

Is that an error in direct_url.py or is the contents of direct_url.json wrong?

[pfmoore]

is the contents of direct_url.json wrong?

Yes. It's using a colon instead of an equal sign. See the spec which says

A deprecated hash key (type string) MAY be present for backwards compatibility purposes, with value <hash-algorithm>=<expected-hash>.

It looks like Poetry isn't following the spec.

[pfmoore]

The problem looks like it might be here: https://github.com/python-poetry/poetry/blob/master/src/poetry/installation/executor.py#L671

[pradyunsg]

None the less, we should probably still have pip not fail with a hard-crash in these cases; and just note that the package has invalid metadata.

(one of the problems with standards that people opt-in to implementing!! :P)

[sbidoul]

A fix is in #11779

[sbidoul]

Only the environment created with Poetry contains file direct_url.json

Noting that if the direct_url.json is missing when installing with pip, it either means you are using an old pip version that does not implement it, or you have not installed from a direct URL.

Thanks for reporting, BTW. Sorry for the inconvenience.

[chopeen]

Thanks @sbidoul!

I submitted PR python-poetry/poetry#7475 to fix the issue in Poetry. Can you have a look my PR description? Please share your thoughts on that and comment whether it makes sense to have both keys, hashes and the deprecated hash.