[BUG] WorkingSet no longer prefers latest version of a dist

[cottsay]

setuptools version

setuptools==66.0.0

Python version

Python 3.11

OS

Fedora 37

Additional environment information

No response

Description

Given that there are multiple versions of a distribution installed under a single path entry, calling WorkingSet.require with the latest version of that distribution results in a VersionConflict where it previously succeeded.

On my Fedora system, during WorkingSet._build_master, this consequently results in a call to WorkingSet._build_from_requirements where previously the call to WorkingSet.require succeeded, which in turn results in a new InvalidVersion exception which would otherwise not have occurred (unfortunately in an unrelated system package). I understand that the system package should be fixed, but this change in behavior exacerbates that issue where I would otherwise not have been impacted.

Here is the PR which modified the order of the distributions: https://github.com/pypa/setuptools/pull/2822/files#diff-e3d3d454fa3a072c9f46f8affa27513892fbc2d245e87f57a5927a7be851de05L2134-R2083

Expected behavior

When there are multiple versions of a dist present under a path entry, pkg_resources should prefer the newer one, and a call to WorkingSet.require should activate it successfully.

How to Reproduce

def test_version_priority(tmpdir):
    """
    The best version of a package should be usable.
    """
    egg_info = tmpdir / 'proj-24.egg-info'
    egg_info.mkdir()
    pkg_info = egg_info / 'PKG-INFO'
    pkg_info.write_text(
        f'Metadata-Version: 1.0\nName: proj\nVersion: 24\n',
        encoding='utf-8')

    egg_info = tmpdir / 'proj-42.egg-info'
    egg_info.mkdir()
    pkg_info = egg_info / 'PKG-INFO'
    pkg_info.write_text(
        f'Metadata-Version: 1.0\nName: proj\nVersion: 42\n',
        encoding='utf-8')

    ws = pkg_resources.WorkingSet([tmpdir])
    ws.require('proj==42')

Output

Where the reproducer passed before #2822, current HEAD raises:

pkg_resources.VersionConflict: (proj 24 (/tmp/pytest-of-cottsay/pytest-60/popen-gw0/test_version_priority0), Requirement.parse('proj==42'))

[pradyunsg]

Ah, this needs a reverse=True on that call.

[cottsay]

Ah, this needs a reverse=True on that call.

Unfortunately that isn't sufficient to completely restore the established behavior, which sorted based on the version numbers and not alphabetically. Some of the sorting logic that was removed will need to be restored.

I extended the reproducer to cover this case:

def test_version_priority(tmpdir):
    """
    The best version of a package should be usable.
    """
    egg_info = tmpdir / 'proj-24.egg-info'
    egg_info.mkdir()
    pkg_info = egg_info / 'PKG-INFO'
    pkg_info.write_text(
        f'Metadata-Version: 1.0\nName: proj\nVersion: 24\n',
        encoding='utf-8')

    egg_info = tmpdir / 'proj-42.2.egg-info'
    egg_info.mkdir()
    pkg_info = egg_info / 'PKG-INFO'
    pkg_info.write_text(
        f'Metadata-Version: 1.0\nName: proj\nVersion: 42.2\n',
        encoding='utf-8')

    egg_info = tmpdir / 'proj-42.10.egg-info'
    egg_info.mkdir()
    pkg_info = egg_info / 'PKG-INFO'
    pkg_info.write_text(
        f'Metadata-Version: 1.0\nName: proj\nVersion: 42.10\n',
        encoding='utf-8')

    ws = pkg_resources.WorkingSet([tmpdir])
    ws.require('proj==42.10')