Support PEP751 lockfile format in bzlmod

[aignas]

After pip lock lands, this might become a better requirements.txt alternative.

References:

• pip (25.1) will export a lockfile - ETA 2 weeks PEP 751 experimental pip lock command pypa/pip#13213
• packaging has landed the marker resolution (unlocks PDM, poetry, hatch) Support PEP 751 in markers pypa/packaging#885
• uv is tracking it as an export format Add support for PEP 751 lockfiles astral-sh/uv#12584
• poetry still thinking about it https://github.com/orgs/python-poetry/discussions/10322
• pip-tools keen Implement PEP 751 "A file format to list Python dependencies for installation reproducibility" (a.k.a. lock files | pylock.toml | pylock.*.toml) jazzband/pip-tools#2124

This most likely depends on #2786.

Making this work needs:

• Extra marker evaluation support (Support extras and dependency_groups in the PEP508 env marker evaluation #2786).
• Parse toml to JSON using some predefined interpreter (set with Improve the pip.parse API to allow for incremental building of the configuration #2747 .defaults).
• Map to the output like what parse_requirements is returning for easy plumbing.
• Call the API for parsing py.lock something like pip.from_lock.

[aignas]

#3088 is a duplicate for this.

[aignas]

I had a go at this and the number of pieces that have to fall together is roughly as below:

• Add support for outputing a pylock.toml file in our lock rule. Fairly trivial, just need to work around the quirk that uv will refuse to output the pylock.toml file if the filename is not pylock.toml.
• Add support for converting toml to json. Can be done using the toml2json wheels from PyPI, that are pretty lightweight and we could select the host-compatible one at runtime, then we could just mctx.execute directly the binary. This or we could execute a python script to do the conversion.
• Parse the lockfile JSON as per PEP751. This is also trivial.
• Do not require extras in the lock file to correctly deduce the dependency graph. The pylock.toml files do not have this information. We should generate targets for each provides-extra in the whl_library_targets macro and then expose some of them through the hub repo. Which extra targets we should expose is unclear.

The last one is a blocker. I think we have a ticket for that somewhere. I think the same problem would exist with uv.lock.

The only way around that I can see is to:

• Download the wheels (or their METADATA beforehand).
• Write the Requires-Dist and Provides-Extra contents into the bzlmod lockfile through the new facts mechanism so that this downloading is not needed all the time.
• For each Provides-Extra, add an extra alias.

Alternative could be to use transitions on py_library, something like the following:

py_library_with_extras(
    name = "my_dep_with_extras",
    dep = "@pypi//actual_dep",
    extras = ["foo", "bar"]
)

which would be defined at call site (e.g. the whl_library that depends on a package with extras).

This may potentially still break bazel query. I am wondering if the conclusion that we have from this is that we either need a dependency graph or the extras for each package in order to use a lock file.

This means that we can use the requirements.txt because we can fetch the whl paths via the SimpleAPI. And we can use the uv.lock file because we would not need to use the METADATA files within the wheels to construct the dependency graph and we could just use the dependency graph within the uv.lock file.

[aignas]

Some extra thoughts on what I said above.

The two alternative ways to define the dep graph is as follows.

1. Define the dependencies at each node

This is what we do today by parsing the wheel METADATA. The good part is that any wheel that has correct metadata can be loaded into bazel and we can get the list of others nodes that we have to depend on.

The problematic parts: multiplatform wheel support - what happens when there are multiple wheels for different platforms? What happens when there are cyclic dependencies? Because at parsing time all you know is the scope of the package, it is tough to address these global concerns. What we tried doing in rules_python is to use the hub to skirt these two problems but one issue remains. The packages may depend on extras within other packages and there is a problem that the hub does not know about the extras that exist within each package without eagerly fetching the packages or getting that info from a lock file or user suplying that info.

The only way I see for everything to work is to load the list and to not go through the hub repo. This means if the user is targeting 2 platforms, the whl library will know which wheels to include depending on which platform we have.

Passing around the info to the whl libraries is a design that is a bit hard to work with.

2. Lay out the dep graph within the hub repo

Here we would need some way to get the dep graph in the hub repo.

Alternatives:

1. Get it from uv.lock. However, the lock file is not standard and there is no spec for it, so a little hard to work with it.
2. Use METADATA files from each wheel. This would require us to download all of the wheels (or just metadata files if they are available on the index).

I am inclined to go number 2 and then we can continue supporting py.lock, requirements and others. But I don't know when is the best time to do that.

We could invent our own lock file format to have all of the data that we need, but I am still not sure about the added value of having it. pylock.toml does not support defining extras requested for a particular package, so we would have to deviate from the spec there by adding an extra field. And we would have to populate the dependencies field as well.

[rickeylev]

Can be done using the toml2json wheels from PyPI

The stdlib has both toml and json parsers, so nothing from pypi should be needed.

[groodt]

Can be done using the toml2json wheels from PyPI

The stdlib has both toml and json parsers, so nothing from pypi should be needed.

toml is only included 3.11+

Raising for awareness in case it’s useful (as a reference or as a tool), packaging recently spawned a nice standards compliant parser of pylock for downstream packaging tools.

pypa/packaging#900

[aignas]

Ok, so my thinking right now aboutwthis is that we can fetch the metadata files for all wheels that we want to use as there are references for those in Simple API. Those would be fetched at module evaluation time.

Alternatively, users can call some pin command that does that as a build action and fetches/writes the information from the wheels to a special lock file.

The first works with out of the box cases, the second means that the no network requirement can be still satisfied when non root modules define dependencies.

Initially the lock file that we generate can just contain the provided extras for each package.

Later deps and or cyclic deps.

[rickeylev]

toml is 3.11+

That's ok. We can just use a 3.11+ interpreter.

I vibe-coded up a greenfield thing a few months ago to poke at what a uv thing had and how to get it into starlark: main...rickeylev:rules_python:parse.uvlock

Key parts being:

• use_repo() for python_3_14_host. Used to run toml2json.py
• toml2json.py (called using the existing pypi_repo_utils.execute_checked_stdout helper)
• uv_external_deps.bzl%convert_uv_lock_to_json() (called during bzlmod phase)

My general thinking is: the pylock (and uv lock) are basically like precomputed files of the distribution -> urls information that pipstar has to manually fetch. So convert their toml->json, then extract out that dist->url info and then pass it along to pipstar as-if it had looked them all up via the simple api.

At the least, should I send out a PR for the toml2json code?