Support python_version on modules_mapping and/or gazelle_manifest rule

[chrisirhc]

🚀 feature request

Relevant Rules

• gazelle_python_manifest
• modules_mapping

Description

When in the middle of working on a Python version upgrade, it's valuable to generate gazelle manifests with different module maps because you the requirements.in may include additional requirements for the new python version:

gradio; python_version >= "3.11"

When running the manifest rule with all_whl_requirements, it'll fail with something like:

No matching wheel for current configuration's Python version.

In order to get the desired behavior, we need to run the gazelle manifest update by specifying a different python version via command line --@rules_python//python/config_settings:python_version=3.11.

Describe the solution you'd like

Adding a `python_version attribute to the manifest or modules_mapping rule, so that we can run it with the expected python_version as needed without a command-line flag.

Describe alternatives you've considered

We currently customize our scripts so that we add the command line flag, but I don't think this is feasible.

Other alternatives:

• Generate a modules mapping per different environment tag satisfied in the requirements.in . This might be heavy-handed, but could work for all other environment markers beyond python_version . And there's no need to maintain multiple manifest files.

[aignas]

Just to acknowledge that I've seen this and was thinking about it.

In some sense, the Python code should gather wheels from all versions and make a union of the imports to packages.

If gradio is only available for python_version >= 3.11, then it is fine to have gradio in the gazelle manifest and let gazelle include @pypi//gradio in the py_library - the build will fail for any configurations that try to use the target on different python versions, so I think that would work as expected.

If you also want to make gazelle add target_compatible_with flags to your python targets, then it would be a little bit more complicated.

TLDR: I think just considering the modules_mapping macro and how to make it work to pull all of the wheels for all of the target python versions could be an option, IMHO. @dougthor42, what do you think?

[chrisirhc]

In some sense, the Python code should gather wheels from all versions and make a union of the imports to packages.

If gradio is only available for python_version >= 3.11, then it is fine to have gradio in the gazelle manifest and let gazelle include @pypi//gradio in the py_library - the build will fail for any configurations that try to use the target on different python versions, so I think that would work as expected.

Yes, this is sufficient for this use case.

If you also want to make gazelle add target_compatible_with flags to your python targets, then it would be a little bit more complicated.

Agreed. Supporting python_version or target_compatible_with can be a future enhancement.
We may want this capability in our internal implementation (as a gazelle plugin perhaps) but having a general deterministic implementation of this (which needs to support arbitrary number of environments and python versions) can be tricky.

[dougthor42]

@dougthor42, what do you think?

I think that makes sense to me. Especially given #2048, the manifest file is mostly indifferent to package version changes.

The only case that I can think of where it's not indifferent to versions is the case where a package changes the import name between versions. Imagine package foo that gets imported as foo_package for version 1, but imported as foo for version 2 and above:

try:
    import foo_package
except ImportError:
    # using foo >= 2
    import foo as foo_package

foo_package.frobnicate()

The manifest file would have to have two entries:

foo_package: foo
foo: foo