Add build_extras requirements field to recipe to allow adding extra build-requires

[hoodmane]

No description provided.

[agriyakhetarpal]

Without having looked at this in detail, is this something we require on a frequent basis when building recipes, since I assume we can always add build-time requirements by patching the packages' pyproject.toml or setup.py files? It looks like it's coming more out of a need for convenience – something which I am not opposed to; I just want to know how many packages in our distribution require this.

[hoodmane]

Currently just scipy.

we can always add build-time requirements by patching the packages' pyproject.toml or setup.py files?

Yes but this is much more annoying.

[hoodmane]

We need often to pin or cap build requirements or transitive requirements for packages. Sometimes we need to substitute a fork of a build requirement. All these things can be done much more conveniently with this field than with a patch.

[hoodmane]

Especially in the common case where e.g., a setuptools release breaks a bunch of our packages and we want to add a cap everywhere. Adding one patch per recipe is frustrating.

[hoodmane]

Using it here to substitute a fork of gast that supports Python 3.15:
https://github.com/pyodide/pyodide/pull/6355/changes#diff-d6d5fb2d234d6c6bafa1cff3848a6f49e87a62111e50baab21b4e954ae2b85beR154-R159

[agriyakhetarpal]

Agreed, the use case makes sense, and I am aligned with it. But we should note the way this is currently implemented:

Sometimes we need to substitute a fork of a build requirement. All these things can be done much more conveniently with this field than with a patch.

This is additive, and doesn't replace existing build requirements. How do we make it replace the build requirements entirely? Perhaps another field override-requirements where we completely ignore the build requirements coming from the package, and add our own?

[hoodmane]

This is additive, and doesn't replace existing build requirements.

It seems to work fine? If it doesn't work in the future, then we can modify it. But you will note that pyodide/pyodide#6355 is using this change and CI is green.

[agriyakhetarpal]

Ah, your new comment came up just as I was typing mine. I would suggest making this clearer in the recipe format docs. pyodide/pyodide#6355 looks good to me, then.

[agriyakhetarpal]

i.e., what I mean is that requirements.build from a bird's eye view seems to me as if it overrides everything but it's actually adding extra requirements. Could we rename the field to something better...

[hoodmane]

How about build_extras

[agriyakhetarpal]

Works for me!

[agriyakhetarpal]

This reads better to me now. I have a few questions and suggestions, but overall LGTM.

Could you please add:

• an integration recipe test that exercises this
• a unit test that adds extra build requirements with PEP 508 markers (I checked locally that these already work, but we've already had a regression for this before, so one test would actually be great)
• a CHANGELOG entry?

Thanks @hoodmane!

P.S. I was going to suggest that we should add a note about this new field to the "The meta.yaml specification" page but I couldn't find it. As it turns out, it's gone with pyodide/pyodide#6181 and we never added that page to pyodide-build's docs. We should add it at some point.

[ryanking13]

Thanks, yeah this is often annoying when we need to deal with some old packages or when there is a breaking changes in the build system.

Since we are slowly deprecating the recipes system (at least in my opinion), I am okay with adding some adhoc features to the recipes to make our life easier. People who are building packages out of tree would rarely use recipes anyways.

[hoodmane]

I still think the recipes play an important role in testing new emscripten versions and new abis and finding Emscripten bugs and other problems. If we reduce the number too much, the risk is that problems are discovered downstream much later by people without the resources to fix them.

Especially since maintaining the recipes is quite a lot easier with langage models, so the cost has gone down.