PEP 656 musllinux support

[uranusjr]

PEP 656 proposed musllinux, a counterpart of manylinux for Linux distros running on musl libc. I’ve recently implemented support for installing musllinux wheels (pypa/packaging#411), and are now looking for a place to implement wheel generation logic.

Do you think auditwheel is a good place for this? Many things are the same auditing manylinux and musllinux wheels (the Linux part), but from my (very little) understanding, many parts in auditwheel assume glibc and would need some refactoring to work with musl, so there’s case to create a new tool for musllinux wheel auditing as well. My main end goal is to add musl to the support matrix of cibuildwheel (pypa/cibuildwheel#627), so either approach would ultimately be fine.

Edit from maintainers:
List of tasks that would need to be done in order to add support for musllinux.
This list is subject to changes and will be updated with feedback from comments.

• Rename manylinux policy auditwheel/policy/policy.json -> auditwheel/policy/manylinux-policy.json refactor: rename policy.json to manylinux-policy.json #311
• Prepare a musllinux policy auditwheel/policy/musllinux-policy.json following the same schema. Add support for musllinux #315
• Detect if running a musl or glibc linux Add support for musllinux #315
• Select the correct policy file based on previous detection Add support for musllinux #315
• Filter musllinux policies to only retain the one matching the system's musl version Add support for musllinux #315 PEP 656 musllinux support #305 (comment)
• Fix ldtree to behave properly on musl Add support for musllinux #315

This seems to be what's needed for now.
Be warned, most of the work is probably musllinux-policy.json.

[mayeut]

Hi @uranusjr,

Sorry it took this long for someone to answer.

I think it makes sense to add musllinux support to auditwheel.
As you said, many things should be the same between musllinux & manylinux. Of course, there are also differences for glibc vs musl that need to be taken care of but I wouldn't say that auditwheel assumes glibc in its code.

One a of the major difference I foresee is that, because glibc uses symbol versioning, everything is done through the policy wether we're talking about glibc libraries, gcc/g++ libraries. Symbol versions are used to compute the effective policy and thus wether or not it's compatible with the requested one.
I do hope that symbol versioning is still a thing for gcc/g++ libraries on musl distros, that way, at least that part of the code can be reused. Detecting the minimum musl version a wheel is eligible for might be a bit trickier and require some new mechanisms in auditwheel (I've not read the full thread on discourse so don't know where the discussions stopped on that matter). A first step could be "check I'm valid for the current musl & graft needed libraries".

As I see it, it would require at least a new json policy file specific to musllinux, a way to switch between policies at startup & the minimum "musl" detection code.

@lkollar, any other inputs/ideas/remarks/objections ?

I do not have the time to look into it myself but will do my best to review any proposition to add such support.

[uranusjr]

The musllinux spec was intentionally designed to follow the new “perennial” manylinux design, so I’m kind of expecting to reuse much of the same logic. musl does not support the same symbol versioning functionalities as glibc, only allows pulling the default version and not the latest version, but I’m expecting users to mainly use containers to produce musllinux wheels, so they can use an older musl version in their image to get the musllinux tag they want; i.e. a wheel built against musl 1.2 can only become musllinux_1_2 or later, no matter what symbols you use.

[mayeut]

I’m expecting users to mainly use containers to produce musllinux wheels, so they can use an older musl version in their image to get the musllinux tag they want i.e. a wheel built against musl 1.2 can only become musllinux_1_2 or later, no matter what symbols you use.

That was my - probably unclear - comment:

A first step could be "check I'm valid for the current musl & graft needed libraries".

You build & repair on musl 1.2, you get musllinux_1_2.

I was more thinking about symbol introduction rather than symbol compatibility and trying to detect which minimum musl version satisfies the symbols used by the wheel being repaired. Maybe that's a bit too complicated, not wanted or whatever.

The option you propose is a perfectly valid one although it differs a bit from my comment which doesn't add "or later". If you get "later", then, it becomes a hassle to test the wheel you just built in the same image because pip will refuse - rightfully so - to install it.

[uranusjr]

I think we are on the same page, just not communicating very well 🙂

I was more thinking about symbol introduction rather than symbol compatibility and trying to detect which minimum musl version satisfies the symbols used by the wheel being repaired. Maybe that's a bit too complicated, not wanted or whatever.

I don’t think it’s practical. musl does not expose the information itself, so to achieve this we’ll need to maintain the symbol-minver lookup ourselves. So yeah, too complicated etc.

The option you propose is a perfectly valid one although it differs a bit from my comment which doesn't add "or later".

I probably shouldn’t have added the “or later” part, it only made things more complicated. I was trying to say that the user theoratically can choose to repair a wheel into musllinux_1_3 against musl 1.2 (assuming other dependencies can be correctly grafted). I don’t see anyone would actually want to do that in practice though, so let’s not worry about that.

[mayeut]

@uranusjr,

do you mind if auditwheel maintainers edit your first post to add a checklist of what we would expect to see in a PR ?

I think it would help potential contributors to understand what's expected without browsing through all comments.

[uranusjr]

No problem at all, please go ahead!

[mayeut]

At first glance, most of the work should be in creating a musllinux-policy.json.

For manylinux1, manylinux2010 & manylinux2014, this was done semi-manually by running the scripts/calculate_symbol_versions.py script on different distros.

For PEP 600 manylinux, I automated this process in https://github.com/mayeut/pep600_compliance (which still relies on "run the script on a bunch of distros")

For PEP 656 musllinux, the same kind of work must happen.

I do hope that symbol versioning is still a thing for gcc/g++ libraries on musl distros, that way, at least that part of the code can be reused.

this has to be checked. If it's not the case, something else shall be implemented.

[uranusjr]

How do you determine whether a distro is compliant to a PEP 600 manylinux tag? From my understanding of PEP 600 (which PEP 656 follows), whether a library (except libc) is allowed depends on whether it is provided by popular distros. This means the policy is inheritantly defined by distros, and it seems circular to me if distros can be categorised by that policy.

[mayeut]

How do you determine whether a distro is compliant to a PEP 600 manylinux tag?

Just looking at its glibc version.
The PEP 600 tag policy should conform to that, not the other way around.

From my understanding of PEP 600 (which PEP 656 follows), whether a library (except libc) is allowed depends on whether it is provided by popular distros.

For auditing, it's not only wether a library is allowed. It's also wether or not its version (or more exactly symbols it uses) are allowed. This is only enforced for gcc libraries in auditwheel as of now (& glibc but we already talked about how to handle musl in a previous comment).

Let's take a look at simple sample:
distros with glibc 2.28, trying to determine what the manylinux_2_28 policy should be
for the sake of simplicity, let's just consider there are just CentOS 8 & Photon 3 in "popular" distros with glibc 2.28
If you compare the package list, you'll see that they're not using the same gcc package version.
If I were to build a wheel on CentOS 8 with the latest additions to the C++ standard, I'd probably be building a wheel that's not compatible with Photon 3 even though they're using the same glibc and that libstdc++.so.6 is a whitelisted library. Thus, CentOS 8 built wheels cannot be marked manylinux_2_28 just by looking at the glibc version and the whitelisted library, we have to take into account symbols used in libstdc++.so.6 (and symbol versioning makes that easy for gcc libraries)

In other words, every popular glibc 2.28 distro is compliant for consuming manylinux_2_28 wheels but every popular glibc 2.28 distro might not produce manylinux_2_28 compliant wheels.

This means the policy is inheritantly defined by distros, and it seems circular to me if distros can be categorised by that policy.

Well, I think you had a look at the https://github.com/mayeut/pep600_compliance and that the wording there is a bit misleading.
The "distro compatibility" section is just a mapping of the glibc version i.e. policy column manylinux_{glibc_major}_{glibc_minor} and distro column, just the list of popular distros with this glibc version.
It's mostly a centralized way for me to find the glibc version of a given popular distro.
It's thus not circular, it's a strict equivalence as defined in PEP 600.

The 2 other sections are a bit more interesting:
"Known compatibility issues":
The automation uses the policy file from auditwheel to detect if already defined policies are breaking some rule.
Again, the wording is probably bad, it's not the distro that's not compatible, it's the policy that's buggy. In any case, that's a compatibility issue.

The "Acceptable distros to build wheels" section is a bit more interesting here.
It reports distros which libraries/symbols versions matches exactly the policy.

The way the policies are built is even more interesting for the purpose of this thread and it exactly matches your comment "policy is inheritantly defined by distros":

next_version = None
for glibc_version in sorted(versions, reverse=True):
    policy_{glibc_version} = intersection([symbol_versions(distro, whitelist_libraries) for distro in get_distros(glibc_version)])
    if next_version:
        # A policy defined for glibc (x1, y1) shall be a subset of the policy defined for glibc (x2, y2) where (x1, y1) < (x2, y2)
        policy_{glibc_version} &= policy_{next_version}
    next_version = glibc_version

[uranusjr]

Thanks a lot for the detailed explaination, indeed I was misunderstanding what pep600_compliance is showing 🙂

Since musl does not cover C++ in the first place and I believe distros just use libstdc++ built against musl, can the above PEP 600 implementation be used for PEP 656? Where can I find it? I quickly searched but couldn’t find the above code snippet in either pep600_compliance or auditwheel.

[mayeut]

can the above PEP 600 implementation be used for PEP 656?

If gcc libraries do use symbol versioning (as opposed to musl), sure you can use it. If not....

I quickly searched but couldn’t find the above code snippet in either pep600_compliance or auditwheel.

I wrote this as pseudo-code for it to be a bit clearer.

The actual way it works in the pep600_compliance repo is:

1. dump symbol versions for every distro in cache/{machine}/{distro_name}-{distro_version}.json
   It does a bit more than just symbol versions in light of We should probably allow libz linkage, and maybe libexpat #152.
   The content of the json is built by https://github.com/mayeut/pep600_compliance/blob/83b1afd0b44610f81b484021d3fa5ff0ec73b70b/pep600_compliance/images/base.py#L268-L271
   One function just calls the "symbol finding script"
   The other finds which "not whitelisted" libraries are dependencies of the python interpreter. The whitelist is not read from the policy file yet. For a new standard such as PEP 656 (and maybe even for manylinux), I think it would make sense to blacklist anything that appears in "extra", c.f. We should probably allow libz linkage, and maybe libexpat #152 for the rationale.
2. Analyse all the cache/*/*.json files to update README.md (and some other markdown files)
   The policies are never written to disk in the repo. The code is there though, you'd be interested in https://github.com/mayeut/pep600_compliance/blob/83b1afd0b44610f81b484021d3fa5ff0ec73b70b/pep600_compliance/make_policies.py#L182-L190. It calls the make_policies which is closest to the pseudo-code from the previous comment.

PEP 656 starts from scratch so there's no pre-existing policies which should simplify a bit the code (look for "official") and policies can be built cleanly.
For PEP 600, we need to know if existing policies are breaking any rules and that's why those policies are immutable and incompatibilities can be reported.

[uranusjr]

I know Alpine’s toolchain does support symbol versioning, and at least some libs are compiled with it (not sure if all of them do), so that should be a good starting point (again, I expect PEP 656 users to mainly use containers for compilation, so having one working distro should be enough). I’ll find some time to dig into the compliance logic and se what I can come up with.

(But thinking this some more, Alpine doesn’t really come with many libraries in the first place, maybe this whole discussion would be pretty moot and we end up not including any system libraries at all… I’ll need to check that.)

[mayeut]

maybe this whole discussion would be pretty moot and we end up not including any system libraries at all… I’ll need to check that.

I need to find references for that but libstdc++.so.6 can't be grafted so if it's not whitelisted, it shall be blacklisted. With pybind11 wrappers for example, this would require some specific link options. There's a rather long discussion about those options in the manylinux repo. edited per #305 (comment)