wheel 0.36.0 fails on macOS 11 (Big Sur) when MACOSX_DEPLOYMENT_TARGET=11

[fxcoudert]

On previously macOS versions, the major version were 10.15, 10.14, etc. But on macOS Big Sur (11.x.y) the major version is now 11, and the next major version will be 12, etc.

Therefore, because MACOSX_DEPLOYMENT_TARGET typically specifies a major macOS version, it is now natural to have MACOSX_DEPLOYMENT_TARGET=11. This is accepted by all system build tools and compilers.

This triggered several build failures in cpython 3.9.0, which are now fixed (see python/cpython#23556). This is also triggering problems now in wheel. We are seeing this error as part of Homebrew build and testing (Homebrew/homebrew-core#66063 (comment)).

---

I install Python 3.9.0, with an upstream patch (python/cpython#23556), and wheel 0.36.0. Then I run pip3 install markupsafe, which gives the following error:

    File "/usr/local/Cellar/python@3.9/3.9.0_5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/usr/local/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 337, in run
      impl_tag, abi_tag, plat_tag = self.get_tag()
    File "/usr/local/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 260, in get_tag
      plat_name = get_platform(self.bdist_dir)
    File "/usr/local/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 52, in get_platform
      result = calculate_macosx_platform_tag(archive_root, result)
    File "/usr/local/lib/python3.9/site-packages/wheel/macosx_libfile.py", line 356, in calculate_macosx_platform_tag
      assert len(base_version) == 2
  AssertionError
  ----------------------------------------
  ERROR: Failed building wheel for markupsafe

[fxcoudert]

What's happening? calculate_macosx_platform_tag is called with platform_tag being macosx-11-x86_64. Therefore base_version becomes (11, ), and this triggers the assert on line 356. I'm not actually sure why that assert is there, it seems contradictory with the check on len(base_version) above.

Even if we remove that assert, or the platform_tag were different: if MACOSX_DEPLOYMENT_TARGET=11 is set in the environment, then the code below will set base_version to deploy_target, which will be (11, ). That's not wrong, but it will trigger another assert on line 371.

If I simply remove the two asserts, then I can run pip3 install markupsafe and I get this behaviour:

Collecting markupsafe
  Downloading MarkupSafe-1.1.1.tar.gz (19 kB)
Building wheels for collected packages: markupsafe
  Building wheel for markupsafe (setup.py) ... done
  Created wheel for markupsafe: filename=MarkupSafe-1.1.1-cp39-cp39-macosx_11_0_x86_64.whl size=16482 sha256=da3dc29e1dbf01ee611984e85a58e822211c599221d5e2cafe8ce5e15694ba53
  Stored in directory: /Users/fx/Library/Caches/pip/wheels/e0/19/6f/6ba857621f50dc08e084312746ed3ebc14211ba30037d5e44e
Successfully built markupsafe
Installing collected packages: markupsafe
Successfully installed markupsafe-1.1.1

I am not sure whether the platform tag macosx_11_0_x86_64 is what is expected here, or if it should be macosx_11_x86_64 (i.e., only recording the major macOS version).

[fxcoudert]

On further thought, I think since distutils.util.get_platform() returns macosx-11-x86_64, the platform tag should be macosx_11_x86_64 according to PEP 425.

[agronholm]

@Czaki comments? It's your code.

[agronholm]

Also, how come the existing tests don't fail when run on macOS 11? Does this only happen when the env variable is set?

[fxcoudert]

@agronholm I don't know how you run your tests. This happens when MACOSX_DEPLOYMENT_TARGET=11 at runtime, or when Python was built with MACOSX_DEPLOYMENT_TARGET=11.

[Czaki]

@fxcoudert could You check MACOSX_DEPLOYMENT_TARGET=11.0? Every time when I work with MACOSX_DEPLOYMENT_TARGET I set <mayor>.<minor>.

I also check pypi warehouse source code: https://github.com/pypa/warehouse/blob/0b6c95b1318a1572507d33f8abf220eef2984b19/warehouse/forklift/legacy.py#L127 and its requirements to have mayor and minor provided. So tag should be macosx_11_0_x86_64

I do not upgrade to BigSur yet.

When I working on this code there is no option to build a wheel for the system below 10.6. So I do not expect to provide only major version.

[fxcoudert]

@Czaki the numbering of macOS has changed from 10.major.minor to major.minor.minor2. Like you said, it is usual (but not mandatory) to set MACOSX_DEPLOYMENT_TARGET to the major macOS version: before, that was 10.15, 10.14, etc. Now it is 11, and the next major version will be 12.

In any case, MACOSX_DEPLOYMENT_TARGET=11 is valid and should not lead to failure.

[Czaki]

@Czaki the numbering of macOS has changed from 10.major.minor to major.minor.minor2. Like you said, it is usual (but not mandatory) to set MACOSX_DEPLOYMENT_TARGET to the major macOS version: before, that was 10.15, 10.14, etc. Now it is 11, and the next major version will be 12.

In any case, MACOSX_DEPLOYMENT_TARGET=11 is valid and should not lead to failure.

Ok. I understand. At this moment I do not have access to a machine with BigSur. I stuck on the 10.x series. I could create PR with a patch but I cannot test it locally.
If MACOSX_DEPLOYMENT_TARGET=11.0 cause build to pass the fix will be short. I only request your help.

During writing this code it is impossible to predict that Apple will change the numbering scheme.

But to solve this problem there should be a decision done in the warehouse. Currently, the warehouse will accept macosx_11_0_x86_64 and decline macosx_11_x86_64. As you pointed, it should accept macosx_11_x86_64, because 0 is the patch number. And this decision should be made by a proper person from pypa (@agronholm - did you have an idea who should be called)

To prepare the patch I also need some .so or .dylib file to check the results of binary file inspection.

@fxcoudert If I prepare a patch did you have time to test it?

[fxcoudert]

According to pypa/packaging#319 (comment):

In order to actually remove the minor version we'd have to coordinate with wheel (and maybe create a PEP?) Instead we just pin the minor version to 0.

It seems there is a need for a coordinated discussion about this.

[Czaki]

Ok. So I will prepare bugfix patch. But I still have a problem with testing it.

[ronaldoussoren]

Please pin the minor version to 0 and treat MACOSX_DEPLOYMENT_TARGET=N as MACOSX_DEPLOYMENT_TARGET=N.0.

There are already wheels for macosx_11_0_universal2 on PyPI, and PyPI requires that there is a minor number.