Skip subprojects when building the wheel

[tristan957]

WARNING File could not be mapped to an equivalent wheel directory: /tmp/mesonpy-udbdqxaw/install/usr/lib64/libxxhash.a ({libdir_static}/libxxhash.a)

Example project: https://github.com/hse-project/hse-python. Use the tristan957/mesonpy branch.

[dogtopus]

I also have a upcoming project that can benefit from this. Currently I have pure-static external subprojects which there's currently no way to skip the installation on the meson side other than specifying --skip-subprojects. It's fine (although hacky and error prone) when manually invoking meson but with a builder script it becomes a problem.

EDIT: In my case it didn't seem to cause much of a problem since meson-python seems to just look into the site-packages directory.

[FFY00]

Hi @tristan957 and @dogtopus, thank you for opening this issue, and sorry for the late reply. There is no location in wheels for static libraries, and they aren't needed at runtime, so we should probably just ignore them. I will try to look into a PR tomorrow.

[tristan957]

Very cool. Happy to see some movement on this issue

[tristan957]

As an update here is what is currently in the wheel

$ unzip -l dist/hse-3.0.0-cp310-cp310-linux_x86_64.whl 
Archive:  dist/hse-3.0.0-cp310-cp310-linux_x86_64.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
    15162  10-04-2022 11:51   hse-3.0.0.dist-info/METADATA
       88  10-04-2022 11:51   hse-3.0.0.dist-info/WHEEL
    11357  10-04-2022 16:51   hse-3.0.0.dist-info/LICENSE
     1251  10-04-2022 16:51   hse3/__init__.py
        0  10-04-2022 16:51   hse3/py.typed
    29718  10-04-2022 16:51   hse3/hse.pyi
   461072  10-04-2022 16:51   hse3/hse.cpython-310-x86_64-linux-gnu.so
      853  10-04-2022 16:51   hse3/limits.pyi
    31168  10-04-2022 16:51   hse3/limits.cpython-310-x86_64-linux-gnu.so
      310  10-04-2022 16:51   hse3/version.pyi
    27072  10-04-2022 16:51   hse3/version.cpython-310-x86_64-linux-gnu.so
  1515368  10-04-2022 16:51   .hse.mesonpy.libs/libhse-3.so.3.0.0
      243  10-04-2022 16:51   .hse.mesonpy.libs/pkgconfig/libcjson.pc
      308  10-04-2022 16:51   .hse.mesonpy.libs/pkgconfig/libcjson_utils.pc
      465  10-04-2022 16:51   .hse.mesonpy.libs/pkgconfig/hse-3.pc
    15829  08-25-2021 11:15   hse-3.0.0.data/headers/cjson/cJSON.h
     3938  08-25-2021 11:15   hse-3.0.0.data/headers/cjson/cJSON_Utils.h
      701  10-04-2022 16:51   hse-3.0.0.data/headers/hse-3/hse/version.h
      618  10-04-2022 16:51   hse-3.0.0.data/headers/hse-3/hse/flags.h
    38128  10-04-2022 16:51   hse-3.0.0.data/headers/hse-3/hse/hse.h
     1270  10-04-2022 16:51   hse-3.0.0.data/headers/hse-3/hse/limits.h
     3888  10-04-2022 16:51   hse-3.0.0.data/headers/hse-3/hse/types.h
     4883  10-04-2022 16:51   hse-3.0.0.data/headers/hse-3/hse/experimental.h
     2100  10-04-2022 16:51   hse-3.0.0.dist-info/RECORD
---------                     -------
  2165790                     24 files

Seems like the referenced PR will help if we have a way to pass --skip-subprojects. Overall really impressed with this project so far.

[rgommers]

There is no location in wheels for static libraries, and they aren't needed at runtime, so we should probably just ignore them. I will try to look into a PR tomorrow.

I should have commented on this earlier, but: this statement is unfortunately not 100% accurate. NumPy ships static libraries (libnpymath.a and libnpyrandom.a) that live inside site-packages/numpy/. Rebuilding those from source for consuming libraries is highly nontrivial, so I don't think that NumPy will be able to get rid of them very easily.

Shipping static libraries is obviously not the best design, and it makes assumptions about the compiler toolchain that consumers are using (must be MSVC-compatible on Windows, GCC-compatible on Linux and Clang-compatible on macOS).

I suspect that when NumPy moves to Meson, we can probably work around this problem by not using static_library(..., install: true) (which I think is what produces {libdir_static}) but custom_target.

[FFY00]

@tristan957 is this still relevant? We now support passing arguments to Meson, which you can use to pass --skip-subprojects to meson install.

See https://meson-python.readthedocs.io/en/latest/how-to-guides/meson-args.html

I am gonna close this, but feel free to re-open if there's anything you think I missed. Thanks!

[tristan957]

Should be good I think. Got laid off from the project I was using this with, so probably won't be making use of it anytime soon, but thanks anyways :)