Using internal shared libraries with cython, install_dir not respected? Could not map installation path to an equivalent wheel directory

[AndrewAnnex]

Slightly related to #769, and definitely in theme with #410, #556, and #765, I am trying to update my Python library SpiceyPy to use meson-python.

SpiceyPy wraps a C library (that doesn't ship with a cmakelists file) with ctypes and now cython, each linking to a shared library that gets bundled in the wheel and has been built successfully for years with cibuildwheel, a custom build script, and some horrible setup.py stuff I am trying to remove entirely so I can transition fully to pyproject.toml and not maintain a custom build script at the same time. The cython module is much newer and links against numpy as well so slightly further complicating things.

I have been reading those issues and the docs page at https://mesonbuild.com/meson-python/how-to-guides/shared-libraries.html but I am new to meson so I am getting the could not map error mentioned above, and as far as I can tell, I am doing all the things recommended in the docs/those discussions already so I am a bit lost.

Because of the use of ctypes (and well because I know this already works) none of the instructions regarding static libraries seem relevant to me. But I am willing to have my mine changed...

In any case, to experiment with meson-python, I have another public github repository https://github.com/AndrewAnnex/cspice-meson-cyice-test where I have a minimal version of the cython module of SpiceyPy that I hope to get working (and eventually with cibuildwheel) to start.

So far, I have worked out how to compile the C library with meson via a subproject. Because the source code is distributed as different files for different operating systems (and needs a few patches), I have a series of wrap files and patch files that contain the meson.build, the meson_options.txt to support the configuration idea from the docs #700 . This compiles and in previous versions I had a test c file compiled to an executable against it (something I need to put back in probably).

Next, I have a top level meson.build file where I have a subproject defined

https://github.com/AndrewAnnex/cspice-meson-cyice-test/blob/fd516fe5def09b3ad1965624749667a35401e4b8/meson.build#L74-L83

libcspice_sub = subproject(
  cspice_wrap_name,
  default_options: {
    'only_install_main_lib': true,
    'install_dir': py_install_dir,
  }
)
cspice_dep = libcspice_sub.get_variable('cspice_dep')
cspice_shared = libcspice_sub.get_variable('libcspice')
cspice_headers = libcspice_sub.get_variable('cspice_inc')

and where the python install dir is grabbed a few lines above:

https://github.com/AndrewAnnex/cspice-meson-cyice-test/blob/fd516fe5def09b3ad1965624749667a35401e4b8/meson.build#L12-L17

# python stuff
py_mod = import('python')
py = py_mod.find_installation(pure: false)
py_dep = py.dependency(required: true)
py_install_dir = py.get_install_dir(subdir: 'test_cyice/utils', pure: false) 
message('py install dir: ', py_install_dir)

and then I have a meson.build in the src layout parlance in src/test_cyice/meson.build where the sub project dependency actually gets used in the cython compilation step

https://github.com/AndrewAnnex/cspice-meson-cyice-test/blob/fd516fe5def09b3ad1965624749667a35401e4b8/src/test_cyice/meson.build#L2-L36

py.install_sources(
   [
    '__init__.py'
   ],
  pure: false,    # Will be installed next to binaries
  subdir: 'test_cyice'  # Folder relative to site-packages to install to
)


subdir('utils')


# for numpy
incdir_numpy = run_command(py,
  ['-c', 'import os; os.chdir(".."); import numpy; print(numpy.get_include())'],
  check : true
).stdout().strip()
inc_np = include_directories(incdir_numpy)
message('numpy inc dir: ', incdir_numpy)


# extension module stuff for test_cyice


test_cyice = py.extension_module(
  'test_cyice',
  sources: [
    'test_cyice.pyx',
  ],
  dependencies: [
    py_dep, 
    numpy_dep,
    cspice_dep,  
    m_dep
  ],
  include_directories: [inc_np, cspice_headers],
  install: true,
  subdir: 'test_cyice'
)

When I go to do meson build/meson compile, everything works as expected and I see the expected shared libraries in the build/compile folders within the correct subprojects directory.

But when I go to use meson-python via python -m build -w, after the shared library finishes being compiled I get the familiar error mentioned above after linking occurs:

* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
  - cython
  - meson-python
  - numpy>=2.0.0
  - setuptools>=76.0.0
  - wheel
* Getting build dependencies for wheel...
* Building wheel...
+ meson setup /Users/andrew/PycharmProjects/cspice-meson-spiceypy /Users/andrew/PycharmProjects/cspice-meson-spiceypy/.mesonpy-1w9y0ga8 -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/Users/andrew/PycharmProjects/cspice-meson-spiceypy/.mesonpy-1w9y0ga8/meson-python-native-file.ini
The Meson build system
Version: 1.8.2
Source dir: /Users/andrew/PycharmProjects/cspice-meson-spiceypy
Build dir: /Users/andrew/PycharmProjects/cspice-meson-spiceypy/.mesonpy-1w9y0ga8
Build type: native build
Project name: test_cyice
Project version: 7.1.0
C compiler for the host machine: cc (clang 17.0.0 "Apple clang version 17.0.0 (clang-1700.0.13.5)")
C linker for the host machine: cc ld64 1167.5
Cython compiler for the host machine: cython (cython 3.1.2)
Host machine cpu family: aarch64
Host machine cpu: aarch64
Program python found: YES (/private/var/folders/6r/0_ck04691rx1vj2x5knvq8s00000gn/T/build-env-a4nhvttt/bin/python)
Found pkg-config: YES (/opt/homebrew/bin/pkg-config) 2.4.3
Run-time dependency python found: YES 3.13
Message: py install dir:  /usr/local/lib/python3.13/site-packages/test_cyice/utils
Library m found: YES
Compiler for C supports link arguments -Wl,-dead_strip: YES 
Message: install_dir:  /usr/local/lib/python3.13/site-packages/test_cyice/utils

Executing subproject libcspice-macos-arm 

libcspice-macos-arm| Project name: libcspice
libcspice-macos-arm| Project version: 67
libcspice-macos-arm| C compiler for the host machine: cc (clang 17.0.0 "Apple clang version 17.0.0 (clang-1700.0.13.5)")
libcspice-macos-arm| C linker for the host machine: cc ld64 1167.5
libcspice-macos-arm| Library m found: YES
libcspice-macos-arm| Message: only_install_main_lib:  true
libcspice-macos-arm| Message: install_dir:  /usr/local/lib/python3.13/site-packages/test_cyice/utils
libcspice-macos-arm| Message: Registering libcspice override dependency
libcspice-macos-arm| Build targets in project: 1
libcspice-macos-arm| Subproject libcspice-macos-arm finished.

numpy-config found: YES (/private/var/folders/6r/0_ck04691rx1vj2x5knvq8s00000gn/T/build-env-a4nhvttt/bin/numpy-config) 2.3.0
Run-time dependency numpy found: YES 2.3.0
Message: numpy inc dir:  /private/var/folders/6r/0_ck04691rx1vj2x5knvq8s00000gn/T/build-env-a4nhvttt/lib/python3.13/site-packages/numpy/_core/include
Build targets in project: 2

test_cyice 7.1.0

  Subprojects
    libcspice-macos-arm: YES

  User defined options
    Native files       : /Users/andrew/PycharmProjects/cspice-meson-spiceypy/.mesonpy-1w9y0ga8/meson-python-native-file.ini
    b_ndebug           : if-release
    b_vscrt            : md
    buildtype          : release

Found ninja-1.12.1 at /Users/andrew/mambaforge/envs/meson/bin/ninja
+ /Users/andrew/mambaforge/envs/meson/bin/ninja
[2233/2234] Compiling C object src/test_cyice/test_cyice.cpython-313-darwin.so.p/meson-generated_src_test_cyice_test_cyice.pyx.c.o
In file included from src/test_cyice/test_cyice.cpython-313-darwin.so.p/src/test_cyice/test_cyice.pyx.c:1137:
In file included from /private/var/folders/6r/0_ck04691rx1vj2x5knvq8s00000gn/T/build-env-a4nhvttt/lib/python3.13/site-packages/numpy/_core/include/numpy/ufuncobject.h:337:
/private/var/folders/6r/0_ck04691rx1vj2x5knvq8s00000gn/T/build-env-a4nhvttt/lib/python3.13/site-packages/numpy/_core/include/numpy/__ufunc_api.h:333:23: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
  333 | PyUFunc_ImportUFuncAPI()
      |                       ^
      |                        void
1 warning generated.
[2234/2234] Linking target src/test_cyice/test_cyice.cpython-313-darwin.so

meson-python: error: Could not map installation path to an equivalent wheel directory: '{install_dir}/libcspice.1.dylib'

ERROR Backend subprocess exited when trying to invoke build_wheel

From the above you can see places where message functions are called, and seemingly showing that the correct path is being provided (I want to install the C shared library within a "utils" directory of the project, which also has it's own subdir meson.build and that gets called before the python extension function is called anyways) but it looks like it's ignored at this stage or I need to pass in a different path or not try to install the shared library and get it in via some other mechanism (a custom target?)

running introspect as suggested elsewhere reveals:

(meson) PycharmProjects/cspice-meson-spiceypy [ meson introspect build --install-plan -i                                                               main * ] 4:00 PM
{
    "targets": {
        "/Users/andrew/PycharmProjects/cspice-meson-spiceypy/build/subprojects/cspice/libcspice.1.dylib": {
            "destination": "{install_dir}/libcspice.1.dylib",
            "tag": "runtime",
            "subproject": "libcspice-macos-arm",
            "install_rpath": null
        },
        "/Users/andrew/PycharmProjects/cspice-meson-spiceypy/build/src/test_cyice/test_cyice.cpython-313-darwin.so": {
            "destination": "{py_platlib}/test_cyice/test_cyice.cpython-313-darwin.so",
            "tag": "runtime",
            "subproject": null,
            "install_rpath": null
        }
    },
    "python": {
        "/Users/andrew/PycharmProjects/cspice-meson-spiceypy/src/test_cyice/__init__.py": {
            "destination": "{py_platlib}/test_cyice/__init__.py",
            "tag": "python-runtime",
            "subproject": null
        },
        "/Users/andrew/PycharmProjects/cspice-meson-spiceypy/src/test_cyice/utils/__init__.py": {
            "destination": "{py_platlib}/test_cyice/utils/__init__.py",
            "tag": "python-runtime",
            "subproject": null
        }
    }
}

which is confusing considering I do provide the install dir for the shared library. I would expect it to have a path like {py_platlib}/test_cyice/utils/libcspice.1.dylib.

In any case I am not sure if this is a bug/known unsupported thing in meson-build. More likely I am missing some context/knowledge about meson that would provide an obvious fix.

Any help would be very appreciated!

[rgommers]

This seems to be a case where the installation location metadata Meson generates doesn't match the actual install location for libcspice.dylib, probably because of how install_dir is passed on to the subproject. I verified that this:

$ meson setup build --prefix=$PWD/build-install
$ ninja -C build
$ meson install -C build

results in the install location: build-install/lib/python3.13/site-packages/test_cyice/utils/libcspice.dylib. While the metadata says "destination": "{install_dir}/libcspice.1.dylib", so missing the test_cyice/utils component.

I changed subprojects/cspice/meson.build to use install_dir: install_dir / 'test_cyice/utils' for the shared library, but that only changes the error message to:

meson-python: error: Could not map installation path to an equivalent wheel directory: '{install_dir}/test_cyice/utils/libcspice.1.dylib'

We don't actually have a test package for what you're trying to do here with ctypes, so this may be a bug. What we do have is:

1. link-library-in-subproject, which has a regular shared library in a subproject but doesn't use install_dir for it; instead it relies on meson-python to auto-vendor the shared library into the wheel and link against it from an extension module. That wouldn't work with ctypes.
2. sharedlib-in-package, which install the shared library inside the package (which would work for ctypes), but avoids the subproject part.

[eli-schwartz]

1. link-library-in-subproject, which has a regular shared library in a subproject but doesn't use install_dir for it; instead it relies on meson-python to auto-vendor the shared library into the wheel and link against it from an extension module. That wouldn't work with ctypes.

I've done ctypes using the main project plus auto-vendoring into the wheel, e.g. google/liblc3@3f05fcb#diff-bdc5d0018b7c3a32fc959012f201023cc8f754dd252dce2457f5b33974058aafL58

[AndrewAnnex]

@eli-schwartz if I am interpreting that correctly it looks like you are treating the python part as a optional component to the c library? I'd used the term "subproject" here, but it's not literally a meson subproject I suppose.

I don't think that would work well here or it really isn't the way I want to do things, as my project uses cspice as a dependency (and I don't maintain cspice) and I'd like to be able to eventually move the cspice subproject out of my codebase and into WrapDB anyways. That way other projects wanting to depending on cspice could benefit as well. But it is curious that it does work that way

[eli-schwartz]

The point here is that the C parts install "regularly", to /usr/lib64 for example.

The ctypes part has an if/else that checks to see if it can first be found by scanning in the site-packages directory relative to __file__ using meson-python's standard relocation directory (.mesonpy.libs).

When installing liblc3 as a system package it finds the version in /usr/lib64 as expected.

[AndrewAnnex]

ahh okay I think I got it, this is more in response to my use case for conda-forge right?

[AndrewAnnex]

@rgommers should I post an issue to meson?