ENH: support shared libraries on Windows when explicitly enabled

Fixes #525.

Author: dnicolodi

docs/how-to-guides/shared-libraries.rst

@@ -24,22 +24,22 @@ this ``libdir`` in this guide) or to a location in ``site-packages`` within the
 Python package install tree. All these scenarios are (or will be) supported,
 with some caveats:
 
-+-----------------------+------------------+---------+-------+-------+
-| shared library source | install location | Windows | macOS | Linux |
-+=======================+==================+=========+=======+=======+
-| internal              | libdir           | no (1)  | ✓     | ✓     |
-+-----------------------+------------------+---------+-------+-------+
-| internal              | site-packages    | ✓       | ✓     | ✓     |
-+-----------------------+------------------+---------+-------+-------+
-| external              | n/a              | ✓ (2)   | ✓     | ✓     |
-+-----------------------+------------------+---------+-------+-------+
++-----------------------+------------------+------------+-------+-------+
+| shared library source | install location | Windows    | macOS | Linux |
++=======================+==================+============+=======+=======+
+| internal              | libdir           | ✓ :sup:`1` | ✓     | ✓     |
++-----------------------+------------------+------------+-------+-------+
+| internal              | site-packages    | ✓          | ✓     | ✓     |
++-----------------------+------------------+------------+-------+-------+
+| external              | ---              | ✓ :sup:`2` | ✓     | ✓     |
++-----------------------+------------------+------------+-------+-------+
 
 .. TODO: add subproject as a source
 
-1. Internal shared libraries on Windows cannot be automatically handled
-   correctly, and currently ``meson-python`` therefore raises an error for them.
-   `PR meson-python#551 <https://github.com/mesonbuild/meson-python/pull/551>`__
-   may improve that situation in the near future.
+1. Support for internal shared libraries on Windows is enabled with the
+   :option:`allow-windows-internal-shared-libs` option and requires cooperation
+   from the package to extend the DLL search path or pre-load the required
+   libraries. See below for more details.
 
 2. External shared libraries require ``delvewheel`` usage on Windows (or some
    equivalent way, like amending the DLL search path to include the directory
@@ -91,23 +91,26 @@ the lack of RPATH support:
    :start-after: start-literalinclude
    :end-before: end-literalinclude
 
-If an internal shared library is not only used as part of a Python package, but
-for example also as a regular shared library in a C/C++ project or as a
-standalone library, then the method shown above won't work. The library is
-then marked for installation into the system default ``libdir`` location.
-Actually installing into ``libdir`` isn't possible with wheels, hence
-``meson-python`` will instead do the following *on platforms other than
-Windows*:
+If an internal shared library is not only used as part of a Python package,
+but for example also as a regular shared library then the method shown above
+won't work. The library is then marked for installation into the system
+default ``libdir`` location.  Actually installing into ``libdir`` isn't
+possible with wheels, hence ``meson-python`` will instead do the following:
 
-1. Install the shared library to ``<project-name>.mesonpy.libs`` (i.e., a
+1. Install the shared library to the ``.<project-name>.mesonpy.libs``
    top-level directory in the wheel, which on install will end up in
-   ``site-packages``).
-2. Rewrite RPATH entries for install targets that depend on the shared library
-   to point to that new install location instead.
+   ``site-packages``.
+2. On platforms other than Windows, rewrite RPATH entries for install targets
+   that depend on the shared library to point to that new install location
+   instead.
 
-This will make the shared library work automatically, with no other action needed
-from the package author. *However*, currently an error is raised for this situation
-on Windows. This is documented also in :ref:`reference-limitations`.
+On platforms other than Windows, this will make the shared library work
+automatically, with no other action needed from the package author. On
+Windows, due to the lack of RPATH support, the ``.<project-name>.mesonpy.libs``
+location search path needs to be added to the DLL search path, with code
+similar to the one presented above. For this reason, handling of internal
+shared libraries on Windows is conditional to setting the
+:option:`allow-windows-internal-shared-libs` option.
 
 
 External shared libraries
@@ -245,7 +248,7 @@ this will look something like:
     foo_dep = foo_subproj.get_variable('foo_dep')
 
 Now we can use ``foo_dep`` like a normal dependency, ``meson-python`` will
-include it into the wheel in ``<project-name>.mesonpy.libs`` just like an
+include it into the wheel in ``.<project-name>.mesonpy.libs`` just like an
 internal shared library that targets ``libdir`` (see
 :ref:`internal-shared-libraries`).
 *Remember: this method doesn't support Windows (yet)!*

docs/reference/config-settings.rst

@@ -14,6 +14,19 @@ settings you can pass when building the project. Please refer to the
 :ref:`how-to-guides-meson-args` guides for information on how to use
 them.
 
+.. option:: allow-windows-internal-shared-libs
+
+   Enable support for relocating internal shared libraries that would be
+   installed into the system shared library location to the
+   ``.<package-name>.mesonpy.libs`` folder also on Windows. The relocation can
+   be done transparently on UNIX platforms and on macOS, where the shared
+   library load path can be adjusted via RPATH or equivalent mechanisms.
+   Windows lacks a similar facility, thus the Python package is responsible to
+   extend the DLL load path to include this directory or to pre-load the
+   shared libraries. See :ref:`here <internal-shared-libraries>` for detailed
+   documentation. This option ensures that the package authors are aware of
+   this requirement.
+
 .. option:: build-dir
 
    By default ``meson-python`` uses a temporary build directory.

docs/reference/limitations.rst

@@ -31,16 +31,12 @@ Shared libraries on Windows
 On Windows, ``meson-python`` cannot encapsulate shared libraries
 installed as part of the Meson project into the Python wheel for
 Python extension modules or executables, in a way suitable for them to
-be found at run-time.
-
-This limitation can be overcome with static linking or using
-`delvewheel`_ to post-process the Python wheel to bundle the required
-shared libraries and include the setup code to properly set the
-library search path.
+be found at run-time without cooperation from the Python package, see
+the documentation for the :option:`allow-windows-internal-shared-libs`
+and the :ref:`shared-libraries` guide.
 
 
 .. _install_data(): https://mesonbuild.com/Reference-manual_functions.html#install_data
 .. _importlib-resources: https://importlib-resources.readthedocs.io/en/latest/index.html
-.. _delvewheel: https://github.com/adang1345/delvewheel
 
 .. |install_data()| replace:: ``install_data()``

docs/reference/pyproject-settings.rst

@@ -34,6 +34,17 @@ use them and examples.
    ``meson-python`` itself. It can be overridden by the :envvar:`MESON`
    environment variable.
 
+.. option:: tool.meson-python.shared-libs-win32
+
+   A boolean indicating whether shared libraries should be supported on
+   Windows. ``meson-python`` installs shared libraries in a dedicated location
+   and uses RPATH or equivalent mechanisms to have Python modules and native
+   executables load them form there. Windows does not have an equivalent
+   mechanism to set the DLL load path. Supporting shared libraries on Windows
+   requires collaboration from the package. To make sure that package authors
+   are aware of this requirement, ``meson-python`` raises an error if a
+   package contains DLLs and this option is not set.
+
 .. option:: tool.meson-python.args.dist
 
    Extra arguments to be passed to the ``meson dist`` command.

mesonpy/__init__.py

@@ -309,10 +309,12 @@ def __init__(
         metadata: Metadata,
         manifest: Dict[str, List[Tuple[pathlib.Path, str]]],
         limited_api: bool,
+        windows_shared_libs: bool,
     ) -> None:
         self._metadata = metadata
         self._manifest = manifest
         self._limited_api = limited_api
+        self._windows_shared_libs = windows_shared_libs
 
     @property
     def _has_internal_libs(self) -> bool:
@@ -422,6 +424,12 @@ def _install_path(self, wheel_file: mesonpy._wheelfile.WheelFile, origin: Path,
 
         if self._has_internal_libs:
             if _is_native(origin):
+                if sys.platform == 'win32' and not self._windows_shared_libs:
+                    raise NotImplementedError(
+                        'Loading shared libraries bundled in the Python wheel on Windows requires '
+                        'setting the DLL load path or pre-loading. See the documentation for '
+                        'the "tool.meson-pyhton.allow-windows-internal-shared-libs" option.')
+
                 # When an executable, libray, or Python extension module is
                 # dynamically linked to a library built as part of the project,
                 # Meson adds a library load path to it pointing to the build
@@ -567,6 +575,7 @@ def _string_or_path(value: Any, name: str) -> str:
     scheme = _table({
         'meson': _string_or_path,
         'limited-api': _bool,
+        'allow-windows-internal-shared-libs': _bool,
         'args': _table({
             name: _strings for name in _MESON_ARGS_KEYS
         }),
@@ -784,6 +793,10 @@ def __init__(
                 'The package targets Python\'s Limited API, which is not supported by free-threaded CPython. '
                 'The "python.allow_limited_api" Meson build option may be used to override the package default.')
 
+        # Shared library support on Windows requires collaboration
+        # from the package, make sure the developers aknowledge this.
+        self._windows_shared_libs = pyproject_config.get('allow-windows-internal-shared-libs', False)
+
     def _run(self, cmd: Sequence[str]) -> None:
         """Invoke a subprocess."""
         # Flush the line to ensure that the log line with the executed
@@ -988,13 +1001,13 @@ def sdist(self, directory: Path) -> pathlib.Path:
     def wheel(self, directory: Path) -> pathlib.Path:
         """Generates a wheel in the specified directory."""
         self.build()
-        builder = _WheelBuilder(self._metadata, self._manifest, self._limited_api)
+        builder = _WheelBuilder(self._metadata, self._manifest, self._limited_api, self._windows_shared_libs)
         return builder.build(directory)
 
     def editable(self, directory: Path) -> pathlib.Path:
         """Generates an editable wheel in the specified directory."""
         self.build()
-        builder = _EditableWheelBuilder(self._metadata, self._manifest, self._limited_api)
+        builder = _EditableWheelBuilder(self._metadata, self._manifest, self._limited_api, self._windows_shared_libs)
         return builder.build(directory, self._source_dir, self._build_dir, self._build_command, self._editable_verbose)
 
 

mesonpy/_rpath.py

@@ -19,7 +19,7 @@
 if sys.platform == 'win32' or sys.platform == 'cygwin':
 
     def fix_rpath(filepath: Path, libs_relative_path: str) -> None:
-        raise NotImplementedError(f'Bundling libraries in wheel is not supported on {sys.platform}')
+        pass
 
 elif sys.platform == 'darwin':
 

tests/packages/sharedlib-in-package/mypkg/__init__.py

@@ -8,16 +8,15 @@
 
 # start-literalinclude
 def _append_to_sharedlib_load_path():
-    """
-    Ensure the shared libraries in this package can be loaded on Windows.
+    """Ensure the shared libraries in this package can be loaded on Windows.
 
     Windows lacks a concept equivalent to RPATH: Python extension modules
     cannot find DLLs installed outside the DLL search path. This function
     ensures that the location of the shared libraries distributed inside this
     Python package is in the DLL search path of the process.
 
-    The Windows DLL search path includes the object depending on it is located:
-    the DLL search path needs to be augmented only when the Python extension
+    The Windows DLL search path includes the path to the object attempting
+    to load the DLL: it needs to be augmented only when the Python extension
     modules and the DLLs they require are installed in separate directories.
     Cygwin does not have the same default library search path: all locations
     where the shared libraries are installed need to be added to the search

tests/test_tags.py

@@ -77,7 +77,7 @@ def test_python_host_platform(monkeypatch):
 def wheel_builder_test_factory(content, pure=True, limited_api=False):
     manifest = defaultdict(list)
     manifest.update({key: [(pathlib.Path(x), os.path.join('build', x)) for x in value] for key, value in content.items()})
-    return mesonpy._WheelBuilder(None, manifest, limited_api)
+    return mesonpy._WheelBuilder(None, manifest, limited_api, False)
 
 
 def test_tag_empty_wheel():

tests/test_wheel.py

@@ -266,7 +266,8 @@ def test_entrypoints(wheel_full_metadata):
 
 def test_top_level_modules(package_module_types):
     with mesonpy._project() as project:
-        builder = mesonpy._EditableWheelBuilder(project._metadata, project._manifest, project._limited_api)
+        builder = mesonpy._EditableWheelBuilder(
+            project._metadata, project._manifest, project._limited_api, project._shared_libs_win32)
         assert set(builder._top_level_modules) == {
             'file',
             'package',