Failing to compile numpy from source against a uv-managed interpreter (`Cannot compile `Python.h`.`)

[neutrinoceros]

I'm not able to build numpy from source against a uv-managed Python:

../meson.build:44:2: ERROR: Problem encountered: Cannot compile `Python.h`. Perhaps you need to install python-dev|python-devel

this seems similar to #10185, which was transferred from https://github.com/astral-sh/python-build-standalone, so I'm opening this ticket directly in uv's repo. This issue was already present with uv 0.5.13 and appears to reproduce with 0.5.18

setup:

$ git clone https://github.com/numpy/numpy
$ cd numpy
$ git submodule update --init

then

$ uv build
Building source distribution...
+ /Users/clm/.cache/uv/builds-v0/.tmpcUq9Nd/bin/python /private/tmp/numpy/vendored-meson/meson/meson.py setup /private/tmp/numpy /private/tmp/numpy/.mesonpy-xqqxtm9n -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/private/tmp/numpy/.mesonpy-xqqxtm9n/meson-python-native-file.ini
The Meson build system
Version: 1.5.2
Source dir: /private/tmp/numpy
Build dir: /private/tmp/numpy/.mesonpy-xqqxtm9n
Build type: native build
Project name: NumPy
Project version: 2.3.0.dev0+git20241226.a07c6c5
C compiler for the host machine: cc (clang 16.0.0 "Apple clang version 16.0.0 (clang-1600.0.26.6)")
C linker for the host machine: cc ld64 1115.7.3
C++ compiler for the host machine: c++ (clang 16.0.0 "Apple clang version 16.0.0 (clang-1600.0.26.6)")
C++ linker for the host machine: c++ ld64 1115.7.3
Cython compiler for the host machine: cython (cython 3.0.11)
Host machine cpu family: aarch64
Host machine cpu: aarch64
Program python found: YES (/Users/clm/.cache/uv/builds-v0/.tmpcUq9Nd/bin/python)
Found pkg-config: YES (/opt/homebrew/bin/pkg-config) 2.3.0
Run-time dependency python found: YES 3.13
Has header "Python.h" with dependency python-3.13: NO

../meson.build:44:2: ERROR: Problem encountered: Cannot compile `Python.h`. Perhaps you need to install python-dev|python-devel

A full log can be found at /private/tmp/numpy/.mesonpy-xqqxtm9n/meson-logs/meson-log.txt
  × Failed to build `/private/tmp/numpy`
  ├─▶ The build backend returned an error
  ╰─▶ Call to `mesonpy.build_sdist` failed (exit status: 1)
      hint: This usually indicates a problem with the package or the build environment.

A shorter path to the same error

$ uv venv
$ uv pip install git+https://github.com/numpy/numpy

[zanieb]

The header file is available, e.g., on my machine at ~/.local/share/uv/python/cpython-3.13.1-macos-aarch64-none/include/python3.13/Python.h

I wonder how their configure script seaches for it?

[oscarbenjamin]

I've just run into what I assume is the same issue trying to build python-flint with a uv-installed Python. I have made a simple demonstration repo for a project that has meson as build system and builds a Cython extension module:

https://github.com/oscarbenjamin/meson_simple

It builds fine with a system Python or pyenv Python but not with a uv python.

Steps:

git clone https://github.com/oscarbenjamin/meson_simple
cd meson_simple
uv venv
uv pip install -r requirements-dev.txt
source .venv/bin/activate
meson setup build --wipe
meson compile -C build
spin test

I don't know if this is a new issue because I have not previously tried to build anything like this with a uv-installed Python.

This is the meson configure output:

$ meson setup build --wipe
The Meson build system
Version: 1.6.1
Source dir: /stuff/current/active/mesontest
Build dir: /stuff/current/active/mesontest/build
Build type: native build
Project name: meson-test
Project version: undefined
C compiler for the host machine: cc (gcc 11.4.0 "cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
C linker for the host machine: cc ld.bfd 2.38
Cython compiler for the host machine: cython (cython 3.0.11)
Host machine cpu family: x86_64
Host machine cpu: x86_64
Program python3 found: YES (/stuff/current/active/mesontest/.venv/bin/python3)
Found pkg-config: YES (/usr/bin/pkg-config) 0.29.2
Run-time dependency python found: YES 3.13
Build targets in project: 1

Found ninja-1.10.1 at /usr/bin/ninja
INFO: autodetecting backend as ninja
INFO: calculating backend command to run: /usr/bin/ninja -C /stuff/current/active/mesontest/build

The compile step fails with:

$ meson compile -C build
ninja: Entering directory `/stuff/current/active/mesontest/build'
[2/3] Compiling C object meson_test/_meson_test.c....p/meson-generated_meson_test__meson_test.pyx.c.o
FAILED: meson_test/_meson_test.cpython-313-x86_64-linux-gnu.so.p/meson-generated_meson_test__meson_test.pyx.c.o 
cc -Imeson_test/_meson_test.cpython-313-x86_64-linux-gnu.so.p -Imeson_test -I../meson_test -I/install/include/python3.13 -fvisibility=hidden -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -std=c99 -O2 -g -fPIC -MD -MQ meson_test/_meson_test.cpython-313-x86_64-linux-gnu.so.p/meson-generated_meson_test__meson_test.pyx.c.o -MF meson_test/_meson_test.cpython-313-x86_64-linux-gnu.so.p/meson-generated_meson_test__meson_test.pyx.c.o.d -o meson_test/_meson_test.cpython-313-x86_64-linux-gnu.so.p/meson-generated_meson_test__meson_test.pyx.c.o -c meson_test/_meson_test.cpython-313-x86_64-linux-gnu.so.p/meson_test/_meson_test.pyx.c
meson_test/_meson_test.cpython-313-x86_64-linux-gnu.so.p/meson_test/_meson_test.pyx.c:16:10: fatal error: Python.h: No such file or directory
   16 | #include "Python.h"
      |          ^~~~~~~~~~
compilation terminated.
ninja: build stopped: subcommand failed.

The specific compiler command wrapped is:

cc -Imeson_test/_meson_test.cpython-313-x86_64-linux-gnu.so.p -Imeson_test
-I../meson_test -I/install/include/python3.13 -fvisibility=hidden
-fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -std=c99
-O2 -g -fPIC -MD -MQ
meson_test/_meson_test.cpython-313-x86_64-linux-gnu.so.p/meson-generated_meson_test__meson_test.pyx.c.o
-MF
meson_test/_meson_test.cpython-313-x86_64-linux-gnu.so.p/meson-generated_meson_test__meson_test.pyx.c.o.d
-o
meson_test/_meson_test.cpython-313-x86_64-linux-gnu.so.p/meson-generated_meson_test__meson_test.pyx.c.o
-c
meson_test/_meson_test.cpython-313-x86_64-linux-gnu.so.p/meson_test/_meson_test.pyx.c

Running with a venv off of a system Python I get this command instead:

cc -Imeson_test/_meson_test.cpython-310-x86_64-linux-gnu.so.p -Imeson_test
-I../meson_test -I/usr/include/python3.10
-I/usr/include/x86_64-linux-gnu/python3.10 -fvisibility=hidden
-fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -std=c99
-O2 -g -fPIC -MD -MQ
meson_test/_meson_test.cpython-310-x86_64-linux-gnu.so.p/meson-generated_meson_test__meson_test.pyx.c.o
-MF
meson_test/_meson_test.cpython-310-x86_64-linux-gnu.so.p/meson-generated_meson_test__meson_test.pyx.c.o.d
-o
meson_test/_meson_test.cpython-310-x86_64-linux-gnu.so.p/meson-generated_meson_test__meson_test.pyx.c.o
-c
meson_test/_meson_test.cpython-310-x86_64-linux-gnu.so.p/meson_test/_meson_test.pyx.c

Separating only the include paths it is:

# uv python
-Imeson_test/_meson_test.cpython-313-x86_64-linux-gnu.so.p
-Imeson_test
-I../meson_test
-I/install/include/python3.13

# system python
-Imeson_test/_meson_test.cpython-310-x86_64-linux-gnu.so.p
-Imeson_test
-I../meson_test
-I/usr/include/python3.10
-I/usr/include/x86_64-linux-gnu/python3.10

Somehow it seems that meson has been unable to locate the include directories I guess.

[oscarbenjamin]

Possibly @eli-schwartz could help with understanding how meson identifies the include directories.

[zanieb]

Thanks for the additional details.

Looks like this comes from https://github.com/mesonbuild/meson/blob/dfb449d099d6f3066ca646af197bc7af85059a86/mesonbuild/dependencies/python.py#L369-L370

I don't really see a difference between the variables referenced

❯ $(uv python find --python-preference only-system) -m sysconfig | rg "(INCLUDEPY|include|platinclude) ="
	include = "/opt/homebrew/opt/python@3.12/Frameworks/Python.framework/Versions/3.12/include/python3.12"
	platinclude = "/opt/homebrew/opt/python@3.12/Frameworks/Python.framework/Versions/3.12/include/python3.12"
	CONFINCLUDEPY = "/opt/homebrew/opt/python@3.12/Frameworks/Python.framework/Versions/3.12/include/python3.12"
	INCLUDEPY = "/opt/homebrew/opt/python@3.12/Frameworks/Python.framework/Versions/3.12/include/python3.12"
❯ $(uv python find --python-preference only-managed) -m sysconfig | rg "(INCLUDEPY|include|platinclude) ="
	include = "/Users/zb/.local/share/uv/python/cpython-3.12.6-macos-aarch64-none/include/python3.12"
	platinclude = "/Users/zb/.local/share/uv/python/cpython-3.12.6-macos-aarch64-none/include/python3.12"
	CONFINCLUDEPY = "/Users/zb/.local/share/uv/python/cpython-3.12.6-macos-aarch64-none/include/python3.12"
	INCLUDEPY = "/Users/zb/.local/share/uv/python/cpython-3.12.6-macos-aarch64-none/include/python3.12"

Can you see the pkg-config it uses?

[zanieb]

@oscarbenjamin your logs show -I/install/include/python3.13 which is old uv behavior, can you try reinstalling your interpreter? We change those paths at install time now (see #9857)

[oscarbenjamin]

Oh, actually after updating uv and reinstalling Python it works for the demo:

uv self update
uv python install -r 3.12
rm -r .venv
# then it works

[zanieb]

Great to hear!

Yeah the pkg-config file for these look similar too

❯ cat /Users/zb/.local/share/uv/python/cpython-3.12.6-macos-aarch64-none/lib/pkgconfig/python-3.12.pc
# See: man pkg-config
prefix=/Users/zb/.local/share/uv/python/cpython-3.12.6-macos-aarch64-none
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include

Name: Python
Description: Build a C extension for Python
Requires:
Version: 3.12
Libs.private: -ldl  -framework CoreFoundation
Libs: -L${libdir} 
Cflags: -I${includedir}/python3.12%                                                                                                                                                                                                                           

❯ cat /opt/homebrew/opt/python@3.12/lib/pkgconfig/python-3.12.pc
# See: man pkg-config
prefix=/opt/homebrew/opt/python@3.12/Frameworks/Python.framework/Versions/3.12
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include

Name: Python
Description: Build a C extension for Python
Requires:
Version: 3.12
Libs.private: -ldl  -framework CoreFoundation
Libs: -L${libdir} 
Cflags: -I${includedir}/python3.12

[zanieb]

@neutrinoceros can you use uv python install --reinstall <version> and try again?

[oscarbenjamin]

I've confirmed I can build python-flint and run the whole test suite. The problems I was seeing were fixed by updating uv, using the updated uv to reinstall Python and then rebuilding the venv.

Thanks @zanieb !

[eli-schwartz]

Looks like you were able to figure out the critical references we use, and it turns out everything works with current uv?

[oscarbenjamin]

Thanks @eli-schwartz.

everything works with current uv?

Yes in my testing but it would be good for @neutrinoceros to confirm.

I just tried installing numpy from git as described in the OP and it seems to work okay:

$ uv pip install git+https://github.com/numpy/numpy
 Updated https://github.com/numpy/numpy (bbf4836a8)
Resolved 1 package in 34.85s
   Built numpy @ git+https://github.com/numpy/numpy@bbf4836a82d255c2a2161d277abb4b6f5455ca6d
Prepared 1 package in 5m 54s
Installed 1 package in 19ms
 + numpy==2.3.0.dev0 (from git+https://github.com/numpy/numpy@bbf4836a82d255c2a2161d277abb4b6f5455ca6d)
$ python -c 'import numpy; print(numpy.__version__)'
2.3.0.dev0+git20250110.bbf4836

If anyone wants to try that one at home then they would first need to install openblas e.g. something like apt-get libopenblas-dev.

[neutrinoceros]

@neutrinoceros can you use uv python install --reinstall and try again?

Sorry I should have mentioned: I tried this with uv 0.5.14, which if I'm following correctly has the most recent updates in this area, so I didn't think it was necessary to try again with 0.5.18. I just tried now:

uv python install 3.13 --reinstall
uv pip install git+https://github.com/numpy/numpy --no-cache --no-config

(throwing --no-cache and --no-config for good measure)
The error persists :/