Build Python extensions with CMake

[JCGoran]

Context

We currently build NEURON Python modules with distutils (integrated as part of setuptools), which has been marked deprecated for years (see also #2228). It is not at all obvious how any of the Python modules are built, and tracing how it actually works (how the CLI args are constructed) basically requires stepping through setup.py using PDB (the Python debugger). distutils also has very poor compiler support (see for instance #3313), so it would be useful to move the Python compiled modules to another build system. Ideally, we would migrate everything to something like scikit-build-core, which basically just runs CMake and builds the Python module.

Changes

This PR moves the following modules so they get built with CMake instead, and use scikit-build-core as a build backend:

• neuron.hoc
• neuron.rxd.geometry3d
• neuronmusic (not shipped in wheels)

This has the following consequences:

• setup.py and any other files that it used (like MANIFEST.in or setup.cfg) are removed
• all information about the configuration used for building wheels is contained in the top-level pyproject.toml file
• building a local wheel can be achieved using pip wheel --no-deps .
• building a portable wheel can be achieved by installing cibuildwheel and running it in the top-level directory
• as an alternative to the above two build options, the build_wheels.bash script can now be used in almost exactly the same way as before, except the coreneuron option has been replaced by the NRN_ENABLE_CORENEURON env variable (or passing --config-settings=cmake.define.NRN_ENABLE_CORENEURON=ON to pip, see updated dev docs for details)
• due to the inflexibility of PEP 621, we cannot change certain parts of pyproject.toml, notably, the name of the wheel (this has nothing to do with the importable Python package name, just the filename of the wheel and the metadata that is inserted). The change_name.py script is added solely as a workaround for this limitation (otherwise we would need to add yet another CMakeLists.txt somewhere that just configures the pyproject.toml file, and CMake-in-CMake is not a pleasant experience), and runs only when doing a release build (admittedly, it is hard to test that the change in Azure actually works, though I can verify that at least the script itself works locally)
• the seemingly-large CMake changes are mostly due to having to copy the Python files, and I think having an actual list of them (instead of globing for *.py files) is a good idea
• the notable additions to CMake, in PythonCompileHelper.cmake, are the cythonize and add_nrn_python_library functions, which are extensively documented, and now build the above modules
• due to RXD using the NumPy C API, our wrapper for find_package(Python) now has to find NumPy headers as well. This is not a problem when building wheels, but when NRN_PYTHON_DYNAMIC has multiple versions, each Python version must be able to find its own installation of NumPy
• the various Python script wrappers (or rather, their softlinks) are gone, and are just added when building a wheel (since they don't make sense outside of that context)
• the install prefix of Python files is now always defined by NRN_INSTALL_PYTHON_PREFIX, and anything else (scripts, libraries, etc.) is installed under NRN_INSTALL_DATA_PREFIX. The two variables are internal and cannot be set, but default to lib/python/neuron and . (for CMake builds), and neuron and neuron/.data/ (for wheels). This is because we currently ship all of the other non-Python NEURON components under neuron/.data
• update IV submodule to customize install location
• resolves all of the linked issues in the sidebar since we finally get rid of setuptools/distutils as a build backend

EDIT: requires merging neuronsimulator/iv#55 first.

[azure-pipelines]

✔️ 83ba919 -> Azure artifacts URL

[azure-pipelines]

✔️ 6028ba5 -> Azure artifacts URL

[azure-pipelines]

✔️ f118ff2 -> Azure artifacts URL

[azure-pipelines]

✔️ 26f31f0 -> Azure artifacts URL

[azure-pipelines]

✔️ 30b45e7 -> Azure artifacts URL

[nrnhines]

Thanks! This is a great benefit and represents substantial modernization. The main comment clarifies very well. I'm looking forward to working with this new style. Any issues that crop up in the future with my sometimes peculiar build circumstances are best fixed with future PRs on this base. I tried

hines@Michaels-MacBook-Pro-2 temp % bash bldnrnmacpkg.sh python3.9 python3.13 2>&1 | tee ztemp

and of course it fails with a dozen or so

/Users/hines/neuron/temp/build/share/lib/python/neuron/rxd/geometry3d/graphicsPrimitives.cpp:30453:134: error: use of undeclared identifier '_PyGC_FINALIZED'
/Users/hines/neuron/temp/build/share/lib/python/neuron/rxd/geometry3d/graphicsPrimitives.cpp:33500:36: error: no member named 'curexc_traceback' in '_ts'
/Users/hines/neuron/temp/build/share/lib/python/neuron/rxd/geometry3d/graphicsPrimitives.cpp:34259:55: error: no member named 'ob_digit' in '_longobject'
/Users/hines/neuron/temp/build/share/lib/python/neuron/rxd/geometry3d/graphicsPrimitives.cpp:34400:27: error: no matching function for call to '_PyLong_AsByteArray'

Go ahead and merge. I'll deal with this next week.

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[azure-pipelines]

✔️ 537f40f -> Azure artifacts URL

[JCGoran]

@nrnhines I tried running bash bldnrnmacpkg.sh python3.9 in a venv with all of the Python requirements installed, and it built the main target. It stopped later at the macpkg target at the codesign step, but I take it it's possible to work around it.