Current guidance on converting a large autoconf-based project

[AA-Turner]

Hello,

We are currently in the early stages of experimenting with Meson for use as CPython (the reference implementation of the Python programming language)'s build system.

CPython currently uses an autotools-based workflow for Unix, with configure.ac, a checked-in configure, and Makefile.pre.in, converted to the regular Makefile via the configure script. We also have custom scripts currently written in Python for cross-compiling to e.g. WASM and for mobile platforms, and a Windows build-system as a VS solution.

The current guidance on the Meson website is useful, although fairly sparse. I wondered if there might be any other resources recommended by the project that have been generally found to be helpful?

For example, we make extensive use of custom autoconf/m4 macros (AC_DEFUN). Meson lacks (and I believe does not intend to support?) macros or user-defined functions, so what have others relied upon to convert this style of construct to Meson, whilst not increasing line length too much?

I also wondered what the guidance is more generally for file-size management. Our configure.ac and Makefile.pre.in together are nearly 12,000 lines long, and hard to reason about. Is there a nice solution for this in Meson? I couldn't find anything obvious on a quick first look, sorry if I've missed something obvious.

Thanks,
Adam

[bonzini]

what have others relied upon to convert this style of construct to Meson, whilst not increasing line length too much?

Generally the preferred pattern is to use loops. It depends a lot on what the functions are doing, though. Often, the possibility to alternate configure-like content and Makefile-like content, arranged by subdirectory, is enough to get a good organization.

One of the largest projects I know of, for which a one-patch-at-a-time conversion from autotools is available (not bisectable), is Libvirt; see https://patchew.org/Libvirt/[email protected]/ (321 files changed, 9526 insertions, 15979 deletions!)

My experience is that while with Meson you will encounter a few drawbacks or weird choices, the annoyance is quite limited and especially balanced by the readability and hackability of the meson.build files, and by the availability of good data structures (this is #1 reason for me to prefer Meson to CMake or autotools, really). And while Meson's DSL is not Python, it is certainly similar and you should be at home.

One thing I would check is the OSes you support, since it is possible that some are missing in Meson. Also I am not sure of what your stance is on bootstrapping; if you need to escape and do something more than what Meson's DSL allows, the usual thing to do is to... use Python. That may be a problem for you, for obvious reasons, or maybe not. Also if in some cases you feel that you need some extensions to Meson, you're welcome to ask!

[AA-Turner]

Hi Paolo,

Thank you for the very fast reply!

I will look at libvirt, thank you for the pointer.

One thing I would check is the OSes you support

At minimum, we support Windows, macOS, Linux (eg Debian, Ubuntu, Gentoo, Fedora, RHEL), WASM (WASI & emscripten), iOS, Android, FreeBSD, and z/OS. PEP 11 is our official policy here. We have limited support for musl libc, but do accept patches.

Also I am not sure of what your stance is on bootstrapping

We will require that bootstrapping be possible, especially given how foundational Python is for many Linux distributions. Our current plan for this is to restrict ourselves to Muon-compatible Meson. Would you have different advice?

In general, how open would you be to extending the DSL, either for Meson in general, or perhaps a mechanism where we can do so just for CPython? I appreciate that the usual 'escape hatch' would be to reach for Python code, but we're in rather a special case!

If you'd prefer to keep the language free of UDFs or macro-like constructs (entirely fair, it is your language!), we might have a 'wish list' of helper functions that would be useful. We're not at that point yet, to be clear.

Thanks,
Adam

[bonzini]

Ok, z/OS might be the one that needs some work in Meson. I remember seeing PRs.

Our current plan for this is to restrict ourselves to Muon-compatible Meson. Would you have different advice?

Makes sense, though that would also make it harder to add new features to the DSL. But if you plan to have autotools and Meson coexist for a while, adding anything that's missing in Muon (both language features and platform support) can be pipelined with the actual conversion.

In general, how open would you be to extending the DSL, either for Meson in general, or perhaps a mechanism where we can do so just for CPython?

I'm not a maintainer but in my experience, while adding functions and/or macros is not going to happen, reasonable additions do happen, especially if they come as PRs. The filesystem module gets some additions relatively often, for example.

[eli-schwartz]

In general, how open would you be to extending the DSL, either for Meson in general, or perhaps a mechanism where we can do so just for CPython? I appreciate that the usual 'escape hatch' would be to reach for Python code, but we're in rather a special case!

The usual escape hatch is technically to write a script. Python is "recommended" by Meson because it's a pleasant language and likely to be installed if meson is, and it avoids the issue of writing your script twice over, once as a Posix shell script and once as a windows batch file. But of course you can always write it twice over.

As @bonzini said, reasonable additions are very much of interest to us, we're just nervous about functions specifically. I'd like to add to that by highlighting the fact that the meson "modules" system is specifically designed to allow people to suggest new and possibly somewhat fringe functionality dedicated to particular use cases in a stable and non arbitrary-functions way. If you look at for example the gnome module, it largely consists of functions that aren't very useful outside of the gnome ecosystem, but are very useful to the gnome ecosystem. The functions do whatever the gnome people need it to do, and that way they don't need to maintain a giant third-party macros collection copy-pasted into every gnome project. :D

Last time I checked the python build system (a year or so ago?) to ponder the feasibility of a meson port, I got the impression that there was nothing so unique to python that implementing support in meson couldn't be dressed up as a generic helper which many projects could benefit from.

One thing I have kind of wanted for a while is to have better ways to generate a config.h style header for the common cases (which I think autoconf+autoheader is rather good at, especially w.r.t. DRY). A dedicated autoheader-style module might work out very nicely.

[eli-schwartz]

We will require that bootstrapping be possible, especially given how foundational Python is for many Linux distributions. Our current plan for this is to restrict ourselves to Muon-compatible Meson. Would you have different advice?

I would absolutely recommend this.

Bootstrapping can be done by depending on an older version of the thing you are building, e.g. this is how GCC itself is bootstrapped. Current GCC can build with GCC 5.4 (GCC 11 through 14 can build with GCC 4.8.3), and meson does have a somewhat similar style because we still guarantee support for python 3.7. Still, it's very nice to not have to build repeated versions in order to bootstrap. For C/C++ it's sort of unavoidable. Less so for python... And having multiple versions of python in the bootstrap set for a distro is awkward if that distro isn't Gentoo Linux, which already supports multiple python versions via features that are very much not broadly available on other package managers.

Muon is very open to patches implementing missing features from Meson. I have found the project to be delightful to work with. I expect that you won't find restricting yourself to Muon compatible features, to be a practical blocker.

[eli-schwartz]

Hi, this sounds very exciting and I'd love to help however I can. :)

As @bonzini mentioned, it's generally quite practical to use foreach loops to replace the use of "define a macro / function and call it many times with slightly different arguments".

I also wondered what the guidance is more generally for file-size management. Our configure.ac and Makefile.pre.in together are nearly 12,000 lines long, and hard to reason about. Is there a nice solution for this in Meson? I couldn't find anything obvious on a quick first look, sorry if I've missed something obvious.

There has previously been support / sympathy for the idea of an include() function, although the person working on it iirc closed the draft PR out of fading interest. From memory, we refactored the core a bit to allow tracking visited files rather than assuming visited directories has a one to one correspondence to visited files, but never ended up going further. We might want to restrict "includes" to the current directory (because otherwise if one included file defines a build target you have to ask "where should the build output be placed", normally "the directory it is defined in"), but you'd be able to split up your build handling into a kind of "package", e.g. subdir('build-aux') in turn does include('deps.meson'); include('funcs.meson'); ... which sounds like what you'd want.

You can of course very crudely hack it today, by splitting across files and simply placing each file in a distinct subdir. Like the include() proposal but with more subdirectories just for bookkeeping. :P