Read Ninja response files natively in pywasmcross

[agriyakhetarpal]

We currently set a higher threshold for Ninja (to 131072, to be more exact) to make it create response files at a much longer limit, but we should explore if pywasmcross can gain the ability to read such files and reconstruct the command-line arguments from there (and write to them).

I am not sure if this can be done in an elegant manner, but since we don't really support Windows (despite a few users using it), we could even add this as an environment variable for build_env for a start.

xref scipy/scipy#25618

[ryanking13]

How is Windows related to this problem? I am +1 for supporting handling the response files properly anyways. I had a PR for that #70, but it had some regression with meson (I don't remember exactly what the regression was)

[agriyakhetarpal]

Ah, nice, thanks for the link to your PR! Windows is related to the problem from a historical perspective because it's heavily tied to the existence of response files. They stemmed in build systems from the fact that Windows used to have fairly short command-line length limits. It is roughly 32767 characters on Windows nowadays. So while it's much bigger, it's still one-fourth and minus one of our threshold and not enough to succeed in SciPy builds.

[agriyakhetarpal]

After discussion with @ryanking13 in person a few minutes after posting the above comment, we settled on bumping the threshold limit globally for Ninja here as the easier solution, as we already kind of do it for CMake:

pyodide-build/pyodide_build/tools/cmake/Modules/Platform/Emscripten.cmake

Lines 81 to 86 in 2678e04

	set(CMAKE_C_USE_RESPONSE_FILE_FOR_LIBRARIES 0)
	set(CMAKE_CXX_USE_RESPONSE_FILE_FOR_LIBRARIES 0)
	set(CMAKE_C_USE_RESPONSE_FILE_FOR_OBJECTS 1)
	set(CMAKE_CXX_USE_RESPONSE_FILE_FOR_OBJECTS 1)
	set(CMAKE_C_USE_RESPONSE_FILE_FOR_INCLUDES 1)
	set(CMAKE_CXX_USE_RESPONSE_FILE_FOR_INCLUDES 1)

. We can leave handling the response files natively for a later day.