BUG: downstream segfault with Cython >= 3.0a5

[tylerjereddy]

In short, we're seeing a segfault for a pickle-related line of code when using Cython >= 3.0a5, but not stable release or pre-release versions prior to that. The original issue description downstream is scipy/scipy#14732, but I will also paste the problematic code block for quick inspection (from Ralf's original description)--the last line below segfaults:

    def test_pickleable(self):
        # Make sure we can pickle and unpickle the interpolant without any
        # changes in the behavior.
        seq = Halton(1, scramble=False, seed=np.random.RandomState())

        x = 3*seq.random(50)
        xitp = 3*seq.random(50)

        y = _1d_test_function(x)

        interp = self.build(x, y)

        yitp1 = interp(xitp)
        yitp2 = pickle.loads(pickle.dumps(interp))(xitp)

While that doesn't exactly give you an isolated/minimal reproducer since it is buried in the depths of SciPy, perhaps you have suggestions for avoiding the segfault? For now, I'll propose using <3.0a5 in our pre-release tests in the absence of another way to avoid the segfault: scipy/scipy#14801

Although I did bisect Cython versions carefully with the reproducer below, I did not yet check to see if the usage of multiple threads by the test suite was contributing to the problem:

python runtests.py -t "scipy/interpolate/tests/test_rbfinterp.py" -- -n 2

[da-woods]

I'm struggling to reproduce this locally in a Python 3.8 virtualenv with Scipy build from source (5938242bbfedd14fa2fa6ce5a9fa1b1a535585df), Numpy built from source (c37b6af5577584f6ab0a08a161d9d657741341ea), and Cython-3.0.0a9, pythran-0.10.0 from Pip.

Running python runtests.py -t "scipy/interpolate/tests/test_rbfinterp.py" passes the tests while running python runtests.py -t "scipy/interpolate/tests/test_rbfinterp.py" -- -n 2 gives me

ERROR: usage: runtests.py [options] [file_or_dir] [file_or_dir] [...]
runtests.py: error: unrecognized arguments: -n scipy/interpolate/tests/test_rbfinterp.py

[tylerjereddy]

hmm, missing pytest-xdist plugin maybe?

[da-woods]

hmm, missing pytest-xdist plugin maybe?

Yes - that gets -n 2 to work. The test passes on my PC though

[tylerjereddy]

Hmm, it has been happening in our CI for a few weeks. There is some suspicion of an interaction with Pythran maybe according to our current discussion in scipy/scipy#14732. I mostly just wanted to identify the first alpha release of Cython that was involved in that segfaulting interaction. I'm not sure if it is mostly on the Pythran side, but I've certainly been able to reproduce locally.

[da-woods]

If you can run it locally it might be useful to try to get the C traceback from the faulthandler module (just adding import faulthander; faulthandler.enable() to the test function should be enough. It might give a bit more of an indication where it's all going wrong.

[tylerjereddy]

I didn't have any luck with that, though I did at least confirm locally that a single test thread was sufficient to reproduce the segfault. I tried a debug build of SciPy, but still getting the same information back it seems. I guess I could try gdb with Python at some point.

Since it is a fairly old alpha release of Cython where the segfault starts, I think some folks prefer to check the Pythran side of things--I just thought isolating a specific alpha release might hint at something.

[tylerjereddy]

A bit more information using the following debug workflow:

• gdb python
• run runtests.py -t "scipy/interpolate/tests/test_rbfinterp.py"

Program received signal SIGSEGV, Segmentation fault.
__pyx_pf_5scipy_7spatial_7ckdtree_7cKDTree_22__setstate__ (__pyx_v_state=<optimized out>, __pyx_v_self=<optimized out>, this=<optimized out>) at scipy/spatial/ckdtree.cxx:18771
18771	  __pyx_t_18 = __pyx_f_5numpy_7ndarray_4size_size(__pyx_v_tree);

[tylerjereddy]

I also bisected the exact NumPy commit that participates in the segfault with Cython: scipy/scipy#14732 (comment)