Recursion error on s390x

[skriesch]

We need asttokens for openSUSE on s390x.
asttokens version: 3.0.0
We have the following build recursion test failures on openSUSE Tumbleweed:

 =================================== FAILURES ===================================
[   32s] ______________________ TestMarkTokens.test_deep_recursion ______________________
[   32s] 
[   32s] self = <tests.test_mark_tokens.TestMarkTokens testMethod=test_deep_recursion>
[   32s] 
[   32s]     def test_deep_recursion(self):
[   32s]       # This testcase has 1050 strings joined with '+', which causes naive recursions to fail with
[   32s]       # 'maximum recursion depth exceeded' error. We actually handle it just fine, but we can't use
[   32s]       # to_source() on it because it chokes on recursion depth. So we test individual nodes.
[   32s]       source = tools.read_fixture('astroid/joined_strings.py')
[   32s]     
[   32s]       if self.is_astroid_test:
[   32s]         if getattr(astroid, '__version__', '1') >= '2':
[   32s]           # Astroid 2 no longer supports this; see
[   32s]           # https://github.com/PyCQA/astroid/issues/557#issuecomment-396004274
[   32s]           self.skipTest('astroid-2.0 does not support this')
[   32s]     
[   32s]         # Astroid < 2 does support this with optimize_ast set to True
[   32s]         astroid.MANAGER.optimize_ast = True
[   32s]         try:
[   32s]           m = self.create_mark_checker(source, verify=False)
[   32s]         finally:
[   32s]           astroid.MANAGER.optimize_ast = False
[   32s]     
[   32s]         self.assertEqual(len(m.all_nodes), 4)     # This is the result of astroid's optimization
[   32s]         self.assertEqual(m.view_node_types_at(1, 0), {'Module', 'Assign', 'AssignName'})
[   32s]         const = next(n for n in m.all_nodes if isinstance(n, astroid.nodes.Const))
[   32s]         # TODO: Astroid's optimization makes it impossible to get the right start-end information
[   32s]         # for the combined node. So this test fails. To avoid it, don't set 'optimize_ast=True'. To
[   32s]         # fix it, astroid would probably need to record the info from the nodes it's combining. Or
[   32s]         # astroid could avoid the need for the optimization by using an explicit stack like we do.
[   32s]         #self.assertEqual(m.atok.get_text_range(const), (5, len(source) - 1))
[   32s]       else:
[   32s] >       m = self.create_mark_checker(source, verify=False)
[   32s] 
[   32s] tests/test_mark_tokens.py:220: 
[   32s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   32s] tests/test_mark_tokens.py:37: in create_mark_checker
[   32s]     atok = self.create_asttokens(source)
[   32s] tests/test_mark_tokens.py:53: in create_asttokens
[   32s]     return ASTTokens(source, parse=True)
[   32s] asttokens/asttokens.py:109: in __init__
[   32s]     self._tree = ast.parse(source_text, filename) if parse else tree
[   32s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   32s] 
[   32s] source = "x = ('R0lGODlhigJnAef/AAABAAEEAAkCAAMGAg0GBAYJBQoMCBMODQ4QDRITEBkS'\r\n     +'CxsSEhkWDhYYFQ0aJhkaGBweGyccGh8hHiIkIiM...Mh44U0gxMDI5JkM0JjU3NDY6Kjc5Njo7OUE8Ozw+Oz89QTxA'\r\n     +'F1akOFFiRIgPHTZksKBAgMCLGTdGNIAAQgKfDAcgZbj0odOnUA8GBAA7')"
[   32s] filename = '<unknown>', mode = 'exec'
[   32s] 
[   32s]     def parse(source, filename='<unknown>', mode='exec', *,
[   32s]               type_comments=False, feature_version=None):
[   32s]         """
[   32s]         Parse the source into an AST node.
[   32s]         Equivalent to compile(source, filename, mode, PyCF_ONLY_AST).
[   32s]         Pass type_comments=True to get back type comments where the syntax allows.
[   32s]         """
[   32s]         flags = PyCF_ONLY_AST
[   32s]         if type_comments:
[   32s]             flags |= PyCF_TYPE_COMMENTS
[   32s]         if feature_version is None:
[   32s]             feature_version = -1
[   32s]         elif isinstance(feature_version, tuple):
[   32s]             major, minor = feature_version  # Should be a 2-tuple.
[   32s]             if major != 3:
[   32s]                 raise ValueError(f"Unsupported major version: {major}")
[   32s]             feature_version = minor
[   32s]         # Else it should be an int giving the minor version for 3.x.
[   32s] >       return compile(source, filename, mode, flags,
[   32s]                        _feature_version=feature_version)
[   32s] E       RecursionError: maximum recursion depth exceeded during ast construction
[   32s] 
[   32s] /usr/lib64/python3.12/ast.py:52: RecursionError

[alexmojaki]

The error is coming from Python itself trying to compile some source code, not asttokens. See if sys.setrecursionlimit(10000) makes the test pass. Otherwise you could catch the error and skip the test if the message is specifically maximum recursion depth exceeded during ast construction.

[cjwatson]

I also ran into this on Debian, and sys.setrecursionlimit(10000) seems to make no difference. However, the test passes on Python 3.14.2 while it fails on 3.13.11. I haven't attempted to bisect the fix.