Compatibility with Python 3.10

[frenzymadness]

It seems to me that cloudpickle is not compatible with the latest Python 3.10a5. A short reproducer:

>>> import pickle
>>> import cloudpickle
>>> def g(s):
...     return len(s)
... 
>>> d = cloudpickle.dumps(g)
>>> a = pickle.loads(d)
>>> a("aaaa")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line -3, in g
NameError: name 'len' is not defined

According to the following results from tests, there is a problem with builtins:

======================================================================
ERROR: test_dynamic_module_with_unpicklable_builtin (tests.cloudpickle_test.CloudPickleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 605, in test_dynamic_module_with_unpicklable_builtin
    assert depickled_mod.f(-1) == 1
  File "<string>", line 1, in <lambda>
NameError: name 'abs' is not defined

======================================================================
ERROR: test_extended_arg (tests.cloudpickle_test.CloudPickleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 971, in test_extended_arg
    self.assertEqual(f2(), res)
  File "<string>", line -1, in f
NameError: name 'bytes' is not defined

======================================================================
ERROR: test_generator (tests.cloudpickle_test.CloudPickleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 452, in test_generator
    assert list(gen2(3)) == list(range(3))
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 447, in some_generator
    yield i
NameError: name 'range' is not defined

======================================================================
ERROR: test_generic_extensions_final (tests.cloudpickle_test.CloudPickleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 2246, in test_generic_extensions_final
    typing_extensions = pytest.importorskip('typing_extensions')
  File "/home/lbalhar/.virtualenvs/cloudpickle/lib/python3.10/site-packages/_pytest/outcomes.py", line 212, in importorskip
    raise Skipped(reason, allow_module_level=True) from None
Skipped: could not import 'typing_extensions': No module named 'typing_extensions'

======================================================================
ERROR: test_generic_extensions_literal (tests.cloudpickle_test.CloudPickleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 2229, in test_generic_extensions_literal
    typing_extensions = pytest.importorskip('typing_extensions')
  File "/home/lbalhar/.virtualenvs/cloudpickle/lib/python3.10/site-packages/_pytest/outcomes.py", line 212, in importorskip
    raise Skipped(reason, allow_module_level=True) from None
Skipped: could not import 'typing_extensions': No module named 'typing_extensions'

======================================================================
ERROR: test_generic_type (tests.cloudpickle_test.CloudPickleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 2198, in test_generic_type
    assert worker.run(check_generic, C[int], C, int, use_args) == "ok"
  File "/home/lbalhar/Software/cloudpickle/tests/testutils.py", line 134, in run
    raise result
NameError: name 'len' is not defined

======================================================================
ERROR: test_interactive_dynamic_type_and_stored_remote_instances (tests.cloudpickle_test.CloudPickleTest)
Simulate objects stored on workers to check isinstance semantics
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/testutils.py", line 184, in assert_run_python_script
    out = check_output(cmd, **kwargs)
  File "/usr/lib64/python3.10/subprocess.py", line 429, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib64/python3.10/subprocess.py", line 533, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/lbalhar/.virtualenvs/cloudpickle/bin/python', '-W ignore', '/tmp/tmprzcwymxq_src_test_cloudpickle.py']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 1782, in test_interactive_dynamic_type_and_stored_remote_instances
    assert_run_python_script(code)
  File "/home/lbalhar/Software/cloudpickle/tests/testutils.py", line 186, in assert_run_python_script
    raise RuntimeError(u"script errored with output:\n%s"
RuntimeError: script errored with output:
Traceback (most recent call last):
  File "/tmp/tmprzcwymxq_src_test_cloudpickle.py", line 22, in <module>
    id1 = w.run(store, A())
  File "/home/lbalhar/Software/cloudpickle/tests/testutils.py", line 134, in run
    raise result
NameError: name 'getattr' is not defined


======================================================================
ERROR: test_interactive_remote_function_calls_no_memory_leak (tests.cloudpickle_test.CloudPickleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/testutils.py", line 184, in assert_run_python_script
    out = check_output(cmd, **kwargs)
  File "/usr/lib64/python3.10/subprocess.py", line 429, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib64/python3.10/subprocess.py", line 533, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/lbalhar/.virtualenvs/cloudpickle/bin/python', '-W ignore', '/tmp/tmpuwo45i9h_src_test_cloudpickle.py']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 1830, in test_interactive_remote_function_calls_no_memory_leak
    assert_run_python_script(code)
  File "/home/lbalhar/Software/cloudpickle/tests/testutils.py", line 186, in assert_run_python_script
    raise RuntimeError(u"script errored with output:\n%s"
RuntimeError: script errored with output:
Traceback (most recent call last):
  File "/tmp/tmpuwo45i9h_src_test_cloudpickle.py", line 21, in <module>
    result = w.run(func)
  File "/home/lbalhar/Software/cloudpickle/tests/testutils.py", line 134, in run
    raise result
NameError: name 'len' is not defined


======================================================================
ERROR: test_interactive_remote_function_calls_no_side_effect (tests.cloudpickle_test.CloudPickleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/testutils.py", line 184, in assert_run_python_script
    out = check_output(cmd, **kwargs)
  File "/usr/lib64/python3.10/subprocess.py", line 429, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib64/python3.10/subprocess.py", line 533, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/lbalhar/.virtualenvs/cloudpickle/bin/python', '-W ignore', '/tmp/tmppmuuelfh_src_test_cloudpickle.py']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 1665, in test_interactive_remote_function_calls_no_side_effect
    assert_run_python_script(code)
  File "/home/lbalhar/Software/cloudpickle/tests/testutils.py", line 186, in assert_run_python_script
    raise RuntimeError(u"script errored with output:\n%s"
RuntimeError: script errored with output:
Traceback (most recent call last):
  File "/tmp/tmppmuuelfh_src_test_cloudpickle.py", line 36, in <module>
    assert not w.run(is_in_main, "CustomClass")
  File "/home/lbalhar/Software/cloudpickle/tests/testutils.py", line 134, in run
    raise result
NameError: name 'hasattr' is not defined


======================================================================
ERROR: test_locally_defined_function_and_class (tests.cloudpickle_test.CloudPickleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 367, in test_locally_defined_function_and_class
    self.assertEqual(pickle_depickle(some_function, protocol=self.protocol)(41, 1), 1)
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 364, in some_function
    return (x + y) / LOCAL_CONSTANT
NameError: name 'sum' is not defined

======================================================================
ERROR: test_module_locals_behavior (tests.cloudpickle_test.CloudPickleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/testutils.py", line 184, in assert_run_python_script
    out = check_output(cmd, **kwargs)
  File "/usr/lib64/python3.10/subprocess.py", line 429, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib64/python3.10/subprocess.py", line 533, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/lbalhar/.virtualenvs/cloudpickle/bin/python', '-W ignore', '/tmp/tmptogkierp_src_test_cloudpickle.py']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 562, in test_module_locals_behavior
    assert_run_python_script(textwrap.dedent(child_process_script))
  File "/home/lbalhar/Software/cloudpickle/tests/testutils.py", line 186, in assert_run_python_script
    raise RuntimeError(u"script errored with output:\n%s"
RuntimeError: script errored with output:
Traceback (most recent call last):
  File "/tmp/tmptogkierp_src_test_cloudpickle.py", line 7, in <module>
    assert func(range(10)) == 45
  File "/home/lbalhar/Software/cloudpickle/tests/testutils.py", line 25, in g
    return sum(range(10))
NameError: name 'sum' is not defined


======================================================================
ERROR: test_out_of_band_buffers (tests.cloudpickle_test.CloudPickleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 2113, in test_out_of_band_buffers
    np = pytest.importorskip("numpy")
  File "/home/lbalhar/.virtualenvs/cloudpickle/lib/python3.10/site-packages/_pytest/outcomes.py", line 212, in importorskip
    raise Skipped(reason, allow_module_level=True) from None
Skipped: could not import 'numpy': No module named 'numpy'

======================================================================
ERROR: test_relative_import_inside_function (tests.cloudpickle_test.CloudPickleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 2037, in test_relative_import_inside_function
    assert cloned_func() == "hello from a {}!".format(source)
  File "/home/lbalhar/.virtualenvs/cloudpickle/lib/python3.10/site-packages/_cloudpickle_testpkg/__init__.py", line 26, in f
    return module_function()
ImportError: __import__ not found

======================================================================
ERROR: test_tornado_coroutine (tests.cloudpickle_test.CloudPickleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 944, in test_tornado_coroutine
    res = loop.run_sync(functools.partial(g2, 5))
  File "/home/lbalhar/.virtualenvs/cloudpickle/lib64/python3.10/site-packages/tornado/ioloop.py", line 530, in run_sync
    return future_cell[0].result()
  File "/home/lbalhar/.virtualenvs/cloudpickle/lib64/python3.10/site-packages/tornado/ioloop.py", line 492, in run
    result = func()
  File "/home/lbalhar/.virtualenvs/cloudpickle/lib64/python3.10/site-packages/tornado/gen.py", line 233, in wrapper
    try:
NameError: name 'isinstance' is not defined

======================================================================
ERROR: test_unhashable_closure (tests.cloudpickle_test.CloudPickleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 311, in test_unhashable_closure
    self.assertEqual(g(), 2)
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 372, in g
    class SomeClass(object):
NameError: name 'len' is not defined

======================================================================
ERROR: test_dynamic_module_with_unpicklable_builtin (tests.cloudpickle_test.Protocol2CloudPickleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 605, in test_dynamic_module_with_unpicklable_builtin
    assert depickled_mod.f(-1) == 1
  File "<string>", line 1, in <lambda>
NameError: name 'abs' is not defined

======================================================================
ERROR: test_extended_arg (tests.cloudpickle_test.Protocol2CloudPickleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 971, in test_extended_arg
    self.assertEqual(f2(), res)
  File "<string>", line -1, in f
NameError: name 'bytes' is not defined

======================================================================
ERROR: test_generator (tests.cloudpickle_test.Protocol2CloudPickleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 452, in test_generator
    assert list(gen2(3)) == list(range(3))
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 447, in some_generator
    yield i
NameError: name 'range' is not defined

======================================================================
ERROR: test_generic_extensions_final (tests.cloudpickle_test.Protocol2CloudPickleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 2246, in test_generic_extensions_final
    typing_extensions = pytest.importorskip('typing_extensions')
  File "/home/lbalhar/.virtualenvs/cloudpickle/lib/python3.10/site-packages/_pytest/outcomes.py", line 212, in importorskip
    raise Skipped(reason, allow_module_level=True) from None
Skipped: could not import 'typing_extensions': No module named 'typing_extensions'

======================================================================
ERROR: test_generic_extensions_literal (tests.cloudpickle_test.Protocol2CloudPickleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 2229, in test_generic_extensions_literal
    typing_extensions = pytest.importorskip('typing_extensions')
  File "/home/lbalhar/.virtualenvs/cloudpickle/lib/python3.10/site-packages/_pytest/outcomes.py", line 212, in importorskip
    raise Skipped(reason, allow_module_level=True) from None
Skipped: could not import 'typing_extensions': No module named 'typing_extensions'

======================================================================
ERROR: test_generic_type (tests.cloudpickle_test.Protocol2CloudPickleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 2198, in test_generic_type
    assert worker.run(check_generic, C[int], C, int, use_args) == "ok"
  File "/home/lbalhar/Software/cloudpickle/tests/testutils.py", line 134, in run
    raise result
NameError: name 'len' is not defined

======================================================================
ERROR: test_interactive_dynamic_type_and_stored_remote_instances (tests.cloudpickle_test.Protocol2CloudPickleTest)
Simulate objects stored on workers to check isinstance semantics
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/testutils.py", line 184, in assert_run_python_script
    out = check_output(cmd, **kwargs)
  File "/usr/lib64/python3.10/subprocess.py", line 429, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib64/python3.10/subprocess.py", line 533, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/lbalhar/.virtualenvs/cloudpickle/bin/python', '-W ignore', '/tmp/tmpvkkfbn40_src_test_cloudpickle.py']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 1782, in test_interactive_dynamic_type_and_stored_remote_instances
    assert_run_python_script(code)
  File "/home/lbalhar/Software/cloudpickle/tests/testutils.py", line 186, in assert_run_python_script
    raise RuntimeError(u"script errored with output:\n%s"
RuntimeError: script errored with output:
Traceback (most recent call last):
  File "/tmp/tmpvkkfbn40_src_test_cloudpickle.py", line 22, in <module>
    id1 = w.run(store, A())
  File "/home/lbalhar/Software/cloudpickle/tests/testutils.py", line 134, in run
    raise result
NameError: name 'getattr' is not defined


======================================================================
ERROR: test_interactive_remote_function_calls_no_memory_leak (tests.cloudpickle_test.Protocol2CloudPickleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/testutils.py", line 184, in assert_run_python_script
    out = check_output(cmd, **kwargs)
  File "/usr/lib64/python3.10/subprocess.py", line 429, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib64/python3.10/subprocess.py", line 533, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/lbalhar/.virtualenvs/cloudpickle/bin/python', '-W ignore', '/tmp/tmpsnokol80_src_test_cloudpickle.py']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 1830, in test_interactive_remote_function_calls_no_memory_leak
    assert_run_python_script(code)
  File "/home/lbalhar/Software/cloudpickle/tests/testutils.py", line 186, in assert_run_python_script
    raise RuntimeError(u"script errored with output:\n%s"
RuntimeError: script errored with output:
Traceback (most recent call last):
  File "/tmp/tmpsnokol80_src_test_cloudpickle.py", line 21, in <module>
    result = w.run(func)
  File "/home/lbalhar/Software/cloudpickle/tests/testutils.py", line 134, in run
    raise result
NameError: name 'len' is not defined


======================================================================
ERROR: test_interactive_remote_function_calls_no_side_effect (tests.cloudpickle_test.Protocol2CloudPickleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/testutils.py", line 184, in assert_run_python_script
    out = check_output(cmd, **kwargs)
  File "/usr/lib64/python3.10/subprocess.py", line 429, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib64/python3.10/subprocess.py", line 533, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/lbalhar/.virtualenvs/cloudpickle/bin/python', '-W ignore', '/tmp/tmp8uurspoa_src_test_cloudpickle.py']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 1665, in test_interactive_remote_function_calls_no_side_effect
    assert_run_python_script(code)
  File "/home/lbalhar/Software/cloudpickle/tests/testutils.py", line 186, in assert_run_python_script
    raise RuntimeError(u"script errored with output:\n%s"
RuntimeError: script errored with output:
Traceback (most recent call last):
  File "/tmp/tmp8uurspoa_src_test_cloudpickle.py", line 36, in <module>
    assert not w.run(is_in_main, "CustomClass")
  File "/home/lbalhar/Software/cloudpickle/tests/testutils.py", line 134, in run
    raise result
NameError: name 'hasattr' is not defined


======================================================================
ERROR: test_locally_defined_function_and_class (tests.cloudpickle_test.Protocol2CloudPickleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 367, in test_locally_defined_function_and_class
    self.assertEqual(pickle_depickle(some_function, protocol=self.protocol)(41, 1), 1)
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 364, in some_function
    return (x + y) / LOCAL_CONSTANT
NameError: name 'sum' is not defined

======================================================================
ERROR: test_module_locals_behavior (tests.cloudpickle_test.Protocol2CloudPickleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/testutils.py", line 184, in assert_run_python_script
    out = check_output(cmd, **kwargs)
  File "/usr/lib64/python3.10/subprocess.py", line 429, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib64/python3.10/subprocess.py", line 533, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/lbalhar/.virtualenvs/cloudpickle/bin/python', '-W ignore', '/tmp/tmpa3fxyo8c_src_test_cloudpickle.py']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 562, in test_module_locals_behavior
    assert_run_python_script(textwrap.dedent(child_process_script))
  File "/home/lbalhar/Software/cloudpickle/tests/testutils.py", line 186, in assert_run_python_script
    raise RuntimeError(u"script errored with output:\n%s"
RuntimeError: script errored with output:
Traceback (most recent call last):
  File "/tmp/tmpa3fxyo8c_src_test_cloudpickle.py", line 7, in <module>
    assert func(range(10)) == 45
  File "/home/lbalhar/Software/cloudpickle/tests/testutils.py", line -1, in g
NameError: name 'sum' is not defined


======================================================================
ERROR: test_out_of_band_buffers (tests.cloudpickle_test.Protocol2CloudPickleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 2112, in test_out_of_band_buffers
    pytest.skip("Need Pickle Protocol 5 or later")
  File "/home/lbalhar/.virtualenvs/cloudpickle/lib/python3.10/site-packages/_pytest/outcomes.py", line 139, in skip
    raise Skipped(msg=msg, allow_module_level=allow_module_level)
Skipped: Need Pickle Protocol 5 or later

======================================================================
ERROR: test_relative_import_inside_function (tests.cloudpickle_test.Protocol2CloudPickleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 2037, in test_relative_import_inside_function
    assert cloned_func() == "hello from a {}!".format(source)
  File "/home/lbalhar/.virtualenvs/cloudpickle/lib/python3.10/site-packages/_cloudpickle_testpkg/__init__.py", line 26, in f
    return module_function()
ImportError: __import__ not found

======================================================================
ERROR: test_tornado_coroutine (tests.cloudpickle_test.Protocol2CloudPickleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 944, in test_tornado_coroutine
    res = loop.run_sync(functools.partial(g2, 5))
  File "/home/lbalhar/.virtualenvs/cloudpickle/lib64/python3.10/site-packages/tornado/ioloop.py", line 530, in run_sync
    return future_cell[0].result()
  File "/home/lbalhar/.virtualenvs/cloudpickle/lib64/python3.10/site-packages/tornado/ioloop.py", line 492, in run
    result = func()
  File "/home/lbalhar/.virtualenvs/cloudpickle/lib64/python3.10/site-packages/tornado/gen.py", line 233, in wrapper
    try:
NameError: name 'isinstance' is not defined

======================================================================
ERROR: test_unhashable_closure (tests.cloudpickle_test.Protocol2CloudPickleTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 311, in test_unhashable_closure
    self.assertEqual(g(), 2)
  File "/home/lbalhar/Software/cloudpickle/tests/cloudpickle_test.py", line 306, in g
    return len(s)
NameError: name 'len' is not defined

----------------------------------------------------------------------
Ran 219 tests in 6.720s

FAILED (errors=30)
Test failed: <unittest.runner.TextTestResult run=219 errors=30 failures=0>
error: Test failed: <unittest.runner.TextTestResult run=219 errors=30 failures=0>

Some incompatibilities are already fixed in: #400

It would be nice to add Python 3.9 and 3.10 to CI.

[frenzymadness]

I'm not able to find the root cause.

I have two functions:

>>> import cloudpickle as cp
>>> def g():
...     return len("aaa")
... 
>>> a = cp.loads(cp.dumps(g))
>>> g()
3
>>> a()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 17, in g
NameError: name 'len' is not defined

There is a difference in theirs __globals__ but the difference seems to be the same they have in Python 3.9:

>>> g.__globals__["__builtins__"]
<module 'builtins' (built-in)>
>>> a.__globals__["__builtins__"]
{'__name__': 'builtins', '__doc__': "Built-in functions, exceptions, and other objects.\n\nNoteworthy: None is the `nil' object; Ellipsis represents `...' in slices.", '__package__': '', '__loader__': <class '_frozen_importlib.BuiltinImporter'> …

Changing __globals__["__builtins__"] for both functions makes no difference:

>>> import builtins
>>> g.__globals__["__builtins__"] is builtins
True
>>> a.__globals__["__builtins__"] = builtins
>>> a()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in g
NameError: name 'len' is not defined

>>> g.__globals__["__builtins__"] = None
>>> g()
3

@encukou or @vstinner might have a clue where to look next.

[vstinner]

It's a Python regression, I created https://bugs.python.org/issue43228

[vstinner]

The problem is that cloudpickle calls LambaType to build a function with a globals dictionary which doesn't contain a __builtins__ key. In that case, Python 3.10 sets the new internal function builtins dictionary to {'None': None}.

cloudpickle should try to call LambdaType with a globals dictionary which contains a __builtins__ key.

Currently, builtins are only overriden after the function is created, which is too late. In Python 3.10, it's currently not possible to modify the internal function builtins dictionary. But I proposed python/cpython#24559 to let cloudpickle also update this dictionary in its cloudpickle.cloudpicke_fast._function_setstate() function.

[pierreglaser]

Thanks for helping with this issue @vstinner.
I just ran the test suite of cloudpickle using the latest Python master, and the vast majority of the tests are passing (cc @frenzymadness).

If I understood correctly, following python/cpython#24559, when a function is created using types.FunctionType, the function's __builtins__ attribute defaults to some current default set of builtins if the globals argument of the function's constructor does not contain a __builtin__ key (which always is the case in cloudpickle). Moreover, this function's __builtins__ attribute becomes read-only after the said function is created. Thus, the last line of this part of _function_setstate (which changes obj.__globals__['__builtins__']) becomes obsolete. This should be made clear in a comment, or even branched depending on the Python version.

cloudpickle/cloudpickle/cloudpickle_fast.py

Lines 428 to 448 in cdc704d

	def _function_setstate(obj, state):
	"""Update the state of a dynaamic function.
	As __closure__ and __globals__ are readonly attributes of a function, we
	cannot rely on the native setstate routine of pickle.load_build, that calls
	setattr on items of the slotstate. Instead, we have to modify them inplace.
	"""
	state, slotstate = state
	obj.__dict__.update(state)
	obj_globals = slotstate.pop("__globals__")
	obj_closure = slotstate.pop("__closure__")
	# _cloudpickle_subimports is a set of submodules that must be loaded for
	# the pickled function to work correctly at unpickling time. Now that these
	# submodules are depickled (hence imported), they can be removed from the
	# object's state (the object state only served as a reference holder to
	# these submodules)
	slotstate.pop("_cloudpickle_submodules")
	obj.__globals__.update(obj_globals)
	obj.__globals__["__builtins__"] = __builtins__

[vstinner]

I just merged another PR which should fix cloudpickle (but I didn't test yet):

• bpo-42990: Functions inherit current builtins python/cpython#24564
• python/cpython@46496f9

I now plan to add an optional builtins parameter to FunctionType.

[pierreglaser]

python/cpython@46496f9

This is what my previous comment was referring to. cloudpickle mostly works right now on Python master (thus, including python/cpython@46496f9)

I now plan to add an optional builtins parameter to FunctionType.

This will be necessary to let users of cloudpickle hack the __builtins__ of dynamically defined functions (wonder when it is useful...), but cloudpickle does not support that use case explicitly right now, as it does not pickle any __builtins__, and just uses the __builtins__ of the existing environment at unpickling time.

[vstinner]

This issue was a regression of Python 3.10: I fixed it in Python 3.10.0 alpha 6. This cloudpickle issue can now be closed.

• https://bugs.python.org/issue42990
• https://bugs.python.org/issue43228
• python/cpython@46496f9

[jrbourbeau]

Just following up here, based on @vstinner's latest comment (xref #410 (comment)) and the Python 3.10-dev CI build passing on the main branch, it seems like this issue can be closed. cc @pierreglaser

[pierreglaser]

Thanks @jrbourbeau!

[vstinner]

Just following up here, based on @vstinner's latest comment (xref #410 (comment)) and the Python 3.10-dev CI build passing on the main branch, it seems like this issue can be closed. cc @pierreglaser

Thanks for the confirmation!