pdb & multiprocessing.Pool: AttributeError: module '__main__' has no attribute '__spec__'

[macfreek]

BPO	42949
Nosy	@macfreek

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-01-17.22:10:12.759>
labels = ['3.8', 'library', '3.9', '3.10']
title = "pdb & multiprocessing.Pool: AttributeError: module '__main__' has no attribute '__spec__'"
updated_at = <Date 2021-01-17.22:10:12.759>
user = 'https://github.com/macfreek'

bugs.python.org fields:

activity = <Date 2021-01-17.22:10:12.759>
actor = 'macfreek'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2021-01-17.22:10:12.759>
creator = 'macfreek'
dependencies = []
files = []
hgrepos = []
issue_num = 42949
keywords = []
message_count = 1.0
messages = ['385168']
nosy_count = 1.0
nosy_names = ['macfreek']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue42949'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

Linked PRs

• gh-87115: catch exception #106501
• gh-87115: Set __main__.__spec__ to None in pdb #116141
• [3.12] gh-87115: Set __main__.__spec__ to None in pdb (GH-116141) #116154
• [3.11] gh-87115: Set __main__.__spec__ to None in pdb (GH-116141) #116155

[macfreek]

Summary:

multiprocessing.Pool contains a bug when the script is invoked with pdb.

Steps to reproduce:

Consider the following script:

from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == '__main__':
    with Pool(5) as p:
        print(p.map(f, [1, 2, 3]))

When called as python3.10 test.py, this works fine, and returns [1, 4, 9]

When called as python3.10 -m pdb test.py, it gives the following exception:

freek@minnie ~ » python3.9 -m pdb test.py
> /Users/freek/test.py(3)<module>()
-> from multiprocessing import Pool
(Pdb) c
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pdb.py", line 1704, in main
    pdb._runscript(mainpyfile)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pdb.py", line 1573, in _runscript
    self.run(statement)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/bdb.py", line 580, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "/Users/freek/test.py", line 3, in <module>
    from multiprocessing import Pool
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 326, in _repopulate_pool_static
    w.start()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 42, in _launch
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 183, in get_preparation_data
    main_mod_name = getattr(main_module.__spec__, "name", None)
AttributeError: module '__main__' has no attribute '__spec__'

I can reproduce this in Python 3.8.7, 3.9.1, 3.10.0a3. It works fine in Python 3.7.9.

A workaround is to define __spec__ in main:

from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == '__main__':
    __spec__ = None
    with Pool(5) as p:
        print(p.map(f, [1, 2, 3]))

I'd say that multiprocessing/spawn.py", line 183, in get_preparation_data is flawed
main_mod_name = getattr(main_module.__spec__, "name", None)

[qwertyuu]

This is very frustrating. Having the same issue