"spawn" method doesn't work on 2.7 (under Linux)

[pitrou]

The following simple script:

from billiard import get_context

ctx = get_context('spawn')


def f(x):
    return x*x

if __name__ == '__main__':
    with ctx.Pool(2) as p:
        print(p.map(f, list(range(10))))

fails with:

Traceback (most recent call last):
  File "d.py", line 11, in <module>
    with ctx.Pool(2) as p:
  File "/home/antoine/billiard/billiard/context.py", line 165, in Pool
    context=self.get_context())
  File "/home/antoine/billiard/billiard/pool.py", line 1008, in __init__
    self._create_worker_process(i)
  File "/home/antoine/billiard/billiard/pool.py", line 1117, in _create_worker_process
    w.start()
  File "/home/antoine/billiard/billiard/process.py", line 122, in start
    self._popen = self._Popen(self)
  File "/home/antoine/billiard/billiard/context.py", line 341, in _Popen
    return Popen(process_obj)
  File "/home/antoine/billiard/billiard/popen_spawn_posix.py", line 39, in __init__
    super(Popen, self).__init__(process_obj)
  File "/home/antoine/billiard/billiard/popen_fork.py", line 24, in __init__
    self._launch(process_obj)
  File "/home/antoine/billiard/billiard/popen_spawn_posix.py", line 56, in _launch
    reduction.dump(process_obj, fp)
  File "/home/antoine/billiard/billiard/reduction.py", line 99, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "/home/antoine/miniconda3/envs/dask27/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/home/antoine/miniconda3/envs/dask27/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/antoine/miniconda3/envs/dask27/lib/python2.7/pickle.py", line 401, in save_reduce
    save(args)
  File "/home/antoine/miniconda3/envs/dask27/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/antoine/miniconda3/envs/dask27/lib/python2.7/pickle.py", line 568, in save_tuple
    save(element)
  File "/home/antoine/miniconda3/envs/dask27/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/antoine/miniconda3/envs/dask27/lib/python2.7/pickle.py", line 754, in save_global
    (obj, module, name))
pickle.PicklingError: Can't pickle <class 'billiard.pool.Process'>: it's not found as billiard.pool.Process

[auvipy]

can you verify it against new release?

[pitrou]

It still doesn't work. Now it doesn't raise anymore, but it hangs:

No handlers could be found for logger "multiprocessing"
No handlers could be found for logger "multiprocessing"
No handlers could be found for logger "multiprocessing"
No handlers could be found for logger "multiprocessing"
No handlers could be found for logger "multiprocessing"
No handlers could be found for logger "multiprocessing"
[ hangs here ]

[pitrou]

If I use faulthandler to set a process timeout, I see the following traces:

Timeout (0:00:04)!
Thread 0x00007f7a6274b700 <python> (most recent call first):
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/site-packages/billiard/connection.py", line 976 in _poll
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/site-packages/billiard/connection.py", line 998 in wait
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/site-packages/billiard/connection.py", line 465 in _poll
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/site-packages/billiard/connection.py", line 287 in poll
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/site-packages/billiard/pool.py", line 1326 in _poll_result
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/site-packages/billiard/pool.py", line 826 in _process_result
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/site-packages/billiard/pool.py", line 861 in body
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/site-packages/billiard/pool.py", line 478 in run
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x00007f7a62f4c700 <python> (most recent call first):
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/threading.py", line 340 in wait
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/Queue.py", line 168 in get
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/site-packages/billiard/pool.py", line 561 in body
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/site-packages/billiard/pool.py", line 478 in run
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x00007f7a6374d700 <python> (most recent call first):
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/site-packages/billiard/pool.py", line 538 in body
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/site-packages/billiard/pool.py", line 478 in run
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/threading.py", line 774 in __bootstrap

Current thread 0x00007f7a69124700 <python> (most recent call first):
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/threading.py", line 340 in wait
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/threading.py", line 614 in wait
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/site-packages/billiard/pool.py", line 1733 in wait
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/site-packages/billiard/pool.py", line 1736 in get
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/site-packages/billiard/pool.py", line 1372 in map
  File "b.py", line 17 in <module>

[auvipy]

do you have any suggested fix?

[pitrou]

None. I don't use billiard (I evaluated it for Dask/distributed and decided against it) and I don't have time to get acquainted with it.

That said, if the functionality doesn't work and doesn't stand a chance of being fixed, it would be reasonable to remove it so that people don't get lured :-)

[auvipy]

we have a plan to rewrite celeray billiard part with asyncio

[tonal]

see issue #241 and MR #245

[jakirkham]

Still failing it seems. Used the OPs script.

Traceback (most recent call last):
  File "test_fork_exec.py", line 10, in <module>
    with ctx.Pool(2) as p:
  File "/Users/kirkhamj/Developer/PyCharmCE/billiard/billiard/context.py", line 165, in Pool
    context=self.get_context())
  File "/Users/kirkhamj/Developer/PyCharmCE/billiard/billiard/pool.py", line 959, in __init__
    self._setup_queues()
  File "/Users/kirkhamj/Developer/PyCharmCE/billiard/billiard/pool.py", line 1328, in _setup_queues
    self._inqueue = self._ctx.SimpleQueue()
  File "/Users/kirkhamj/Developer/PyCharmCE/billiard/billiard/context.py", line 150, in SimpleQueue
    return SimpleQueue(ctx=self.get_context())
  File "/Users/kirkhamj/Developer/PyCharmCE/billiard/billiard/queues.py", line 377, in __init__
    self._rlock = ctx.Lock()
  File "/Users/kirkhamj/Developer/PyCharmCE/billiard/billiard/context.py", line 105, in Lock
    return Lock(ctx=self.get_context())
  File "/Users/kirkhamj/Developer/PyCharmCE/billiard/billiard/synchronize.py", line 182, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
  File "/Users/kirkhamj/Developer/PyCharmCE/billiard/billiard/synchronize.py", line 101, in __init__
    register(self._semlock.name)
  File "/Users/kirkhamj/Developer/PyCharmCE/billiard/billiard/semaphore_tracker.py", line 72, in register
    self._send('REGISTER', name)
  File "/Users/kirkhamj/Developer/PyCharmCE/billiard/billiard/semaphore_tracker.py", line 79, in _send
    self.ensure_running()
  File "/Users/kirkhamj/Developer/PyCharmCE/billiard/billiard/semaphore_tracker.py", line 61, in ensure_running
    spawnv_passfds(exe, args, fds_to_pass)
  File "/Users/kirkhamj/Developer/PyCharmCE/billiard/billiard/compat.py", line 217, in spawnv_passfds
    False, False, None)
TypeError: fork_exec() argument 4 must be tuple, not list

[tonal]

On https://github.com/celery/billiard/blob/master/billiard/compat.py#L217 only return pid
@jakirkham need use last source.

[tonal]

For latest master bf04adf all tests in 2.7 - Ok.

And start code sample work ok:

from __future__ import absolute_import
from billiard import get_context

ctx = get_context('spawn')

def f(x):
    return x*x

def test_spawn():
    with ctx.Pool(2) as p:
        print(p.map(f, list(range(10))))

if __name__ == '__main__':
    test_spawn()

[jakirkham]

Have tried with the Python 2.7 version and Billiard 3.5.0.4 on the OP's script and it fails as described in this comment. Had to use kill -s 9 <PID> to get rid of the processes. Nothing short of that worked.

[jakirkham]

On Python 3.6.5 and Billiard 3.5.0.4, I get a different error using the OPs script.

Pool process <billiard.pool.Worker object at 0x10e948320> error: AttributeError("Can't get attribute 'f' on <module '__main__' (built-in)>",)
Traceback (most recent call last):
  File "/zopt/conda3/envs/billiard3/lib/python3.6/site-packages/billiard/pool.py", line 289, in __call__
    sys.exit(self.workloop(pid=pid))
  File "/zopt/conda3/envs/billiard3/lib/python3.6/site-packages/billiard/pool.py", line 347, in workloop
    req = wait_for_job()
  File "/zopt/conda3/envs/billiard3/lib/python3.6/site-packages/billiard/pool.py", line 447, in receive
    ready, req = _receive(1.0)
  File "/zopt/conda3/envs/billiard3/lib/python3.6/site-packages/billiard/pool.py", line 419, in _recv
    return True, loads(get_payload())
  File "/zopt/conda3/envs/billiard3/lib/python3.6/site-packages/billiard/common.py", line 107, in pickle_loads
    return load(BytesIO(s))
AttributeError: Can't get attribute 'f' on <module '__main__' (built-in)>
Pool process <billiard.pool.Worker object at 0x100f2a390> error: AttributeError("Can't get attribute 'f' on <module '__main__' (built-in)>",)
Traceback (most recent call last):
  File "/zopt/conda3/envs/billiard3/lib/python3.6/site-packages/billiard/pool.py", line 289, in __call__
    sys.exit(self.workloop(pid=pid))
  File "/zopt/conda3/envs/billiard3/lib/python3.6/site-packages/billiard/pool.py", line 347, in workloop
    req = wait_for_job()
  File "/zopt/conda3/envs/billiard3/lib/python3.6/site-packages/billiard/pool.py", line 447, in receive
    ready, req = _receive(1.0)
  File "/zopt/conda3/envs/billiard3/lib/python3.6/site-packages/billiard/pool.py", line 419, in _recv
    return True, loads(get_payload())
  File "/zopt/conda3/envs/billiard3/lib/python3.6/site-packages/billiard/common.py", line 107, in pickle_loads
    return load(BytesIO(s))
AttributeError: Can't get attribute 'f' on <module '__main__' (built-in)>
Process 'SpawnPoolWorker-2' pid:48250 exited with 'exitcode 1'
Process 'SpawnPoolWorker-1' pid:48249 exited with 'exitcode 1'

It seems to create new spawn pool workers that fail the same way. This continues until the 5th worker is launched and hangs. Though it is at least responsive to ^c.