torchvision sets multiprocessing start_method

[ssnl]

➜  test git:(stft_fft) ✗ python
Python 3.6.5 |Anaconda, Inc.| (default, Apr 29 2018, 16:14:56)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import multiprocessing as mp
>>> mp.set_start_method('spawn')
>>>
➜  test git:(stft_fft) ✗ python
Python 3.6.5 |Anaconda, Inc.| (default, Apr 29 2018, 16:14:56)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torchvision
>>> import multiprocessing as mp
>>> mp.set_start_method('spawn')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ssnl/miniconda3/lib/python3.6/multiprocessing/context.py", line 242, in set_start_method
    raise RuntimeError('context has already been set')
RuntimeError: context has already been set

I used builtin mp. Same thing happens for torch.mp.

[fmassa]

This is very weird!

I did some debugging, and this is actually a problem with tqdm

from tqdm import tqdm
import multiprocessing as mp
mp.set_start_method('spawn')

raises

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-3-36f1a10db7c0> in <module>()
----> 1 mp.set_start_method('spawn')

~/.conda/envs/detectron_v2/lib/python3.6/multiprocessing/context.py in set_start_method(self, method, force)
    240     def set_start_method(self, method, force=False):
    241         if self._actual_context is not None and not force:
--> 242             raise RuntimeError('context has already been set')
    243         if method is None and force:
    244             self._actual_context = None

RuntimeError: context has already been set

If this is a blocker we might want to consider removing tqdm from torchvision

[ssnl]

Actually joblib and sklearn also sets the start method so this is probably happening with a lot of libs. Users can use mp.get_context so it shouldn’t be blocking. But it is annoying indeed.

…

On Fri, Jul 13, 2018 at 19:41 Francisco Massa ***@***.***> wrote: This is very weird! I did some debugging, and this is actually a problem with tqdm from tqdm import tqdmimport multiprocessing as mp mp.set_start_method('spawn') raises --------------------------------------------------------------------------- RuntimeError Traceback (most recent call last) <ipython-input-3-36f1a10db7c0> in <module>() ----> 1 mp.set_start_method('spawn') ~/.conda/envs/detectron_v2/lib/python3.6/multiprocessing/context.py in set_start_method(self, method, force) 240 def set_start_method(self, method, force=False): 241 if self._actual_context is not None and not force: --> 242 raise RuntimeError('context has already been set') 243 if method is None and force: 244 self._actual_context = None RuntimeError: context has already been set If this is a blocker we might want to consider removing tqdm from torchvision — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#544 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFaWZX0m8Gk_dJGI3xlxhrFu8Qn0scHFks5uGNu9gaJpZM4VLNJb> .

[fmassa]

This has been fixed since version 4.29.0 of tqdm https://github.com/tqdm/tqdm/releases

[wj1017090777]

After the update, I still have this problem

[fmassa]

As Simon mentioned above, many libraries set the multiprocessing start method.
torchvision doesn't directly use any of those, and if you are facing this problem it might be good to isolate which library is doing this in torchvision.