-
Notifications
You must be signed in to change notification settings - Fork 3.6k
Closed
Labels
bugSomething isn't workingSomething isn't workingpriority: 0High priority taskHigh priority taskstrategy: ddpDistributedDataParallelDistributedDataParallel
Milestone
Description
🐛 Bug
Since #13744 , the ddp_fork
and ddp_spawn
strategies have find_unused_parameters
set to False
, making them equivalent to ddp_fork_find_unused_parameters_false
and ddp_spawn_find_unused_parameters_false
. This is caused by a mistaken simplification of register_strategies
that removed the distinction between them. Besides being an API break, this can trigger a runtime error when training any model with unused parameters using these strategies
To Reproduce
https://colab.research.google.com/drive/1xxoSnEDc8s6z-j66o1MbtEjVwGJ7K43p
Environment
- CUDA:
- GPU:
- Tesla T4
- available: True
- version: 11.3
- GPU:
- Packages:
- lightning: None
- lightning_app: None
- numpy: 1.21.6
- pyTorch_debug: False
- pyTorch_version: 1.12.0+cu113
- pytorch-lightning: 1.7.0
- tqdm: 4.64.0
- System:
- OS: Linux
- architecture:
- 64bit
- processor: x86_64
- python: 3.7.13
- version: 1 SMP Sun Apr 24 10:03:06 PDT 2022
Please tell me if I can just submit a PR (I can probably have it ready today) or if you'd rather have a go at this yourselves.
cc @tchaton @rohitgr7 @justusschock @kaushikb11 @awaelchli @akihironitta
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't workingpriority: 0High priority taskHigh priority taskstrategy: ddpDistributedDataParallelDistributedDataParallel