Skip to content

Add warning for PyTorch data loader num_workers flag. #726

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Mar 29, 2024
Merged

Add warning for PyTorch data loader num_workers flag. #726

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
135c56a
add warning to flag
priyakasimbeg Mar 26, 2024
e29a2d4
Merge branch 'dev' into num_workers_fix
priyakasimbeg Mar 29, 2024
28adc86
update warning
priyakasimbeg Mar 29, 2024
d492d69
add warning about num_workers
priyakasimbeg Mar 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 12 additions & 3 deletions submission_runner.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -154,9 +154,12 @@
flags.DEFINE_boolean('set_pytorch_max_split_size',
False,
'If true, set pytorch max_split_size_mb to 256')
flags.DEFINE_integer('pytorch_eval_num_workers',
0,
'Number of workers for PyTorch evaluation data loaders.')
flags.DEFINE_integer(
'pytorch_eval_num_workers',
0,
'Number of workers for ImageNet PyTorch evaluation data loaders.'
'WARNING: Setting pytorch_eval_num_workers != 0, will result '
'in incorrect evals currently, see issues/732.')
FLAGS = flags.FLAGS
USE_PYTORCH_DDP, RANK, DEVICE, N_GPUS = pytorch_setup()

Expand Down Expand Up @@ -634,6 +637,12 @@ def main(_):
if FLAGS.framework == 'pytorch':
pytorch_init(USE_PYTORCH_DDP, RANK, profiler)

# TODO: remove once issue resolved.
if FLAGS.pytorch_eval_num_workers != 0:
logging.warning(
'WARNING: Setting pytorch_eval_num_workers != 0, will result '
'in incorrect evals currently, see issues/732.')

workload_metadata = WORKLOADS[FLAGS.workload]

# Prevent OOM on librispeech conformer.
Expand Down