Cannot reproduce the result of ConvNeXt pretraining

[iShohei220]

🐛 Describe the bug

I tried to reproduce the result of ConvNeXt-Tiny's pretraining reported here using the official training recipe. However, the result was worse than the reported score (i.e., 82.52 % top-1 acc and 96.146 % top-5 acc). My result was 81.426 % at top-1 and 95.338 % at top-5. Is it just due to the choice of random seed? If so, please share it.

I also wonder why this training recipe requires so long time (i.e., 600 epochs) for training, while the official implementation of ConvNext can be trained in 300 epochs.

Versions

PyTorch version: 2.1.0a0+fe05266
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.24.1
Libc version: glibc-2.31

Python version: 3.8.10 (default, Mar 13 2023, 10:26:41) [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-4.18.0-193.el8.x86_64-x86_64-with-glibc2.29
Is CUDA available: True
CUDA runtime version: 12.1.66
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA A100-SXM4-40GB
Nvidia driver version: 525.105.17
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 52 bits physical, 57 bits virtual
CPU(s): 144
On-line CPU(s) list: 0-143
Thread(s) per core: 2
Core(s) per socket: 36
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 106
Model name: Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz
Stepping: 6
Frequency boost: enabled
CPU MHz: 3185.813
CPU max MHz: 2401.0000
CPU min MHz: 800.0000
BogoMIPS: 4800.00
Virtualization: VT-x
L1d cache: 3.4 MiB
L1i cache: 2.3 MiB
L2 cache: 90 MiB
L3 cache: 108 MiB
NUMA node0 CPU(s): 0-35,72-107
NUMA node1 CPU(s): 36-71,108-143
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced IBRS, IBPB conditional, RSB filling
Vulnerability Tsx async abort: Not affected
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 invpcid_single ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local wbnoinvd dtherm ida arat pln pts avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid md_clear pconfig flush_l1d arch_capabilities

Versions of relevant libraries:
[pip3] numpy==1.22.2
[pip3] pytorch-fid==0.2.1
[pip3] pytorch-quantization==2.1.2
[pip3] torch==2.1.0a0+fe05266
[pip3] torch-tensorrt==1.4.0.dev0
[pip3] torchtext==0.13.0a0+fae8e8c
[pip3] torchvision==0.15.0a0
[pip3] triton==2.0.0
[conda] Could not collect

[NicolasHug]

We don't set a seed when training, what we usually do is run a bunch of training jobs (with uncontrolled seeds) and we pick the median. It's possible that you got a bit unlucky with your run?

You'll find more details about the training in the original PR #5197

[iShohei220]

Thank you for your quick reply.

We don't set a seed when training, what we usually do is run a bunch of training jobs (with uncontrolled seeds) and we pick the median. It's possible that you got a bit unlucky with your run?

Okay, I will run some more trials, and report the results.

You'll find more details about the training in the original PR #5197

Thank you for your reference. I read the PR, but I am still not sure about why the training takes so long compared to ConvNeXt's official implementation. The committer (@datumbox) mentioned that he followed Torchvision's new recipe, but it seems that the new recipe is not used for the other models (e.g., ResNet) from what I have read in the README. Is there any special reason why only ConvNeXt requires longer training than the others?

[NicolasHug]

We [re]trained a lot of the models with the goal to push the accuracy further, and increasing the number of epochs was part of it.
It's not super obvious but a lot of the older models also use more epochs, it's not just ConvNeXt. The V2 version of resnets also used 600. The way we document the training recipes is a bit scattered, the easiest way is probably to look at the weight table in the docs https://pytorch.org/vision/main/models.html#table-of-all-available-classification-weights and click on the corresponding "recipe" link. For Resnet50 V2 you'll end up here

[iShohei220]

Thanks! I understand the background. I hope the README will be updated to the new recipe for all models, since the current documentation is a little confusing.

[NicolasHug]

That's fair @iShohei220 . Maybe we can add a very visible header at the top of those README to indicate the best way to look at the up-to-date recipes. Would you like to submit a PR for that?

[iShohei220]

Would you like to submit a PR for that?

Sure! I will submit it within a week.

[iShohei220]

Additional report

I've run 3 training jobs, but the results were still worse than the reported accuracy. See the W&B report for more details.

It seems that the models consistently start to overfit in the final 150 epochs. The best accuracy at around the 450th epoch is close to the reported value. If you picked it up as the pretrained model, it makes sense, but it sounds a little tricky.

I think it would be better to strengthen regularization (e.g., increase the weight decay) to prevent the over-fitting.

[iShohei220]

I hope someone else will reproduce the result and confirm that this phenomenon (i.e., the accuracy degradation and over-fitting) is not specific to my environmet.