[QUESTION] In what cases will tp_comm_overlap accelerate training

[Cccei000]

Using H20*8 to finetune Qwen3-32B with tp8 and 16384 packed_seq_len (with padding for fixed seq length as required by tp_comm_overlap userbuffer). The global batchsize is 16.

With tp_comm_overlap, it only speeds up by less than 2s/step from 43s/step, which is minor.

Are there any guides on for what model size, tp size and seq length will tp_comm_overlap be beneficial?

[CodersAcademy006]

This behavior appears to be expected rather than a bug. tp_comm_overlap is most effective when tensor-parallel communication can be hidden behind sufficient per-layer compute, which typically requires larger TP sizes, larger micro-batches, or more MLP-dominated workloads. In this setup (TP=8, micro-batch size=2, long-sequence attention-heavy training on H20 with strong interconnect), per-layer compute is limited and attention dominates, leaving little communication to overlap, so only marginal speedup is achievable. Documenting these expectations may help users better judge when enabling tp_comm_overlap is beneficial.

By doing this you will surely resolve the issue raised above, if any issue persists, please fell free to reach out.

[Cccei000]

This behavior appears to be expected rather than a bug. tp_comm_overlap is most effective when tensor-parallel communication can be hidden behind sufficient per-layer compute, which typically requires larger TP sizes, larger micro-batches, or more MLP-dominated workloads. In this setup (TP=8, micro-batch size=2, long-sequence attention-heavy training on H20 with strong interconnect), per-layer compute is limited and attention dominates, leaving little communication to overlap, so only marginal speedup is achievable. Documenting these expectations may help users better judge when enabling tp_comm_overlap is beneficial.

By doing this you will surely resolve the issue raised above, if any issue persists, please fell free to reach out.

Thank you for your advice.

I understand that tp_comm_overlap will be more effective with MLP-dominated workloads, but in my belief, packing short sequences to long length is not attention-heavy with the help of flash-attn varlen methods.

Anyway, I test on the fixed TP8, larger micro-bsz, shorter single seq and true batching, and here are the results:

mbsz8 gbsz16 seqlen4096 tp8 packing : 19310.1 ms/step
mbsz8 gbsz16 seqlen4096 tp8 packing overlap : 19147.8 ms/step
mbsz16 gbsz32 seqlen2048 tp8 padding : 18991.5 ms/step
mbsz16 gbsz32 seqlen2048 tp8 padding overlap : 18385.1 ms/step

The improvement becomes even more minor. If possible, could you give me an example on the model and the training config mentioned above where you validate tp_comm_overlap?

[CodersAcademy006]

The issue likely stems from the H20's unique hardware profile: it has massive compute power (like an H100) but significantly lower interconnect bandwidth. When running a 32B model on TP=8, the local matrix operations are too small; the GPUs finish the math instantly but get stuck waiting on the slower network, meaning there is no "compute time" long enough to hide the "communication time."

To get better performance on H20s, you effectively need to give each GPU more work to balance out the slower interconnect. I recommend reducing your Tensor Parallelism to TP=2 or TP=4 (handling the rest of the memory with ZeRO/Data Parallelism). This increases the matrix size per GPU, allowing the computation to run longer and better match the communication speed, which should result in higher overall throughput.

If any issue persist, please feel free to reach out.