gradient_accumulation_steps influences scale of the loss

[RuABraun]

🐛 Bug description

Because of the division here, and similar functions, the loss has the wrong scale when using gradient_accumulation_steps. This makes it confusingly low in comparison to the valid loss.

One option is to only divide on the backward call, i.e. doing this:

scaler.scale(loss / gradient_accumulation_steps).backward()

or one could multiply the loss again with gradient_accumulation_steps before returning it.

[vfdev-5]

@RuABraun thanks for reporting !

If i get your point correctly, the fact that loss is scaled by gradient_accumulation_steps and this value is assigned to the output loss, it is confusing that values are not comparable to ones without gradient_accumulation_steps ?
I agree that division is needed for gradients computation.

I think multiplying the output loss as return output_transform(x, y, y_pred, loss * gradient_accumulation_steps) could be good enough

[louis-she]

I'd like to work on this one