Scaling distributed training with adaptive summation

Saeed Maleki et al. Microsoft Research

Key point

$$\vec{g}=\left(1-\frac{{\vec{g}}_1\cdot {\vec{g}}_2}{2|{\vec{g}}_1{|}^2}\right){\vec{g}}_1+\left(1-\frac{{\vec{g}}_1\cdot {\vec{g}}_2}{2|{\vec{g}}_2{|}^2}\right){\vec{g}}_2$$

Reference

• AdaSum with Horovod
• Arxiv