Densifying Assumed-sparse Tensors: Improving Memory Efficiency and MPI Collective Performance during Tensor Accumulation for Parallelized Training of Neural Machine Translation Models

no code implementations • 10 May 2019 • Derya Cavdar, Valeriu Codreanu, Can Karakus, John A. Lockman III, Damian Podareanu, Vikram Saletore, Alexander Sergeev, Don D. Smith II, Victor Suthichai, Quy Ta, Srinivas Varadharajan, Lucas A. Wilson, Rengan Xu, Pei Yang

Neural machine translation - using neural networks to translate human language - is an area of active research exploring new neuron types and network topologies with the goal of dramatically improving machine translation performance.

Machine Translation Translation

Horovod: fast and easy distributed deep learning in TensorFlow

13 code implementations • 15 Feb 2018 • Alexander Sergeev, Mike Del Balso

Depending on the particular methods employed, this communication may entail anywhere from negligible to significant overhead.