Tempo: Accelerating Transformer-Based Model Training through Memory Footprint Reduction

1 code implementation • 19 Oct 2022 • Muralidhar Andoorveedu, Zhanda Zhu, Bojian Zheng, Gennady Pekhimenko

We implement Tempo and evaluate the throughput, memory usage, and accuracy/loss on the BERT Large pre-training task.