Search Results for author:
Found 6 papers, 3 papers with code
Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Models
These features make it necessary to apply 3D parallelism, which integrates data parallelism, pipeline model parallelism and tensor model parallelism, to achieve high training efficiency.
DELTA: Dynamically Optimizing GPU Memory beyond Tensor Recomputation
To the best of our knowledge, we are the first to make a reasonable dynamic runtime scheduler on the combination of tensor swapping and tensor recomputation without user oversight.
EmbRace: Accelerating Sparse Communication for Distributed Training of NLP Neural Networks
EmbRace introduces Sparsity-aware Hybrid Communication, which integrates AlltoAll and model parallelism into data-parallel training, so as to reduce the communication overhead of highly sparse parameters.
S2 Reducer: High-Performance Sparse Communication to Accelerate Distributed Deep Learning
Distributed stochastic gradient descent (SGD) approach has been widely used in large-scale deep learning, and the gradient collective method is vital to ensure the training scalability of the distributed deep learning system.
Hierarchical Adaptive Pooling by Capturing High-order Dependency for Graph Representation Learning
Graph neural networks (GNN) have been proven to be mature enough for handling graph-structured data on node-level graph representation learning tasks.
P-ADMMiRNN: Training RNN with Stable Convergence via An Efficient and Paralleled ADMM Approach
We also provide novel update rules and theoretical convergence analysis.
