Search Results for author:
Found 16 papers, 10 papers with code
TRANSOM: An Efficient Fault-Tolerant System for Training LLMs
As a result, A substantial amount of training time is devoted to task checkpoint saving and loading, task rescheduling and restart, and task manual anomaly checks, which greatly harms the overall training efficiency.
Co-design Hardware and Algorithm for Vector Search
Vector search has emerged as the foundation for large-scale information retrieval and machine learning systems, with search engines like Google and Bing processing tens of thousands of queries per second on petabyte-scale document datasets by evaluating vector similarities between encoded query texts and web documents.
ASDL: A Unified Interface for Gradient Preconditioning in PyTorch
Gradient preconditioning is a key technique to integrate the second-order information into gradients for improving and extending gradient-based learning algorithms.
An End-to-End Network for Upright Adjustment of Panoramic Images
Direction estimation estimates the tilt angle of the panoramic image.
PipeFisher: Efficient Training of Large Language Models Using Pipelining and Fisher Information Matrices
Pipeline parallelism enables efficient training of Large Language Models (LLMs) on large-scale distributed accelerator clusters.
Efficient Quantized Sparse Matrix Operations on Tensor Cores
We propose Magicube, a high-performance sparse-matrix library for low-precision integers on Tensor cores.
HammingMesh: A Network Topology for Large-Scale Deep Learning
Numerous microarchitectural optimizations unlocked tremendous processing power for deep neural networks that in turn fueled the AI revolution.
Near-Optimal Sparse Allreduce for Distributed Deep Learning
However, it is very challenging to obtain real performance improvement because of (1) the difficulty of achieving an scalable and efficient sparse allreduce algorithm and (2) the sparsification overhead.
A Data-Centric Optimization Framework for Machine Learning
Rapid progress in deep learning is leading to a diverse set of quickly changing models, with a dramatically growing demand for compute.
Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines
For a GPT-2 model with 1. 3 billion parameters running on 2, 048 GPU nodes of the Piz Daint supercomputer, Chimera improves the training throughput by 1. 16x-2. 34x over the state-of-the-art synchronous and asynchronous pipeline approaches.
Data Movement Is All You Need: A Case Study on Optimizing Transformers
Transformers are one of the most important machine learning workloads today.
Deep Learning for Post-Processing Ensemble Weather Forecasts
Applied to global data, our mixed models achieve a relative improvement in ensemble forecast skill (CRPS) of over 14%.
Breaking (Global) Barriers in Parallel Stochastic Optimization with Wait-Avoiding Group Averaging
For evaluation, we train ResNet-50 on ImageNet; Transformer for machine translation; and deep reinforcement learning for navigation at scale.
Predicting Weather Uncertainty with Deep Convnets
Modern weather forecast models perform uncertainty quantification using ensemble prediction systems, which collect nonparametric statistics based on multiple perturbed simulations.
Asynchronous Decentralized SGD with Quantized and Local Updates
Perhaps surprisingly, we show that a variant of SGD called \emph{SwarmSGD} still converges in this setting, even if \emph{non-blocking communication}, \emph{quantization}, and \emph{local steps} are all applied \emph{in conjunction}, and even if the node data distributions and underlying graph topology are both \emph{heterogenous}.
Taming Unbalanced Training Workloads in Deep Learning with Partial Collective Operations
Load imbalance pervasively exists in distributed deep learning training systems, either caused by the inherent imbalance in learned tasks or by the system itself.
