Search Results for author:
Found 7 papers, 4 papers with code
Smart-Infinity: Fast Large Language Model Training using Near-Storage Processing on a Real System
Our work, Smart-Infinity, addresses the storage bandwidth bottleneck of storage-offloaded LLM training using near-storage processing devices on a real system.
PeerAiD: Improving Adversarial Distillation from a Specialized Peer Tutor
In this situation, adversarial distillation is a promising option which aims to distill the robustness of the teacher network to improve the robustness of a small student network.
GraNNDis: Efficient Unified Distributed Training Framework for Deep GNNs on Large Clusters
According to the growth in the dataset and the model size used for GNNs, an important problem is that it becomes nearly impossible to keep the whole network on GPU memory.
Pipe-BD: Pipelined Parallel Blockwise Distillation
However, this results in a high overhead of redundant teacher execution, low GPU utilization, and extra data loading.
SGCN: Exploiting Compressed-Sparse Features in Deep Graph Convolutional Network Accelerators
A GCN takes as input an arbitrarily structured graph and executes a series of layers which exploit the graph's structure to calculate their output features.
Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compression
Compressing the communication is one way to mitigate the overhead by reducing the inter-node traffic volume; however, the existing compression techniques have critical limitations to be applied for NLP models with 3D parallelism in that 1) only the data parallelism traffic is targeted, and 2) the existing compression schemes already harm the model quality too much.
Slice-and-Forge: Making Better Use of Caches for Graph Convolutional Network Accelerators
Graph convolutional networks (GCNs) are becoming increasingly popular as they can process a wide variety of data formats that prior deep neural networks cannot easily support.
