Search Results for author:
Found 6 papers, 4 papers with code
Smart-Infinity: Fast Large Language Model Training using Near-Storage Processing on a Real System
Our work, Smart-Infinity, addresses the storage bandwidth bottleneck of storage-offloaded LLM training using near-storage processing devices on a real system.
PeerAiD: Improving Adversarial Distillation from a Specialized Peer Tutor
In this situation, adversarial distillation is a promising option which aims to distill the robustness of the teacher network to improve the robustness of a small student network.
GraNNDis: Efficient Unified Distributed Training Framework for Deep GNNs on Large Clusters
According to the growth in the dataset and the model size used for GNNs, an important problem is that it becomes nearly impossible to keep the whole network on GPU memory.
Pipe-BD: Pipelined Parallel Blockwise Distillation
However, this results in a high overhead of redundant teacher execution, low GPU utilization, and extra data loading.
Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compression
Compressing the communication is one way to mitigate the overhead by reducing the inter-node traffic volume; however, the existing compression techniques have critical limitations to be applied for NLP models with 3D parallelism in that 1) only the data parallelism traffic is targeted, and 2) the existing compression schemes already harm the model quality too much.
Visual Relationship Detection with Language prior and Softmax
Visual relationship detection is an intermediate image understanding task that detects two objects and classifies a predicate that explains the relationship between two objects in an image.
