Search Results for author:
Found 7 papers, 5 papers with code
HGRN2: Gated Linear RNNs with State Expansion
Hierarchically gated linear RNN (HGRN, Qin et al. 2023) has demonstrated competitive training speed and performance in language modeling, while offering efficient inference.
Linear Attention Sequence Parallelism
In this paper, we introduce Linear Attention Sequence Parallel (LASP), an efficient SP method tailored to linear attention-based language models.
MS-Net: A Multi-Path Sparse Model for Motion Prediction in Multi-Scenes
In the training stage, the motion prediction task under differentiated scenes is abstracted as a multi-task learning problem, an evolutionary algorithm is designed to encourage the network search of the optimal parameters for each scene while sharing common knowledge between different scenes.
CO2: Efficient Distributed Training with Full Communication-Computation Overlap
CO2 is able to attain a high scalability even on extensive multi-node clusters constrained by very limited communication bandwidth.
Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models
With its ability to process tokens in linear computational complexities, linear attention, in theory, can handle sequences of unlimited length without sacrificing speed, i. e., maintaining a constant training speed for various sequence lengths with a fixed memory consumption.
TransNormerLLM: A Faster and Better Large Language Model with Improved TransNormer
TransNormerLLM evolves from the previous linear attention architecture TransNormer by making advanced modifications that include positional embedding, linear attention acceleration, gating mechanisms, tensor normalization, and inference acceleration and stabilization.
PowerSGD: Powered Stochastic Gradient Descent Methods for Accelerated Non-Convex Optimization
In this paper, we propose a novel technique for improving the stochastic gradient descent (SGD) method to train deep networks, which we term \emph{PowerSGD}.
