Search Results for author:
Found 8 papers, 0 papers with code
Attention is Naturally Sparse with Gaussian Distributed Input
The computational intensity of Large Language Models (LLMs) is a critical bottleneck, primarily due to the $O(n^2)$ complexity of the attention mechanism in transformer architectures.
Enhancing Stochastic Gradient Descent: A Unified Framework and Novel Acceleration Methods for Faster Convergence
Based on SGD, previous works have proposed many algorithms that have improved convergence speed and generalization in stochastic optimization, such as SGDm, AdaGrad, Adam, etc.
One Pass Streaming Algorithm for Super Long Token Attention Approximation in Sublinear Space
Considering a single-layer self-attention with Query, Key, and Value matrices $Q, K, V \in \mathbb{R}^{n \times d}$, the polynomial method approximates the attention output $T \in \mathbb{R}^{n \times d}$.
A Theoretical Insight into Attack and Defense of Gradient Leakage in Transformer
The Deep Leakage from Gradient (DLG) attack has emerged as a prevalent and highly effective method for extracting sensitive training data by inspecting exchanged gradients.
Unmasking Transformers: A Theoretical Approach to Data Recovery via Attention Weights
In the realm of deep learning, transformers have emerged as a dominant architecture, particularly in natural language processing tasks.
An Automatic Learning Rate Schedule Algorithm for Achieving Faster Convergence and Steeper Descent
The delta-bar-delta algorithm is recognized as a learning rate adaptation technique that enhances the convergence speed of the training process in optimization by dynamically scheduling the learning rate based on the difference between the current and previous weight updates.
Fine-tune Language Models to Approximate Unbiased In-context Learning
To address this issue, we introduce a reweighted algorithm called RICL (Reweighted In-context Learning).
How to Protect Copyright Data in Optimization of Large Language Models?
Large language models (LLMs) and generative AI have played a transformative role in computer research and applications.
