Search Results for author:
Found 3 papers, 1 papers with code
On Difficulties of Attention Factorization through Shared Memory
Transformers have revolutionized deep learning in numerous fields, including natural language processing, computer vision, and audio processing.
Linear Self-Attention Approximation via Trainable Feedforward Kernel
In pursuit of faster computation, Efficient Transformers demonstrate an impressive variety of approaches -- models attaining sub-quadratic attention complexity can utilize a notion of sparsity or a low-rank approximation of inputs to reduce the number of attended keys; other ways to reduce complexity include locality-sensitive hashing, key pooling, additional memory to store information in compacted or hybridization with other architectures, such as CNN.
SimpleTRON: Simple Transformer with O(N) Complexity
In this paper, we propose that the dot product pairwise matching attention layer, which is widely used in Transformer-based models, is redundant for the model performance.
