Attention Mechanisms

Attention Mechanisms are a component used in neural networks to model long-range interaction, for example across a text in NLP. The key idea is to build shortcuts between a context vector and the input, to allow a model to attend to different parts. Below you can find a continuously updating list of attention mechanisms.

METHOD YEAR PAPERS
Scaled Dot-Product Attention
2017 2259
Additive Attention
2014 69
Dot-Product Attention
2015 48
Location-based Attention
2015 23
Content-based Attention
2014 22
Location Sensitive Attention
2015 7
Channel-wise Soft Attention
2017 4
Global and Sliding Window Attention
2020 2
Dilated Sliding Window Attention
2020 2
LSH Attention
2020 2
Strided Attention
2019 2
Sliding Window Attention
2020 2
Fixed Factorized Attention
2019 2
Dense Synthesized Attention
2020 1
SortCut Sinkhorn Attention
2020 1
Random Synthesized Attention
2020 1
Routing Attention
2020 1
Factorized Dense Synthesized Attention
2020 1
Factorized Random Synthesized Attention
2020 1
Sparse Sinkhorn Attention
2020 1
Multiplicative Attention
2015 1
Adaptive Masking
2019 1