On the Relationship between Self-Attention and Convolutional Layers

ICLR 2020 Jean-Baptiste CordonnierAndreas LoukasMartin Jaggi

Recent trends of incorporating attention mechanisms in vision have led researchers to reconsider the supremacy of convolutional layers as a primary building block. Beyond helping CNNs to handle long-range dependencies, Ramachandran et al. (2019) showed that attention can completely replace convolution and achieve state-of-the-art performance on vision tasks... (read more)

PDF Abstract

Results from the Paper


TASK DATASET MODEL METRIC NAME METRIC VALUE GLOBAL RANK RESULT LEADERBOARD
Image Classification CIFAR-10 SA quadratic embedding Percentage correct 93.8 # 45

Methods used in the Paper