Sandwich Transformer

Introduced by Press et al. in Improving Transformer Models by Reordering their Sublayers

A Sandwich Transformer is a variant of a Transformer that reorders sublayers in the architecture to achieve better performance. The reordering is based on the authors' analysis that models with more self-attention toward the bottom and more feedforward sublayers toward the top tend to perform better in general.

Source: Improving Transformer Models by Reordering their Sublayers

Read Paper See Code

Papers

Paper	Code	Results	Date	Stars

Tasks

Task	Papers	Share
Language Modelling	1	33.33%
Machine Translation	1	33.33%
Translation	1	33.33%

Usage Over Time

This feature is experimental; we are continuously improving our matching algorithm.

Components

Component	Type	Add Remove
Adaptive Masking	Attention Mechanisms
Dense Connections	Feedforward Networks
Dropout	Regularization
Layer Normalization	Normalization
Multi-Head Attention	Attention Modules
ReLU	Activation Functions
Residual Connection	Skip Connections
Scaled Dot-Product Attention	Attention Mechanisms
Softmax	Output Functions

Categories

Add Remove

Transformers

Autoregressive Transformers

Language Models