Single Headed Attention RNN

Introduced by Merity in Single Headed Attention RNN: Stop Thinking With Your Head

SHA-RNN, or Single Headed Attention RNN, is a recurrent neural network, and language model when combined with an embedding input and softmax classifier, based on a core LSTM component and a single-headed attention module. Other design choices include a Boom feedforward layer and the use of layer normalization. The guiding principles of the author were to ensure simplicity in the architecture and to keep computational costs bounded (the model was originally trained with a single GPU).

Source: Single Headed Attention RNN: Stop Thinking With Your Head

Read Paper See Code

Papers

Paper	Code	Results	Date	Stars

Tasks

Task	Papers	Share
Language Modelling	1	100.00%

Usage Over Time

This feature is experimental; we are continuously improving our matching algorithm.

Components

Component	Type	Add Remove
Boom Layer	Feedforward Networks
Dropout	Regularization
Embedding Dropout	Regularization
Layer Normalization	Normalization
LSTM	Recurrent Neural Networks
Single-Headed Attention	Attention Modules
Softmax	Output Functions

Categories

Add Remove

Language Models

Recurrent Neural Networks