TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK	REMOVE

Stabilizing Transformers for Reinforcement Learning

ICML 2020 • Emilio Parisotto • H. Francis Song • Jack W. Rae • Razvan Pascanu • Caglar Gulcehre • Siddhant M. Jayakumar • Max Jaderberg • Raphael Lopez Kaufman • Aidan Clark • Seb Noury • Matthew M. Botvinick • Nicolas Heess • Raia Hadsell

Owing to their ability to both effectively integrate information over long time horizons and scale to massive amounts of data, self-attention architectures have recently shown breakthrough success in natural language processing (NLP), achieving state-of-the-art results in domains such as language modeling and machine translation. Harnessing the transformer's ability to process long time horizons of information could provide a similar performance boost in partially observable reinforcement learning (RL) domains, but the large-scale transformers used in NLP have yet to be successfully applied to the RL setting... (read more)

PDF Abstract ICML 2020 PDF

Code

Add Remove Mark official

lucidrains/x-transformers

196

lucidrains/compressive-transformer-pytorch

jerrodparker20/adaptive-transformers-in-rl

dhruvramani/Transformers-RL

Tasks

Add Remove

GENERAL REINFORCEMENT LEARNING

LANGUAGE MODELLING

MACHINE TRANSLATION

Datasets

Add Datasets introduced or used in this paper

Results from the Paper

Edit

Submit results from this paper to get state-of-the-art GitHub badges and help the community compare results to other papers.

Methods used in the Paper

Add Remove

METHOD	TYPE
Cosine Annealing	Learning Rate Schedules
Sigmoid Activation	Activation Functions
Tanh Activation	Activation Functions
Variational Dropout	Regularization
Residual Connection	Skip Connections
Adaptive Input Representations	Input Embedding Factorization
Adaptive Softmax	Output Functions
Linear Warmup With Cosine Annealing	Learning Rate Schedules
Transformer-XL	Transformers
BPE	Subword Segmentation
Dense Connections	Feedforward Networks
Label Smoothing	Regularization
ReLU	Activation Functions
Adam	Stochastic Optimization
Softmax	Output Functions
Dropout	Regularization
Multi-Head Attention	Attention Modules
Layer Normalization	Normalization
Scaled Dot-Product Attention	Attention Mechanisms
LSTM	Recurrent Neural Networks
Transformer	Transformers

Edit Social Preview

Stabilizing Transformers for Reinforcement Learning Edit social preview

Code Edit Add Remove Mark official

Tasks Edit Add Remove

Datasets Edit

Results from the Paper Edit

Methods used in the Paper Edit Add Remove