Logic and the $2$-Simplicial Transformer

2 Sep 2019 · James Clift, Dmitry Doryn, Daniel Murfet, James Wallbridge ·

We introduce the $2$-simplicial Transformer, an extension of the Transformer which includes a form of higher-dimensional attention generalising the dot-product attention, and uses this attention to update entity representations with tensor products of value vectors. We show that this architecture is a useful inductive bias for logical reasoning in the context of deep reinforcement learning.

Code

Add Remove Mark official

No code implementations yet. Submit your code now

Tasks

Add Remove

Logical Reasoning

reinforcement-learning

Reinforcement Learning (RL)

Datasets

AbstractReasoning

Results from the Paper

Edit

Submit results from this paper to get state-of-the-art GitHub badges and help the community compare results to other papers.

Methods

Add Remove

Absolute Position Encodings • Adam • BPE • Dense Connections • Dropout • Label Smoothing • Layer Normalization • Linear Layer • Multi-Head Attention • Position-Wise Feed-Forward Layer • ReLU • Residual Connection • Scaled Dot-Product Attention • Softmax • Transformer