Transformers

Feedback Transformer

Introduced by Fan et al. in Addressing Some Limitations of Transformers with Feedback Memory

A Feedback Transformer is a type of sequential transformer that exposes all previous representations to all future representations, meaning the lowest representation of the current timestep is formed from the highest-level abstract representation of the past. This feedback nature allows this architecture to perform recursive computation, building stronger representations iteratively upon previous states. To achieve this, the self-attention mechanism of the standard Transformer is modified so it attends to higher level representations rather than lower ones.

Source: Addressing Some Limitations of Transformers with Feedback Memory

Papers


Paper Code Results Date Stars

Tasks


Task Papers Share
Point Cloud Registration 1 25.00%
Language Modelling 1 25.00%
Machine Translation 1 25.00%
Translation 1 25.00%

Categories