Transformers as Transducers

no code implementations • 2 Apr 2024 • Lena Strobl, Dana Angluin, David Chiang, Jonathan Rawski, Ashish Sabharwal

We study the sequence-to-sequence mapping capacity of transformers by relating them to finite transducers, and find that they can express surprisingly large classes of transductions.

Hard Attention POS

Transformers as Recognizers of Formal Languages: A Survey on Expressivity

no code implementations • 1 Nov 2023 • Lena Strobl, William Merrill, Gail Weiss, David Chiang, Dana Angluin

As transformers have gained prominence in natural language processing, some researchers have investigated theoretically what problems they can and cannot solve, by treating problems as formal languages.

Average-Hard Attention Transformers are Constant-Depth Uniform Threshold Circuits

no code implementations • 6 Aug 2023 • Lena Strobl

Transformers have emerged as a widely used neural network model for various natural language processing tasks.

Hard Attention