How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformers

1 code implementation • 7 Nov 2022 • Michael Hassid, Hao Peng, Daniel Rotem, Jungo Kasai, Ivan Montero, Noah A. Smith, Roy Schwartz

Our results motivate research on simpler alternatives to input-dependent attention, as well as on methods for better utilization of this mechanism in the Transformer architecture.

Sentence Bottleneck Autoencoders from Transformer Language Models

1 code implementation • EMNLP 2021 • Ivan Montero, Nikolaos Pappas, Noah A. Smith

Representation learning for text via pretraining a language model on a large corpus has become a standard starting point for building NLP systems.

Denoising Language Modelling +6

Pivot Through English: Reliably Answering Multilingual Questions without Document Retrieval

no code implementations • NAACL (MIA) 2022 • Ivan Montero, Shayne Longpre, Ni Lao, Andrew J. Frank, Christopher DuBois

Existing methods for open-retrieval question answering in lower resource languages (LRLs) lag significantly behind English.

Answer Generation Question Answering +4

Plug and Play Autoencoders for Conditional Text Generation

1 code implementation • EMNLP 2020 • Florian Mai, Nikolaos Pappas, Ivan Montero, Noah A. Smith, James Henderson

Text autoencoders are commonly used for conditional generation tasks such as style transfer.

Conditional Text Generation Navigate +1