no code implementations • 1 Nov 2023 • Lena Strobl, William Merrill, Gail Weiss, David Chiang, Dana Angluin
As transformers have gained prominence in natural language processing, some researchers have investigated theoretically what problems they can and cannot solve, by treating problems as formal languages.
no code implementations • 4 Oct 2023 • Deniz Bayazit, Negar Foroutan, Zeming Chen, Gail Weiss, Antoine Bosselut
In this work, we investigate whether pretrained language models contain various knowledge-critical subnetworks: particular sparse computational subgraphs responsible for encoding specific knowledge the model has memorized.
no code implementations • NeurIPS 2023 • Zeming Chen, Gail Weiss, Eric Mitchell, Asli Celikyilmaz, Antoine Bosselut
In the outer loop, the model learns to use the updated weights to reproduce and answer reasoning questions about the memorized knowledge.
4 code implementations • 13 Jun 2021 • Gail Weiss, Yoav Goldberg, Eran Yahav
In this paper we aim to change that, proposing a computational model for the transformer-encoder in the form of a programming language.
no code implementations • 20 Jan 2021 • Daniel M. Yellin, Gail Weiss
We present an algorithm for extracting a subclass of the context free grammars (CFGs) from a trained recurrent neural network (RNN).
no code implementations • ACL 2020 • William Merrill, Gail Weiss, Yoav Goldberg, Roy Schwartz, Noah A. Smith, Eran Yahav
While formally extending these findings to unsaturated RNNs is left to future work, we hypothesize that the practical learnable capacity of unsaturated RNNs obeys a similar hierarchy.
1 code implementation • NeurIPS 2019 • Gail Weiss, Yoav Goldberg, Eran Yahav
We present an algorithm for extraction of a probabilistic deterministic finite automaton (PDFA) from a given black-box language model, such as a recurrent neural network (RNN).
1 code implementation • ACL 2018 • Gail Weiss, Yoav Goldberg, Eran Yahav
While Recurrent Neural Networks (RNNs) are famously known to be Turing complete, this relies on infinite precision in the states and unbounded computation time.
1 code implementation • ICML 2018 • Gail Weiss, Yoav Goldberg, Eran Yahav
We do this using Angluin's L* algorithm as a learner and the trained RNN as an oracle.