Search Results for author:
Found 8 papers, 3 papers with code
Transformers Can Represent $n$-gram Language Models
This provides a first step towards understanding the mechanisms that transformer LMs can use to represent probability distributions over strings.
The Role of $n$-gram Smoothing in the Age of Neural Networks
For nearly three decades, language models derived from the $n$-gram assumption held the state of the art on the task.
A Theoretical Result on the Inductive Bias of RNN Language Models
However, a closer inspection of Hewitt et al.'s (2020) construction shows that it is not limited to hierarchical LMs, posing the question of what \emph{other classes} of LMs can be efficiently represented by RNNs.
Formal Aspects of Language Modeling
Large language models have become one of the most commonly deployed NLP inventions.
On the Representational Capacity of Recurrent Neural Language Models
We extend the Turing completeness result to the probabilistic case, showing how a rationally weighted RLM with unbounded computation time can simulate any deterministic probabilistic Turing machine (PTM) with rationally weighted transitions.
Recurrent Neural Language Models as Probabilistic Finite-state Automata
These results present a first step towards characterizing the classes of distributions RNN LMs can represent and thus help us understand their capabilities and limitations.
A Geometric Notion of Causal Probing
The linear subspace hypothesis (Bolukbasi et al., 2016) states that, in a language model's representation space, all information about a concept such as verbal number is encoded in a linear subspace.
Algorithms for Acyclic Weighted Finite-State Automata with Failure Arcs
The pathsum in ordinary acyclic WFSAs is efficiently computed by the backward algorithm in time $O(|E|)$, where $E$ is the set of transitions.
