Search Results for author:
Found 10 papers, 7 papers with code
The Illusion of State in State-Space Models
Our analysis reveals that the expressive power of SSMs is limited very similarly to transformers: SSMs cannot express computation outside the complexity class $\mathsf{TC}^0$.
GPQA: A Graduate-Level Google-Proof Q&A Benchmark
We present GPQA, a challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry.
Debate Helps Supervise Unreliable Experts
Comparing debate to a baseline we call consultancy, where a single expert argues for only one answer which is correct half of the time, we find that debate performs significantly better, with 84% judge accuracy compared to consultancy's 74%.
In-context Learning Generalizes, But Not Always Robustly: The Case of Syntax
In-context learning (ICL) is now a common method for teaching large language models (LLMs) new tasks: given labeled examples in the input context, the LLM learns to perform the task without weight updates.
How Abstract Is Linguistic Generalization in Large Language Models? Experiments with Argument Structure
We find that LLMs perform well in generalizing the distribution of a novel noun argument between related contexts that were seen during pre-training (e. g., the active object and passive subject of the verb spray), succeeding by making use of the semantically-organized structure of the embedding space for word embeddings.
The Impact of Depth on Compositional Generalization in Transformer Language Models
Because model latency is approximately linear in the number of layers, these results lead us to the recommendation that, with a given total parameter budget, transformers can be made shallower than is typical without sacrificing performance.
(QA)$^2$: Question Answering with Questionable Assumptions
Naturally occurring information-seeking questions often contain questionable assumptions -- assumptions that are false or unverifiable.
Do Language Models Learn Position-Role Mappings?
How is knowledge of position-role mappings in natural language learned?
Transformers Generalize Linearly
Natural language exhibits patterns of hierarchically governed dependencies, in which relations between words are sensitive to syntactic structure rather than linear ordering.
Sequence-to-Sequence Networks Learn the Meaning of Reflexive Anaphora
Reflexive anaphora present a challenge for semantic interpretation: their meaning varies depending on context in a way that appears to require abstract variables.
