Search Results for author:
Found 21 papers, 15 papers with code
Overconfidence in the Face of Ambiguity with Adversarial Data
Adversarial data collection has shown promise as a method for building models which are more robust to the spurious correlations that generally appear in naturalistic data.
Bias-Augmented Consistency Training Reduces Biased Reasoning in Chain-of-Thought
Moreover, this model generalizes to other forms of bias, reducing biased reasoning on held-out biases by an average of 37%.
The Case for Scalable, Data-Driven Theory: A Paradigm for Scientific Progress in NLP
I propose a paradigm for scientific progress in NLP centered around developing scalable, data-driven theories of linguistic structure.
GPQA: A Graduate-Level Google-Proof Q&A Benchmark
We present GPQA, a challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry.
Debate Helps Supervise Unreliable Experts
Comparing debate to a baseline we call consultancy, where a single expert argues for only one answer which is correct half of the time, we find that debate performs significantly better, with 84% judge accuracy compared to consultancy's 74%.
Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
We demonstrate that CoT explanations can be heavily influenced by adding biasing features to model inputs--e. g., by reordering the multiple-choice options in a few-shot prompt to make the answer always "(A)"--which models systematically fail to mention in their explanations.
We're Afraid Language Models Aren't Modeling Ambiguity
We find that the task remains extremely challenging, including for GPT-4, whose generated disambiguations are considered correct only 32% of the time in human evaluation, compared to 90% for disambiguations in our dataset.
What Do NLP Researchers Believe? Results of the NLP Community Metasurvey
We present the results of the NLP Community Metasurvey.
kNN-Prompt: Nearest Neighbor Zero-Shot Inference
Retrieval-augmented language models (LMs) use non-parametric memory to substantially outperform their non-retrieval counterparts on perplexity-based evaluations, but it is an open question whether they achieve similar gains in few- and zero-shot end-task accuracy.
Asking It All: Generating Contextualized Questions for any Semantic Role
We develop a two-stage model for this task, which first produces a context-independent question prototype for each role and then revises it to be contextually appropriate for the passage.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
Many commonsense reasoning NLP tasks involve choosing between one or more possible answers to a question or prompt based on knowledge that is often implicit.
Asking without Telling: Exploring Latent Ontologies in Contextual Representations
The success of pretrained contextual encoders, such as ELMo and BERT, has brought a great deal of interest in what these models learn: do they, without explicit supervision, learn to encode meaningful notions of linguistic structure?
AmbigQA: Answering Ambiguous Open-domain Questions
Ambiguity is inherent to open-domain question answering; especially when exploring new topics, it can be difficult to ask questions that have a single, unambiguous answer.
Controlled Crowdsourcing for High-Quality QA-SRL Annotation
Question-answer driven Semantic Role Labeling (QA-SRL) was proposed as an attractive open and natural flavour of SRL, potentially attainable from laymen.
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
In the last year, new models and methods for pretraining and transfer learning have driven striking performance improvements across a range of language understanding tasks.
Supervised Open Information Extraction
We present data and methods that enable a supervised learning approach to Open Information Extraction (Open IE).
Large-Scale QA-SRL Parsing
We present a new large-scale corpus of Question-Answer driven Semantic Role Labeling (QA-SRL) annotations, and the first high-quality QA-SRL parser.
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
For natural language understanding (NLU) technology to be maximally useful, both practically and as a scientific object of study, it must be general: it must be able to process language in a way that is not exclusively tailored to any one specific task or dataset.
Ranked #46 on
Natural Language Inference
on MultiNLI
Natural Language Inference
Natural Language Understanding
+2
Crowdsourcing Question-Answer Meaning Representations
We introduce Question-Answer Meaning Representations (QAMRs), which represent the predicate-argument structure of a sentence as a set of question-answer pairs.
