Search Results for author:
Found 255 papers, 121 papers with code
Expected Validation Performance and Estimation of a Random Variable’s Maximum
We find that the two biased estimators lead to the fewest incorrect conclusions, which hints at the importance of minimizing variance and MSE.
The Engage Corpus: A Social Media Dataset for Text-Based Recommender Systems
Social media platforms play an increasingly important role as forums for public discourse.
Writing Strategies for Science Communication: Data and Computational Analysis
We collect a corpus of 128k science writing documents in English and annotate a subset of this corpus.
Domain Mismatch Doesn’t Always Prevent Cross-lingual Transfer Learning
Cross-lingual transfer learning without labeled target language data or parallel text has been surprisingly effective in zero-shot cross-lingual classification, question answering, unsupervised machine translation, etc.
A Taxonomy of Ambiguity Types for NLP
Ambiguity is an critical component of language that allows for more effective communication between speakers, but is often ignored in NLP.
RewardBench: Evaluating Reward Models for Language Modeling
In this paper, we present RewardBench, a benchmark dataset and code-base for evaluation, to enhance scientific understanding of reward models.
Third-Party Language Model Performance Prediction from Instruction
Language model-based instruction-following systems have lately shown increasing performance on many benchmark tasks, demonstrating the capability of adapting to a broad variety of instructions.
Encode Once and Decode in Parallel: Efficient Transformer Decoding
Transformer-based NLP models are powerful but have high computational costs that limit deployment scenarios.
Set the Clock: Temporal Alignment of Pretrained Language Models
We then develop several methods, from prompting to finetuning, to align LMs to use their most recent knowledge when answering questions, and investigate various factors in this alignment.
OLMo: Accelerating the Science of Language Models
Given the importance of these details in scientifically studying these models, including their biases and potential risks, we believe it is essential for the research community to have access to powerful, truly open LMs.
Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research
Language models have become a critical technology to tackling a wide range of natural language processing tasks, yet many details about how the best-performing language models were developed are not reported.
Breaking the Curse of Multilinguality with Cross-lingual Expert Language Models
Despite their popularity in non-English NLP, multilingual language models often underperform monolingual ones due to inter-language competition for model parameters.
Tuning Language Models by Proxy
Despite the general capabilities of large pretrained language models, they consistently benefit from further adaptation to better achieve desired behaviors.
Time is Encoded in the Weights of Finetuned Language Models
We present time vectors, a simple tool to customize language models to new time periods.
Paloma: A Benchmark for Evaluating Language Model Fit
We invite submissions to our benchmark and organize results by comparability based on compliance with guidelines such as removal of benchmark contamination from pretraining.
Language Models: A Guide for the Perplexed
Given the growing importance of AI literacy, we decided to write this tutorial to help narrow the gap between the discourse among those who study language models -- the core technology underlying ChatGPT and similar products -- and those who are intrigued and want to learn more about them.
Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2
Since the release of T\"ULU [Wang et al., 2023b], open resources for instruction tuning have developed quickly, from better base models to new finetuning techniques.
Measuring and Improving Attentiveness to Partial Inputs with Counterfactuals
Previous work has found that datasets with paired inputs are prone to correlations between a specific part of the input (e. g., the hypothesis in NLI) and the label; consequently, models trained only on those outperform chance.
ACID: Abstractive, Content-Based IDs for Document Retrieval with Language Models
Our results demonstrate the effectiveness of human-readable, natural-language IDs in generative retrieval with LMs.
What's In My Big Data?
We open-source WIMBD's code and artifacts to provide a standard set of evaluations for new text-based corpora and to encourage more analyses and transparency around them.
That was the last straw, we need more: Are Translation Systems Sensitive to Disambiguating Context?
In experiments, we compare MT-specific models and language models for (i) their preference when given an ambiguous subsentence, (ii) their sensitivity to disambiguating context, and (iii) the performance disparity between figurative and literal source sentences.
In-Context Pretraining: Language Modeling Beyond Document Boundaries
Large language models (LMs) are currently trained to predict tokens given document prefixes, enabling them to directly perform long-form generation and prompting-style tasks which can be reduced to document completion.
SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore
SILO is built by (1) training a parametric LM on Open License Corpus (OLC), a new corpus we curate with 228B tokens of public domain and permissively licensed text and (2) augmenting it with a more general and easily modifiable nonparametric datastore (e. g., containing copyrighted books or news) that is only queried during inference.
Efficiency Pentathlon: A Standardized Arena for Efficiency Evaluation
In response, we introduce Pentathlon, a benchmark for holistic and realistic evaluation of model efficiency.
Does Collaborative Human-LM Dialogue Generation Help Information Extraction from Human Dialogues?
The capabilities of pretrained language models have opened opportunities to explore new application areas, but applications involving human-human interaction are limited by the fact that most data is protected from public release for privacy reasons.
Estimating the Causal Effect of Early ArXiving on Paper Acceptance
However, since quality is a challenging construct to estimate, we use the negative outcome control method, using paper citation count as a control variable to debias the quality confounding effect.
Reproducibility in NLP: What Have We Learned from the Checklist?
Scientific progress in NLP rests on the reproducibility of researchers' claims.
Morphosyntactic probing of multilingual BERT models
We introduce an extensive dataset for multilingual probing of morphological information in language models (247 tasks across 42 languages from 10 families), each consisting of a sentence with a target word and a morphological tag as the desired label, derived from the Universal Dependencies treebanks.
How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources
Our evaluations show that the best model in any given evaluation reaches on average 87% of ChatGPT performance, and 73% of GPT-4 performance, suggesting that further investment in building better base models and instruction-tuning data is required to close the gap.
Stubborn Lexical Bias in Data and Models
Using a new statistical method, we examine whether such spurious patterns in data appear in models trained on the data.
Fine-Grained Human Feedback Gives Better Rewards for Language Model Training
We introduce Fine-Grained RLHF, a framework that enables training and learning from reward functions that are fine-grained in two respects: (1) density, providing a reward after every segment (e. g., a sentence) is generated; and (2) incorporating multiple reward models associated with different feedback types (e. g., factual incorrectness, irrelevance, and information incompleteness).
Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models
Language models have graduated from being research prototypes to commercialized products offered as web APIs, and recent works have highlighted the multilingual capabilities of these products.
How Language Model Hallucinations Can Snowball
A major risk of using language models in practical applications is their tendency to hallucinate incorrect statements.
Vera: A General-Purpose Plausibility Estimation Model for Commonsense Statements
Despite the much discussed capabilities of today's language models, they are still prone to silly and unexpected commonsense failures.
We're Afraid Language Models Aren't Modeling Ambiguity
We find that the task remains extremely challenging, including for GPT-4, whose generated disambiguations are considered correct only 32% of the time in human evaluation, compared to 90% for disambiguations in our dataset.
Scaling Expert Language Models with Unsupervised Domain Discovery
Large language models are typically trained densely: all parameters are updated with respect to all inputs.
NarrowBERT: Accelerating Masked Language Model Pretraining and Inference
We propose NarrowBERT, a modified transformer encoder that increases the throughput for masked language model pretraining by more than $2\times$.
PromptCap: Prompt-Guided Image Captioning for VQA with GPT-3
PromptCap outperforms generic captions by a large margin and achieves state-of-the-art accuracy on knowledge-based VQA tasks (60. 4% on OK-VQA and 59. 6% on A-OKVQA).
Self-Instruct: Aligning Language Models with Self-Generated Instructions
Applying our method to the vanilla GPT3, we demonstrate a 33% absolute improvement over the original model on Super-NaturalInstructions, on par with the performance of InstructGPT-001, which was trained with private user data and human annotations.
One Embedder, Any Task: Instruction-Finetuned Text Embeddings
Our analysis suggests that INSTRUCTOR is robust to changes in instructions, and that instruction finetuning mitigates the challenge of training a single model on diverse datasets.
Demystifying Prompts in Language Models via Perplexity Estimation
Language models can be prompted to perform a wide variety of zero- and few-shot learning problems.
Data-Efficient Finetuning Using Cross-Task Nearest Neighbors
Obtaining labeled data to train a model for a task of interest is often expensive.
Domain Mismatch Doesn't Always Prevent Cross-Lingual Transfer Learning
Cross-lingual transfer learning without labeled target language data or parallel text has been surprisingly effective in zero-shot cross-lingual classification, question answering, unsupervised machine translation, etc.
How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformers
Our results motivate research on simpler alternatives to input-dependent attention, as well as on methods for better utilization of this mechanism in the Transformer architecture.
Modeling Context With Linear Attention for Scalable Document-Level Translation
Document-level machine translation leverages inter-sentence dependencies to produce more coherent and consistent translations.
Transparency Helps Reveal When Language Models Learn Meaning
Many current NLP systems are built from language models trained to optimize unsupervised objectives on large amounts of raw text.
Measuring and Narrowing the Compositionality Gap in Language Models
We investigate the ability of language models to perform compositional reasoning tasks where the overall solution depends on correctly composing the answers to sub-problems.
Ranked #4 on
Question Answering
on Bamboogle
Binding Language Models in Symbolic Languages
We propose Binder, a training-free neural-symbolic framework that maps the task input to a program, which (1) allows binding a unified API of language model (LM) functionalities to a programming language (e. g., SQL, Python) to extend its grammar coverage and thus tackle more diverse questions, (2) adopts an LM as both the program parser and the underlying model called by the API during execution, and (3) requires only a few in-context exemplar annotations.
Ranked #4 on
Table-based Fact Verification
on TabFact
Selective Annotation Makes Language Models Better Few-Shot Learners
Departing from recent in-context learning methods, we formulate an annotation-efficient, two-step framework: selective annotation that chooses a pool of examples to annotate from unlabeled data in advance, followed by prompt retrieval that retrieves task examples from the annotated pool at test time.
Elaboration-Generating Commonsense Question Answering at Scale
In question answering requiring common sense, language models (e. g., GPT-3) have been used to generate text expressing background knowledge that helps improve performance.
Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models
New ELMs are learned by branching from (mixtures of) ELMs in the current set, further training the parameters on data for the new domain, and then merging the resulting model back into the set for future use.
RealTime QA: What's the Answer Right Now?
We introduce REALTIME QA, a dynamic question answering (QA) platform that announces questions and evaluates systems on a regular basis (weekly in this version).
Measuring the Carbon Intensity of AI in Cloud Instances
By providing unprecedented access to computational resources, cloud computing has enabled rapid growth in technologies such as machine learning, the computational demands of which incur a high energy cost and a commensurate carbon footprint.
Unsupervised Learning of Hierarchical Conversation Structure
Human conversations can evolve in many different ways, creating challenges for automatic understanding and summarization.
Twist Decoding: Diverse Generators Guide Each Other
Our extensive evaluations on machine translation and scientific paper summarization demonstrate that Twist decoding substantially outperforms each model decoded in isolation over various scenarios, including cases where domain-specific and general-purpose models are both available.
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
This large and diverse collection of tasks enables rigorous benchmarking of cross-task generalization under instructions -- training models to follow instructions on a subset of tasks and evaluating them on the remaining unseen ones.
A Call for Clarity in Beam Search: How It Works and When It Stops
Based on this finding, we introduce a patience factor, a simple modification to this beam decoding implementation, that generalizes the stopping criterion and provides flexibility to the depth of search.
In-Context Learning for Few-Shot Dialogue State Tracking
In this work, we propose an in-context learning (ICL) framework for zero-shot and few-shot learning DST, where a large pre-trained language model (LM) takes a test instance and a few exemplars as input, and directly decodes the dialogue state without any parameter updates.
Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection
Language models increasingly rely on massive web dumps for diverse text data.
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models
Structured knowledge grounding (SKG) leverages structured knowledge to complete user requests, such as semantic parsing over databases and question answering over knowledge bases.
Ranked #1 on
Task-Oriented Dialogue Systems
on KVRET
WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation
Starting with an existing dataset, MultiNLI for natural language inference (NLI), our approach uses dataset cartography to automatically identify examples that demonstrate challenging reasoning patterns, and instructs GPT-3 to compose new examples with similar patterns.
Imagined versus Remembered Stories: Quantifying Differences in Narrative Flow
We quantify the differences between autobiographical and imagined stories by introducing sequentiality, a measure of narrative flow of events, drawing probabilistic inferences from a cutting-edge large language model (GPT-3).
NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristics
To enable constrained generation, we build on NeuroLogic decoding (Lu et al., 2021), combining its flexibility in incorporating logical constraints with A*esque estimates of future constraint satisfaction.
Ranked #1 on
Text Generation
on ROCStories
Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand
We therefore propose a generalization of leaderboards, bidimensional leaderboards (Billboards), that simultaneously tracks progress in language generation models and metrics for their evaluation.
Transparent Human Evaluation for Image Captioning
We establish THumB, a rubric-based human evaluation protocol for image captioning models.
Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection
The perceived toxicity of language can vary based on someone's identity and beliefs, but this variation is often ignored when collecting toxic language datasets, resulting in dataset and model biases.
Time Waits for No One! Analysis and Challenges of Temporal Misalignment
When an NLP model is trained on text data from one time period and tested or deployed on data from another, the resulting temporal misalignment can degrade end-task performance.
ABC: Attention with Bounded-memory Control
One way to improve the efficiency is to bound the memory size.
Expected Validation Performance and Estimation of a Random Variable's Maximum
We find that the two biased estimators lead to the fewest incorrect conclusions, which hints at the importance of minimizing variance and MSE.
Sentence Bottleneck Autoencoders from Transformer Language Models
Representation learning for text via pretraining a language model on a large corpus has become a standard starting point for building NLP systems.
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
Since the introduction of the transformer model by Vaswani et al. (2017), a fundamental question has yet to be answered: how does a model achieve extrapolation at inference time for sequences that are longer than it saw during training?
DEMix Layers: Disentangling Domains for Modular Language Modeling
We introduce a new domain expert mixture (DEMix) layer that enables conditioning a language model (LM) on the domain of the input text.
All That's `Human' Is Not Gold: Evaluating Human Evaluation of Generated Text
Human evaluations are typically considered the gold standard in natural language generation, but as models{'} fluency improves, how well can evaluators detect and judge machine-generated text?
Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Text
To support the broad range of real machine errors that can be identified by laypeople, the ten error categories of Scarecrow -- such as redundancy, commonsense errors, and incoherence -- are identified through several rounds of crowd annotation experiments without a predefined ontology.
All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text
Human evaluations are typically considered the gold standard in natural language generation, but as models' fluency improves, how well can evaluators detect and judge machine-generated text?
Saturated Transformers are Constant-Depth Threshold Circuits
Transformers have become a standard neural network architecture for many NLP problems, motivating theoretical analysis of their power in terms of formal languages.
Scientific Language Models for Biomedical Knowledge Base Completion: An Empirical Study
Biomedical knowledge graphs (KGs) hold rich information on entities such as diseases, drugs, and genes.
Specializing Multilingual Language Models: An Empirical Study
Pretrained multilingual language models have become a common tool in transferring NLP capabilities to low-resource languages, often with adaptations.
Choose Your Own Adventure: Paired Suggestions in Collaborative Writing for Evaluating Story Generation Models
Story generation is an open-ended and subjective task, which poses a challenge for evaluating story generation models.
DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts
Despite recent advances in natural language generation, it remains challenging to control attributes of generated text.
A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers
Readers of academic research papers often read with the goal of answering specific questions.
Ranked #1 on
Evidence Selection
on QASPER
Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand?
We study whether assertions enable a system to emulate representations preserving semantic relations like equivalence.
Go Forth and Prosper: Language Modeling with Ancient Textual History
This method can improve perplexity of pretrained LMs with no updates to the LM's own parameters.
Competency Problems: On Finding and Removing Artifacts in Language Data
In this work we argue that for complex language understanding tasks, all simple feature correlations are spurious, and we formalize this notion into a class of problems which we call competency problems.
Probing Across Time: What Does RoBERTa Know and When?
Models of language trained on very large corpora have been demonstrated useful for NLP.
Finetuning Pretrained Transformers into RNNs
Specifically, we propose a swap-then-finetune procedure: in an off-the-shelf pretrained transformer, we replace the softmax attention with its linear-complexity recurrent alternative and then finetune.
Ranked #2 on
Machine Translation
on WMT2017 Chinese-English
Random Feature Attention
RFA can be used as a drop-in replacement for conventional softmax attention and offers a straightforward way of learning with recency bias through an optional gating mechanism.
Ranked #27 on
Machine Translation
on IWSLT2014 German-English
Challenges in Automated Debiasing for Toxic Language Detection
Overall, our findings show that debiasing a model trained on biased toxic language data is not as effective as simply relabeling the data to remove existing biases.
GENIE: Toward Reproducible and Standardized Human Evaluation for Text Generation
While often assumed a gold standard, effective human evaluation of text generation remains an important, open area for research.
Shortformer: Better Language Modeling using Shorter Inputs
Increasing the input length has been a driver of progress in language modeling with transformers.
Ranked #26 on
Language Modelling
on WikiText-103
Infusing Finetuning with Semantic Dependencies
For natural language processing systems, two kinds of evidence support the use of text representations from neural language models "pretrained" on large unannotated corpora: performance on application-inspired benchmarks (Peters et al., 2018, inter alia), and the emergence of syntactic abstractions in those representations (Tenney et al., 2019, inter alia).
Thinking Like a Skeptic: Defeasible Inference in Natural Language
Defeasible inference is a mode of reasoning in which an inference (X is a bird, therefore X flies) may be weakened or overturned in light of new evidence (X is a penguin).
Measuring Association Between Labels and Free-Text Rationales
In interpretable NLP, we require faithful rationales that reflect the model's decision-making process for an explained instance.
Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphs
Natural language rationales could provide intuitive, higher-level explanations that are easily understandable by humans, complementing the more broadly studied lower-level explanations based on gradients or attention weights.
Unsupervised Bitext Mining and Translation via Self-trained Contextual Embeddings
We then improve an XLM-based unsupervised neural MT system pre-trained on Wikipedia by supplementing it with pseudo-parallel text mined from the same corpus, boosting unsupervised translation performance by up to 3. 5 BLEU on the WMT'14 French-English and WMT'16 German-English tasks and outperforming the previous state-of-the-art.
The Multilingual Amazon Reviews Corpus
We present the Multilingual Amazon Reviews Corpus (MARC), a large-scale collection of Amazon reviews for multilingual text classification.
Plug and Play Autoencoders for Conditional Text Generation
Text autoencoders are commonly used for conditional generation tasks such as style transfer.
Multilevel Text Alignment with Cross-Document Attention
Text alignment finds application in tasks such as citation recommendation and plagiarism detection.
Evaluating NLP Models via Contrast Sets
Unfortunately, when a dataset has systematic gaps (e. g., annotation artifacts), these evaluations are misleading: a model can learn simple decision rules that perform well on the test set but do not capture a dataset's intended capabilities.
Parsing with Multilingual BERT, a Small Corpus, and a Small Treebank
Pretrained multilingual contextual representations have shown great success, but due to the limits of their pretraining data, their benefits do not apply equally to all language varieties.
Grounded Compositional Outputs for Adaptive Language Modeling
To our knowledge, the result is the first word-level language model with a size that does not depend on the training vocabulary.
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
We investigate the extent to which pretrained LMs can be prompted to generate toxic language, and the effectiveness of controllable text generation algorithms at preventing such toxic degeneration.
Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics
Experiments across four datasets show that these model-dependent measures reveal three distinct regions in the data map, each with pronounced characteristics.
A Mixture of h - 1 Heads is Better than h Heads
Multi-head attentive neural architectures have achieved state-of-the-art results on a variety of natural language processing tasks.
Exploring the Effect of Author and Reader Identity in Online Story Writing: the STORIESINTHEWILD Corpus.
We analyze the effect of author and reader characteristics and story writing setup on the quality of stories in a short storytelling task.
Recollection versus Imagination: Exploring Human Memory and Cognition via Neural Language Models
We introduce a measure of narrative flow and use this to examine the narratives for imagined and recalled events.
Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation
We show that the speed disadvantage for autoregressive baselines compared to non-autoregressive methods has been overestimated in three aspects: suboptimal layer allocation, insufficient speed measurement, and lack of knowledge distillation.
Multilingual and Interlingual Semantic Representations for Natural Language Processing: A Brief Introduction
We introduce the Computational Linguistics special issue on Multilingual and Interlingual Semantic Representations for Natural Language Processing.
A Mixture of $h-1$ Heads is Better than $h$ Heads
Multi-head attentive neural architectures have achieved state-of-the-art results on a variety of natural language processing tasks.
Don't Stop Pretraining: Adapt Language Models to Domains and Tasks
Language models pretrained on text from a wide variety of sources form the foundation of today's NLP.
A Formal Hierarchy of RNN Architectures
While formally extending these findings to unsaturated RNNs is left to future work, we hypothesize that the practical learnable capacity of unsaturated RNNs obeys a similar hierarchy.
The Right Tool for the Job: Matching Model and Instance Complexities
Our method presents a favorable speed/accuracy tradeoff in almost all cases, producing models which are up to five times faster than the state of the art, while preserving their accuracy.
Evaluating Models' Local Decision Boundaries via Contrast Sets
Unfortunately, when a dataset has systematic gaps (e. g., annotation artifacts), these evaluations are misleading: a model can learn simple decision rules that perform well on the test set but do not capture a dataset's intended capabilities.
Multi-View Learning for Vision-and-Language Navigation
Learning to navigate in a visual environment following natural language instructions is a challenging task because natural language instructions are highly variable, ambiguous, and under-specified.
Explaining Relationships Between Scientific Documents
We address the task of explaining relationships between two scientific documents using natural language text.
On Consequentialism and Fairness
In this paper we provide a consequentialist critique of common definitions of fairness within machine learning, as well as a machine learning perspective on consequentialism.
Improving Transformer Models by Reordering their Sublayers
Multilayer transformer networks consist of interleaved self-attention and feedforward sublayers.
Ranked #7 on
Language Modelling
on enwik8
Social Bias Frames: Reasoning about Social and Power Implications of Language
We introduce Social Bias Frames, a new conceptual formalism that aims to model the pragmatic frames in which people project social biases and stereotypes onto others.
Situating Sentence Embedders with Nearest Neighbor Overlap
As distributed approaches to natural language semantics have developed and diversified, embedders for linguistic units larger than words have come to play an increasingly important role.
Low-Resource Parsing with Crosslingual Contextualized Representations
Despite advances in dependency parsing, languages with small treebanks still present challenges.
Improving Natural Language Inference with a Pretrained Parser
We introduce a novel approach to incorporate syntax into natural language inference (NLI) models.
Knowledge Enhanced Contextual Word Representations
Contextual word representations, typically trained on unstructured, unlabeled text, do not contain any explicit grounding to real world entities and are often unable to remember facts about those entities.
Ranked #9 on
Relation Classification
on TACRED
Show Your Work: Improved Reporting of Experimental Results
Research in natural language processing proceeds, in part, by demonstrating that new models achieve superior performance (e. g., accuracy) on held-out test data, compared to previous results.
RNN Architecture Learning with Sparse Regularization
Our method also highlights the interpretable properties of rational RNNs.
PaLM: A Hybrid Parser and Language Model
We present PaLM, a hybrid parser and neural language model.
Topics to Avoid: Demoting Latent Confounds in Text Classification
Despite impressive performance on many text classification tasks, deep neural networks tend to learn frequent superficial patterns that are specific to the training data and do not always generalize well.
Shallow Syntax in Deep Water
Shallow syntax provides an approximation of phrase-syntactic structure of sentences; it can be produced with high accuracy, and is computationally cheap to obtain.
Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning
Machine comprehension of texts longer than a single sentence often requires coreference resolution.
Sentence Mover's Similarity: Automatic Evaluation for Multi-Sentence Texts
For evaluating machine-generated texts, automatic methods hold the promise of avoiding collection of human judgments, which can be expensive and time-consuming.
The Risk of Racial Bias in Hate Speech Detection
We investigate how annotators{'} insensitivity to differences in dialect can lead to racial bias in automatic hate speech detection models, potentially amplifying harm against minority populations.
Is Attention Interpretable?
Attention mechanisms have recently boosted performance on a range of NLP tasks.
Variational Pretraining for Semi-supervised Text Classification
We accompany this paper with code to pretrain and use VAMPIRE embeddings in downstream tasks.
Evaluating Gender Bias in Machine Translation
We present the first challenge set and evaluation protocol for the analysis of gender bias in machine translation (MT).
Inoculation by Fine-Tuning: A Method for Analyzing Challenge Datasets
Several datasets have recently been constructed to expose brittleness in models trained on existing benchmarks.
Linguistic Knowledge and Transferability of Contextual Representations
Contextual word representations derived from large-scale neural language models are successful across a diverse set of NLP tasks, suggesting that they encode useful and transferable features of language.
To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks
While most previous work has focused on different pretraining objectives and architectures for transfer learning, we ask how to best adapt the pretrained model to a given target task.
Measuring Online Debaters' Persuasive Skill from Text over Time
We build on a widely used model of skill in two-player games and augment it with linguistic features of a debater{'}s content.
Polyglot Contextual Representations Improve Crosslingual Transfer
We introduce Rosita, a method to produce multilingual contextual word representations by training a single language model on text from multiple languages.
Contextual Word Representations: A Contextual Introduction
This introduction aims to tell the story of how we put words into computers.
Deep Weighted Averaging Classifiers
Recent advances in deep learning have achieved impressive gains in classification accuracy on a variety of types of data, including images and text.
You May Not Need Attention
In NMT, how far can we get without attention and without separate encoding and decoding?
ATOMIC: An Atlas of Machine Commonsense for If-Then Reasoning
We present ATOMIC, an atlas of everyday commonsense reasoning, organized through 877k textual descriptions of inferential knowledge.
Syntactic Scaffolds for Semantic Structures
We introduce the syntactic scaffold, an approach to incorporating syntactic information into semantic tasks.
Neural Cross-Lingual Named Entity Recognition with Minimal Resources
To improve robustness to word order differences, we propose to use self-attention, which allows for a degree of flexibility with respect to word order.
Semantic Matching Against a Corpus: New Applications and Methods
We consider the case of a domain expert who wishes to explore the extent to which a particular idea is expressed in a text collection.
Rational Recurrences
We characterize this connection formally, defining rational recurrences to be recurrent hidden state update functions that can be written as the Forward calculation of a finite set of WFSAs.
Bridging CNNs, RNNs, and Weighted Finite-State Machines
Recurrent and convolutional neural networks comprise two distinct families of models that have proven to be useful for encoding natural language utterances.
Discovering Phonesthemes with Sparse Regularization
We introduce a simple method for extracting non-arbitrary form-meaning representations from a collection of semantic vectors.
Neural Text Generation in Stories Using Entity Representations as Context
We introduce an approach to neural text generation that explicitly represents entities mentioned in the text.
The Importance of Calibration for Estimating Proportions from Annotations
Estimating label proportions in a target corpus is a type of measurement that is useful for answering certain types of social-scientific questions.
LSTMs Exploit Linguistic Attributes of Data
While recurrent neural networks have found success in a variety of natural language processing applications, they are general models of sequential data.
Toward Abstractive Summarization Using Semantic Representations
We present a novel abstractive summarization framework that draws on the recent development of a treebank for the Abstract Meaning Representation (AMR).
Event2Mind: Commonsense Inference on Events, Intents, and Reactions
We investigate a new commonsense inference task: given an event described in a short free-form text ("X drinks coffee in the morning"), a system reasons about the likely intents ("X wants to stay awake") and reactions ("X feels alert") of the event's participants.
Ranked #1 on
Common Sense Reasoning
on Event2Mind test
SoPa: Bridging CNNs, RNNs, and Weighted Finite-State Machines
Recurrent and convolutional neural networks comprise two distinct families of models that have proven to be useful for encoding natural language utterances.
Explainable artificial intelligence
General Classification
+3
Backpropagating through Structured Argmax using a SPIGOT
We introduce the structured projection of intermediate gradients optimization technique (SPIGOT), a new method for backpropagating through neural networks that include hard-decision structured predictions (e. g., parsing) in intermediate layers.
Sounding Board: A User-Centric and Content-Driven Social Chatbot
We present Sounding Board, a social chatbot that won the 2017 Amazon Alexa Prize.
Parsing Tweets into Universal Dependencies
Nonetheless, using the new treebank, we build a pipeline system to parse raw tweets into UD.
Ranked #2 on
Dependency Parsing
on Tweebank
Learning Joint Semantic Parsers from Disjoint Data
We present a new approach to learning semantic parsers from multiple datasets, even when the target semantic formalisms are drastically different, and the underlying corpora do not overlap.
Annotation Artifacts in Natural Language Inference Data
Large-scale datasets for natural language inference are created by presenting crowd workers with a sentence (premise), and asking them to generate three new sentences (hypotheses) that it entails, contradicts, or is logically neutral with respect to.
"You are no Jack Kennedy": On Media Selection of Highlights from Presidential Debates
We first examine the effect of wording and propose a binary classification framework that controls for both the speaker and the debate situation.
Dynamic Entity Representations in Neural Language Models
Understanding a long document requires tracking how entities are introduced and evolve over time.
End-to-End Neural Segmental Models for Speech Recognition
Segmental models are an alternative to frame-based models for sequence prediction, where hypothesized path weights are based on entire segment scores rather than a single frame at a time.
Frame-Semantic Parsing with Softmax-Margin Segmental RNNs and a Syntactic Scaffold
We present a new, efficient frame-semantic parser that labels semantic arguments to FrameNet predicates.
Open Loop Hyperparameter Optimization and Determinantal Point Processes
Driven by the need for parallelizable hyperparameter optimization methods, this paper studies \emph{open loop} search methods: sequences that are predetermined and can be generated before a single configuration is evaluated.
Greedy Transition-Based Dependency Parsing with Stack LSTMs
During training, dynamic oracles alternate between sampling parser states from the training data and from the model as it is being learned, making the model more robust to the kinds of errors that will be made at test time.
Neural Models for Documents with Metadata
Most real-world document collections involve various types of metadata, such as author, source, and date, and yet the most commonly-used approaches to modeling text corpora ignore this information.
Friendships, Rivalries, and Trysts: Characterizing Relations between Ideas in Texts
Combining two statistics --- cooccurrence within documents and prevalence correlation over time --- our approach reveals a number of different ways in which ideas can cooperate and compete.
Deep Multitask Learning for Semantic Dependency Parsing
We present a deep neural architecture that parses sentences into three semantic dependency graph formalisms.
Story Cloze Task: UW NLP System
This paper describes University of Washington NLP{'}s submission for the Linking Models of Lexical, Sentential and Discourse-level Semantics (LSDSem 2017) shared task{---}the Story Cloze Task.
Multitask Learning with CTC and Segmental CRF for Speech Recognition
Segmental conditional random fields (SCRFs) and connectionist temporal classification (CTC) are two sequence labeling methods used for end-to-end training of speech recognition models.
The Effect of Different Writing Tasks on Linguistic Style: A Case Study of the ROC Story Cloze Task
A writer's style depends not just on personal traits but also on her intent and mental state.
What Do Recurrent Neural Network Grammars Learn About Syntax?
We investigate what information they learn, from a linguistic perspective, through various ablations to the model and the data, and by augmenting the model with an attention mechanism (GA-RNNG) to enable closer inspection.
Ranked #20 on
Constituency Parsing
on Penn Treebank
Character Sequence Models for ColorfulWords
We present a neural network architecture to predict a point in color space from the sequence of characters in the color's name.
Distilling an Ensemble of Greedy Dependency Parsers into One MST Parser
We introduce two first-order graph-based dependency parsers achieving a new state of the art.
Ranked #17 on
Dependency Parsing
on Penn Treebank
Hierarchical Character-Word Models for Language Identification
Social media messages' brevity and unconventional spelling pose a challenge to language identification.
Greedy, Joint Syntactic-Semantic Parsing with Stack LSTMs
We present a transition-based parser that jointly produces syntactic and semantic dependencies.
Training with Exploration Improves a Greedy Stack-LSTM Parser
We adapt the greedy Stack-LSTM dependency parser of Dyer et al. (2015) to support a training-with-exploration procedure using dynamic oracles(Goldberg and Nivre, 2013) instead of cross-entropy minimization.
Ranked #2 on
Chinese Dependency Parsing
on Chinese Pennbank
Segmental Recurrent Neural Networks for End-to-end Speech Recognition
This model connects the segmental conditional random field (CRF) with a recurrent neural network (RNN) used for feature extraction.
Ranked #16 on
Speech Recognition
on TIMIT
Recurrent Neural Network Grammars
We introduce recurrent neural network grammars, probabilistic models of sentences with explicit phrase structure.
Ranked #25 on
Constituency Parsing
on Penn Treebank
Massively Multilingual Word Embeddings
We introduce new methods for estimating and evaluating embeddings of words in more than fifty languages in a single shared embedding space.
Many Languages, One Parser
We train one multilingual model for dependency parsing and use it to parse sentences in several languages.
Annotating Character Relationships in Literary Texts
We present a dataset of manually annotated relationships between characters in literary texts, in order to support the training and evaluation of automatic methods for relation type prediction in this domain (Makazhanov et al., 2014; Kokkinakis, 2013) and the broader computational analysis of literary character (Elson et al., 2010; Bamman et al., 2014; Vala et al., 2015; Flekova and Gurevych, 2015).
Segmental Recurrent Neural Networks
Representations of the input segments (i. e., contiguous subsequences of the input) are computed by encoding their constituent tokens using bidirectional recurrent neural nets, and these "segment embeddings" are used to define compatibility scores with output labels.
Improved Transition-Based Parsing by Modeling Characters instead of Words with LSTMs
We present extensions to a continuous-state dependency parsing method that makes it applicable to morphologically rich languages.
