Search Results for author:
Found 356 papers, 216 papers with code
Systematic Inequalities in Language Technology Performance across the World’s Languages
Natural language processing (NLP) systems have become a central technology in communication, education, medicine, artificial intelligence, and many other domains of research and development.
When is Wall a Pared and when a Muro?: Extracting Rules Governing Lexical Selection
Learning fine-grained distinctions between vocabulary items is a key challenge in learning a new language.
Findings of the WMT 2020 Shared Task on Machine Translation Robustness
We report the findings of the second edition of the shared task on improving robustness in Machine Translation (MT).
Findings of the AmericasNLP 2021 Shared Task on Open Machine Translation for Indigenous Languages of the Americas
This paper presents the results of the 2021 Shared Task on Open Machine Translation for Indigenous Languages of the Americas.
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalisation
However, these broad-coverage benchmarks have been mostly limited to English, and despite an increasing interest in multilingual models, a benchmark that enables the comprehensive evaluation of such methods on a diverse range of languages and tasks is still missing.
Ranked #1 on
Zero-Shot Cross-Lingual Transfer
on XTREME
(AVG metric)
CMU’s IWSLT 2022 Dialect Speech Translation System
We use additional paired Modern Standard Arabic data (MSA) to directly improve the speech recognition (ASR) and machine translation (MT) components of our cascaded systems.
Project MAIA: Multilingual AI Agent Assistant
This paper presents the Multilingual Artificial Intelligence Agent Assistant (MAIA), a project led by Unbabel with the collaboration of CMU, INESC-ID and IT Lisbon.
Better Synthetic Data by Retrieving and Transforming Existing Datasets
Recent work has studied prompt-driven synthetic data generation using large language models, but these generated datasets tend to lack complexity and diversity.
VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?
Multimodal Large Language models (MLLMs) have shown promise in web-related tasks, but evaluating their performance in the web domain remains a challenge due to the lack of comprehensive benchmarks.
CMULAB: An Open-Source Framework for Training and Deployment of Natural Language Processing Models
Effectively using Natural Language Processing (NLP) tools in under-resourced languages requires a thorough understanding of the language itself, familiarity with the latest models and training methodologies, and technical expertise to deploy these models.
An Incomplete Loop: Deductive, Inductive, and Abductive Learning in Large Language Models
Modern language models (LMs) can learn to perform new tasks in different ways: in instruction following, the target task is described explicitly in natural language; in few-shot prompting, the task is specified implicitly with a small number of examples; in instruction inference, LMs are presented with in-context examples and are then prompted to generate a natural language task description before making predictions.
Evaluating Text-to-Visual Generation with Image-to-Text Generation
For instance, the widely-used CLIPScore measures the alignment between a (generated) image and text prompt, but it fails to produce reliable scores for complex prompts involving compositions of objects, attributes, and relations.
An image speaks a thousand words, but can everyone listen? On translating images for cultural relevance
First, we build three pipelines comprising state-of-the-art generative models to do the task.
Wav2Gloss: Generating Interlinear Glossed Text from Speech
Thousands of the world's languages are in danger of extinction--a tremendous threat to cultural identities and human language diversity.
What Are Tools Anyway? A Survey from the Language Model Perspective
Language models (LMs) are powerful yet mostly for text generation tasks.
RAGGED: Towards Informed Design of Retrieval Augmented Generation Systems
RAGGED offers further insights into LMs' context utilization habits, where we find that encoder-decoder models rely more on contexts and are thus more sensitive to retrieval quality, while decoder-only models tend to rely on knowledge memorized during training.
SOTOPIA-$π$: Interactive Learning of Socially Intelligent Language Agents
Motivated by this gap, we propose an interactive learning method, SOTOPIA-$\pi$, improving the social intelligence of language agents.
GlossLM: Multilingual Pretraining for Low-Resource Interlinear Glossing
A key aspect of language documentation is the creation of annotated text in a format such as interlinear glossed text (IGT), which captures fine-grained morphosyntactic analyses in a morpheme-by-morpheme format.
What Is Missing in Multilingual Visual Reasoning and How to Fix It
NLP models today strive for supporting multiple languages and modalities, improving accessibility for diverse users.
Repetition Improves Language Model Embeddings
In this work, we address an architectural limitation of autoregressive models: token embeddings cannot contain information from tokens that appear later in the input.
Instruction-tuned Language Models are Better Knowledge Learners
The standard recipe for doing so involves continued pre-training on new documents followed by instruction-tuning on question-answer (QA) pairs.
Everybody Prune Now: Structured Pruning of LLMs with only Forward Passes
Given the generational gap in available hardware between lay practitioners and the most endowed institutions, LLMs are becoming increasingly inaccessible as they grow in size.
Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate
Despite the utility of Large Language Models (LLMs) across a wide range of tasks and scenarios, developing a method for reliably evaluating LLMs across varied contexts continues to be challenging.
VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks
Through extensive quantitative and qualitative analysis, we identify several limitations of text-only LLM agents, and reveal gaps in the capabilities of state-of-the-art multimodal language agents.
TroVE: Inducing Verifiable and Efficient Toolboxes for Solving Programmatic Tasks
Language models (LMs) can solve tasks such as answering questions about tables or images by writing programs.
Fine-grained Hallucination Detection and Editing for Language Models
On our benchmark, our automatic and human evaluations show that FAVA significantly outperforms ChatGPT and GPT-4 on fine-grained hallucination detection, and edits suggested by FAVA improve the factuality of LM-generated text.
An In-depth Look at Gemini's Language Abilities
The recently released Google Gemini class of models are the first to comprehensively report results that rival the OpenAI GPT series across a wide variety of tasks.
Alignment for Honesty
Recent research has made significant strides in applying alignment techniques to enhance the helpfulness and harmlessness of large language models (LLMs) in accordance with human intentions.
Multitask Learning Can Improve Worst-Group Outcomes
We primarily consider the standard setting of fine-tuning a pre-trained model, where, following recent work \citep{gururangan2020don, dery2023aang}, we multitask the end task with the pre-training objective constructed from the end task data itself.
Program-Aided Reasoners (better) Know What They Know
Our analysis uncovers that prompting styles that produce lesser diversity in generations also have more calibrated results, and thus we also experiment with inducing lower generation diversity using temperature scaling and find that for certain temperatures, PAL is not only more accurate but is also more calibrated than COT.
Divergences between Language Models and Human Brains
In this work, we systematically explore the divergences between human and machine language processing by examining the differences between LM representations and human brain responses to language as measured by Magnetoencephalography (MEG) across two datasets in which subjects read and listened to narrative stories.
Learning to Filter Context for Retrieval-Augmented Generation
To alleviate these problems, we propose FILCO, a method that improves the quality of the context provided to the generator by (1) identifying useful context based on lexical and information-theoretic approaches, and (2) training context filtering models that can filter retrieved contexts at test time.
DeMuX: Data-efficient Multilingual Learning
In this paper, we introduce DEMUX, a framework that prescribes the exact data-points to label from vast amounts of unlabelled multilingual data, having unknown degrees of overlap with the target set.
Do LLMs exhibit human-like response biases? A case study in survey design
As large language models (LLMs) become more capable, there is growing excitement about the possibility of using LLMs as proxies for humans in real-world tasks where subjective labels are desired, such as in surveys and opinion polling.
Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
This is challenging because i) it requires that such experts be accessible and have the necessary resources, and ii) describing all the intricacies of a language is time-consuming and prone to omission.
SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents
We present SOTOPIA, an open-ended environment to simulate complex social interactions between artificial agents and evaluate their social intelligence.
Crossing the Threshold: Idiomatic Machine Translation through Retrieval Augmentation and Loss Weighting
Idioms are common in everyday language, but often pose a challenge to translators because their meanings do not follow from the meanings of their parts.
It's MBR All the Way Down: Modern Generation Techniques Through the Lens of Minimum Bayes Risk
Minimum Bayes Risk (MBR) decoding is a method for choosing the outputs of a machine learning system based not on the output with the highest probability, but the output with the lowest risk (expected error) among multiple candidates.
ChatGPT MT: Competitive for High- (but not Low-) Resource Languages
Without published experimental evidence on the matter, it is difficult for speakers of the world's diverse languages to know how and whether they can use LLMs for their languages.
Prompt2Model: Generating Deployable Models from Natural Language Instructions
In this paper, we propose Prompt2Model, a general-purpose method that takes a natural language task description like the prompts provided to LLMs, and uses it to train a special-purpose model that is conducive to deployment.
The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation
Automatic evaluation of machine translation (MT) is a critical tool driving the rapid iterative development of MT systems.
WebArena: A Realistic Web Environment for Building Autonomous Agents
Building upon our environment, we release a set of benchmark tasks focusing on evaluating the functional correctness of task completions.
FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios
With the above challenges in mind, in this paper, we propose FacTool, a task and domain agnostic framework for detecting factual errors of texts generated by large language models (e. g., ChatGPT).
Improving Factuality of Abstractive Summarization via Contrastive Reward Learning
Modern abstractive summarization models often generate summaries that contain hallucinated or contradictory information.
Large Language Models Enable Few-Shot Clustering
In this paper, we ask whether a large language model can amplify an expert's guidance to enable query-efficient, few-shot semi-supervised text clustering.
Neural Machine Translation for the Indigenous Languages of the Americas: An Introduction
Neural models have drastically advanced state of the art for machine translation (MT) between high-resource languages.
Multi-Dimensional Evaluation of Text Summarization with In-Context Learning
Most frameworks that perform such multi-dimensional evaluation require training on large manually or synthetically generated datasets.
Syntax and Semantics Meet in the "Middle": Probing the Syntax-Semantics Interface of LMs Through Agentivity
Recent advances in large language models have prompted researchers to examine their abilities across a variety of linguistic tasks, but little has been done to investigate how models handle the interactions in meaning across words and larger syntactic forms -- i. e. phenomena at the intersection of syntax and semantics.
DataFinder: Scientific Dataset Recommendation from Natural Language Descriptions
Using this data, we compare various information retrieval algorithms on our test set and present a superior bi-encoder retriever for text-based dataset recommendation.
Multi-lingual and Multi-cultural Figurative Language Understanding
Figurative language permeates human communication, but at the same time is relatively understudied in NLP.
GlobalBench: A Benchmark for Global Progress in Natural Language Processing
Despite the major advances in NLP, significant disparities in NLP system performance across languages still exist.
Solving NLP Problems through Human-System Collaboration: A Discussion-based Approach
Humans work together to solve common problems by having discussions, explaining, and agreeing or disagreeing with each other.
Active Retrieval Augmented Generation
In this work, we provide a generalized view of active retrieval augmented generation, methods that actively decide when and what to retrieve across the course of the generation.
Unlimiformer: Long-Range Transformers with Unlimited Length Input
This kNN index can be kept on either the GPU or CPU memory and queried in sub-linear time; this way, we can index practically unlimited input sequences, while every attention head in every decoder layer retrieves its top-k keys, instead of attending to every key.
Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation
Many recent advances in natural language generation have been fueled by training large language models on internet-scale data.
A Gold Standard Dataset for the Reviewer Assignment Problem
We address this challenge by collecting a novel dataset of similarity scores that we release to the research community.
Computational Language Acquisition with Theory of Mind
We also find some evidence that increasing task difficulty in the training process results in more fluent and precise utterances in evaluation.
User-Centric Evaluation of OCR Systems for Kwak'wala
There has been recent interest in improving optical character recognition (OCR) for endangered languages, particularly because a large number of documents and books in these languages are not in machine-readable formats.
Optical Character Recognition
Optical Character Recognition (OCR)
Learning Performance-Improving Code Edits
Next, we propose a broad range of adaptation strategies for code optimization; for prompting, these include retrieval-based few-shot prompting and chain-of-thought, and for finetuning, these include performance-conditioned generation and synthetic data augmentation based on self-play.
Cross-Modal Fine-Tuning: Align then Refine
Fine-tuning large-scale pretrained models has led to tremendous progress in well-studied modalities such as vision and NLP.
CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code
We release five language-specific pretrained models to use with our publicly available code.
Why do Nearest Neighbor Language Models Work?
Language models (LMs) compute the probability of a text by sequentially computing a representation of an already-seen context and using this representation to predict the next word.
EXCALIBUR: Encouraging and Evaluating Embodied Exploration
Humans constantly explore and learn about their environment out of curiosity, gather information, and update their models of the world.
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
Contrastive learning has been successfully used for retrieval of semantically aligned sentences, but it often requires large batch sizes or careful engineering to work well.
Execution-Based Evaluation for Open-Domain Code Generation
To extend the scope of coding queries to more realistic settings, we propose ODEX, the first Open-Domain EXecution-based natural language (NL) to Python code generation dataset.
NusaCrowd: Open Source Initiative for Indonesian NLP Resources
We present NusaCrowd, a collaborative initiative to collect and unify existing resources for Indonesian languages, including opening access to previously non-public resources.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Searching for Effective Multilingual Fine-Tuning Methods: A Case Study in Summarization
Recently, a large number of tuning strategies have been proposed to adapt pre-trained language models to downstream tasks.
T5Score: Discriminative Fine-tuning of Generative Evaluation Metrics
Both have their advantages; discriminative metrics are able to directly optimize for the problem of distinguishing between good and bad outputs, while generative metrics can be trained using abundant raw text.
Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformer
Systems for knowledge-intensive tasks such as open-domain question answering (QA) usually consist of two stages: efficient retrieval of relevant documents from a large corpus and detailed reading of the selected documents to generate answers.
Ranked #1 on
Passage Retrieval
on Natural Questions
PAL: Program-aided Language Models
Much of this success can be attributed to prompting methods such as "chain-of-thought'', which employ LLMs for both understanding the problem description by decomposing it into steps, as well as solving each step of the problem.
Ranked #17 on
Arithmetic Reasoning
on GSM8K
English Contrastive Learning Can Learn Universal Cross-lingual Sentence Embeddings
The performance can be further enhanced when cross-lingual NLI data is available.
DiffusER: Discrete Diffusion via Edit-based Reconstruction
In text generation, models that generate text from scratch one token at a time are currently the dominant paradigm.
He Said, She Said: Style Transfer for Shifting the Perspective of Dialogues
As a sample application, we demonstrate that applying perspective shifting to a dialogue summarization dataset (SAMSum) substantially improves the zero-shot performance of extractive news summarization models on this data.
MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognition
African languages are spoken by over a billion people, but are underrepresented in NLP research and development.
Language Models of Code are Few-Shot Commonsense Learners
In all these natural language tasks, we show that using our approach, a code generation LM (CODEX) outperforms natural-LMs that are fine-tuned on the target task (e. g., T5) and other strong LMs such as GPT-3 in the few-shot setting.
A Multi-dimensional Evaluation of Tokenizer-free Multilingual Pretrained Models
Recent work on tokenizer-free multilingual pretrained models show promising results in improving cross-lingual transfer and reducing engineering overhead (Clark et al., 2022; Xue et al., 2022).
CTC Alignments Improve Autoregressive Translation
Connectionist Temporal Classification (CTC) is a widely used approach for automatic speech recognition (ASR) that performs conditionally independent monotonic alignment.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Understanding and Improving Zero-shot Multi-hop Reasoning in Generative Question Answering
In sum, these results demonstrate that multi-hop reasoning does not emerge naturally in generative QA models, but can be encouraged by advances in training or modeling techniques.
Are Representations Built from the Ground Up? An Empirical Examination of Local Composition in Language Models
We find that the representation of a parent phrase can be predicted with some accuracy given an affine transformation of its children.
Mega: Moving Average Equipped Gated Attention
The design choices in the Transformer attention mechanism, including weak inductive bias and quadratic computational complexity, have limited its application for modeling long sequences.
Ranked #1 on
Long-range modeling
on LRA
KGxBoard: Explainable and Interactive Leaderboard for Evaluation of Knowledge Graph Completion Models
Knowledge Graphs (KGs) store information in the form of (head, predicate, tail)-triples.
DocPrompting: Generating Code by Retrieving the Docs
Publicly available source-code libraries are continuously growing and changing.
OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering
The information in tables can be an important complement to text, making table-based question answering (QA) systems of great value.
Ranked #7 on
Semantic Parsing
on WikiTableQuestions
Building African Voices
This paper focuses on speech synthesis for low-resourced African languages, from corpus creation to sharing and deploying the Text-to-Speech (TTS) systems.
Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning
This process is challenging because i) it requires that such experts be accessible and have the necessary resources, and ii) even if there are such experts, describing all the intricacies of a language is time-consuming and prone to omission.
AANG: Automating Auxiliary Learning
Auxiliary objectives, supplementary learning signals that are introduced to help aid learning on data-starved or highly complex end-tasks, are commonplace in machine learning.
Learning to Model Editing Processes
We introduce baseline results and metrics on this task, finding that modeling editing processes improves performance on a variety of axes on both our proposed task and related downstream tasks compared to previous single-step models of edits.
Table Retrieval May Not Necessitate Table-specific Model Design
In this work, we focus on the task of table retrieval, and ask: "is table-specific model design necessary for table retrieval, or can a simpler text-based model be effectively used to achieve a similar result?"
Quality-Aware Decoding for Neural Machine Translation
Despite the progress in machine translation quality estimation and evaluation in the last years, decoding in neural machine translation (NMT) is mostly oblivious to this and centers around finding the most probable translation according to the model (MAP decoding), approximated with beam search.
Prompt Consistency for Zero-Shot Task Generalization
One of the most impressive results of recent NLP history is the ability of pre-trained language models to solve new tasks in a zero-shot setting.
Testing the Ability of Language Models to Interpret Figurative Language
Figurative and metaphorical language are commonplace in discourse, and figurative expressions play an important role in communication and cognition.
Learning to Scaffold: Optimizing Model Explanations for Teaching
In this work, leveraging meta-learning techniques, we extend this idea to improve the quality of the explanations themselves, specifically by optimizing explanations such that student models more effectively learn to simulate the original model.
Distributionally Robust Models with Parametric Likelihood Ratios
As machine learning models are deployed ever more broadly, it becomes increasingly important that they are not only able to perform well on their training distribution, but also yield accurate predictions when confronted with distribution shift.
BRIO: Bringing Order to Abstractive Summarization
Abstractive summarization models are commonly trained using maximum likelihood estimation, which assumes a deterministic (one-point) target distribution in which an ideal model will assign all the probability mass to the reference summary.
Ranked #2 on
Text Summarization
on X-Sum
AUTOLEX: An Automatic Framework for Linguistic Exploration
Each language has its own complex systems of word, phrase, and sentence construction, the guiding principles of which are often summarized in grammar descriptions for the consumption of linguists or language learners.
Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation
The performance of multilingual pretrained models is highly dependent on the availability of monolingual or parallel text present in a target language.
MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages
While there has been a recent burgeoning of applications at the intersection of natural and programming languages, such as code generation and code summarization, these applications are usually English-centric.
Show Me More Details: Discovering Hierarchies of Procedures from Semi-structured Web Data
To this end, we develop a simple and efficient method that links steps (e. g., "purchase a camera") in an article to other articles with similar goals (e. g., "how to choose a camera"), recursively constructing the KB.
A Systematic Evaluation of Large Language Models of Code
We aim to fill in some of these blanks through a systematic evaluation of the largest existing models: Codex, GPT-J, GPT-Neo, GPT-NeoX-20B, and CodeParrot, across various programming languages.
DataLab: A Platform for Data Analysis and Intervention
Despite data's crucial role in machine learning, most existing tools and research tend to focus on systems on top of existing data rather than how to interpret and manipulate data.
Interpreting Language Models with Contrastive Explanations
Model interpretability methods are often used to explain NLP model decisions on tasks such as text classification, where the output space is relatively small.
Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval
Retrieval-based language models (R-LM) model the probability of natural language text by combining a standard language model (LM) with examples retrieved from an external datastore at test time.
Explain, Edit, and Understand: Rethinking User Study Design for Evaluating Model Explanations
Through our evaluation, we observe that for a linear bag-of-words model, participants with access to the feature coefficients during training are able to cause a larger reduction in model confidence in the testing phase when compared to the no-explanation control.
VarCLR: Variable Semantic Representation Pre-training via Contrastive Learning
Machine learning-based program analysis methods use variable name representations for a wide range of tasks, such as suggesting new variable names and bug detection.
DEEP: DEnoising Entity Pre-training for Neural Machine Translation
It has been shown that machine translation models usually generate poor translations for named entities that are infrequent in the training corpus.
Lexically Aware Semi-Supervised Learning for OCR Post-Correction
In addition, to enforce consistency in the recognized vocabulary, we introduce a lexically-aware decoding method that augments the neural post-correction model with a count-based language model constructed from the recognized texts, implemented using weighted finite-state automata (WFSA) for efficient and effective decoding.
On The Ingredients of an Effective Zero-shot Semantic Parser
Semantic parsers map natural language utterances into meaning representations (e. g., programs).
Breaking Down Multilingual Machine Translation
While multilingual training is now an essential ingredient in machine translation (MT) systems, recent work has demonstrated that it has different effects in different multilingual settings, such as many-to-one, one-to-many, and many-to-many learning.
Systematic Inequalities in Language Technology Performance across the World's Languages
Natural language processing (NLP) systems have become a central technology in communication, education, medicine, artificial intelligence, and many other domains of research and development.
Towards a Unified View of Parameter-Efficient Transfer Learning
Furthermore, our unified framework enables the transfer of design elements across different approaches, and as a result we are able to instantiate new parameter-efficient fine-tuning methods that tune less parameters than previous methods while being more effective, achieving comparable results to fine-tuning all parameters on all four tasks.
Capturing Structural Locality in Non-parametric Language Models
Structural locality is a ubiquitous feature of real-world datasets, wherein data points are organized into local hierarchies.
Symmetric Machine Theory of Mind
Theory of mind (ToM), the ability to understand others' thoughts and desires, is a cornerstone of human intelligence.
Learning to Superoptimize Real-world Programs
We also report that SILO's rate of superoptimization on our test set is over five times that of a standard policy gradient approach and a model pre-trained on compiler optimization demonstration.
Dependency Induction Through the Lens of Visual Perception
Our experiments find that concreteness is a strong indicator for learning dependency grammars, improving the direct attachment score (DAS) by over 50\% as compared to state-of-the-art models trained on pure text.
Procedures as Programs: Hierarchical Control of Situated Agents through Natural Language
When humans conceive how to perform a particular task, they do so hierarchically: splitting higher-level tasks into smaller sub-tasks.
When Does Translation Require Context? A Data-driven, Multilingual Exploration
Although proper handling of discourse significantly contributes to the quality of machine translation (MT), these improvements are not adequately measured in common translation quality metrics.
Should We Be Pre-training? An Argument for End-task Aware Training as an Alternative
In most settings of practical concern, machine learning practitioners know in advance what end-task they wish to boost with auxiliary tasks.
When is Wall a Pared and when a Muro? -- Extracting Rules Governing Lexical Selection
Learning fine-grained distinctions between vocabulary items is a key challenge in learning a new language.
Efficient Test Time Adapter Ensembling for Low-resource Language Varieties
Adapters are light-weight modules that allow parameter-efficient fine-tuning of pretrained models.
AfroMT: Pretraining Strategies and Reproducible Benchmarks for Translation of 8 African Languages
Reproducible benchmarks are crucial in driving progress of machine translation research.
Distributionally Robust Multilingual Machine Translation
Multilingual neural machine translation (MNMT) learns to translate multiple language pairs with a single model, potentially improving both the accuracy and the memory-efficiency of deployed models.
Efficient Nearest Neighbor Language Models
Non-parametric neural language models (NLMs) learn predictive distributions of text utilizing an external datastore, which allows them to learn through explicitly memorizing the training datapoints.
Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing
This paper surveys and organizes research works in a new paradigm in natural language processing, which we dub "prompt-based learning".
Few-shot Language Coordination by Modeling Theory of Mind
Positive results from our experiments hint at the importance of explicitly modeling communication as a socio-pragmatic progress.
BARTScore: Evaluating Generated Text as Text Generation
In this work, we conceptualize the evaluation of generated text as a text generation problem, modeled using pre-trained sequence-to-sequence models.
Phrase-level Active Learning for Neural Machine Translation
Neural machine translation (NMT) is sensitive to domain shift.
Examining and Combating Spurious Features under Distribution Shift
Group distributionally robust optimization (DRO) provides an effective tool to alleviate covariate shift by minimizing the worst-case training loss over a set of pre-defined groups.
CitationIE: Leveraging the Citation Graph for Scientific Information Extraction
Automatically extracting key information from scientific documents has the potential to help scientists work more efficiently and accelerate the pace of scientific progress.
Compositional Generalization for Neural Semantic Parsing via Span-level Supervised Attention
We describe a span-level supervised attention loss that improves compositional generalization in semantic parsers.
Data Augmentation for Sign Language Gloss Translation
Sign language translation (SLT) is often decomposed into video-to-gloss recognition and gloss-to-text translation, where a gloss is a sequence of transcribed spoken-language words in the order in which they are signed.
Data Augmentation
Low-Resource Neural Machine Translation
+3
Do Context-Aware Translation Models Pay the Right Attention?
Are models paying large amounts of attention to the same context?
Measuring and Increasing Context Usage in Context-Aware Machine Translation
Recent work in neural machine translation has demonstrated both the necessity and feasibility of using inter-sentential context -- context from sentences other than those currently being translated.
Paraphrastic Representations at Scale
We train these models on large amounts of data, achieving significantly improved performance from the original papers proposing the methods on a suite of monolingual semantic similarity, cross-lingual semantic similarity, and bitext mining tasks.
AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languages
Continued pretraining offers improvements, with an average accuracy of 44. 05%.
MetaXL: Meta Representation Transformation for Low-resource Cross-lingual Learning
Extensive experiments on real-world low-resource languages - without access to large-scale monolingual corpora or large amounts of labeled data - for tasks like cross-lingual sentiment analysis and named entity recognition show the effectiveness of our approach.
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
While a sizeable gap to human-level performance remains, improvements have been easier to achieve in some tasks than in others.
ExplainaBoard: An Explainable Leaderboard for NLP
In this paper, we present a new conceptualization and implementation of NLP evaluation: the ExplainaBoard, which in addition to inheriting the functionality of the standard leaderboard, also allows researchers to (i) diagnose strengths and weaknesses of a single system (e. g.~what is the best-performing system bad at?)
Phoneme Recognition through Fine Tuning of Phonetic Representations: a Case Study on Luhya Language Varieties
Models pre-trained on multiple languages have shown significant promise for improving speech recognition, particularly for low-resource languages.
Evaluating the Morphosyntactic Well-formedness of Generated Texts
Text generation systems are ubiquitous in natural language processing applications.
MasakhaNER: Named Entity Recognition for African Languages
We take a step towards addressing the under-representation of the African continent in NLP research by creating the first large publicly available high-quality dataset for named entity recognition (NER) in ten African languages, bringing together a variety of stakeholders.
Modeling the Second Player in Distributionally Robust Optimization
Distributionally robust optimization (DRO) provides a framework for training machine learning models that are able to perform well on a collection of related data distributions (the "uncertainty set").
Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models
Specifically, we focus on multilingual text-to-video search and propose a Transformer-based model that learns contextualized multilingual multimodal embeddings.
Multi-view Subword Regularization
Multilingual pretrained representations generally rely on subword segmentation algorithms to create a shared multilingual vocabulary.
Meta Back-translation
Back-translation is an effective strategy to improve the performance of Neural Machine Translation~(NMT) by generating pseudo-parallel data.
Towards More Fine-grained and Reliable NLP Performance Prediction
We perform an analysis of four types of NLP tasks, and both demonstrate the feasibility of fine-grained performance prediction and the necessity to perform reliability analysis for performance prediction methods in the future.
Can We Automate Scientific Reviewing?
The rapid development of science and technology has been accompanied by an exponential growth in peer-reviewed scientific publications.
Learning Structural Edits via Incremental Tree Transformations
To show the unique benefits of modeling tree edits directly, we further propose a novel edit encoder for learning to represent edits, as well as an imitation learning method that allows the editor to be more robust.
In-IDE Code Generation from Natural Language: Promise and Challenges
A great part of software development involves conceptualizing or communicating the underlying procedures and logic that needs to be expressed in programs.
Code Generation
Data Visualization
Software Engineering
Word Alignment by Fine-tuning Embeddings on Parallel Corpora
In addition, we demonstrate that we are able to train multilingual word aligners that can obtain robust performance on different language pairs.
How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering
We examine this question from the point of view of calibration, the property of a probabilistic model's predicted probabilities actually being well correlated with the probabilities of correctness.
Endangered Languages meet Modern NLP
This tutorial will focus on NLP for endangered languages documentation and revitalization.
Evaluating Explanations: How much do explanations from the teacher aid students?
While many methods purport to explain predictions by highlighting salient features, what aims these explanations serve and how they ought to be evaluated often go unstated.
Automatic Interlinear Glossing for Under-Resourced Languages Leveraging Translations
Interlinear Glossed Text (IGT) is a widely used format for encoding linguistic information in language documentation projects and scholarly papers.
Decoding and Diversity in Machine Translation
This makes human-level BLEU a misleading benchmark in that modern MT systems cannot approach human-level BLEU while simultaneously maintaining human-level translation diversity.
WikiAsp: A Dataset for Multi-domain Aspect-based Summarization
In this paper, we propose WikiAsp, a large-scale dataset for multi-domain aspect-based summarization that attempts to spur research in the direction of open-domain aspect-based summarization.
Interpretable Multi-dataset Evaluation for Named Entity Recognition
With the proliferation of models for natural language processing tasks, it is even harder to understand the differences between models and their relative merits.
OCR Post Correction for Endangered Language Texts
There is little to no data available to build natural language processing models for most endangered languages.
Detecting Hallucinated Content in Conditional Neural Sequence Generation
Neural sequence models can generate highly fluent sentences, but recent studies have also shown that they are also prone to hallucinate additional content not supported by the input.
Weakly- and Semi-supervised Evidence Extraction
For many prediction tasks, stakeholders desire not only predictions but also supporting evidence that a human can use to verify its correctness.
Reducing Confusion in Active Learning for Part-Of-Speech Tagging
Active learning (AL) uses a data selection algorithm to select useful training samples to minimize annotation cost.
On Learning Text Style Transfer with Direct Rewards
In most cases, the lack of parallel corpora makes it impossible to directly train supervised models for the text style transfer task.
NeuSpell: A Neural Spelling Correction Toolkit
We introduce NeuSpell, an open-source toolkit for spelling correction in English.
GSum: A General Framework for Guided Neural Abstractive Summarization
Neural abstractive summarization models are flexible and can produce coherent summaries, but they are sometimes unfaithful and can be difficult to control.
Explicit Alignment Objectives for Multilingual Bidirectional Encoders
Pre-trained cross-lingual encoders such as mBERT (Devlin et al., 2019) and XLMR (Conneau et al., 2020) have proven to be impressively effective at enabling transfer-learning of NLP systems from high-resource languages to low-resource languages.
Re-evaluating Evaluation in Text Summarization
Automated evaluation metrics as a stand-in for manual evaluation are an essential part of the development of text-generation tasks such as text summarization.
X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models
We further propose a code-switching-based method to improve the ability of multilingual LMs to access knowledge, and verify its effectiveness on several benchmark languages.
Improving Target-side Lexical Transfer in Multilingual Neural Machine Translation
To improve the performance of Neural Machine Translation~(NMT) for low-resource languages~(LRL), one effective strategy is to leverage parallel data from a related high-resource language~(HRL).
Automatic Extraction of Rules Governing Morphological Agreement
Using cross-lingual transfer, even with no expert annotations in the language of interest, our framework extracts a grammatical specification which is nearly equivalent to those created with large amounts of gold-standard annotated data.
The Return of Lexical Dependencies: Neural Lexicalized PCFGs
In this paper we demonstrate that $\textit{context free grammar (CFG) based methods for grammar induction benefit from modeling lexical dependencies}$.
TICO-19: the Translation Initiative for Covid-19
Further, the team is converting the test and development data into translation memories (TMXs) that can be used by localizers from and to any of the languages.
Transliteration for Cross-Lingual Morphological Inflection
Cross-lingual transfer between typologically related languages has been proven successful for the task of morphological inflection.
Weight Poisoning Attacks on Pretrained Models
Recently, NLP has seen a surge in the usage of large pre-trained models.
Findings of the Fourth Workshop on Neural Generation and Translation
We describe the finding of the Fourth Workshop on Neural Generation and Translation, held in concert with the annual conference of the Association for Computational Linguistics (ACL 2020).
Learning Sparse Prototypes for Text Generation
While effective, these methods are inefficient at test time as a result of needing to store and index the entire training corpus.
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Recent years have witnessed the burgeoning of pretrained language models (LMs) for text-based natural language (NL) understanding tasks.
Ranked #10 on
Text-To-SQL
on spider
(Exact Match Accuracy (Dev) metric)
Soft Gazetteers for Low-Resource Named Entity Recognition
However, designing such features for low-resource languages is challenging, because exhaustive entity gazetteers do not exist in these languages.
Predicting Performance for Natural Language Processing Tasks
Given the complexity of combinations of tasks, languages, and domains in natural language processing (NLP) research, it is computationally prohibitive to exhaustively test newly proposed models on each possible experimental setting.
A Benchmark for Structured Procedural Knowledge Extraction from Cooking Videos
Watching instructional videos are often used to learn about procedures.
Politeness Transfer: A Tag and Generate Approach
This paper introduces a new task of politeness transfer which involves converting non-polite sentences to polite sentences while preserving the meaning.
A Summary of the First Workshop on Language Technology for Language Documentation and Revitalization
Despite recent advances in natural language processing and other language technology, the application of such technology to language documentation and conservation has been limited.
Practical Comparable Data Collection for Low-Resource Languages via Images
We propose a method of curating high-quality comparable training data for low-resource languages with monolingual annotators.
Incorporating External Knowledge through Pre-training for Natural Language to Code Generation
Open-domain code generation aims to generate code in a general-purpose programming language (such as Python) from natural language (NL) intents.
Ranked #3 on
Code Generation
on CoNaLa-Ext
AlloVera: A Multilingual Allophone Database
While phonemic representations are language specific, phonetic representations (stated in terms of (allo)phones) are much closer to a universal (language-independent) transcription.
Balancing Training for Multilingual Neural Machine Translation
When training multilingual machine translation (MT) models that can translate to/from multiple languages, we are faced with imbalanced training sets: some languages have much more training data than others.
Weight Poisoning Attacks on Pre-trained Models
We show that by applying a regularization method, which we call RIPPLe, and an initialization procedure, which we call Embedding Surgery, such attacks are possible even with limited knowledge of the dataset and fine-tuning procedure.
Dynamic Data Selection and Weighting for Iterative Back-Translation
Back-translation has proven to be an effective method to utilize monolingual data in neural machine translation (NMT), and iteratively conducting back-translation can further improve the model performance.
A Set of Recommendations for Assessing Human-Machine Parity in Language Translation
The quality of machine translation has increased remarkably over the past years, to the degree that it was found to be indistinguishable from professional human translation in a number of empirical investigations.
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
However, these broad-coverage benchmarks have been mostly limited to English, and despite an increasing interest in multilingual models, a benchmark that enables the comprehensive evaluation of such methods on a diverse range of languages and tasks is still missing.
Improving Candidate Generation for Low-resource Cross-lingual Entity Linking
Cross-lingual entity linking (XEL) is the task of finding referents in a target-language knowledge base (KB) for mentions extracted from source-language texts.
Universal Phone Recognition with a Multilingual Allophone System
Multilingual models can improve language processing, particularly for low resource situations, by sharing parameters across languages.
Differentiable Reasoning over a Virtual Knowledge Base
In particular, we describe a neural module, DrKIT, that traverses textual data like a KB, softly following paths of relations between mentions of entities in the corpus.
Learning Relation Entailment with Structured and Textual Information
We propose several methods that incorporate both structured and textual information to represent relations for this task.
A Probabilistic Formulation of Unsupervised Text Style Transfer
Across all style transfer tasks, our approach yields substantial gains over state-of-the-art non-generative baselines, including the state-of-the-art unsupervised machine translation techniques that our approach generalizes.
Merging Weak and Active Supervision for Semantic Parsing
Experiments on WikiTableQuestions with human annotators show that our method can improve the performance with only 100 active queries, especially for weakly-supervised parsers learnt from a cold start.
How Can We Know What Language Models Know?
Recent work has presented intriguing results examining the knowledge contained in language models (LM) by having the LM fill in the blanks of prompts such as "Obama is a _ by profession".
Optimizing Data Usage via Differentiable Rewards
To acquire a new skill, humans learn better and faster if a tutor, based on their current knowledge level, informs them of how much attention they should pay to particular content or practice problems.
A Bilingual Generative Transformer for Semantic Sentence Embedding
Semantic sentence embedding models encode natural language sentences into vectors, such that closeness in embedding space indicates closeness in the semantics between the sentences.
Generalizing Natural Language Analysis through Span-relation Representations
Natural language processing covers a wide variety of tasks predicting syntax, semantics, and information content, and usually each type of output is generated with specially designed architectures.
Ranked #1 on
Relation Extraction
on WLPC
Aspect-Based Sentiment Analysis
Aspect-Based Sentiment Analysis (ABSA)
+8
Should All Cross-Lingual Embeddings Speak English?
Most of recent work in cross-lingual word embeddings is severely Anglocentric.
Understanding Knowledge Distillation in Non-autoregressive Machine Translation
We find that knowledge distillation can reduce the complexity of data sets and help NAT to model the variations in the output data.
