Search Results for author:
Found 100 papers, 61 papers with code
Unlearning Reveals the Influential Training Data of Language Models
This paper presents UnTrac, which estimates the influence of a training dataset by unlearning it from the trained model.
Compositional Generalization for Data-to-Text Generation
Data-to-text generation involves transforming structured data, often represented as predicate-argument tuples, into coherent textual descriptions.
Latent Feature-based Data Splits to Improve Generalisation Evaluation: A Hate Speech Detection Case Study
We challenge hate speech models via new train-test splits of existing datasets that rely on the clustering of models' hidden representations.
Memorisation Cartography: Mapping out the Memorisation-Generalisation Continuum in Neural Machine Translation
When training a neural network, it will quickly memorise some source-target mappings from your dataset but never learn some others.
Subspace Chronicles: How Linguistic Information Emerges, Shifts and Interacts during Language Model Training
We identify critical learning phases across tasks and time, during which subspaces emerge, share information, and later disentangle to specialize.
Cross-Modal Conceptualization in Bottleneck Models
Concept Bottleneck Models (CBMs) assume that training examples (e. g., x-ray images) are annotated with high-level concepts (e. g., types of abnormalities), and perform classification by first predicting the concepts, followed by predicting the label relying on these concepts.
On the Transferability of Visually Grounded PCFGs
We consider a zero-shot transfer learning setting where a model is trained on the source domain and is directly applied to target domains, without any further training.
Cache & Distil: Optimising API Calls to Large Language Models
To curtail the frequency of these calls, one can employ a smaller language model -- a student -- which is continuously trained on the responses of the LLM.
Injecting a Structural Inductive Bias into a Seq2Seq Model by Simulation
Strong inductive biases enable learning from little data and help generalization outside of the training distribution.
Autoencoding Conditional Neural Processes for Representation Learning
Conditional neural processes (CNPs) are a flexible and efficient family of models that learn to learn a stochastic process from data.
Compositional Generalization without Trees using Multiset Tagging and Latent Permutations
Our model outperforms pretrained seq2seq models and prior work on realistic semantic parsing tasks that require generalization to longer examples.
Recursive Neural Networks with Bottlenecks Diagnose (Non-)Compositionality
We illustrate that comparing data's representations in models with and without the bottleneck can be used to produce a compositionality metric.
Hierarchical Phrase-based Sequence-to-Sequence Learning
We describe a neural transducer that maintains the flexibility of standard sequence-to-sequence (seq2seq) models while incorporating hierarchical phrases as a source of inductive bias during training and as explicit constraints during inference.
Compositional Generalisation with Structured Reordering and Fertility Layers
Seq2seq models have been shown to struggle with compositional generalisation, i. e. generalising to new and potentially more complex structures than seen during training.
Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation
In this work, we investigate whether the non-compositionality of idioms is reflected in the mechanics of the dominant NMT model, Transformer, by analysing the hidden states and attention patterns for models with English as source language and one of seven European languages as target language.
Sparse Interventions in Language Models with Differentiable Masking
Typically, interpretation methods i) do not guarantee that the model actually uses the encoded information, and ii) do not discover small subsets of neurons responsible for a considered phenomenon.
Learning Opinion Summarizers by Selecting Informative Reviews
Opinion summarization has been traditionally approached with unsupervised, weakly-supervised and few-shot learning techniques.
Highly Parallel Autoregressive Entity Linking with Discriminative Correction
Generative approaches have been recently shown to be effective for both Entity Disambiguation and Entity Linking (i. e., joint mention detection and disambiguation).
Ranked #5 on
Entity Linking
on AIDA-CoNLL
Language Modeling, Lexical Translation, Reordering: The Training Process of NMT through the Lens of Classical SMT
Differently from the traditional statistical MT that decomposes the translation task into distinct separately learned components, neural machine translation uses a single neural network to model the entire translation process.
Beyond Sentence-Level End-to-End Speech Translation: Context Helps
Document-level contextual information has shown benefits to text-based machine translation, but whether and how context helps end-to-end (E2E) speech translation (ST) is still under-studied.
Exploring Unsupervised Pretraining Objectives for Machine Translation
Unsupervised cross-lingual pretraining has achieved strong results in neural machine translation (NMT), by drastically reducing the need for large parallel data.
Meta-Learning to Compositionally Generalize
Natural language is compositional; the meaning of a sentence is a function of the meaning of its parts.
Structured Reordering for Modeling Latent Alignments in Sequence Transduction
Despite success in many domains, neural models struggle in settings where train and test examples are drawn from different distributions.
Editing Factual Knowledge in Language Models
We present KnowledgeEditor, a method which can be used to edit this knowledge and, thus, fix 'bugs' or unexpected predictions without the need for expensive re-training or fine-tuning.
Sparse Attention with Linear Units
Recently, it has been argued that encoder-decoder models can be made more interpretable by replacing the softmax function in the attention with its sparse variants.
Learning from Executions for Semantic Parsing
Based on the observation that programs which correspond to NL utterances must be always executable, we propose to encourage a parser to generate executable programs for unlabeled utterances.
An Empirical Study of Compound PCFGs
Compound probabilistic context-free grammars (C-PCFGs) have recently established a new state of the art for unsupervised phrase-structure grammar induction.
Neutron study of magnetic correlations in rare-earth-free Mn-Bi magnets
We report the results of an unpolarized small-angle neutron scattering (SANS) study on Mn-Bi-based rare-earth-free permanent magnets.
Materials Science
Detecting Word Sense Disambiguation Biases in Machine Translation for Model-Agnostic Adversarial Attacks
Word sense disambiguation is a well-known source of translation errors in NMT.
Fast Interleaved Bidirectional Sequence Generation
Instead of assuming independence between neighbouring tokens (semi-autoregressive decoding, SA), we take inspiration from bidirectional sequence generation and introduce a decoder that generates target words from the left-to-right and right-to-left directions simultaneously.
A Differentiable Relaxation of Graph Segmentation and Alignment for AMR Parsing
In contrast, we treat both alignment and segmentation as latent variables in our model and induce them as part of end-to-end training.
Ranked #22 on
AMR Parsing
on LDC2017T10
Meta-Learning for Domain Generalization in Semantic Parsing
The importance of building semantic parsers which can be applied to new domains and generate programs unseen at training has long been acknowledged, and datasets testing out-of-domain performance are becoming increasingly available.
Analyzing the Source and Target Contributions to Predictions in Neural Machine Translation
We find that models trained with more data tend to rely on source information more and to have more sharp token contributions; the training process is non-monotonic with several stages of different nature.
Adaptive Feature Selection for End-to-End Speech Translation
Information in speech signals is not evenly distributed, making it an additional challenge for end-to-end (E2E) speech translation (ST) to learn to focus on informative features.
Interpreting Graph Neural Networks for NLP With Differentiable Edge Masking
In this work, we introduce a post-hoc method for interpreting the predictions of GNNs which identifies unnecessary edges.
Visually Grounded Compound PCFGs
In this work, we study visually grounded grammar induction and learn a constituency parser from both unlabeled text and its visual groundings.
Unsupervised Transfer of Semantic Role Models from Verbal to Nominal Domain
Nominal roles are not labeled in the training data, and the learning objective instead pushes the labeler to assign roles predictive of the arguments.
Preventing Posterior Collapse with Levenshtein Variational Autoencoder
Variational autoencoders (VAEs) are a standard framework for inducing latent variable models that have been shown effective in learning text representations as well as in text generation.
How do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Masking
Attribution methods assess the contribution of inputs to the model prediction.
Few-Shot Learning for Opinion Summarization
In this work, we show that even a handful of summaries is sufficient to bootstrap generation of the summary text with all expected properties, such as writing style, informativeness, fluency, and sentiment preservation.
Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
Massively multilingual models for neural machine translation (NMT) are theoretically attractive, but often underperform bilingual models and deliver poor zero-shot translations.
On Sparsifying Encoder Outputs in Sequence-to-Sequence Models
Inspired by these observations, we explore the feasibility of specifying rule-based patterns that mask out encoder outputs based on information such as part-of-speech tags, word frequency and word position.
Information-Theoretic Probing with Minimum Description Length
Instead, we propose an alternative to the standard probes, information-theoretic probing with minimum description length (MDL).
Unsupervised Opinion Summarization as Copycat-Review Generation
At test time, when generating summaries, we force the novelty to be minimal, and produce a text reflecting consensus opinions.
The Emergence of Compositional Languages for Numeric Concepts Through Iterated Learning in Neural Agents
Since first introduced, computer simulation has been an increasingly important tool in evolutionary linguistics.
Capturing Argument Interaction in Semantic Role Labeling with Capsule Networks
In every network layer, the capsules interact with each other and with representations of words in the sentence.
Graph Convolutions over Constituent Trees for Syntax-Aware Semantic Role Labeling
Semantic role labeling (SRL) is the task of identifying predicates and labeling argument spans with semantic roles.
Learning Semantic Parsers from Denotations with Latent Structured Alignments and Abstract Programs
Semantic parsing aims to map natural language utterances onto machine interpretable meaning representations, aka programs whose execution against a real-world environment produces a denotation.
Ranked #13 on
Semantic Parsing
on WikiTableQuestions
Semantic Role Labeling with Iterative Structure Refinement
Modern state-of-the-art Semantic Role Labeling (SRL) methods rely on expressive sentence encoders (e. g., multi-layer LSTMs) but tend to model only local (if any) interactions between individual argument labeling decisions.
The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives
In this work, we use canonical correlation analysis and mutual information estimators to study how information flows across Transformer layers and how this process depends on the choice of learning objective.
Context-Aware Monolingual Repair for Neural Machine Translation
For training, the DocRepair model requires only monolingual document-level data in the target language.
Improving Deep Transformer with Depth-Scaled Initialization and Merged Attention
The general trend in NLP is towards increasing model capacity and performance via deeper neural networks.
Widening the Representation Bottleneck in Neural Machine Translation with Lexical Shortcuts
The transformer is a state-of-the-art neural translation model that uses attention to iteratively refine lexical representations with information drawn from the surrounding context.
Learning Latent Trees with Stochastic Perturbations and Differentiable Dynamic Programming
We treat projective dependency trees as latent variables in our probabilistic model and induce them in such a way as to be beneficial for a downstream task, without relying on any direct tree supervision.
Boosting Entity Linking Performance by Leveraging Unlabeled Documents
First, we construct a high recall list of candidate entities for each mention in an unlabeled document.
Ranked #17 on
Entity Disambiguation
on AIDA-CoNLL
Single Document Summarization as Tree Induction
In this paper, we conceptualize single-document extractive summarization as a tree induction problem.
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
Multi-head self-attention is a key component of the Transformer, a state-of-the-art architecture for neural machine translation.
Interpretable Neural Predictions with Differentiable Binary Variables
The success of neural networks comes hand in hand with a desire for more interpretability.
Distant Learning for Entity Linking with Automatic Noise Detection
As the learning signal is weak and our surrogate labels are noisy, we introduce a noise detection component in our model: it lets the model detect and disregard examples which are likely to be noisy.
When a Good Translation is Wrong in Context: Context-Aware Machine Translation Improves on Deixis, Ellipsis, and Lexical Cohesion
Though machine translation errors caused by the lack of context beyond one sentence have long been acknowledged, the development of context-aware NMT systems is hampered by several problems.
Obfuscation for Privacy-preserving Syntactic Parsing
We introduce an idea for a privacy-preserving transformation on natural language data, inspired by homomorphic encryption.
Block Neural Autoregressive Flow
Recently, as an alternative to hand-crafted bijections, Huang et al. (2018) proposed neural autoregressive flow (NAF) which is a universal approximator for density functions.
Ranked #1 on
Density Estimation
on UCI MINIBOONE
Modeling Latent Sentence Structure in Neural Machine Translation
Recently it was shown that linguistic structure predicted by a supervised parser can be beneficial for neural machine translation (NMT).
Question Answering by Reasoning Across Documents with Graph Convolutional Networks
Most research in reading comprehension has focused on answering questions based on individual documents or even single paragraphs.
Differentiable Perturb-and-Parse: Semi-Supervised Parsing with a Structured Variational Autoencoder
Human annotation for syntactic parsing is expensive, and large resources are available only for a fraction of languages.
Context-Aware Neural Machine Translation Learns Anaphora Resolution
Standard machine translation systems process sentences in isolation and hence ignore extra-sentential information, even though extended context can both prevent mistakes in ambiguous cases and improve translation coherence.
AMR Parsing as Graph Prediction with Latent Alignment
AMR parsing is challenging partly due to the lack of annotated alignments between nodes in the graphs and words in the corresponding sentences.
Ranked #1 on
AMR Parsing
on LDC2015E86
Improving Entity Linking by Modeling Latent Relations between Mentions
Entity linking involves aligning textual mentions of named entities to their corresponding entries in a knowledge base.
Exploiting Semantics in Neural Machine Translation with Graph Convolutional Networks
Semantic representations have long been argued as potentially useful for enforcing meaning preservation and improving generalization performance of machine translation methods.
Ranked #8 on
Machine Translation
on WMT2016 English-German
Embedding Words as Distributions with a Bayesian Skip-gram Model
Rather than assuming that a word embedding is fixed across the entire text collection, as in standard word embedding methods, in our Bayesian model we generate it from a word-specific prior density for each occurrence of a given word.
Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols
Learning to communicate through interaction, rather than relying on explicit supervision, is often considered a prerequisite for developing a general AI.
Graph Convolutional Encoders for Syntax-aware Neural Machine Translation
We present a simple and effective approach to incorporating syntactic structure into neural attention-based encoder-decoder models for machine translation.
Optimizing Differentiable Relaxations of Coreference Evaluation Metrics
Coreference evaluation metrics are hard to optimize directly as they are non-differentiable functions, not easily decomposable into elementary decisions.
Modeling Relational Data with Graph Convolutional Networks
We demonstrate the effectiveness of R-GCNs as a stand-alone model for entity classification.
Ranked #1 on
Node Classification
on AIFB
Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling
GCNs over syntactic dependency trees are used as sentence encoders, producing latent feature representations of words in a sentence.
Ranked #2 on
Chinese Semantic Role Labeling
on CoNLL-2009
Modeling Semantic Expectation: Using Script Knowledge for Referent Prediction
Recent research in psycholinguistics has provided increasing evidence that humans predict upcoming content.
A Simple and Accurate Syntax-Agnostic Neural Model for Dependency-based Semantic Role Labeling
However, when automatically predicted part-of-speech tags are provided as input, it substantially outperforms all previous local models and approaches the best reported results on the English CoNLL-2009 dataset.
Bilingual Learning of Multi-sense Embeddings with Discrete Autoencoders
We present an approach to learning multi-sense word embeddings relying both on monolingual and bilingual information.
Adapting to All Domains at Once: Rewarding Domain Invariance in SMT
Existing work on domain adaptation for statistical machine translation has consistently assumed access to a small sample from the test distribution (target domain) at training time.
Discrete-State Variational Autoencoders for Joint Discovery and Factorization of Relations
We present a method for unsupervised open-domain relation discovery.
Word Representations, Tree Models and Syntactic Functions
Word representations induced from models with discrete latent variables (e. g.\ HMMs) have been shown to be beneficial in many NLP applications.
Inducing Semantic Representation from Text by Jointly Predicting and Factorizing Relations
In this work, we propose a new method to integrate two recent lines of work: unsupervised induction of shallow semantics (e. g., semantic roles) and factorization of relations in text and knowledge bases.
Unsupervised Induction of Semantic Roles within a Reconstruction-Error Minimization Framework
We introduce a new approach to unsupervised estimation of feature-rich semantic role labeling models.
Learning Semantic Script Knowledge with Event Embeddings
Induction of common sense knowledge about prototypical sequences of events has recently received much attention.
