Search Results for author:
Found 211 papers, 98 papers with code
SIGMORPHON–UniMorph 2022 Shared Task 0: Generalization and Typologically Diverse Morphological Inflection
The 2022 SIGMORPHON–UniMorph shared task on large scale morphological inflection generation included a wide range of typologically diverse languages: 33 languages from 11 top-level language families: Arabic (Modern Standard), Assamese, Braj, Chukchi, Eastern Armenian, Evenki, Georgian, Gothic, Gujarati, Hebrew, Hungarian, Itelmen, Karelian, Kazakh, Ket, Khalkha Mongolian, Kholosi, Korean, Lamahalot, Low German, Ludic, Magahi, Middle Low German, Old English, Old High German, Old Norse, Polish, Pomak, Slovak, Turkish, Upper Sorbian, Veps, and Xibe.
Conditional Poisson Stochastic Beams
Beam search is the default decoding strategy for many sequence generation tasks in NLP.
A surprisal–duration trade-off across and within the world’s languages
We thus conclude that there is strong evidence of a surprisal–duration trade-off in operation, both across and within the world’s languages.
Efficient Sampling of Dependency Structure
In this paper, we adapt two spanning tree sampling algorithms to faithfully sample dependency trees from a graph subject to the root constraint.
The SIGTYP 2022 Shared Task on the Prediction of Cognate Reflexes
This study describes the structure and the results of the SIGTYP 2022 shared task on the prediction of cognate reflexes from multilingual wordlists.
High probability or low information? The probability–quality paradox in language generation
When generating natural language from neural probabilistic models, high probability does not always coincide with high quality.
SIGMORPHON 2021 Shared Task on Morphological Reinflection: Generalization Across Languages
This year's iteration of the SIGMORPHON Shared Task on morphological reinflection focuses on typological diversity and cross-lingual variation of morphosyntactic features.
On the Machine Learning of Ethical Judgments from Natural Language
Ethics is one of the longest standing intellectual endeavors of humanity.
Measuring the Similarity of Grammatical Gender Systems by Comparing Partitions
Of particular interest, languages on the same branch of our phylogenetic tree are notably similar, whereas languages from separate branches are no more similar than chance.
Low-Resource Named Entity Recognition with Cross-Lingual, Character-Level Neural Conditional Random Fields
Low-resource named entity recognition is still an open problem in NLP.
Low Resource Named Entity Recognition
named-entity-recognition
+2
Labeled Morphological Segmentation with Semi-Markov Models
We present labeled morphological segmentation, an alternative view of morphological processing that unifies several tasks.
[Call for Papers] The 2nd BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus
The big changes for this year's competition are as follows: First, we replace the loose track with a paper track, which allows (for example) non-model-based submissions, novel cognitively-inspired benchmarks, or analysis techniques.
Context versus Prior Knowledge in Language Models
To answer a question, language models often need to integrate prior knowledge learned during pretraining and new information presented in context.
The Role of $n$-gram Smoothing in the Age of Neural Networks
For nearly three decades, language models derived from the $n$-gram assumption held the state of the art on the task.
Towards Explainability in Legal Outcome Prediction Models
Current legal outcome prediction models - a staple of legal NLP - do not explain their reasoning.
On the Challenges and Opportunities in Generative AI
The field of deep generative modeling has grown rapidly and consistently over the years.
A Theoretical Result on the Inductive Bias of RNN Language Models
However, a closer inspection of Hewitt et al.'s (2020) construction shows that it is not limited to hierarchical LMs, posing the question of what \emph{other classes} of LMs can be efficiently represented by RNNs.
What Changed? Converting Representational Interventions to Natural Language
Interventions targeting the representation space of language models (LMs) have emerged as effective means to influence model behavior.
Direct Preference Optimization with an Offset
DPO, as originally formulated, relies on binary preference data and fine-tunes a language model to increase the likelihood of a preferred response over a dispreferred response.
MiMiC: Minimally Modified Counterfactuals in the Representation Space
We demonstrate the effectiveness of the proposed approaches in mitigating bias in multiclass classification and in reducing the generation of toxic language, outperforming strong baselines.
Do Language Models Exhibit the Same Cognitive Biases in Problem Solving as Human Learners?
We find evidence that LLMs, with and without instruction-tuning, exhibit human-like biases in both the text-comprehension and the solution-planning steps of the solving process, but not during the final step which relies on the problem's arithmetic expressions (solution execution).
Principled Gradient-based Markov Chain Monte Carlo for Text Generation
Recent papers have demonstrated the possibility of energy-based text generation by adapting gradient-based sampling algorithms, a paradigm of MCMC algorithms that promises fast convergence.
Revisiting the Optimality of Word Lengths
Under this method, we find that a language's word lengths should instead be proportional to the surprisal's expectation plus its variance-to-mean ratio.
The Ethics of Automating Legal Actors
The introduction of large public legal datasets has brought about a renaissance in legal NLP.
Grammatical Gender's Influence on Distributional Semantics: A Causal Perspective
However, when we control for the meaning of the noun, we find that grammatical gender has a near-zero effect on adjective choice, thereby calling the neo-Whorfian hypothesis into question.
Quantifying the redundancy between prosody and text
Using a large spoken corpus of English audiobooks, we extract prosodic features aligned to individual words and test how well they can be predicted from LLM embeddings, compared to non-contextual word embeddings.
An Exploration of Left-Corner Transformations
The left-corner transformation (Rosenkrantz and Lewis, 1970) is used to remove left recursion from context-free grammars, which is an important step towards making the grammar parsable top-down with simple techniques.
Formal Aspects of Language Modeling
Large language models have become one of the most commonly deployed NLP inventions.
Efficient Algorithms for Recognizing Weighted Tree-Adjoining Languages
From these, we also immediately obtain stringsum and allsum algorithms for TAG, LIG, PAA, and EPDA.
On the Representational Capacity of Recurrent Neural Language Models
We extend the Turing completeness result to the probabilistic case, showing how a rationally weighted RLM with unbounded computation time can simulate any deterministic probabilistic Turing machine (PTM) with rationally weighted transitions.
Recurrent Neural Language Models as Probabilistic Finite-state Automata
These results present a first step towards characterizing the classes of distributions RNN LMs can represent and thus help us understand their capabilities and limitations.
An Analysis of On-the-fly Determinization of Finite-state Automata
In this paper we establish an abstraction of on-the-fly determinization of finite-state automata using transition monoids and demonstrate how it can be applied to bound the asymptotics.
A Geometric Notion of Causal Probing
The linear subspace hypothesis (Bolukbasi et al., 2016) states that, in a language model's representation space, all information about a concept such as verbal number is encoded in a linear subspace.
Testing the Predictions of Surprisal Theory in 11 Languages
We address this gap in the current literature by investigating the relationship between surprisal and reading times in eleven different languages, distributed across five language families.
On the Efficacy of Sampling Adapters
While this trade-off is not reflected in standard metrics of distribution quality (such as perplexity), we find that several precision-emphasizing measures indeed indicate that sampling adapters can lead to probability distributions more aligned with the true distribution.
Efficient Semiring-Weighted Earley Parsing
This paper provides a reference description, in the form of a deduction system, of Earley's (1970) context-free parsing algorithm with various speed-ups.
Generalizing Backpropagation for Gradient-Based Interpretability
Many popular feature-attribution methods for interpreting deep neural networks rely on computing the gradients of a model's output with respect to its inputs.
A Formal Perspective on Byte-Pair Encoding
Via submodular functions, we prove that the iterative greedy version is a $\frac{1}{{\sigma(\boldsymbol{\mu}^\star)}}(1-e^{-{\sigma(\boldsymbol{\mu}^\star)}})$-approximation of an optimal merge sequence, where ${\sigma(\boldsymbol{\mu}^\star)}$ is the total backward curvature with respect to the optimal merge sequence $\boldsymbol{\mu}^\star$.
Hexatagging: Projective Dependency Parsing as Tagging
We introduce a novel dependency parser, the hexatagger, that constructs dependency trees by tagging the words in a sentence with elements from a finite set of possible tags.
Convergence and Diversity in the Control Hierarchy
Furthermore, using an even stricter notion of equivalence called d-strong equivalence, we make precise the intuition that a CFG controlling a CFG is a TAG, a PDA controlling a PDA is an embedded PDA, and a PDA controlling a CFG is a LIG.
A Cross-Linguistic Pressure for Uniform Information Density in Word Order
Here, we ask whether a pressure for UID may have influenced word order patterns cross-linguistically.
LEACE: Perfect linear concept erasure in closed form
Concept erasure aims to remove specified features from a representation.
Structured Voronoi Sampling
In this paper, we take an important step toward building a principled approach for sampling from language models with gradient-based methods.
Linear-Time Modeling of Linguistic Structure: An Order-Theoretic Perspective
We show that these exhaustive comparisons can be avoided, and, moreover, the complexity of such tasks can be reduced to linear by casting the relation between tokens as a partial order over the string.
All Roads Lead to Rome? Exploring the Invariance of Transformers' Representations
Transformer models bring propelling advances in various NLP tasks, thus inducing lots of interpretability research on the learned representations of the models.
RecurrentGPT: Interactive Generation of (Arbitrarily) Long Text
In addition to producing AI-generated content (AIGC), we also demonstrate the possibility of using RecurrentGPT as an interactive fiction that directly interacts with consumers.
Efficient Prompting via Dynamic In-Context Learning
To achieve this, we train a meta controller that predicts the number of in-context examples suitable for the generalist model to make a good prediction based on the performance-efficiency trade-off for a specific input.
Discourse Centric Evaluation of Machine Translation with a Densely Annotated Parallel Corpus
Several recent papers claim human parity at sentence-level Machine Translation (MT), especially in high-resource languages.
Controlled Text Generation with Natural Language Instructions
Large language models generate fluent texts and can follow natural language instructions to solve a wide range of tasks without task-specific training.
Discriminative Class Tokens for Text-to-Image Diffusion Models
This approach has two disadvantages: (i) supervised datasets are generally small compared to large-scale scraped text-image datasets on which text-to-image models are trained, affecting the quality and diversity of the generated images, or (ii) the input is a hard-coded label, as opposed to free-form text, limiting the control over the generated images.
Algorithms for Acyclic Weighted Finite-State Automata with Failure Arcs
The pathsum in ordinary acyclic WFSAs is efficiently computed by the backward algorithm in time $O(|E|)$, where $E$ is the set of transitions.
A Measure-Theoretic Characterization of Tight Language Models
Language modeling, a central task in natural language processing, involves estimating a probability distribution over strings.
The Ordered Matrix Dirichlet for State-Space Models
For discrete data, SSMs commonly do so through a state-to-action emission matrix and a state-to-state transition matrix.
On the Effect of Anticipation on Reading Times
We assess the effect of anticipation on reading by comparing how well surprisal and contextual entropy predict reading times on four naturalistic reading datasets: two self-paced and two eye-tracking.
Schrödinger's Bat: Diffusion Models Sometimes Generate Polysemous Words in Superposition
Recent work has shown that despite their impressive capabilities, text-to-image diffusion models such as DALL-E 2 (Ramesh et al., 2022) can display strange behaviours when a prompt contains a word with multiple possible meanings, often generating images containing both senses of the word (Rassin et al., 2022).
On Parsing as Tagging
There have been many proposals to reduce constituency parsing to tagging in the literature.
The Architectural Bottleneck Principle
This shift in perspective leads us to propose a new principle for probing, the architectural bottleneck principle: In order to estimate how much information a given component could extract, a probe should look exactly like the component.
Autoregressive Structured Prediction with Language Models
Recent years have seen a paradigm shift in NLP towards using pretrained language models ({PLM}) for a wide range of tasks.
Ranked #1 on
Relation Extraction
on CoNLL04
(RE+ Micro F1 metric)
A Bilingual Parallel Corpus with Discourse Annotations
The BWB corpus consists of Chinese novels translated by experts into English, and the annotated test set is designed to probe the ability of machine translation systems to model various discourse phenomena.
Investigating the Role of Centering Theory in the Context of Neural Coreference Resolution Systems
Our analysis further shows that contextualized embeddings contain much of the coherence information, which helps explain why CT can only provide little gains to modern neural coreference resolvers which make use of pretrained representations.
Mutual Information Alleviates Hallucinations in Abstractive Summarization
Despite significant progress in the quality of language generated from abstractive summarization models, these models still exhibit the tendency to hallucinate, i. e., output content not supported by the source document.
Log-linear Guardedness and its Implications
Methods for erasing human-interpretable concepts from neural representations that assume linearity have been found to be tractable and useful.
Algorithms for Weighted Pushdown Automata
Weighted pushdown automata (WPDAs) are at the core of many natural language processing tasks, like syntax-based statistical machine translation and transition-based dependency parsing.
An Ordinal Latent Variable Model of Conflict Intensity
It is based only on the action category ("what") and disregards the subject ("who") and object ("to whom") of an event, as well as contextual information, like associated casualty count, that should contribute to the perception of an event's "intensity".
State-of-the-art generalisation research in NLP: A taxonomy and review
We present a taxonomy for characterising and understanding generalisation research in NLP.
Equivariant Transduction through Invariant Alignment
The ability to generalize compositionally is key to understanding the potentially infinite number of sentences that can be constructed in a human language from only a finite number of words.
On the Intersection of Context-Free and Regular Languages
It shows, by a simple construction, that the intersection of a context-free language and a regular language is itself context-free.
On the Role of Negative Precedent in Legal Outcome Prediction
In contrast, we turn our focus to negative outcomes here, and introduce a new task of negative outcome prediction.
Benchmarking Compositionality with Formal Languages
Recombining known primitive concepts into larger novel combinations is a quintessentially human cognitive capability.
Visual Comparison of Language Model Adaptation
The simplicity of adapter training and composition comes along with new challenges, such as maintaining an overview of adapter properties and effectively comparing their produced embedding spaces.
Probing via Prompting
We then examine the usefulness of a specific linguistic property for pre-training by removing the heads that are essential to that property and evaluating the resulting model's performance on language modeling.
The SIGMORPHON 2022 Shared Task on Morpheme Segmentation
The SIGMORPHON 2022 shared task on morpheme segmentation challenged systems to decompose a word into a sequence of morphemes and covered most types of morphology: compounds, derivations, and inflections.
Ranked #8 on
Morpheme Segmentaiton
on UniMorph 4.0
On the Usefulness of Embeddings, Clusters and Strings for Text Generator Evaluation
As we show, however, this is not a tight approximation -- in either theory or practice.
Naturalistic Causal Probing for Morpho-Syntax
Probing has become a go-to methodology for interpreting and analyzing deep neural models in natural language processing.
A Structured Span Selector
Many natural language processing tasks, e. g., coreference resolution and semantic role labeling, require selecting text spans and making decisions about them.
UniMorph 4.0: Universal Morphology
The project comprises two major thrusts: a language-independent feature schema for rich morphological annotation and a type-level resource of annotated data in diverse languages realizing that schema.
Same Neurons, Different Languages: Probing Morphosyntax in Multilingual Pre-trained Models
The success of multilingual pre-trained models is underpinned by their ability to learn representations shared by multiple languages even in absence of any explicit supervision.
Exact Paired-Permutation Testing for Structured Test Statistics
However, practitioners rely on Monte Carlo approximation to perform this test due to a lack of a suitable exact algorithm.
Probing for the Usage of Grammatical Number
We also find that BERT uses a separate encoding of grammatical number for nouns and verbs.
Estimating the Entropy of Linguistic Distributions
Shannon entropy is often a quantity of interest to linguists studying the communicative capacity of human language.
Analyzing Wrap-Up Effects through an Information-Theoretic Lens
Numerous analyses of reading time (RT) data have been implemented -- all in an effort to better understand the cognitive processes driving reading comprehension.
On the probability-quality paradox in language generation
Specifically, we posit that human-like language should contain an amount of information (quantified as negative log-probability) that is close to the entropy of the distribution over natural strings.
On Decoding Strategies for Neural Text Generators
For example, the nature of the diversity-quality trade-off in language generation is very task-specific; the length bias often attributed to beam search is not constant across tasks.
Locally Typical Sampling
Automatic and human evaluations show that, in comparison to nucleus and top-k sampling, locally typical sampling offers competitive performance (in both abstractive summarization and story generation) in terms of quality while consistently reducing degenerate repetitions.
Linear Adversarial Concept Erasure
Modern neural models trained on textual data rely on pre-trained representations that emerge without direct supervision.
Kernelized Concept Erasure
One prominent approach for the identification of concepts in neural representations is searching for a linear subspace whose erasure prevents the prediction of the concept from the representations.
A Latent-Variable Model for Intrinsic Probing
The success of pre-trained contextualized representations has prompted researchers to analyze them for the presence of linguistic information.
A Word on Machine Ethics: A Response to Jiang et al. (2021)
Ethics is one of the longest standing intellectual endeavors of humanity.
Probing as Quantifying Inductive Bias
Such performance improvements have motivated researchers to quantify and understand the linguistic information encoded in these representations.
A surprisal--duration trade-off across and within the world's languages
We thus conclude that there is strong evidence of a surprisal--duration trade-off in operation, both across and within the world's languages.
On Homophony and Rényi Entropy
Homophony's widespread presence in natural languages is a controversial topic.
Classifying Dyads for Militarized Conflict Analysis
We do this by devising a set of textual and graph-based features which represent each of the causes.
Revisiting the Uniform Information Density Hypothesis
The uniform information density (UID) hypothesis posits a preference among language users for utterances structured such that information is distributed uniformly across a signal.
Conditional Poisson Stochastic Beam Search
In this work, we propose a new method for turning beam search into a stochastic process: Conditional Poisson stochastic beam search.
A Plug-and-Play Method for Controlled Text Generation
Large pre-trained language models have repeatedly shown their ability to produce fluent text.
Efficient Sampling of Dependency Structures
Colbourn (1996)'s sampling algorithm has a running time of $\mathcal{O}(N^3)$, which is often greater than the mean hitting time of a directed graph.
Searching for More Efficient Dynamic Programs
To this end, we describe a set of program transformations, a simple metric for assessing the efficiency of a transformed program, and a heuristic search procedure to improve this metric.
A Bayesian Framework for Information-Theoretic Probing
Pimentel et al. (2020) recently analysed probing from an information-theoretic perspective.
Differentiable Subset Pruning of Transformer Heads
Multi-head attention, a collection of several attention mechanisms that independently attend to different parts of the input, is the key ingredient in the Transformer.
Towards Zero-shot Language Modeling
Motivated by this question, we aim at constructing an informative prior over neural weights, in order to adapt quickly to held-out languages in the task of character-level language modeling.
On Finding the K-best Non-projective Dependency Trees
Furthermore, we present a novel extension of the algorithm for decoding the K-best dependency trees of a graph which are subject to a root constraint.
SIGTYP 2021 Shared Task: Robust Spoken Language Identification
While language identification is a fundamental speech and language processing task, for many languages and language families it remains a challenging task.
Do Syntactic Probes Probe Syntax? Experiments with Jabberwocky Probing
One method of doing so, which is frequently cited to support the claim that models like BERT encode syntax, is called probing; probes are small supervised models trained to extract linguistic information from another model's output.
Modeling the Unigram Distribution
The unigram distribution is the non-contextual probability of finding a specific word form in a corpus.
Examining the Inductive Bias of Neural Language Models with Artificial Languages
Since language models are used to model a wide variety of languages, it is natural to ask whether the neural architectures used for the task have inductive biases towards modeling particular types of languages.
Is Sparse Attention more Interpretable?
Sparse attention has been claimed to increase model interpretability under the assumption that it highlights influential inputs.
Higher-order Derivatives of Weighted Finite-state Machines
In the case of second-order derivatives, our scheme runs in the optimal $\mathcal{O}(A^2 N^4)$ time where $A$ is the alphabet size and $N$ is the number of states.
On Finding the $K$-best Non-projective Dependency Trees
Furthermore, we present a novel extension of the algorithm for decoding the $K$-best dependency trees of a graph which are subject to a root constraint.
Language Model Evaluation Beyond Perplexity
As concrete examples, text generated under the nucleus sampling scheme adheres more closely to the type--token relationship of natural language than text produced using standard ancestral sampling; text from LSTMs reflects the natural language distributions over length, stopwords, and symbols surprisingly well.
A Non-Linear Structural Probe
Probes are models devised to investigate the encoding of knowledge -- e. g. syntactic structure -- in contextual representations.
A Cognitive Regularizer for Language Modeling
The uniform information density (UID) hypothesis, which posits that speakers behaving optimally tend to distribute information uniformly across a linguistic signal, has gained traction in psycholinguistics as an explanation for certain syntactic, morphological, and prosodic choices.
How (Non-)Optimal is the Lexicon?
Examining corpora from 7 typologically diverse languages, we use those upper bounds to quantify the lexicon's optimality and to explore the relative costs of major constraints on natural codes.
Quantifying Gender Bias Towards Politicians in Cross-Lingual Language Models
Recent research has demonstrated that large pre-trained language models reflect societal biases expressed in natural language.
Finding Concept-specific Biases in Form--Meaning Associations
It is not a new idea that there are small, cross-linguistic associations between the forms and meanings of words.
BlonDe: An Automatic Evaluation Metric for Document-level Machine Translation
Standard automatic metrics, e. g. BLEU, are not reliable for document-level MT evaluation.
Searching for Search Errors in Neural Morphological Inflection
Yet, on word-level tasks, exact inference of these models reveals the empty string is often the global optimum.
Differentiable Generative Phonology
As the main contribution of our work, we implement the phonological generative system as a neural model differentiable end-to-end, rather than as a set of rules or constraints.
Disambiguatory Signals are Stronger in Word-initial Positions
Psycholinguistic studies of human word processing and lexical access provide ample evidence of the preferred nature of word-initial versus word-final segments, e. g., in terms of attention paid by listeners (greater) or the likelihood of reduction by speakers (lower).
Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTs
Large-scale pretraining and task-specific fine-tuning is now the standard methodology for many tasks in computer vision and natural language processing.
Morphologically Aware Word-Level Translation
We propose a novel morphologically aware probability model for bilingual lexicon induction, which jointly models lexeme translation and inflectional morphology in a structured way.
SIGTYP 2020 Shared Task: Prediction of Typological Features
Typological knowledge bases (KBs) such as WALS (Dryer and Haspelmath, 2013) contain information about linguistic properties of the world's languages.
Investigating Cross-Linguistic Adjective Ordering Tendencies with a Latent-Variable Model
Across languages, multiple consecutive adjectives modifying a noun (e. g. "the big red dog") follow certain unmarked ordering rules.
Intrinsic Probing through Dimension Selection
Most modern NLP systems make use of pre-trained contextual representations that attain astonishingly high performance on a variety of tasks.
Please Mind the Root: Decoding Arborescences for Dependency Parsing
The connection between dependency trees and spanning trees is exploited by the NLP community to train and to decode graph-based dependency parsers.
If beam search is the answer, what was the question?
This implies that the MAP objective alone does not express the properties we desire in text, which merits the question: if beam search is the answer, what was the question?
Speakers Fill Lexical Semantic Gaps with Context
For a language to be clear and efficiently encoded, we posit that the lexical ambiguity of a word type should correlate with how much information context provides about it, on average.
Pareto Probing: Trading Off Accuracy for Complexity
In our contribution to this discussion, we argue for a probe metric that reflects the fundamental trade-off between probe complexity and performance: the Pareto hypervolume.
Exploring the Linear Subspace Hypothesis in Gender Bias Mitigation
Their method takes pre-trained word embeddings as input and attempts to isolate a linear subspace that captures most of the gender bias in the embeddings.
Efficient Computation of Expectations under Spanning Tree Distributions
We propose unified algorithms for the important cases of first-order expectations and second-order expectations in edge-factored, non-projective spanning-tree models.
Best-First Beam Search
Decoding for many NLP tasks requires an effective heuristic algorithm for approximating exact search since the problem of searching the full output space is often intractable, or impractical in many settings.
Metaphor Detection using Context and Concreteness
We report the results of our system on the Metaphor Detection Shared Task at the Second Workshop on Figurative Language Processing 2020.
SIGMORPHON 2020 Shared Task 0: Typologically Diverse Morphological Inflection
Systems were developed using data from 45 languages and just 5 language families, fine-tuned with data from an additional 45 languages and 10 language families (13 in total), and evaluated on all 90 languages.
A Corpus for Large-Scale Phonetic Typology
A major hurdle in data-driven research on typology is having sufficient data in many languages to draw meaningful conclusions.
Applying the Transformer to Character-level Transduction
The transformer has been shown to outperform recurrent neural network-based sequence-to-sequence models in various word-level NLP tasks.
Phonotactic Complexity and its Trade-offs
We present methods for calculating a measure of phonotactic complexity---bits per phoneme---that permits a straightforward cross-linguistic comparison.
It's Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Information
The performance of neural machine translation systems is commonly evaluated in terms of BLEU.
The Paradigm Discovery Problem
Our benchmark system first makes use of word embeddings and string similarity to cluster forms by cell and by paradigm.
A Tale of a Probe and a Parser
One such probe is the structural probe (Hewitt and Manning, 2019), designed to quantify the extent to which syntactic information is encoded in contextualised word representations.
On the Relationships Between the Grammatical Genders of Inanimate Nouns and Their Co-Occurring Adjectives and Verbs
We also find that there are statistically significant relationships between the grammatical genders of inanimate nouns and the verbs that take those nouns as direct objects, as indirect objects, and as subjects.
Generalized Entropy Regularization or: There's Nothing Special about Label Smoothing
Prior work has explored directly regularizing the output distributions of probabilistic models to alleviate peaky (i. e. over-confident) predictions, a common sign of overfitting.
UniMorph 3.0: Universal Morphology
The project comprises two major thrusts: a language-independent feature schema for rich morphological annotation and a type-level resource of annotated data in diverse languages realizing that schema.
Predicting Declension Class from Form and Meaning
We find for two Indo-European languages (Czech and German) that form and meaning respectively share significant amounts of information with class (and contribute additional information above and beyond gender).
Information-Theoretic Probing for Linguistic Structure
The success of neural networks on a diverse set of NLP tasks has led researchers to question how much these networks actually ``know'' about natural language.
Parameter Space Factorization for Zero-Shot Learning across Tasks and Languages
In this work, we propose a Bayesian generative model for the space of neural parameters.
Morphological Segmentation Inside-Out
Morphological segmentation has traditionally been modeled with non-hierarchical models, which yield flat segmentations as output.
Weird Inflects but OK: Making Sense of Morphological Generation Errors
We conduct a manual error analysis of the CoNLL-SIGMORPHON Shared Task on Morphological Reinflection.
Quantifying the Semantic Core of Gender Systems
To that end, we use canonical correlation analysis to correlate the grammatical gender of inanimate nouns with an externally grounded definition of their lexical semantics.
The SIGMORPHON 2019 Shared Task: Morphological Analysis in Context and Cross-Lingual Transfer for Inflection
The SIGMORPHON 2019 shared task on cross-lingual transfer and contextual analysis in morphology examined transfer learning of inflection between 100 language pairs, as well as contextual lemmatization and morphosyntactic description in 66 languages.
Don't Forget the Long Tail! A Comprehensive Analysis of Morphological Generalization in Bilingual Lexicon Induction
Human translators routinely have to translate rare inflections of words - due to the Zipfian distribution of words in a language.
Examining Gender Bias in Languages with Grammatical Gender
Recent studies have shown that word embeddings exhibit gender bias inherited from the training corpora.
It's All in the Name: Mitigating Gender Bias with Name-Based Counterfactual Data Substitution
An alternative approach is Counterfactual Data Augmentation (CDA), in which a corpus is duplicated and augmented to remove bias, e. g. by swapping all inherently-gendered words in the copy.
Rethinking Phonotactic Complexity
In this work, we propose the use of phone-level language models to estimate phonotactic complexity{---}measured in bits per phoneme{---}which makes cross-linguistic comparison straightforward.
On the Distribution of Deep Clausal Embeddings: A Large Cross-linguistic Study
Embedding a clause inside another ({``}the girl [who likes cars [that run fast]] has arrived{''}) is a fundamental resource that has been argued to be a key driver of linguistic expressiveness.
Morphological Irregularity Correlates with Frequency
We present a study of morphological irregularity.
Uncovering Probabilistic Implications in Typological Knowledge Bases
The study of linguistic typology is rooted in the implications we find between linguistic features, such as the fact that languages with object-verb word ordering tend to have post-positions.
Meaning to Form: Measuring Systematicity as Information
A longstanding debate in semiotics centers on the relationship between linguistic signs and their corresponding semantics: is there an arbitrary relationship between a word form and its meaning, or does some systematic phenomenon pervade?
Unsupervised Discovery of Gendered Language through Latent-Variable Modeling
Studying the ways in which language is gendered has long been an area of interest in sociolinguistics.
Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology
Gender stereotypes are manifest in most of the world's languages and are consequently propagated or amplified by NLP systems.
What Kind of Language Is Hard to Language-Model?
Trying to answer the question of what features difficult languages have in common, we try and fail to reproduce our earlier (Cotterell et al., 2018) observation about morphological complexity and instead reveal far simpler statistics of the data that seem to drive complexity in a much larger sample.
Exact Hard Monotonic Attention for Character-Level Transduction
Our models achieve state-of-the-art performance on morphological inflection.
Contextualization of Morphological Inflection
Critical to natural language generation is the production of correctly inflected text.
Gender Bias in Contextualized Word Embeddings
In this paper, we quantify, analyze and mitigate gender bias exhibited in ELMo's contextualized word vectors.
Combining Sentiment Lexica with a Multi-View Variational Autoencoder
When assigning quantitative labels to a dataset, different methodologies may rely on different scales.
A Simple Joint Model for Improved Contextual Neural Lemmatization
English verbs have multiple forms.
A Probabilistic Generative Model of Linguistic Typology
In the principles-and-parameters framework, the structural features of languages depend on parameters that may be toggled on or off, with a single parameter often dictating the status of multiple features.
On the Idiosyncrasies of the Mandarin Chinese Classifier System
While idiosyncrasies of the Chinese classifier system have been a richly studied topic among linguists (Adams and Conklin, 1973; Erbaugh, 1986; Lakoff, 1986), not much work has been done to quantify them with statistical methods.
UniMorph 2.0: Universal Morphology
The Universal Morphology UniMorph project is a collaborative effort to improve how NLP handles complex morphology across the world's languages.
The CoNLL--SIGMORPHON 2018 Shared Task: Universal Morphological Reinflection
Apart from extending the number of languages involved in earlier supervised tasks of generating inflected forms, this year the shared task also featured a new second task which asked participants to inflect words in sentential context, similar to a cloze task.
Marrying Universal Dependencies and Universal Morphology
The Universal Dependencies (UD) and Universal Morphology (UniMorph) projects each present schemata for annotating the morphosyntactic details of language.
Generalizing Procrustes Analysis for Better Bilingual Dictionary Induction
Most recent approaches to bilingual dictionary induction find a linear alignment between the word vector spaces of two languages.
Hard Non-Monotonic Attention for Character-Level Transduction
We compare soft and hard non-monotonic attention experimentally and find that the exact algorithm significantly improves performance over the stochastic approximation and outperforms soft attention.
A Discriminative Latent-Variable Model for Bilingual Lexicon Induction
We introduce a novel discriminative latent variable model for bilingual lexicon induction.
Recurrent Neural Networks in Linguistic Theory: Revisiting Pinker and Prince (1988) and the Past Tense Debate
We suggest that the empirical performance of modern networks warrants a re-examination of their utility in linguistic and cognitive modeling.
On the Complexity and Typology of Inflectional Morphological Systems
We quantify the linguistic complexity of different languages' morphological systems.
A Deep Generative Model of Vowel Formant Typology
What makes some types of languages more probable than others?
Explaining and Generalizing Back-Translation through Wake-Sleep
Back-translation has become a commonly employed heuristic for semi-supervised neural machine translation.
A Structured Variational Autoencoder for Contextual Morphological Inflection
Statistical morphological inflectors are typically trained on fully supervised, type-level data.
Are All Languages Equally Hard to Language-Model?
For general modeling methods applied to diverse languages, a natural question is: how well should we expect our models to work on languages with differing typological profiles?
Unsupervised Disambiguation of Syncretism in Inflected Lexicons
Lexical ambiguity makes it difficult to compute various useful statistics of a corpus.
On the Diachronic Stability of Irregularity in Inflectional Morphology
Many languages' inflectional morphological systems are replete with irregulars, i. e., words that do not seem to follow standard inflectional rules.
Cross-lingual Character-Level Neural Morphological Tagging
Even for common NLP tasks, sufficient supervision is not available in many languages {--} morphological tagging is no exception.
Paradigm Completion for Derivational Morphology
The generation of complex derived word forms has been an overlooked problem in NLP; we fill this gap by applying neural sequence-to-sequence models to the task.
Cross-lingual, Character-Level Neural Morphological Tagging
Even for common NLP tasks, sufficient supervision is not available in many languages -- morphological tagging is no exception.
Frame-Based Continuous Lexical Semantics through Exponential Family Tensor Factorization and Semantic Proto-Roles
We study how different frame annotations complement one another when learning continuous lexical semantics.
CoNLL-SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection in 52 Languages
In sub-task 2, systems were given a lemma and some of its specific inflected forms, and asked to complete the inflectional paradigm by predicting all of the remaining inflected forms.
Probabilistic Typology: Deep Generative Models of Vowel Inventories
Linguistic typology studies the range of structures present in human language.
Morphological Analysis of the Dravidian Language Family
The Dravidian languages are one of the most widely spoken language families in the world, yet there are very few annotated resources available to NLP researchers.
A Rich Morphological Tagger for English: Exploring the Cross-Linguistic Tradeoff Between Morphology and Syntax
A traditional claim in linguistics is that all human languages are equally expressive{---}able to convey the same wide range of meanings.
Neural Graphical Models over Strings for Principal Parts Morphological Paradigm Completion
Many of the world{'}s languages contain an abundance of inflected forms for each lexeme.
Explaining and Generalizing Skip-Gram through Exponential Family Principal Component Analysis
The popular skip-gram model induces word embeddings by exploiting the signal from word-context coocurrence.
One-Shot Neural Cross-Lingual Transfer for Paradigm Completion
We present a novel cross-lingual transfer method for paradigm completion, the task of mapping a lemma to its inflected forms, using a neural encoder-decoder model, the state of the art for the monolingual task.
Context-Aware Prediction of Derivational Word-forms
Derivational morphology is a fundamental and complex characteristic of language.
Joint Semantic Synthesis and Morphological Analysis of the Derived Word
Since morphology obeys the principle of compositionality, the semantics of the word can be systematically derived from the meaning of its parts.
Neural Multi-Source Morphological Reinflection
We explore the task of multi-source morphological reinflection, which generalizes the standard, single-source version.
An Analysis of Lemmatization on Topic Models of Morphologically Rich Language
Topic models are typically represented by top-$m$ word lists for human interpretation.
