Search Results for author:
Found 53 papers, 24 papers with code
PARADISE”:" Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretraining
Despite the success of multilingual sequence-to-sequence pretraining, most existing approaches rely on monolingual corpora and do not make use of the strong cross-lingual signal contained in parallel data.
Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Models
On text benchmarks, Core not only performs competitively to other frontier models on a set of well-established benchmarks (e. g. MMLU, GSM8K) but also outperforms GPT4-0613 on human evaluation.
Latxa: An Open Language Model and Evaluation Suite for Basque
We introduce Latxa, a family of large language models for Basque ranging from 7 to 70 billion parameters.
Gender-specific Machine Translation with Large Language Models
While machine translation (MT) systems have seen significant improvements, it is still common for translations to reflect societal biases, such as gender bias.
The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants
We use this dataset to evaluate the capabilities of multilingual masked language models (MLMs) and large language models (LLMs).
Evaluation of Faithfulness Using the Longest Supported Subsequence
Our proposed metric demonstrates an 18% enhancement over the prevailing state-of-the-art metric for faithfulness on our dataset.
Do Multilingual Language Models Think Better in English?
In this work, we introduce a new approach called self-translate, which overcomes the need of an external translation system by leveraging the few-shot translation capabilities of multilingual language models.
Common Sense Reasoning
Cross-Lingual Natural Language Inference
+6
Improving Language Plasticity via Pretraining with Active Forgetting
Pretrained language models (PLMs) are today the primary model for natural language processing.
CombLM: Adapting Black-Box Language Models through Small Fine-Tuned Models
Methods for adapting language models (LMs) to new tasks and domains have traditionally assumed white-box access to the model, and work by modifying its parameters.
Revisiting Machine Translation for Cross-lingual Classification
Machine Translation (MT) has been widely used for cross-lingual classification, either by translating the test set into English and running inference with a monolingual model (translate-test), or translating the training set into the target languages and finetuning a multilingual model (translate-train).
Mini-Model Adaptation: Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Training
Prior work shows that it is possible to expand pretrained Masked Language Models (MLMs) to new languages by learning a new set of embeddings, while keeping the transformer body frozen.
On the Role of Parallel Data in Cross-lingual Transfer Learning
While prior work has established that the use of parallel data is conducive for cross-lingual learning, it is unclear if the improvements come from the data itself, or if it is the modeling of parallel interactions that matters.
Training Trajectories of Language Models Across Scales
Why do larger language models demonstrate more desirable behaviors?
Don't Prompt, Search! Mining-based Zero-Shot Learning with Language Models
Our results suggest that the success of prompting can partly be explained by the model being exposed to similar examples during pretraining, which can be directly retrieved through regular expressions.
State-of-the-art generalisation research in NLP: A taxonomy and review
We present a taxonomy for characterising and understanding generalisation research in NLP.
Prompting ELECTRA: Few-Shot Learning with Discriminative Pre-Trained Models
In this work, we adapt prompt-based few-shot learning to ELECTRA and show that it outperforms masked language models in a wide range of tasks.
On the Role of Bidirectionality in Language Model Pre-Training
Prior work on language model pre-training has explored different architectures and learning objectives, but differences in data, hyperparameters and evaluation make a principled comparison difficult.
PoeLM: A Meter- and Rhyme-Controllable Language Model for Unsupervised Poetry Generation
During inference, we build control codes for the desired meter and rhyme scheme, and condition our language model on them to generate formal verse poetry.
Principled Paraphrase Generation with Parallel Corpora
Round-trip Machine Translation (MT) is a popular choice for paraphrase generation, which leverages readily available parallel corpora for supervision.
Multilingual Machine Translation with Hyper-Adapters
We find that hyper-adapters are more parameter efficient than regular adapters, reaching the same performance with up to 12 times less parameters.
Lifting the Curse of Multilinguality by Pre-training Modular Transformers
Multilingual pre-trained models are known to suffer from the curse of multilinguality, which causes per-language performance to drop as they cover more languages.
OPT: Open Pre-trained Transformer Language Models
Large language models, which are often trained for hundreds of thousands of compute days, have shown remarkable capabilities for zero- and few-shot learning.
Ranked #2 on
Stereotypical Bias Analysis
on CrowS-Pairs
Does Corpus Quality Really Matter for Low-Resource Languages?
For instance, 66% of documents are rated as high-quality for EusCrawl, in contrast with <33% for both mC4 and CC100.
Efficient Language Modeling with Sparse all-MLP
All-MLP architectures have attracted increasing interest as an alternative to attention-based models.
Ranked #17 on
Question Answering
on StoryCloze
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Large language models (LMs) are able to in-context learn -- perform a new task via inference alone by conditioning on a few input-label pairs (demonstrations) and making predictions for new inputs.
Efficient Large Scale Language Modeling with Mixtures of Experts
This paper presents a detailed empirical study of how autoregressive MoE language models scale in comparison with dense models in a wide range of settings: in- and out-of-domain language modeling, zero- and few-shot priming, and full-shot fine-tuning.
Few-shot Learning with Multilingual Language Models
Large-scale generative language models such as GPT-3 are competitive few-shot learners.
PARADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretraining
Despite the success of multilingual sequence-to-sequence pretraining, most existing approaches rely on monolingual corpora, and do not make use of the strong cross-lingual signal contained in parallel data.
Beyond Offline Mapping: Learning Cross-lingual Word Embeddings through Context Anchoring
Recent research on cross-lingual word embeddings has been dominated by unsupervised mapping approaches that align monolingual embeddings.
Bilingual Lexicon Induction
Cross-Lingual Word Embeddings
+2
Unsupervised Multilingual Sentence Embeddings for Parallel Corpus Mining
Existing models of multilingual sentence embeddings require large parallel data resources which are not available for low-resource languages.
Multilingual Autoregressive Entity Linking
Moreover, in a zero-shot setting on languages with no training data at all, mGENRE treats the target language as a latent variable that is marginalized at prediction time.
Ranked #2 on
Entity Disambiguation
on Mewsli-9
(using extra training data)
Beyond Offline Mapping: Learning Cross Lingual Word Embeddings through Context Anchoring
Recent research on cross-lingual word embeddings has been dominated by unsupervised mapping approaches that align monolingual embeddings.
Bilingual Lexicon Induction
Cross-Lingual Word Embeddings
+2
Training Multilingual Machine Translation by Alternately Freezing Language-Specific Encoders-Decoders
We propose a modular architecture of language-specific encoder-decoders that constitutes a multilingual machine translation system that can be incrementally extended to new languages without the need for retraining the existing system when adding new languages.
A Call for More Rigor in Unsupervised Cross-lingual Learning
We review motivations, definition, approaches, and methodology for unsupervised cross-lingual learning and call for a more rigorous position in each of them.
Multilingual Machine Translation: Closing the Gap between Shared and Language-specific Encoder-Decoders
State-of-the-art multilingual machine translation relies on a universal encoder-decoder, which requires retraining the entire system to add new languages.
Translation Artifacts in Cross-lingual Transfer Learning
Both human and machine translation play a central role in cross-lingual transfer learning: many multilingual datasets have been created through professional translation services, and using machine translation to translate either the test set or the training set is a widely used transfer technique.
Do all Roads Lead to Rome? Understanding the Role of Initialization in Iterative Back-Translation
In this paper, we analyze the role that such initialization plays in iterative back-translation.
On the Cross-lingual Transferability of Monolingual Representations
This generalization ability has been attributed to the use of a shared subword vocabulary and joint training across multiple languages giving rise to deep multilingual abstractions.
Contextualized Translations of Phrasal Verbs with Distributional Compositional Semantics and Monolingual Corpora
The contextualization of meaning is carried out by means of distributional composition within a structured vector space with syntactic dependencies, and the bilingual space is created by means of transfer rules and a bilingual dictionary.
Bilingual Lexicon Induction through Unsupervised Machine Translation
A recent research line has obtained strong results on bilingual lexicon induction by aligning independently trained word embeddings in two languages and using the resulting cross-lingual embeddings to induce word translation pairs through nearest neighbor or related retrieval methods.
Analyzing the Limitations of Cross-lingual Word Embedding Mappings
Recent research in cross-lingual word embeddings has almost exclusively focused on offline methods, which independently train word embeddings in different languages and map them to a shared space through linear transformations.
Bilingual Lexicon Induction
Cross-Lingual Word Embeddings
+1
An Effective Approach to Unsupervised Machine Translation
While machine translation has traditionally relied on large amounts of parallel corpora, a recent research line has managed to train both Neural Machine Translation (NMT) and Statistical Machine Translation (SMT) systems using monolingual corpora only.
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
We introduce an architecture to learn joint multilingual sentence representations for 93 languages, belonging to more than 30 different families and written in 28 different scripts.
Cross-Lingual Bitext Mining
Cross-Lingual Document Classification
+6
Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings
Machine translation is highly sensitive to the size and quality of the training data, which has led to an increasing interest in collecting and filtering large parallel corpora.
Uncovering divergent linguistic information in word embeddings with lessons for intrinsic and extrinsic evaluation
Following the recent success of word embeddings, it has been argued that there is no such thing as an ideal representation for words, as different models tend to capture divergent and often mutually incompatible aspects like semantics/syntax and similarity/relatedness.
Unsupervised Statistical Machine Translation
While modern machine translation has relied on large parallel corpora, a recent line of work has managed to train Neural Machine Translation (NMT) systems from monolingual corpora only (Artetxe et al., 2018c; Lample et al., 2018).
Ranked #3 on
Machine Translation
on WMT2014 French-English
A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings
Recent work has managed to learn cross-lingual word embeddings without parallel data by mapping monolingual embeddings to a shared space through adversarial training.
Unsupervised Neural Machine Translation
In spite of the recent success of neural machine translation (NMT) in standard benchmarks, the lack of large parallel corpora poses a major practical problem for many language pairs.
Ranked #6 on
Machine Translation
on WMT2015 English-German
Learning bilingual word embeddings with (almost) no bilingual data
Most methods to learn bilingual word embeddings rely on large parallel corpora, which is difficult to obtain for most language pairs.
