Search Results for author:
Found 14 papers, 7 papers with code
RoAST: Robustifying Language Models via Adversarial Perturbation with Selective Training
Under a unified evaluation of fine-tuned LMs by incorporating four representative perspectives of model robustness, we demonstrate the effectiveness of RoAST compared to state-of-the-art fine-tuning methods on six different types of LMs, which indicates its usefulness in practice.
Co-training and Co-distillation for Quality Improvement and Compression of Language Models
2) The enhanced performance of the larger model further boosts the performance of the smaller model.
The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants
We use this dataset to evaluate the capabilities of multilingual masked language models (MLMs) and large language models (LLMs).
A Study on Knowledge Distillation from Weak Teacher for Scaling Up Pre-trained Language Models
Distillation from Weak Teacher (DWT) is a method of transferring knowledge from a smaller, weaker teacher model to a larger student model to improve its performance.
XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models
Large multilingual language models typically rely on a single vocabulary shared across 100+ languages.
Attention-guided Generative Models for Extractive Question Answering
We propose a simple strategy to obtain an extractive answer span from the generative model by leveraging the decoder cross-attention patterns.
Multiplicative Position-aware Transformer Models for Language Understanding
Transformer models, which leverage architectural improvements like self-attention, perform remarkably well on Natural Language Processing (NLP) tasks.
Decoding and Diversity in Machine Translation
This makes human-level BLEU a misleading benchmark in that modern MT systems cannot approach human-level BLEU while simultaneously maintaining human-level translation diversity.
Improve Transformer Models with Better Relative Position Embeddings
In this paper, we first review absolute position embeddings and existing methods for relative position embeddings.
Embedding-based Zero-shot Retrieval through Query Generation
In some cases, our model trained on synthetic data can even outperform the same model trained on real data
TRANS-BLSTM: Transformer with Bidirectional LSTM for Language Understanding
Prior to the transformer era, bidirectional Long Short-Term Memory (BLSTM) has been the dominant modeling architecture for neural machine translation and question answering.
Ranked #1 on
Text Classification
on GLUE MRPC
Masked Language Model Scoring
Instead, we evaluate MLMs out of the box via their pseudo-log-likelihood scores (PLLs), which are computed by masking tokens one by one.
Learning Noise-Invariant Representations for Robust Speech Recognition
Despite rapid advances in speech recognition, current models remain brittle to superficial perturbations to their inputs.
Deep Automated Multi-task Learning
Our results show that training on a primary task in parallel with a secondary automated task improves both the convergence speed and accuracy for the primary task.
