Search Results for author:
Found 33 papers, 11 papers with code
Is margin all you need? An extensive empirical study of active learning on tabular data
Given a labeled training set and a collection of unlabeled data, the goal of active learning (AL) is to identify the best unlabeled points to label.
Confident Adaptive Language Modeling
Recent advances in Transformer-based large language models (LLMs) have led to significant performance improvements across many tasks.
UL2: Unifying Language Learning Paradigms
Our model also achieve strong results at in-context learning, outperforming 175B GPT-3 on zero-shot SuperGLUE and tripling the performance of T5-XXL on one-shot summarization.
Ranked #1 on
Long-range modeling
on SCROLLS
(CNLI metric)
ED2LM: Encoder-Decoder to Language Model for Faster Document Re-ranking Inference
This results in significant inference time speedups since the decoder-only architecture only needs to learn to interpret static encoder embeddings during inference.
Transformer Memory as a Differentiable Search Index
In this paper, we demonstrate that information retrieval can be accomplished with a single Transformer, in which all information about the corpus is encoded in the parameters of the model.
ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning
Despite the recent success of multi-task learning and transfer learning for natural language processing (NLP), few works have systematically studied the effect of scaling up the number of tasks during pre-training.
Sharpness-Aware Minimization Improves Language Model Generalization
The allure of superhuman-level capabilities has led to considerable interest in language models like GPT-3 and T5, wherein the research has, by and large, revolved around new model architectures, training tasks, and loss objectives, along with substantial engineering efforts to scale up model capacity and dataset size.
An Empirical Study of Pre-trained Models on Out-of-distribution Generalization
Concretely, we show that larger models and larger datasets need to be simultaneously leveraged to improve OOD performance.
Are Pretrained Convolutions Better than Pretrained Transformers?
In the context of language models, are convolutional models competitive to Transformers when pre-trained?
SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption
Self-supervised contrastive representation learning has proved incredibly successful in the vision and natural language domains, enabling state-of-the-art performance with orders of magnitude less labeled data.
Charformer: Fast Character Transformers via Gradient-based Subword Tokenization
In this paper, we propose a new model inductive bias that learns a subword tokenization end-to-end as part of the model.
Ranked #3 on
Paraphrase Identification
on Quora Question Pairs
Churn Reduction via Distillation
In real-world systems, models are frequently updated as more data becomes available, and in addition to achieving high accuracy, the goal is to also maintain a low difference in predictions compared to the base model (i. e. predictive "churn").
Are Pre-trained Convolutions Better than Pre-trained Transformers?
In the context of language models, are convolutional models competitive to Transformers when pre-trained?
Rethinking Search: Making Domain Experts out of Dilettantes
When experiencing an information need, users want to engage with a domain expert, but often turn to an information retrieval system, such as a search engine, instead.
OmniNet: Omnidirectional Representations from Transformers
In OmniNet, instead of maintaining a strictly horizontal receptive field, each token is allowed to attend to all tokens in the entire network.
Ranked #1 on
Machine Translation
on WMT2017 Russian-English
Locally Adaptive Label Smoothing for Predictive Churn
Training modern neural networks is an inherently noisy process that can lead to high \emph{prediction churn} -- disagreements between re-trainings of the same model due to factors such as randomization in the parameter initialization and mini-batches -- even when the trained models all attain similar accuracies.
Label Smoothed Embedding Hypothesis for Out-of-Distribution Detection
Detecting out-of-distribution (OOD) examples is critical in many applications.
Out-of-Distribution Detection
Out of Distribution (OOD) Detection
Long Range Arena : A Benchmark for Efficient Transformers
Transformers do not scale very well to long sequence lengths largely because of quadratic self-attention complexity.
HyperGrid Transformers: Towards A Single Model for Multiple Tasks
Specifically, we propose a decomposable hypernetwork that learns grid-wise projections that help to specialize regions in weight matrices for different tasks.
Synthesizer: Rethinking Self-Attention for Transformer Models
The dot product self-attention is known to be central and indispensable to state-of-the-art Transformer models.
Deep $k$-NN Label Smoothing Improves Reproducibility of Neural Network Predictions
Training modern neural networks is an inherently noisy process that can lead to high \emph{prediction churn}-- disagreements between re-trainings of the same model due to factors such as randomization in the parameter initialization and mini-batches-- even when the trained models all attain high accuracies.
StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modeling
There are two major classes of natural language grammar -- the dependency grammar that models one-to-one correspondences between words and the constituency grammar that models the assembly of one or several corresponded words.
Long Range Arena: A Benchmark for Efficient Transformers
In the recent months, a wide spectrum of efficient, fast Transformers have been proposed to tackle this problem, more often than not claiming superior or comparable model quality to vanilla Transformer models.
Ranked #18 on
Long-range modeling
on LRA
(Pathfinder metric)
Surprise: Result List Truncation via Extreme Value Theory
Work in information retrieval has largely been centered around ranking and relevance: given a query, return some number of results ordered by relevance to the user.
Efficient Transformers: A Survey
Transformer model architectures have garnered immense interest lately due to their effectiveness across a range of domains like language, vision and reinforcement learning.
Generative Models are Unsupervised Predictors of Page Quality: A Colossal-Scale Study
Large generative language models such as GPT-2 are well-known for their ability to generate text as well as their utility in supervised downstream tasks via fine-tuning.
HyperGrid: Efficient Multi-Task Transformers with Grid-wise Decomposable Hyper Projections
The proposed approach is based on a decomposable hypernetwork that learns grid-wise projections that help to specialize regions in weight matrices for different tasks.
Synthesizer: Rethinking Self-Attention in Transformer Models
The dot product self-attention is known to be central and indispensable to state-of-the-art Transformer models.
Ranked #1 on
Dialogue Generation
on Persona-Chat
(BLEU-1 metric, using extra
training data)
Deep k-NN for Noisy Labels
Modern machine learning models are often trained on examples with noisy labels that hurt performance and are hard to identify.
Choppy: Cut Transformer For Ranked List Truncation
Work in information retrieval has traditionally focused on ranking and relevance: given a query, return some number of results ordered by relevance to the user.
Reverse Engineering Configurations of Neural Text Generation Models
This paper seeks to develop a deeper understanding of the fundamental properties of neural text generations models.
Sparse Sinkhorn Attention
We propose Sparse Sinkhorn Attention, a new efficient and sparse method for learning to attend.
Diminishing Returns Shape Constraints for Interpretability and Regularization
We investigate machine learning models that can provide diminishing returns and accelerating returns guarantees to capture prior knowledge or policies about how outputs should depend on inputs.
