Search Results for author:
Found 123 papers, 20 papers with code
Language Model Cascades: Token-level uncertainty and beyond
While the principles underpinning cascading are well-studied for classification tasks - with deferral based on predicted class uncertainty favored theoretically and practically - a similar understanding is lacking for generative LM tasks.
Towards Fast Inference: Exploring and Improving Blockwise Parallel Drafts
Despite the remarkable strides made by autoregressive language models, their potential is often hampered by the slow inference speeds inherent in sequential token generation.
SOAR: Improved Indexing for Approximate Nearest Neighbor Search
This paper introduces SOAR: Spilling with Orthogonality-Amplified Residuals, a novel data indexing technique for approximate nearest neighbor (ANN) search.
Metric-aware LLM inference for regression and scoring
Large language models (LLMs) have demonstrated strong results on a range of NLP tasks.
HiRE: High Recall Approximate Top-$k$ Estimation for Efficient LLM Inference
Autoregressive decoding with generative Large Language Models (LLMs) on accelerators (GPUs/TPUs) is often memory-bound where most of the time is spent on transferring model parameters from high bandwidth memory (HBM) to cache.
Tandem Transformers for Inference Efficient LLMs
On the PaLM2 pretraining dataset, a tandem of PaLM2-Bison and PaLM2-Gecko demonstrates a 3. 3% improvement in next-token prediction accuracy over a standalone PaLM2-Gecko, offering a 1. 16x speedup compared to a PaLM2-Otter model with comparable downstream performance.
Efficient Stagewise Pretraining via Progressive Subnetworks
RaPTr achieves better pre-training loss for BERT and UL2 language models while requiring 20-33% fewer FLOPs compared to standard training, and is competitive or better than other efficient training methods.
SpacTor-T5: Pre-training T5 Models with Span Corruption and Replaced Token Detection
In this paper, we present SpacTor, a new training procedure consisting of (1) a hybrid objective combining span corruption (SC) and token replacement detection (RTD), and (2) a two-stage curriculum that optimizes the hybrid objective over the initial $\tau$ iterations, then transitions to standard SC loss.
A Weighted K-Center Algorithm for Data Subset Selection
The success of deep learning hinges on enormous data and large models, which require labor-intensive annotations and heavy computation costs.
ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agent
Answering complex natural language questions often necessitates multi-step reasoning and integrating external information.
Ranked #1 on
Question Answering
on Bamboogle
Rethinking FID: Towards a Better Evaluation Metric for Image Generation
It is an unbiased estimator that does not make any assumptions on the probability distribution of the embeddings and is sample efficient.
It's an Alignment, Not a Trade-off: Revisiting Bias and Variance in Deep Models
Classical wisdom in machine learning holds that the generalization error can be decomposed into bias and variance, and these two terms exhibit a \emph{trade-off}.
DistillSpec: Improving Speculative Decoding via Knowledge Distillation
Finally, in practical scenarios with models of varying sizes, first using distillation to boost the performance of the target model and then applying DistillSpec to train a well-aligned draft model can reduce decoding latency by 6-10x with minimal performance drop, compared to standard decoding without distillation.
What do larger image classifiers memorise?
The success of modern neural networks has prompted study of the connection between memorisation and generalisation: overparameterised models generalise well, despite being able to perfectly fit (memorise) completely random labels.
Functional Interpolation for Relative Positions Improves Long Context Transformers
Preventing the performance decay of Transformers on inputs longer than those used for training has been an important challenge in extending the context length of these models.
Think before you speak: Training Language Models With Pause Tokens
Language models generate responses by producing a series of tokens in immediate succession: the $(K+1)^{th}$ token is an outcome of manipulating $K$ hidden vectors per layer, one vector per preceding token.
MarkovGen: Structured Prediction for Efficient Text-to-Image Generation
Modern text-to-image generation models produce high-quality images that are both photorealistic and faithful to the text prompts.
When Does Confidence-Based Cascade Deferral Suffice?
Cascades are a classical strategy to enable inference cost to vary adaptively across samples, wherein a sequence of classifiers are invoked in turn.
Depth Dependence of $μ$P Learning Rates in ReLU MLPs
In this short note we consider random fully connected ReLU networks of width $n$ and depth $L$ equipped with a mean-field weight initialization.
On student-teacher deviations in distillation: does it pay to disobey?
Knowledge distillation (KD) has been widely used to improve the test accuracy of a "student" network, by training it to mimic the soft probabilities of a trained "teacher" network.
Plugin estimators for selective classification with out-of-distribution detection
Recent work on selective classification with OOD detection (SCOD) has argued for the unified study of these problems; however, the formal underpinnings of this problem are still nascent, and existing techniques are heuristic in nature.
Out-of-Distribution Detection
Out of Distribution (OOD) Detection
Supervision Complexity and its Role in Knowledge Distillation
Despite the popularity and efficacy of knowledge distillation, there is limited understanding of why it helps.
Leveraging Importance Weights in Subset Selection
In such a setting, an algorithm can sample examples one at a time but, in order to limit overhead costs, is only able to update its state (i. e. further train model weights) once a large enough batch of examples is selected.
EmbedDistill: A Geometric Knowledge Distillation for Information Retrieval
Large neural models (such as Transformers) achieve state-of-the-art performance for information retrieval (IR).
Automating Nearest Neighbor Search Configuration with Constrained Optimization
The approximate nearest neighbor (ANN) search problem is fundamental to efficiently serving many real-world machine learning applications.
Large Language Models with Controllable Working Memory
By contrast, when the context is irrelevant to the task, the model should ignore it and fall back on its internal knowledge.
Two-stage LLM Fine-tuning with Less Specialization and More Generalization
Pretrained large language models (LLMs) are general purpose problem solvers applicable to a diverse set of tasks with prompts.
When does mixup promote local linearity in learned representations?
Towards this, we study two questions: (1) how does the Mixup loss that enforces linearity in the \emph{last} network layer propagate the linearity to the \emph{earlier} layers?
The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers
This paper studies the curious phenomenon for machine learning models with Transformer architectures that their activation maps are sparse.
Decoupled Context Processing for Context Augmented Language Modeling
Language models can be augmented with a context retriever to incorporate knowledge from large external databases.
Teacher Guided Training: An Efficient Framework for Knowledge Transfer
In this paper, we propose the teacher-guided training (TGT) framework for training a high-quality compact model that leverages the knowledge acquired by pretrained generative models, while obviating the need to go through a large volume of data.
TPU-KNN: K Nearest Neighbor Search at Peak FLOP/s
This paper presents a novel nearest neighbor search algorithm achieving TPU (Google Tensor Processing Unit) peak performance, outperforming state-of-the-art GPU algorithms with similar level of recall.
ELM: Embedding and Logit Margins for Long-Tail Learning
Long-tail learning is the problem of learning under skewed label distributions, which pose a challenge for standard learners.
Predicting on the Edge: Identifying Where a Larger Model Does Better
Much effort has been devoted to making large and more accurate models, but relatively little has been put into understanding which examples are benefiting from the added complexity.
Robust Training of Neural Networks Using Scale Invariant Architectures
In contrast to SGD, adaptive gradient methods like Adam allow robust training of modern deep networks, especially large language models.
Efficient Training of Retrieval Models using Negative Cache
These models are typically trained by optimizing the model parameters to score relevant positive" pairs higher than the irrelevantnegative" ones.
When in Doubt, Summon the Titans: Efficient Inference with Large Models
In a nutshell, we use the large teacher models to guide the lightweight student models to only make correct predictions on a subset of "easy" examples; for the "hard" examples, we fall-back to the teacher.
Leveraging redundancy in attention with Reuse Transformers
Pairwise dot product-based attention allows Transformers to exchange information between tokens in an input-dependent way, and is key to their success across diverse applications in language and vision.
In defense of dual-encoders for neural ranking
Transformer-based models such as BERT have proven successful in information retrieval problem, which seek to identify relevant documents for a given query.
Less data is more: Selecting informative and diverse subsets with balancing constraints
Deep learning has yielded extraordinary results in vision and natural language processing, but this achievement comes at a cost.
When in Doubt, Summon the Titans: A Framework for Efficient Inference with Large Models
In a nutshell, we use the large teacher models to guide the lightweight student models to only make correct predictions on a subset of "easy" examples; for the "hard" examples, we fall-back to the teacher.
Model-Efficient Deep Learning with Kernelized Classification
We investigate the possibility of using the embeddings produced by a lightweight network more effectively with a nonlinear classification layer.
Batch Active Learning at Scale
The ability to train complex and highly effective models often requires an abundance of training data, which can easily become a bottleneck in cost, time, and computational resources.
Teacher's pet: understanding and mitigating biases in distillation
Knowledge distillation is widely used as a means of improving the performance of a relatively simple student model using the predictions from a complex teacher model.
Eigen Analysis of Self-Attention and its Reconstruction from Partial Computation
State-of-the-art transformer models use pairwise dot-product based self-attention, which comes at a computational cost quadratic in the input sequence length.
Scaling Hierarchical Agglomerative Clustering to Billion-sized Datasets
Hierarchical Agglomerative Clustering (HAC) is one of the oldest but still most widely used clustering methods.
Balancing Robustness and Sensitivity using Feature Contrastive Learning
It is generally believed that robust training of extremely large networks is critical to their success in real-world applications.
Disentangling Sampling and Labeling Bias for Learning in Large-Output Spaces
Negative sampling schemes enable efficient training given a large number of classes, by offering a means to approximate a computationally expensive loss function that takes all labels into account.
Less is more: Selecting informative and diverse subsets with balancing constraints
Deep learning has yielded extraordinary results in vision and natural language processing, but this achievement comes at a cost.
RankDistil: Knowledge Distillation for Ranking
Knowledge distillation is an approach to improve the performance of a student model by using the knowledge of a complex teacher. Despite its success in several deep learning applications, the study of distillation is mostly confined to classification settings.
On the Reproducibility of Neural Network Predictions
By analyzing the relationship between churn and prediction confidences, we pursue an approach with two components for churn reduction.
Overparameterisation and worst-case generalisation: friend or foe?
Overparameterised neural networks have demonstrated the remarkable ability to perfectly fit training samples, while still generalising to unseen test samples.
Kernelized Classification in Deep Networks
We propose a kernelized classification layer for deep networks.
O(n) Connections are Expressive Enough: Universal Approximability of Sparse Transformers
We propose sufficient conditions under which we prove that a sparse attention model can universally approximate any sequence-to-sequence function.
Modifying Memories in Transformer Models
In this paper, we propose a new task of \emph{explicitly modifying specific factual knowledge in Transformer models while ensuring the model performance does not degrade on the unmodified facts}.
Coping with Label Shift via Distributionally Robust Optimisation
The label shift problem refers to the supervised learning setting where the train and test label distributions do not match.
Semantic Label Smoothing for Sequence to Sequence Problems
Label smoothing has been shown to be an effective regularization strategy in classification, that prevents overfitting and helps in label de-noising.
Learning discrete distributions: user vs item-level privacy
If each user has $m$ samples, we show that straightforward applications of Laplace or Gaussian mechanisms require the number of users to be $\mathcal{O}(k/(m\alpha^2) + k/\epsilon\alpha)$ to achieve an $\ell_1$ distance of $\alpha$ between the true and estimated distributions, with the privacy-induced penalty $k/\epsilon\alpha$ independent of the number of samples per user $m$.
Multi-Stage Influence Function
With this score, we can identify the pretraining examples in the pretraining task that contribute most to a prediction in the finetuning task.
Long-tail learning via logit adjustment
Real-world classification problems typically exhibit an imbalanced or long-tailed label distribution, wherein many labels are associated with only a few samples.
Ranked #48 on
Long-tail Learning
on ImageNet-LT
$O(n)$ Connections are Expressive Enough: Universal Approximability of Sparse Transformers
We propose sufficient conditions under which we prove that a sparse attention model can universally approximate any sequence-to-sequence function.
Evaluations and Methods for Explanation through Robustness Analysis
In this paper, we establish a novel set of evaluation criteria for such feature based explanations by robustness analysis.
Why distillation helps: a statistical perspective
In this paper, we present a statistical perspective on distillation which addresses this question, and provides a novel connection to extreme multiclass retrieval techniques.
Can gradient clipping mitigate label noise?
Gradient clipping is a widely-used technique in the training of deep networks, and is generally motivated from an optimisation lens: informally, it controls the dynamics of iterates, thus enhancing the rate of convergence to a local minimum.
Doubly-stochastic mining for heterogeneous retrieval
Modern retrieval problems are characterised by training sets with potentially billions of labels, and heterogeneous data distributions across subpopulations (e. g., users of a retrieval system may be from different countries), each of which poses a challenge.
Federated Learning with Only Positive Labels
We consider learning a multi-class classification model in the federated setting, where each user has access to the positive data associated with only a single class.
Robust Large-Margin Learning in Hyperbolic Space
In this paper, we present, to our knowledge, the first theoretical guarantees for learning a classifier in hyperbolic rather than Euclidean space.
Does label smoothing mitigate label noise?
Label smoothing is commonly used in training deep learning models, wherein one-hot training labels are mixed with uniform label vectors.
Ranked #12 on
Learning with noisy labels
on CIFAR-10N-Random3
Adaptive Federated Optimization
Federated learning is a distributed machine learning paradigm in which a large number of clients coordinate with a central server to learn a model without sharing their own training data.
Low-Rank Bottleneck in Multi-head Attention Models
Attention based Transformer architecture has enabled significant advances in the field of natural language processing.
Pre-training Tasks for Embedding-based Large-scale Retrieval
We consider the large-scale query-document retrieval problem: given a query (e. g., a question), return the set of relevant documents (e. g., paragraphs containing the answer) from a large document corpus.
New Loss Functions for Fast Maximum Inner Product Search
In this work, we focus directly on minimizing error in inner product approximation and derive a new class of quantization loss functions.
Are Transformers universal approximators of sequence-to-sequence functions?
In this paper, we establish that Transformer models are universal approximators of continuous permutation equivariant sequence-to-sequence functions with compact support, which is quite surprising given the amount of shared parameters in these models.
Why are Adaptive Methods Good for Attention Models?
While stochastic gradient descent (SGD) is still the \emph{de facto} algorithm in deep learning, adaptive methods like Clipped SGD/Adam have been observed to outperform SGD across important tasks, such as attention models.
Multilabel reductions: what is my loss optimising?
Multilabel classification is a challenging problem arising in applications ranging from information retrieval to image tagging.
Breaking the Glass Ceiling for Embedding-Based Classifiers for Large Output Spaces
In extreme classification settings, embedding-based neural network models are currently not competitive with sparse linear and tree-based methods in terms of accuracy.
Learning to Learn by Zeroth-Order Oracle
In the learning to learn (L2L) framework, we cast the design of optimization algorithms as a machine learning problem and use deep neural networks to learn the update rules.
Concise Multi-head Attention Models
Attention based Transformer architecture has enabled significant advances in the field of natural language processing.
LEARNING TO LEARN WITH BETTER CONVERGENCE
We consider the learning to learn problem, where the goal is to leverage deeplearning models to automatically learn (iterative) optimization algorithms for training machine learning models.
Why ADAM Beats SGD for Attention Models
While stochastic gradient descent (SGD) is still the de facto algorithm in deep learning, adaptive methods like Adam have been observed to outperform SGD across important tasks, such as attention models.
Online Hierarchical Clustering Approximations
The second algorithm, OHAC, is an online counterpart to offline HAC, which is known to yield a 1/3-approximation to the MW revenue, and produce good quality clusters in practice.
Accelerating Large-Scale Inference with Anisotropic Vector Quantization
Based on the observation that for a given query, the database points that have the largest inner products are more relevant, we develop a family of anisotropic quantization loss functions.
AdaCliP: Adaptive Clipping for Private SGD
Motivated by this, differentially private stochastic gradient descent (SGD) algorithms for training machine learning models have been proposed.
Sampled Softmax with Random Fourier Features
For the settings where a large number of classes are involved, a common method to speed up training is to sample a subset of classes and utilize an estimate of the loss gradient based on these classes, known as the sampled softmax method.
Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise
In this paper, we propose a new continuous neural network framework called Neural Stochastic Differential Equation (Neural SDE) network, which naturally incorporates various commonly used regularization mechanisms based on random noise injection.
On the Convergence of Adam and Beyond
Several recently proposed stochastic optimization methods that have been successfully used in training deep networks such as RMSProp, Adam, Adadelta, Nadam are based on using gradient updates scaled by square roots of exponential moving averages of squared past gradients.
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
In this paper, we first study a principled layerwise adaptation strategy to accelerate training of deep neural networks using large mini-batches.
Ranked #11 on
Question Answering
on SQuAD1.1 dev
(F1 metric)
Local Orthogonal Decomposition for Maximum Inner Product Search
More specifically, we decompose a residual vector locally into two orthogonal components and perform uniform quantization and multiscale quantization to each component respectively.
Efficient Inner Product Approximation in Hybrid Spaces
In this paper, we propose a technique that approximates the inner product computation in hybrid vectors, leading to substantial speedup in search while maintaining high accuracy.
Escaping Saddle Points with Adaptive Gradient Methods
Adaptive methods such as Adam and RMSProp are widely used in deep learning but are not well understood.
Adaptive Methods for Nonconvex Optimization
In this work, we provide a new analysis of such methods applied to nonconvex stochastic optimization problems, characterizing the effect of increasing minibatch size.
Learning to Screen for Fast Softmax Inference on Large Vocabulary Neural Networks
The algorithm achieves an order of magnitude faster inference than the original softmax layer for predicting top-$k$ words in various tasks such as beam search in machine translation or next words prediction.
Stochastic Negative Mining for Learning with Large Output Spaces
Furthermore, we identify a particularly intuitive class of loss functions in the aforementioned family and show that they are amenable to practical implementation in the large output space setting (i. e. computation is possible without evaluating scores of all labels) by developing a technique called Stochastic Negative Mining.
Privacy and Utility Tradeoff in Approximate Differential Privacy
We show that the multiplicative gap of the lower bounds and upper bounds goes to zero in various high privacy regimes, proving the tightness of the lower and upper bounds and thus establishing the optimality of the truncated Laplacian mechanism.
Optimal Noise-Adding Mechanism in Additive Differential Privacy
We derive the optimal $(0, \delta)$-differentially private query-output independent noise-adding mechanism for single real-valued query function under a general cost-minimization framework.
Loss Decomposition for Fast Learning in Large Output Spaces
For problems with large output spaces, evaluation of the loss function and its gradient are expensive, typically taking linear time in the size of the output space.
Learning a Compressed Sensing Measurement Matrix via Gradient Unrolling
Our experiments show that there is indeed additional structure beyond sparsity in the real datasets; our method is able to discover it and exploit it to create excellent reconstructions with fewer measurements (by a factor of 1. 1-3x) compared to the previous state-of-the-art methods.
cpSGD: Communication-efficient and differentially-private distributed SGD
Distributed stochastic gradient descent is an important subroutine in distributed learning.
Nonlinear Online Learning with Adaptive Nyström Approximation
Use of nonlinear feature maps via kernel approximation has led to success in many online learning tasks.
Multiscale Quantization for Fast Similarity Search
We propose a multiscale quantization approach for fast similarity search on large, high-dimensional datasets.
Now Playing: Continuous low-power music recognition
To reduce battery consumption, a small music detector runs continuously on the mobile device's DSP chip and wakes up the main application processor only when it is confident that music is present.
Learning Spread-out Local Feature Descriptors
We propose a simple, yet powerful regularization technique that can be used to significantly improve both the pairwise and triplet losses in learning local feature descriptors.
Efficient Natural Language Response Suggestion for Smart Reply
This paper presents a computationally efficient machine-learned method for natural language response suggestion.
Stochastic Generative Hashing
Learning-based binary hashing has become a powerful paradigm for fast search and retrieval in massive databases.
Distributed Mean Estimation with Limited Communication
Motivated by the need for distributed learning and optimization algorithms with low communication cost, we study communication efficient algorithms for distributed mean estimation.
Orthogonal Random Features
We present an intriguing discovery related to Random Fourier Features: in Gaussian kernel approximation, replacing the random Gaussian matrix by a properly scaled random orthogonal matrix significantly decreases kernel approximation error.
Fast Orthogonal Projection Based on Kronecker Product
We propose a family of structured matrices to speed up orthogonal projections for high-dimensional data commonly seen in computer vision applications.
Spherical Random Features for Polynomial Kernels
Among the commonly used kernels for nonlinear classification are polynomial kernels, for which low approximation error has thus far necessitated explicit feature maps of large dimensionality, especially for higher-order polynomials.
On Binary Embedding using Circulant Matrices
To address this problem, we propose Circulant Binary Embedding (CBE) which generates binary codes by projecting the data with a circulant matrix.
Binary embeddings with structured hashed projections
We prove several theoretical results showing that projections via various structured matrices followed by nonlinear mappings accurately preserve the angular distance between input high-dimensional vectors.
Structured Transforms for Small-Footprint Deep Learning
We consider the task of building compact deep learning pipelines suitable for deployment on storage and power constrained mobile devices.
Learning to Hash for Indexing Big Data - A Survey
Such learning to hash methods exploit information such as data distributions or class labels when optimizing the hash codes or functions.
Quantization based Fast Inner Product Search
We propose a quantization based approach for fast approximate Maximum Inner Product Search (MIPS).
Fast Online Clustering with Randomized Skeleton Sets
To achieve fast clustering, we propose to represent each cluster by a skeleton set which is updated continuously as new data is seen.
Compact Nonlinear Maps and Circulant Extensions
This leads to much more compact maps without hurting the performance.
An exploration of parameter redundancy in deep networks with circulant projections
We explore the redundancy of parameters in deep neural networks by replacing the conventional linear projection in fully-connected layers with the circulant projection.
Discrete Graph Hashing
Hashing has emerged as a popular technique for fast nearest neighbor search in gigantic databases.
Circulant Binary Embedding
To address this problem, we propose Circulant Binary Embedding (CBE) which generates binary codes by projecting the data with a circulant matrix.
On Learning from Label Proportions
Learning from Label Proportions (LLP) is a learning setting, where the training data is provided in groups, or "bags", and only the proportion of each class in each bag is known.
$\propto$SVM for learning with label proportions
We study the problem of learning with label proportions in which the training data is provided in groups and only the proportion of each class in each group is known.
Learning Binary Codes for High-Dimensional Data Using Bilinear Projections
Recent advances in visual recognition indicate that to achieve good retrieval and classification accuracy on largescale datasets like ImageNet, extremely high-dimensional visual descriptors, e. g., Fisher Vectors, are needed.
Angular Quantization-based Binary Codes for Fast Similarity Search
Such data typically arises in a large number of vision and text applications where counts or frequencies are used as features.
Ensemble Nystrom Method
A crucial technique for scaling kernel methods to very large data sets reaching or exceeding millions of instances is based on low-rank approximation of kernel matrices.
