Search Results for author:
Found 54 papers, 29 papers with code
Long-form factuality in large language models
Empirically, we demonstrate that LLM agents can outperform crowdsourced human annotators - on a set of ~16k individual facts, SAFE agrees with crowdsourced human annotators 72% of the time, and on a random subset of 100 disagreement cases, SAFE wins 76% of the time.
Gemini: A Family of Highly Capable Multimodal Models
This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding.
Ranked #1 on
Multi-task Language Understanding
on MMLU
(using extra training data)
Larger language models do in-context learning differently
We next study semantically-unrelated label ICL (SUL-ICL), in which labels are semantically unrelated to their inputs (e. g., foo/bar instead of negative/positive), thereby forcing language models to learn the input-label mappings shown in in-context exemplars in order to perform the task.
A Simple Zero-shot Prompt Weighting Technique to Improve Prompt Ensembling in Text-Image Models
In particular, we ask "Given a large pool of prompts, can we automatically score the prompts and ensemble those that are most suitable for a particular downstream dataset, without needing access to labeled validation data?".
Scaling Vision Transformers to 22 Billion Parameters
The scaling of Transformers has driven breakthrough capabilities for language models.
Ranked #1 on
Zero-Shot Transfer Image Classification
on ObjectNet
Benchmarking Bayesian Deep Learning on Diabetic Retinopathy Detection Tasks
We use these tasks to benchmark well-established and state-of-the-art Bayesian deep learning methods on task-specific evaluation metrics.
Plex: Towards Reliability using Pretrained Large Model Extensions
A recent trend in artificial intelligence is the use of pretrained models for language and vision tasks, which have achieved extraordinary performance but also puzzling failures.
A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awareness
The most popular approaches to estimate predictive uncertainty in deep learning are methods that combine predictions from multiple neural networks, such as Bayesian neural networks (BNNs) and deep ensembles.
Sparse MoEs meet Efficient Ensembles
Machine learning models based on the aggregated outputs of submodels, either at the activation or prediction levels, often exhibit strong performance compared to individual models.
Deep Classifiers with Label Noise Modeling and Distance Awareness
Uncertainty estimation in deep learning has recently emerged as a crucial area of interest to advance reliability and robustness in safety-critical applications.
Soft Calibration Objectives for Neural Networks
When incorporated into training, these soft calibration losses achieve state-of-the-art single-model ECE across multiple datasets with less than 1% decrease in accuracy.
Revisiting the Calibration of Modern Neural Networks
Accurate estimation of predictive uncertainty (model calibration) is essential for the safe application of neural networks.
Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning
In this paper we introduce Uncertainty Baselines: high-quality implementations of standard and state-of-the-art deep learning methods on a variety of tasks.
RecSim NG: Toward Principled Uncertainty Modeling for Recommender Ecosystems
The development of recommender systems that optimize multi-turn interaction with users, and model the interactions of different agents (e. g., users, content providers, vendors) in the recommender ecosystem have drawn increasing attention in recent years.
Distilling Ensembles Improves Uncertainty Estimates
We seek to bridge the performance gap between batch ensembles (ensembles of deep networks with shared parameters) and deep ensembles on tasks which require not only predictions, but also uncertainty estimates for these predictions.
Why Are Bootstrapped Deep Ensembles Not Better?
Ensemble methods have consistently reached state of the art across predictive, uncertainty, and out-of-distribution robustness benchmarks.
Combining Ensembles and Data Augmentation can Harm your Calibration
Ensemble methods which average over multiple neural network predictions are a simple approach to improve a model's calibration and robustness.
Training independent subnetworks for robust prediction
Recent approaches to efficiently ensemble neural networks have shown that strong robustness and uncertainty performance can be achieved with a negligible gain in parameters over the original network.
Hyperparameter Ensembles for Robustness and Uncertainty Quantification
Ensembles over neural network weights trained from different random initialization, known as deep ensembles, achieve state-of-the-art accuracy and calibration.
Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness
Bayesian neural networks (BNN) and deep ensembles are principled approaches to estimate the predictive uncertainty of a deep learning model.
Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors
Bayesian neural networks (BNNs) demonstrate promising success in improving the robustness and uncertainty quantification of modern deep learning.
On the Discrepancy between Density Estimation and Sequence Generation
In this paper, by comparing several density estimators on five machine translation tasks, we find that the correlation between rankings of models based on log-likelihood and BLEU varies significantly depending on the range of the model families being compared.
BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning
We also apply BatchEnsemble to lifelong learning, where on Split-CIFAR-100, BatchEnsemble yields comparable performance to progressive neural networks while having a much lower computational and memory costs.
Refining the variational posterior through iterative optimization
Variational inference (VI) is a popular approach for approximate Bayesian inference that is particularly promising for highly parameterized models such as deep neural networks.
Analyzing the Role of Model Uncertainty for Electronic Health Records
We further show that RNNs with only Bayesian embeddings can be a more efficient way to capture model uncertainty compared to ensembles, and we analyze how model uncertainty is impacted across individual input features and patient subgroups.
Discrete Flows: Invertible Generative Models of Discrete Data
While normalizing flows have led to significant advances in modeling high-dimensional continuous distributions, their applicability to discrete distributions remains unknown.
Ranked #16 on
Language Modelling
on Text8
Reliable Uncertainty Estimates in Deep Neural Networks using Noise Contrastive Priors
NCPs are compatible with any model that can output uncertainty estimates, are easy to scale, and yield reliable uncertainty estimates throughout training.
Measuring Calibration in Deep Learning
In this paper, we perform a comprehensive empirical study of choices in calibration measures including measuring all probabilities rather than just the maximum prediction, thresholding probability values, class conditionality, number of bins, bins that are adaptive to the datapoint density, and the norm used to compare accuracies to confidences.
NeuTra-lizing Bad Geometry in Hamiltonian Monte Carlo Using Neural Transport
Hamiltonian Monte Carlo is a powerful algorithm for sampling from difficult-to-normalize posterior distributions.
Bayesian Layers: A Module for Neural Network Uncertainty
We describe Bayesian Layers, a module designed for fast experimentation with neural network uncertainty.
Autoconj: Recognizing and Exploiting Conjugacy Without a Domain-Specific Language
Deriving conditional and marginal distributions using conjugacy relationships can be time consuming and error prone.
Simple, Distributed, and Accelerated Probabilistic Programming
For both a state-of-the-art VAE on 64x64 ImageNet and Image Transformer on 256x256 CelebA-HQ, our approach achieves an optimal linear speedup from 1 to 256 TPUv2 chips.
Mesh-TensorFlow: Deep Learning for Supercomputers
We use Mesh-TensorFlow to implement an efficient data-parallel, model-parallel version of the Transformer sequence-to-sequence model.
Ranked #10 on
Language Modelling
on One Billion Word
Noise Contrastive Priors for Functional Uncertainty
NCPs are compatible with any model that can output uncertainty estimates, are easy to scale, and yield reliable uncertainty estimates throughout training.
Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches
Stochastic neural net weights are used in a variety of contexts, including regularization, Bayesian neural nets, exploration in reinforcement learning, and evolution strategies.
Image Transformer
Image generation has been successfully cast as an autoregressive sequence generation or transformation problem.
Ranked #3 on
Density Estimation
on CIFAR-10
Generative Models for Alignment and Data Efficiency in Language
We examine how learning from unaligned data can improve both the data efficiency of supervised tasks as well as enable alignments without any supervision.
Variational Inference via \chi Upper Bound Minimization
In this paper we propose CHIVI, a black-box variational inference algorithm that minimizes $D_{\chi}(p || q)$, the $\chi$-divergence from $p$ to $q$.
TensorFlow Distributions
The TensorFlow Distributions library implements a vision of probability theory adapted to the modern deep-learning paradigm of end-to-end differentiable computation.
Implicit Causal Models for Genome-wide Association Studies
For the first, we describe implicit causal models, a class of causal models that leverages neural architectures with an implicit density.
Hierarchical Implicit Models and Likelihood-Free Variational Inference
Implicit probabilistic models are a flexible class of models defined by a simulation process for data.
Deep Probabilistic Programming
By treating inference as a first class citizen, on a par with modeling, we show that probabilistic programming can be as flexible and computationally efficient as traditional deep learning.
Variational Inference via $χ$-Upper Bound Minimization
In this paper we propose CHIVI, a black-box variational inference algorithm that minimizes $D_{\chi}(p || q)$, the $\chi$-divergence from $p$ to $q$.
Edward: A library for probabilistic modeling, inference, and criticism
Probabilistic modeling is a powerful approach for analyzing empirical information.
Operator Variational Inference
Though this divergence has been widely used, the resultant posterior approximation can suffer from undesirable statistical properties.
Spectral M-estimation with Applications to Hidden Markov Models
Method of moment estimators exhibit appealing statistical properties, such as asymptotic unbiasedness, for nonconvex problems.
The Variational Gaussian Process
Variational inference is a powerful tool for approximate inference, and it has been recently applied for representation learning with deep generative models.
Stochastic gradient descent methods for estimation with large data sets
When the update is based on a noisy gradient, the stochastic approximation is known as standard stochastic gradient descent, which has been fundamental in modern applications with large data sets.
Copula variational inference
We develop a general variational inference method that preserves dependency among the latent variables.
Towards stability and optimality in stochastic gradient descent
For statistical efficiency, AI-SGD employs averaging of the iterates, which achieves the optimal Cram\'{e}r-Rao bound under strong convexity, i. e., it is an optimal unbiased estimator of the true parameter value.
Expectation propagation as a way of life: A framework for Bayesian inference on partitioned data
A common divide-and-conquer approach for Bayesian computation with big data is to partition the data, perform local inference for each piece separately, and combine the results to obtain a global posterior approximation.
Convex Techniques for Model Selection
We develop a robust convex algorithm to select the regularization parameter in model selection.
