Search Results for author:
Found 39 papers, 16 papers with code
Training LLMs over Neurally Compressed Text
In this paper, we explore the idea of training large language models (LLMs) over highly compressed text.
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
In this report, we present the latest model of the Gemini family, Gemini 1. 5 Pro, a highly compute-efficient multimodal mixture-of-experts model capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio.
Ranked #19 on
Code Generation
on HumanEval
Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models
To do so, we investigate a simple self-training method based on expectation-maximization, which we call ReST$^{EM}$, where we (1) generate samples from the model and filter them using binary feedback, (2) fine-tune the model on these samples, and (3) repeat this process a few times.
Frontier Language Models are not Robust to Adversarial Arithmetic, or "What do I need to say so you agree 2+2=5?
We introduce and study the problem of adversarial arithmetic, which provides a simple yet challenging testbed for language model alignment.
Small-scale proxies for large-scale Transformer training instabilities
In this work, we seek ways to reproduce and study training stability and instability at smaller scales.
Replacing softmax with ReLU in Vision Transformers
Previous research observed accuracy degradation when replacing the attention softmax with a point-wise activation such as ReLU.
MadSGM: Multivariate Anomaly Detection with Score-based Generative Models
The time-series anomaly detection is one of the most fundamental tasks for time-series.
Precursor-of-Anomaly Detection for Irregular Time Series
Unlike conventional anomaly detection, which focuses on determining whether a given time series observation is an anomaly or not, PoA detection aims to detect future anomalies before they happen.
Kernel Regression with Infinite-Width Neural Networks on Millions of Examples
Using this approach, we study scaling laws of several neural kernels across many orders of magnitude for the CIFAR-5m dataset.
Time Series Forecasting with Hypernetworks Generating Parameters in Advance
Forecasting future outcomes from recent time series data is not easy, especially when the future data are different from the past (i. e. time series are under temporal drifts).
Fast Neural Kernel Embeddings for General Activations
Moreover, most prior works on neural kernels have focused on the ReLU activation, mainly due to its popularity but also due to the difficulty of computing such kernels for general activations.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
EXIT: Extrapolation and Interpolation-based Neural Controlled Differential Equations for Time-series Classification and Forecasting
Deep learning inspired by differential equations is a recent research trend and has marked the state of the art performance for many machine learning tasks.
LORD: Lower-Dimensional Embedding of Log-Signature in Neural Rough Differential Equations
The problem of processing very long time-series data (e. g., a length of more than 10, 000) is a long-standing research problem in machine learning.
Invertible Tabular GANs: Killing Two Birds with OneStone for Tabular Data Synthesis
First, we can further improve the synthesis quality, by decreasing the negative log-density of real records in the process of adversarial training.
Invertible Tabular GANs: Killing Two Birds with One Stone for Tabular Data Synthesis
First, we can further improve the synthesis quality, by decreasing the negative log-density of real records in the process of adversarial training.
Decoupled Kernel Neural Processes: Neural Network-Parameterized Stochastic Processes using Explicit Data-driven Kernel
To address this, we introduce a new neural stochastic processes, Decoupled Kernel Neural Processes (DKNPs), which explicitly learn a separate mean and kernel function to directly model the covariance between output variables in a data-driven manner.
Explaining Scaling Laws of Neural Network Generalization
In the large width limit, this can be equivalently obtained from the spectrum of certain kernels, and we present evidence that large width and large dataset resolution-limited scaling exponents are related by a duality.
Dataset Distillation with Infinitely Wide Convolutional Networks
The effectiveness of machine learning algorithms arises from being able to extract useful features from large amounts of data.
OCT-GAN: Neural ODE-based Conditional Tabular GANs
Synthesizing tabular data is attracting much attention these days for various purposes.
Explaining Neural Scaling Laws
In the large width limit, this can be equivalently obtained from the spectrum of certain kernels, and we present evidence that large width and large dataset resolution-limited scaling exponents are related by a duality.
Dataset Meta-Learning from Kernel-Ridge Regression
One of the most fundamental aspects of any machine learning algorithm is the training data used by the algorithm.
Exploring the Uncertainty Properties of Neural Networks’ Implicit Priors in the Infinite-Width Limit
This gives us a better understanding of the implicit prior NNs place on function space and allows a direct comparison of the calibration of the NNGP and its finite-width analogue.
Towards NNGP-guided Neural Architecture Search
Since NNGP inference provides a cheap measure of performance of a network architecture, we investigate its potential as a signal for neural architecture search (NAS).
Dataset Meta-Learning from Kernel Ridge-Regression
One of the most fundamental aspects of any machine learning algorithm is the training data used by the algorithm.
Exploring the Uncertainty Properties of Neural Networks' Implicit Priors in the Infinite-Width Limit
This gives us a better understanding of the implicit prior NNs place on function space and allows a direct comparison of the calibration of the NNGP and its finite-width analogue.
Finite Versus Infinite Neural Networks: an Empirical Study
We perform a careful, thorough, and large scale empirical study of the correspondence between wide neural networks and kernel methods.
Data-Efficient Deep Learning Method for Image Classification Using Data Augmentation, Focal Cosine Loss, and Ensemble
In general, sufficient data is essential for the better performance and generalization of deep-learning models.
NTIRE 2020 Challenge on Video Quality Mapping: Methods and Results
For track 2, some existing methods are evaluated, showing promising solutions to the weakly-supervised video quality mapping problem.
On the infinite width limit of neural networks with a standard parameterization
However, the extrapolation of both of these parameterizations to infinite width is problematic.
Neural Tangents: Fast and Easy Infinite Neural Networks in Python
Neural Tangents is a library designed to enable research into infinite-width neural networks.
On Empirical Comparisons of Optimizers for Deep Learning
In particular, we find that the popular adaptive gradient methods never underperform momentum or gradient descent.
Deep Bayesian Convolutional Networks with Many Channels are Gaussian Processes
There is a previously identified equivalence between wide fully connected neural networks (FCNs) and Gaussian processes (GPs).
Physics Enhanced Artificial Intelligence
We propose that intelligently combining models from the domains of Artificial Intelligence or Machine Learning with Physical and Expert models will yield a more "trustworthy" model than any one model from a single domain, given a complex and narrow enough problem.
Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
A longstanding goal in deep learning research has been to precisely characterize training and generalization.
Measuring the Effects of Data Parallelism on Neural Network Training
Along the way, we show that disagreements in the literature on how batch size affects model quality can largely be explained by differences in metaparameter tuning and compute budgets at different batch sizes.
Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
There is a previously identified equivalence between wide fully connected neural networks (FCNs) and Gaussian processes (GPs).
Deep Neural Networks as Gaussian Processes
As such, previous work has not identified that these kernels can be used as covariance functions for GPs and allow fully Bayesian prediction with a deep neural network.
3d $\mathcal{N}=2$ minimal SCFTs from Wrapped M5-branes
We numerically evaluate the central charges for several closed 3-manifolds with small hyperbolic volume.
High Energy Physics - Theory Mathematical Physics Mathematical Physics
