Search Results for author:
Found 16 papers, 7 papers with code
The Elements of Differentiable Programming
Artificial intelligence has recently experienced remarkable advances, fueled by large models, vast datasets, accelerated hardware, and, last but not least, the transformative power of differentiable programming.
On the Interplay Between Stepsize Tuning and Progressive Sharpening
Recent empirical work has revealed an intriguing property of deep learning models by which the sharpness (largest eigenvalue of the Hessian) increases throughout optimization until it stabilizes around a critical value at which the optimizer operates at the edge of stability, given a fixed stepsize (Cohen et al, 2022).
Distributionally Robust Optimization with Bias and Variance Reduction
We consider the distributionally robust optimization (DRO) problem with spectral risk-based uncertainty set and $f$-divergence penalty.
Dual Gauss-Newton Directions for Deep Learning
Inspired by Gauss-Newton-like methods, we study the benefit of leveraging the structure of deep learning objectives, namely, the composition of a convex loss function and of a nonlinear network, in order to derive better direction oracles than stochastic gradients, based on the idea of partial linearization.
Modified Gauss-Newton Algorithms under Noise
Gauss-Newton methods and their stochastic version have been widely used in machine learning and signal processing.
Stochastic Optimization for Spectral Risk Measures
Spectral risk objectives - also called $L$-risks - allow for learning systems to interpolate between optimizing average-case performance (as in empirical risk minimization) and worst-case performance on a task.
Iterative Linear Quadratic Optimization for Nonlinear Control: Differentiable Programming Algorithmic Templates
We present the implementation of nonlinear control algorithms based on linear and quadratic approximations of the objective from a functional viewpoint.
Target Propagation via Regularized Inversion
Target Propagation (TP) algorithms compute targets instead of gradients along neural networks and propagate them backward in a way that is similar yet different than gradient back-propagation (BP).
Differentiable Programming à la Moreau
The notion of a Moreau envelope is central to the analysis of first-order optimization algorithms for machine learning.
An Elementary Approach to Convergence Guarantees of Optimization Algorithms for Deep Networks
We present an approach to obtain convergence guarantees of optimization algorithms for deep networks based on elementary arguments and computations.
Discriminative Clustering with Representation Learning with any Ratio of Labeled to Unlabeled Data
We present a discriminative clustering approach in which the feature representation can be learned from data and moreover leverage labeled data.
Kernel-based Translations of Convolutional Networks
Convolutional Neural Networks, as most artificial neural networks, are commonly viewed as methods different in essence from kernel-based methods.
A Smoother Way to Train Structured Prediction Models
We present a framework to train a structured prediction model by performing smoothing on the inference algorithm it builds upon.
Sharpness, Restart and Acceleration
The {\L}ojasiewicz inequality shows that H\"olderian error bounds on the minimum of convex optimization problems hold almost generically.
Integration Methods and Optimization Algorithms
We show that accelerated optimization methods can be seen as particular instances of multi-step integration schemes from numerical analysis, applied to the gradient flow equation.
Learning with Clustering Structure
We study supervised learning problems using clustering constraints to impose structure on either features or samples, seeking to help both prediction and interpretation.
