Search Results for author:
Found 38 papers, 5 papers with code
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
We introduce phi-3-mini, a 3. 8 billion parameter language model trained on 3. 3 trillion tokens, whose overall performance, as measured by both academic benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and GPT-3. 5 (e. g., phi-3-mini achieves 69% on MMLU and 8. 38 on MT-bench), despite being small enough to be deployed on a phone.
TinyGSM: achieving >80% on GSM8k with small language models
Specifically for solving grade school math, the smallest model size so far required to break the 80\% barrier on the GSM8K benchmark remains to be 34B.
Ranked #58 on
Arithmetic Reasoning
on GSM8K
Convergence of Alternating Gradient Descent for Matrix Factorization
We show that, for a rank-$r$ matrix $\mathbf{A} \in \mathbb{R}^{m \times n}$, $T = C (\frac{\sigma_1(\mathbf{A})}{\sigma_r(\mathbf{A})})^2 \log(1/\epsilon)$ iterations of alternating gradient descent suffice to reach an $\epsilon$-optimal factorization $\| \mathbf{A} - \mathbf{X} \mathbf{Y}^{T} \|^2 \leq \epsilon \| \mathbf{A}\|^2$ with high probability starting from an atypical random initialization.
Robust Implicit Regularization via Weight Normalization
By analyzing key invariants of the gradient flow and using Lojasiewicz Theorem, we show that weight normalization also has an implicit bias towards sparse solutions in the diagonal linear model, but that in contrast to plain gradient flow, weight normalization enables a robust bias that persists even if the weights are initialized at practically large scale.
Adaptively Weighted Data Augmentation Consistency Regularization for Robust Optimization under Concept Shift
At the saddle point of the underlying objective, the weights assign label-dense samples to the supervised loss and label-sparse samples to the unsupervised consistency regularization.
On the fast convergence of minibatch heavy ball momentum
Simple stochastic momentum methods are widely used in machine learning optimization, but their good practical performance is at odds with an absence of theoretical guarantees of acceleration in the literature.
How catastrophic can catastrophic forgetting be in linear regression?
In specific settings, we highlight differences between forgetting and convergence to the offline solution as studied in those areas.
An Exponentially Increasing Step-size for Parameter Estimation in Statistical Models
Therefore, the total computational complexity of the EGD algorithm is \emph{optimal} and exponentially cheaper than that of the GD for solving parameter estimation in non-regular statistical models while being comparable to that of the GD in regular statistical settings.
Concentration of Random Feature Matrices in High-Dimensions
The spectra of random feature matrices provide essential information on the conditioning of the linear system used in random feature regression problems and are thus connected to the consistency and generalization of random feature models.
Side Effects of Learning from Low-dimensional Data Embedded in a Euclidean Space
In this setting, a typical neural network defines a function that takes a finite number of vectors in the embedding space as input.
Sample Efficiency of Data Augmentation Consistency Regularization
Data augmentation is popular in the training of large neural networks; currently, however, there is no clear theoretical comparison between different algorithmic choices on how to use augmented data.
The Power of Adaptivity in SGD: Self-Tuning Step Sizes with Unbounded Gradients and Affine Variance
We study convergence rates of AdaGrad-Norm as an exemplar of adaptive stochastic gradient methods (SGD), where the step sizes change based on observed stochastic gradients, for minimizing non-convex, smooth objectives.
SHRIMP: Sparser Random Feature Models via Iterative Magnitude Pruning
Inspired by the success of the iterative magnitude pruning technique in finding lottery tickets of neural networks, we propose a new method -- Sparser Random Feature Models via IMP (ShRIMP) -- to efficiently fit high-dimensional data with inherent low-dimensional structure in the form of sparse variable dependencies.
Theoretical Analysis of Consistency Regularization with Limited Augmented Data
Data augmentation is popular in the training of large neural networks; currently, however, there is no clear theoretical comparison between different algorithmic choices on how to use augmented data.
Learning to Forecast Dynamical Systems from Streaming Data
This algorithm dramatically reduces the costs of training and prediction without sacrificing forecasting skill.
AdaLoss: A computationally-efficient and provably convergent adaptive gradient method
We propose a computationally-friendly adaptive learning rate schedule, "AdaLoss", which directly uses the information of the loss function to adjust the stepsize in gradient descent methods.
Bootstrapping the error of Oja's algorithm
We consider the problem of quantifying uncertainty for the estimation error of the leading eigenvector from Oja's algorithm for streaming principal component analysis, where the data are generated IID from some unknown distribution.
Generalization Bounds for Sparse Random Feature Expansions
In particular, we provide generalization bounds for functions in a certain class (that is dense in a reproducing kernel Hilbert space) depending on the number of samples and the distribution of features.
Streaming k-PCA: Efficient guarantees for Oja's algorithm, beyond rank-one updates
We analyze Oja's algorithm for streaming $k$-PCA and prove that it achieves performance nearly matching that of an optimal offline algorithm.
Overparameterization and generalization error: weighted trigonometric interpolation
Motivated by surprisingly good generalization properties of learned deep neural networks in overparameterized scenarios and by the related double descent phenomenon, this paper analyzes the relation between smoothness and low generalization error in an overparameterized linear learning problem.
Implicit Regularization and Convergence for Weight Normalization
For certain stepsizes of g and w , we show that they can converge close to the minimum norm solution.
Linear Convergence of Adaptive Stochastic Gradient Descent
We prove that the norm version of the adaptive stochastic gradient method (AdaGrad-Norm) achieves a linear convergence rate for a subset of either strongly convex functions or non-convex functions that satisfy the Polyak Lojasiewicz (PL) inequality.
Bias of Homotopic Gradient Descent for the Hinge Loss
Gradient descent is a simple and widely used optimization method for machine learning.
AdaOja: Adaptive Learning Rates for Streaming PCA
We also show that AdaOja performs comparably to state-of-the-art algorithms (History PCA and Streaming Power Method) in the same streaming PCA setting.
Global Convergence of Adaptive Gradient Methods for An Over-parameterized Neural Network
Adaptive gradient methods like AdaGrad are widely used in optimizing neural networks.
Recovery guarantees for polynomial approximation from dependent data with outliers
In this work, we study the problem of learning nonlinear functions from corrupted and dependent data.
AdaGrad stepsizes: Sharp convergence over nonconvex landscapes
Adaptive gradient methods such as AdaGrad and its variants update the stepsize in stochastic gradient descent on the fly according to the gradients received along the way; such methods have gained widespread use in large-scale optimization for their ability to converge robustly, without the need to fine-tune the stepsize schedule.
WNGrad: Learn the Learning Rate in Gradient Descent
Adjusting the learning rate schedule in stochastic gradient methods is an important unresolved problem which requires tuning in practice.
A polynomial-time relaxation of the Gromov-Hausdorff distance
The Gromov-Hausdorff distance provides a metric on the set of isometry classes of compact metric spaces.
Clustering subgaussian mixtures by semidefinite programming
We introduce a model-free relax-and-round algorithm for k-means clustering based on a semidefinite relaxation due to Peng and Wei.
The local convexity of solving systems of quadratic equations
This paper considers the recovery of a rank $r$ positive semidefinite matrix $X X^T\in\mathbb{R}^{n\times n}$ from $m$ scalar measurements of the form $y_i := a_i^T X X^T a_i$ (i. e., quadratic measurements of $X$).
Relax, no need to round: integrality of clustering formulations
Under the same distributional model, the $k$-means LP relaxation fails to recover such clusters at separation as large as $\Delta = 4$.
One-bit compressive sensing with norm estimation
Consider the recovery of an unknown signal ${x}$ from quantized linear measurements.
Stochastic Gradient Descent, Weighted Sampling, and the Randomized Kaczmarz algorithm
Furthermore, we show how reweighting the sampling distribution (i. e. importance sampling) is necessary in order to further improve convergence, and obtain a linear dependence in the average smoothness, dominating previous results.
Recovery guarantees for exemplar-based clustering
For a certain class of distributions, we prove that the linear programming relaxation of $k$-medoids clustering---a variant of $k$-means clustering where means are replaced by exemplars from within the dataset---distinguishes points drawn from nonoverlapping balls with high probability once the number of points drawn and the separation distance between any two balls are sufficiently large.
Completing Any Low-rank Matrix, Provably
Matrix completion, i. e., the exact and provable recovery of a low-rank matrix from a small subset of its elements, is currently only known to be possible if the matrix satisfies a restrictive structural constraint---known as {\em incoherence}---on its row and column spaces.
Near-optimal compressed sensing guarantees for total variation minimization
Consider the problem of reconstructing a multidimensional signal from an underdetermined set of measurements, as in the setting of compressed sensing.
Stable and robust sampling strategies for compressive imaging
For Fourier measurements and Haar wavelet sparsity, the local coherence can be controlled and bounded explicitly, so for matrices comprised of frequencies sampled from a suitable inverse square power-law density, we can prove the restricted isometry property with near-optimal embedding dimensions.
