Search Results for author:
Found 6 papers, 4 papers with code
Simple and Scalable Strategies to Continually Pre-train Large Language Models
In this work, we show that a simple and scalable combination of learning rate (LR) re-warming, LR re-decaying, and replay of previous data is sufficient to match the performance of fully re-training from scratch on all available data, as measured by the final loss and the average score on several language model (LM) evaluation benchmarks.
Continual Pre-Training of Large Language Models: How to (re)warm your model?
We study the warmup phase of models pre-trained on the Pile (upstream data, 300B tokens) as we continue to pre-train on SlimPajama (downstream data, 297B tokens), following a linear warmup and cosine decay schedule.
Towards Out-of-Distribution Adversarial Robustness
Compared to existing methods, we obtain similar or superior worst-case adversarial robustness on attacks seen during training.
Learning Robust Kernel Ensembles with Kernel Average Pooling
Model ensembles have long been used in machine learning to reduce the variance in individual model predictions, making them more robust to input perturbations.
Adversarial Feature Desensitization
Our method, called Adversarial Feature Desensitization (AFD), aims at learning features that are invariant towards adversarial perturbations of the inputs.
Linear Lower Bounds and Conditioning of Differentiable Games
In this work, we approach the question of fundamental iteration complexity by providing lower bounds to complement the linear (i. e. geometric) upper bounds observed in the literature on a wide class of problems.
