Search Results for author:
Found 9 papers, 6 papers with code
Simple and Scalable Strategies to Continually Pre-train Large Language Models
In this work, we show that a simple and scalable combination of learning rate (LR) re-warming, LR re-decaying, and replay of previous data is sufficient to match the performance of fully re-training from scratch on all available data, as measured by the final loss and the average score on several language model (LM) evaluation benchmarks.
Continual Pre-Training of Large Language Models: How to (re)warm your model?
We study the warmup phase of models pre-trained on the Pile (upstream data, 300B tokens) as we continue to pre-train on SlimPajama (downstream data, 297B tokens), following a linear warmup and cosine decay schedule.
Wuerstchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Models
This highly compressed representation of an image provides much more detailed guidance compared to latent representations of language and this significantly reduces the computational requirements to achieve state-of-the-art results.
Receptive Field Refinement for Convolutional Neural Networks Reliably Improves Predictive Performance
By further developing and formalizing the analysis of receptive field expansion in convolutional neural networks, we can predict unproductive layers in an automated manner before ever training a model.
Should You Go Deeper? Optimizing Convolutional Neural Network Architectures without Training by Receptive Field Analysis
When optimizing convolutional neural networks (CNN) for a specific image-based task, specialists commonly overshoot the number of convolutional layers in their designs.
Exploring the Properties and Evolution of Neural Network Eigenspaces during Training
In this work we explore the information processing inside neural networks using logistic regression probes \cite{probes} and the saturation metric \cite{featurespace_saturation}.
Size Matters
Fully convolutional neural networks can process input of arbitrary size by applying a combination of downsampling and pooling.
Feature Space Saturation during Training
First, we show that a layer's output can be restricted to the eigenspace of its variance matrix without performance loss.
Spectral Analysis of Latent Representations
We propose a metric, Layer Saturation, defined as the proportion of the number of eigenvalues needed to explain 99% of the variance of the latent representations, for analyzing the learned representations of neural network layers.
