Search Results for author:
Found 21 papers, 16 papers with code
DropCompute: simple and more robust distributed synchronous training via compute variance reduction
Thus, these methods are limited by the delays caused by straggling workers.
Optimal Fine-Grained N:M sparsity for Activations and Neural Gradients
We show that while minimization of the MSE works fine for pruning the activations, it catastrophically fails for the neural gradients.
Energy awareness in low precision neural networks
Power consumption is a major obstacle in the deployment of deep neural networks (DNNs) on end devices.
Graph Representation Learning via Aggregation Enhancement
Graph neural networks (GNNs) have become a powerful tool for processing graph-structured data but still face challenges in effectively aggregating and propagating information between layers, which limits their performance.
Logarithmic Unbiased Quantization: Simple 4-bit Training in Deep Learning
Based on this, we suggest a \textit{logarithmic unbiased quantization} (LUQ) method to quantize all both the forward and backward phase to 4-bit, achieving state-of-the-art results in 4-bit training without overhead.
Beyond Quantization: Power aware neural networks
Power consumption is a major obstacle in the deployment of deep neural networks (DNNs) on end devices.
Logarithmic Unbiased Quantization: Practical 4-bit Training in Deep Learning
Based on this, we suggest a logarithmic unbiased quantization (LUQ) method to quantize both the forward and backward phase to 4-bit, achieving state-of-the-art results in 4-bit training.
Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks
Finally, to solve the problem of switching between different structure constraints, we suggest a method to convert a pre-trained model with unstructured sparsity to an N:M fine-grained block sparsity model with little to no training.
GAN "Steerability" without optimization
However, all existing techniques rely on an optimization procedure to expose those directions, and offer no control over the degree of allowed interaction between different transformations.
Neural gradients are near-lognormal: improved quantized and sparse training
While training can mostly be accelerated by reducing the time needed to propagate neural gradients back throughout the model, most previous works focus on the quantization/pruning of weights and activations.
Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programming
Instead, these methods only use the calibration set to set the activations' dynamic ranges.
Robust Quantization: One Model to Rule Them All
Neural network quantization methods often involve simulating the quantization process during training, making the trained model highly dependent on the target bit-width and precise way quantization is performed.
Post training 4-bit quantization of convolutional networks for rapid-deployment
Convolutional neural networks require significant memory bandwidth and storage for intermediate computations, apart from substantial computing resources.
Loss Aware Post-training Quantization
We show that with more aggressive quantization, the loss landscape becomes highly non-separable with steep curvature, making the selection of quantization parameters more challenging.
CAT: Compression-Aware Training for bandwidth reduction
Our method trains the model to achieve low-entropy feature maps, which enables efficient compression at inference time using classical transform coding methods.
Thanks for Nothing: Predicting Zero-Valued Activations with Lightweight Convolutional Neural Networks
Convolutional neural networks (CNNs) introduce state-of-the-art results for various tasks with the price of high computational demands.
Feature Map Transform Coding for Energy-Efficient CNN Inference
We analyze the performance of our approach on a variety of CNN architectures and demonstrate that FPGA implementation of ResNet-18 with our approach results in a reduction of around 40% in the memory energy footprint, compared to quantized network, with negligible impact on accuracy.
ACIQ: Analytical Clipping for Integer Quantization of neural networks
We analyze the trade-off between quantization noise and clipping distortion in low precision networks.
Post-training 4-bit quantization of convolution networks for rapid-deployment
Convolutional neural networks require significant memory bandwidth and storage for intermediate computations, apart from substantial computing resources.
Scalable Methods for 8-bit Training of Neural Networks
Armed with this knowledge, we quantize the model parameters, activations and layer gradients to 8-bit, leaving at a higher precision only the final step in the computation of the weight gradients.
Norm matters: efficient and accurate normalization schemes in deep networks
Over the past few years, Batch-Normalization has been commonly used in deep networks, allowing faster training and high performance for a wide variety of applications.
