Network Pruning
213 papers with code • 5 benchmarks • 5 datasets
Network Pruning is a popular approach to reduce a heavy network to obtain a light-weight form by removing redundancy in the heavy network. In this approach, a complex over-parameterized network is first trained, then pruned based on come criterions, and finally fine-tuned to achieve comparable performance with reduced parameters.
Source: Ensemble Knowledge Distillation for Learning Improved and Efficient Networks
Benchmarks
These leaderboards are used to track progress in Network Pruning
| Trend | Dataset | Best Model | Paper | Code | Compare |
|---|---|---|---|---|---|
|
|||||
|
|||||
|
|||||
|
|||||
|
Libraries
Use these libraries to find Network Pruning models and implementations
CIFAR-10
ImageNet
CIFAR-100
MNIST
Netzschleuder
Most implemented papers
Dynamic Channel Pruning: Feature Boosting and Suppression
Making deep convolutional neural networks more accurate typically comes at the cost of increased computational and memory resources.
GASL: Guided Attention for Sparsity Learning in Deep Neural Networks
astorfi/attention-guided-sparsity
•
•
The main goal of network pruning is imposing sparsity on the neural network by increasing the number of parameters with zero value in order to reduce the architecture size and the computational speedup.
Towards Learning of Filter-Level Heterogeneous Compression of Convolutional Neural Networks
While mainstream deep learning methods train the neural networks weights while keeping the network architecture fixed, the emerging neural architecture search (NAS) techniques make the latter also amenable to training.
Learning Sparse Networks Using Targeted Dropout
for-ai/TD
•
•
Before computing the gradients for each weight update, targeted dropout stochastically selects a set of units or weights to be dropped using a simple self-reinforcing sparsity criterion and then computes the gradients for the remaining weights.
What Do Compressed Deep Neural Networks Forget?
google-research/google-research
•
•
However, this measure of performance conceals significant differences in how different classes and images are impacted by model compression techniques.
Winning the Lottery with Continuous Sparsification
lolemacs/continuous-sparsification
•
•
NeurIPS 2020
Additionally, the recent Lottery Ticket Hypothesis conjectures that, for a typically-sized neural network, it is possible to find small sub-networks which, when trained from scratch on a comparable budget, match the performance of the original dense counterpart.
Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion
NVlabs/DeepInversion
•
•
CVPR 2020
We introduce DeepInversion, a new method for synthesizing images from the image distribution used to train a deep neural network.
HRank: Filter Pruning using High-Rank Feature Map
The principle behind our pruning is that low-rank feature maps contain less information, and thus pruned results can be easily reproduced.
Comparing Rewinding and Fine-tuning in Neural Network Pruning
lottery-ticket/rewinding-iclr20-public
•
•
ICLR 2020
Learning rate rewinding (which we propose) trains the unpruned weights from their final values using the same learning rate schedule as weight rewinding.
DHP: Differentiable Meta Pruning via HyperNetworks
Passing the sparsified latent vectors through the hypernetworks, the corresponding slices of the generated weight parameters can be removed, achieving the effect of network pruning.
