Distilled Gradual Pruning with Pruned Fine-tuning
Neural Networks (NNs) have been driving machine learning progress in recent years, but their larger models present challenges in resource-limited environments. Weight pruning reduces the computational demand, often with performance degradation and long training procedures. This work introduces Distilled Gradual Pruning with Pruned Fine-tuning (DG2PF), a comprehensive algorithm that iteratively prunes pre-trained neural networks using knowledge distillation. We employ a magnitude-based unstructured pruning function that selectively removes a specified proportion of unimportant weights from the network. This function also leads to an efficient compression of the model size while minimizing classification accuracy loss. Additionally, we introduce a simulated pruning strategy with the same effects of weight recovery but while maintaining stable convergence. Furthermore, we propose a multi-step self-knowledge distillation strategy to effectively transfer the knowledge of the full, unpruned network to the pruned counterpart. We validate the performance of our algorithm through extensive experimentation on diverse benchmark datasets, including CIFAR-10 and ImageNet, as well as a set of model architectures. The results highlight how our algorithm prunes and optimizes pretrained neural networks without substantially degrading their classification accuracy while delivering significantly faster and more compact models.
PDF AbstractCode
CIFAR-10
ImageNet
Results from the Paper
Ranked #25 on
Image Classification
on CIFAR-10
(Top-1 Accuracy metric)
| Task | Dataset | Model | Metric Name | Metric Value | Global Rank | Benchmark |
|---|---|---|---|---|---|---|
| Image Classification | CIFAR-10 | DGPPF-ResNet18 | PARAMS | 0.57M | # 173 | |
| Top-1 Accuracy | 92.90 | # 25 | ||||
| Image Classification | ImageNet | DGPPF-ResNet18 | Top 1 Accuracy | 65.22 | # 969 | |
| Number of params | 1.15M | # 349 | ||||
| GFLOPs | 0.2 | # 12 | ||||
| Image Classification | ImageNet | DGPPF-MobileNetV2 | Top 1 Accuracy | 65.59% | # 968 | |
| Number of params | 1.03M | # 348 | ||||
| GFLOPs | 0.1 | # 2 | ||||
| Image Classification | ImageNet | DGPPF-ResNet50 | Top 1 Accuracy | 73.66% | # 913 | |
| Number of params | 2.56M | # 361 | ||||
| GFLOPs | 0.4 | # 43 |
