Search Results for author:
Found 21 papers, 13 papers with code
Knowledge Translation: A New Pathway for Model Compression
Deep learning has witnessed significant advancements in recent years at the cost of increasing training, inference, and model storage overhead.
Fast ODE-based Sampling for Diffusion Models in Around 5 Steps
Sampling from diffusion models can be treated as solving the corresponding ordinary differential equations (ODEs), with the aim of obtaining an accurate solution with as few number of function evaluations (NFE) as possible.
Customizing Synthetic Data for Data-Free Student Learning
Existing works generally synthesize data from the pre-trained teacher model to replace the original training data for student learning.
Adaptive Multi-Teacher Knowledge Distillation with Meta-Learning
Multi-Teacher knowledge distillation provides students with additional supervision from multiple pre-trained teachers with diverse information sources.
A Geometric Perspective on Diffusion Models
Recent years have witnessed significant progress in developing effective training and fast sampling techniques for diffusion models.
Accelerating Diffusion Sampling with Classifier-based Feature Distillation
Instead of aligning output images, we distill teacher's sharpened feature distribution into the student with a dataset-independent classifier, making the student focus on those important features to improve performance.
Online Cross-Layer Knowledge Distillation on Graph Neural Networks with Deep Supervision
Alignahead++ transfers structure and feature information in a student layer to the previous layer of another simultaneously trained student model in an alternating training procedure.
Label-Efficient Domain Generalization via Collaborative Exploration and Generalization
To escape from the dilemma between domain generalization and annotation costs, in this paper, we introduce a novel task named label-efficient domain generalization (LEDG) to enable model generalization with label-limited source domains.
Improving Knowledge Graph Embedding via Iterative Self-Semantic Knowledge Distillation
Iteratively incorporating and accumulating iteration-based semantic information enables the low-dimensional model to be more expressive for better link prediction in KGs.
Alignahead: Online Cross-Layer Knowledge Extraction on Graph Neural Networks
Existing knowledge distillation methods on graph neural networks (GNNs) are almost offline, where the student model extracts knowledge from a powerful teacher model to improve its performance.
Knowledge Distillation with the Reused Teacher Classifier
Knowledge distillation aims to compress a powerful yet cumbersome teacher model into a lightweight student model without much sacrifice of performance.
Ranked #3 on
Knowledge Distillation
on CIFAR-100
Knowledge Distillation with Deep Supervision
Knowledge distillation aims to enhance the performance of a lightweight student model by exploiting the knowledge from a pre-trained cumbersome teacher model.
Confidence-Aware Multi-Teacher Knowledge Distillation
Knowledge distillation is initially introduced to utilize additional supervision from a single teacher model for the student model training.
Online Adversarial Distillation for Graph Neural Networks
However, its effect on graph neural networks is less than satisfactory since the graph topology and node attributes are likely to change in a dynamic way and in this case a static teacher model is insufficient in guiding student training.
Collaborative Semantic Aggregation and Calibration for Federated Domain Generalization
Domain generalization (DG) aims to learn from multiple known source domains a model that can generalize well to unknown target domains.
Domain-Specific Bias Filtering for Single Labeled Domain Generalization
In this paper, we investigate a Single Labeled Domain Generalization (SLDG) task with only one source domain being labeled, which is more practical and challenging than the CDG task.
Exploring the Connection between Knowledge Distillation and Logits Matching
Knowledge distillation is a generalized logits matching technique for model compression.
Distilling Holistic Knowledge with Graph Neural Networks
The holistic knowledge is represented as a unified graph-based embedding by aggregating individual knowledge from relational neighborhood samples with graph neural networks, the student network is learned by distilling the holistic knowledge in a contrastive manner.
Cross-Layer Distillation with Semantic Calibration
Knowledge distillation is a technique to enhance the generalization ability of a student model by exploiting outputs from a teacher model.
Online Knowledge Distillation via Multi-branch Diversity Enhancement
Extensive experiments proved that our method significantly enhances the diversity among student models and brings better distillation performance.
Online Knowledge Distillation with Diverse Peers
The second-level distillation is performed to transfer the knowledge in the ensemble of auxiliary peers further to the group leader, i. e., the model used for inference.
