Self-Supervised Deep Supervision

The method exploits the finding that high correlation of segmentation performance among each U-Net's decoder layer -- with discriminative layer attached -- tends to have higher segmentation performance in the final segmentation map. By introducing an "Inter-layer Divergence Loss", based on Kulback-Liebler Divergence, to promotes the consistency between each discriminative output from decoder layers by minimizing the divergence.

If we assume that each decoder layer is equivalent to PDE function parameterized by weight parameter $\theta$:

$Decoder_i(x;\theta_i) \equiv PDE(x;\theta_i)$

Then our objective is trying to make each discriminative output similar to each other:

$PDE(x; \theta_d) \sim PDE(x; \theta_i);\text{ } 0 \leq i < d$

Hence the objective is to $\text{minimize} \sum_{i=0}^{d} D_{KL}(\hat{y} || Decoder_i)$.