Weakly Supervised Co-training with Swapping Assignments for Semantic Segmentation
Class activation maps (CAMs) are commonly employed in weakly supervised semantic segmentation (WSSS) to produce pseudo-labels. Due to incomplete or excessive class activation, existing studies often resort to offline CAM refinement, introducing additional stages or proposing offline modules. This can cause optimization difficulties for single-stage methods and limit generalizability. In this study, we aim to reduce the observed CAM inconsistency and error to mitigate reliance on refinement processes. We propose an end-to-end WSSS model incorporating guided CAMs, wherein our segmentation model is trained while concurrently optimizing CAMs online. Our method, Co-training with Swapping Assignments (CoSA), leverages a dual-stream framework, where one sub-network learns from the swapped assignments generated by the other. We introduce three techniques: i) soft perplexity-based regularization to penalize uncertain regions; ii) a threshold-searching approach to dynamically revise the confidence threshold; and iii) contrastive separation to address the coexistence problem. CoSA demonstrates exceptional performance, achieving mIoU of 76.2\% and 51.0\% on VOC and COCO validation datasets, respectively, surpassing existing baselines by a substantial margin. Notably, CoSA is the first single-stage approach to outperform all existing multi-stage methods including those with additional supervision. Code is avilable at \url{https://github.com/youshyee/CoSA}.
PDF AbstractCode
Datasets
Results from the Paper
Task | Dataset | Model | Metric Name | Metric Value | Global Rank | Benchmark |
---|---|---|---|---|---|---|
Weakly-Supervised Semantic Segmentation | COCO 2014 val | CoSA (ViT-B, single-stage) | mIoU | 51.1 | # 6 | |
Weakly-Supervised Semantic Segmentation | COCO 2014 val | CoSA (SWIN-B, multi-stage) | mIoU | 53.7 | # 5 | |
Weakly-Supervised Semantic Segmentation | PASCAL VOC 2012 val | CoSA (SWIN-B, multi-stage) | Mean IoU | 81.4 | # 4 | |
Weakly-Supervised Semantic Segmentation | PASCAL VOC 2012 val | CoSA (ViT-B, single-stage) | Mean IoU | 76.4 | # 8 |