SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers

We present SegFormer, a simple, efficient yet powerful semantic segmentation framework which unifies Transformers with lightweight multilayer perception (MLP) decoders. SegFormer has two appealing features: 1) SegFormer comprises a novel hierarchically structured Transformer encoder which outputs multiscale features. It does not need positional encoding, thereby avoiding the interpolation of positional codes which leads to decreased performance when the testing resolution differs from training. 2) SegFormer avoids complex decoders. The proposed MLP decoder aggregates information from different layers, and thus combining both local attention and global attention to render powerful representations. We show that this simple and lightweight design is the key to efficient segmentation on Transformers. We scale our approach up to obtain a series of models from SegFormer-B0 to SegFormer-B5, reaching significantly better performance and efficiency than previous counterparts. For example, SegFormer-B4 achieves 50.3% mIoU on ADE20K with 64M parameters, being 5x smaller and 2.2% better than the previous best method. Our best model, SegFormer-B5, achieves 84.0% mIoU on Cityscapes validation set and shows excellent zero-shot robustness on Cityscapes-C. Code will be released at: github.com/NVlabs/SegFormer.

PDF Abstract NeurIPS 2021 PDF NeurIPS 2021 Abstract
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Semantic Segmentation ADE20K SegFormer-B5 Validation mIoU 51.8 # 84
Params (M) 84.7 # 31
Semantic Segmentation ADE20K SegFormer-B0 Validation mIoU 37.4 # 216
Params (M) 3.8 # 64
Semantic Segmentation ADE20K SegFormer-B4 Validation mIoU 51.1 # 93
Params (M) 64.1 # 39
Semantic Segmentation ADE20K val SegFormer-B5(MS, 87M #Params, ImageNet-1K pretrain) mIoU 51.8 # 40
Semantic Segmentation Cityscapes test SegFormer (MiT-B5, Mapillary) Mean IoU (class) 83.1% # 19
Semantic Segmentation Cityscapes val SegFormer-B0 Validation mIoU 76.2 # 2
Semantic Segmentation Cityscapes val SegFormer (MiT-B5, Mapillary) mIoU 84.0 # 16
Semantic Segmentation COCO-Stuff full SegFormer-B5 (Single Scale) Mean IoU (class) 46.7 # 1
Semantic Segmentation DADA-seg SegFormer (MiT-B3) mIoU 27.0 # 10
Semantic Segmentation DADA-seg SegFormer (MiT-B2) mIoU 21.2 # 18
Semantic Segmentation DADA-seg SegFormer (MiT-B1) mIoU 16.6 # 26
Semantic Segmentation DELIVER SegFormer mIoU 57.20 # 7
Semantic Segmentation DensePASS SegFormer (MiT-B2) mIoU 42.4% # 12
Semantic Segmentation DensePASS SegFormer (MiT-B1) mIoU 38.5% # 16
Semantic Segmentation EventScape SegFormer-B4 mIoU 59.86 # 3
Semantic Segmentation EventScape SegFormer-B2 mIoU 58.69 # 4
Thermal Image Segmentation MFN Dataset SegFormer (B2) mIOU 53.2 # 30
Thermal Image Segmentation MFN Dataset SegFormer (B4) mIOU 54.8 # 26
Thermal Image Segmentation RGB-T-Glass-Segmentation SegFormer MAE 0.053 # 11
Semantic Segmentation SELMA SegFormer mIoU 77.2 # 2
Semantic Segmentation SynPASS SegFomrer mIoU 37.24% # 3
Semantic Segmentation UPLight SegFormer-B2 (RGB) mIoU 89.60 # 4
Semantic Segmentation UrbanLF SegFormer mIoU (Real) 82.20 # 4
mIoU (Syn) 78.53 # 8
Semantic Segmentation ZJU-RGB-P SegFormer-B2 (RGB) mIoU 89.6 # 5

Methods