ViP-DeepLab: Learning Visual Perception with Depth-aware Video Panoptic Segmentation
In this paper, we present ViP-DeepLab, a unified model attempting to tackle the long-standing and challenging inverse projection problem in vision, which we model as restoring the point clouds from perspective image sequences while providing each point with instance-level semantic interpretations. Solving this problem requires the vision models to predict the spatial location, semantic class, and temporally consistent instance label for each 3D point. ViP-DeepLab approaches it by jointly performing monocular depth estimation and video panoptic segmentation. We name this joint task as Depth-aware Video Panoptic Segmentation, and propose a new evaluation metric along with two derived datasets for it, which will be made available to the public. On the individual sub-tasks, ViP-DeepLab also achieves state-of-the-art results, outperforming previous methods by 5.1% VPQ on Cityscapes-VPS, ranking 1st on the KITTI monocular depth estimation benchmark, and 1st on KITTI MOTS pedestrian. The datasets and the evaluation codes are made publicly available.
PDF Abstract CVPR 2021 PDF CVPR 2021 AbstractCode
Datasets
Introduced in the Paper:
SemKITTI-DVPS Cityscapes-DVPSUsed in the Paper:
Cityscapes
KITTI
SemanticKITTI
Mapillary Vistas Dataset
KITTI MOTS
Cityscapes-VPS
Results from the Paper
Ranked #1 on
Video Panoptic Segmentation
on Cityscapes-VPS
(using extra training data)
