OpenMask3D: Open-Vocabulary 3D Instance Segmentation
We introduce the task of open-vocabulary 3D instance segmentation. Current approaches for 3D instance segmentation can typically only recognize object categories from a pre-defined closed set of classes that are annotated in the training datasets. This results in important limitations for real-world applications where one might need to perform tasks guided by novel, open-vocabulary queries related to a wide variety of objects. Recently, open-vocabulary 3D scene understanding methods have emerged to address this problem by learning queryable features for each point in the scene. While such a representation can be directly employed to perform semantic segmentation, existing methods cannot separate multiple object instances. In this work, we address this limitation, and propose OpenMask3D, which is a zero-shot approach for open-vocabulary 3D instance segmentation. Guided by predicted class-agnostic 3D instance masks, our model aggregates per-mask features via multi-view fusion of CLIP-based image embeddings. Experiments and ablation studies on ScanNet200 and Replica show that OpenMask3D outperforms other open-vocabulary methods, especially on the long-tail distribution. Qualitative experiments further showcase OpenMask3D's ability to segment object properties based on free-form queries describing geometry, affordances, and materials.
PDF Abstract NeurIPS 2023 PDF NeurIPS 2023 AbstractCode
Datasets
Results from the Paper
Task | Dataset | Model | Metric Name | Metric Value | Global Rank | Benchmark |
---|---|---|---|---|---|---|
3D Open-Vocabulary Instance Segmentation | Replica | OpenMask3D | mAP | 13.1 | # 3 | |
3D Open-Vocabulary Instance Segmentation | ScanNet200 | OpenMask3D | mAP | 15.4 | # 2 | |
AP50 | 19.9 | # 2 | ||||
AP25 | 23.1 | # 2 | ||||
AP Head | 17.1 | # 2 | ||||
AP Common | 14.1 | # 2 | ||||
AP Tail | 14.9 | # 2 |