GLIPv2: Unifying Localization and Vision-Language Understanding
We present GLIPv2, a grounded VL understanding model, that serves both localization tasks (e.g., object detection, instance segmentation) and Vision-Language (VL) understanding tasks (e.g., VQA, image captioning). GLIPv2 elegantly unifies localization pre-training and Vision-Language Pre-training (VLP) with three pre-training tasks: phrase grounding as a VL reformulation of the detection task, region-word contrastive learning as a novel region-word level contrastive learning task, and the masked language modeling. This unification not only simplifies the previous multi-stage VLP procedure but also achieves mutual benefits between localization and understanding tasks. Experimental results show that a single GLIPv2 model (all model weights are shared) achieves near SoTA performance on various localization and understanding tasks. The model also shows (1) strong zero-shot and few-shot adaption performance on open-vocabulary object detection tasks and (2) superior grounding capability on VL understanding tasks. Code will be released at https://github.com/microsoft/GLIP.
PDF AbstractCode
Colab
Spaces
Datasets
MS COCO
Visual Genome
LVIS
Flickr30K Entities
PhraseCut
Results from the Paper
Ranked #1 on
Phrase Grounding
on Flickr30k Entities Test
(using extra training data)
| Task | Dataset | Model | Metric Name | Metric Value | Global Rank | Uses Extra Training Data |
Benchmark |
|---|---|---|---|---|---|---|---|
| Object Detection | COCO test-dev | GLIPv2 (CoSwin-H, multi-scale) | box mAP | 62.4 | # 18 | ||
| Phrase Grounding | Flickr30k Entities Test | GLIPv2 | R@1 | 87.7 | # 1 | ||
| Object Detection | LVIS v1.0 minival | GLIPv2 | box AP | 59.8 | # 4 | ||
| Referring Expression Segmentation | PhraseCut | GLIPv2 | Mean IoU | 61.3 | # 1 |
