GROUNDHOG: Grounding Large Language Models to Holistic Segmentation
Most multimodal large language models (MLLMs) learn language-to-object grounding through causal language modeling where grounded objects are captured by bounding boxes as sequences of location tokens. This paradigm lacks pixel-level representations that are important for fine-grained visual understanding and diagnosis. In this work, we introduce GROUNDHOG, an MLLM developed by grounding Large Language Models to holistic segmentation. GROUNDHOG incorporates a masked feature extractor and converts extracted features into visual entity tokens for the MLLM backbone, which then connects groundable phrases to unified grounding masks by retrieving and merging the entity masks. To train GROUNDHOG, we carefully curated M3G2, a grounded visual instruction tuning dataset with Multi-Modal Multi-Grained Grounding, by harvesting a collection of segmentation-grounded datasets with rich annotations. Our experimental results show that GROUNDHOG achieves superior performance on various language grounding tasks without task-specific fine-tuning, and significantly reduces object hallucination. GROUNDHOG also demonstrates better grounding towards complex forms of visual input and provides easy-to-understand diagnosis in failure cases.
PDF Abstract
RefCOCO
Flickr30K Entities
Visual7W
PhraseCut
gRefCOCO
PointQA
Results from the Paper
Ranked #1 on
Generalized Referring Expression Segmentation
on gRefCOCO
(using extra training data)
| Task | Dataset | Model | Metric Name | Metric Value | Global Rank | Uses Extra Training Data |
Benchmark |
|---|---|---|---|---|---|---|---|
| Generalized Referring Expression Segmentation | gRefCOCO | GROUNDHOG | gIoU | 66.70 | # 1 | ||
| Referring Expression Segmentation | PhraseCut | GROUNDHOG | Mean IoU | 54.5 | # 2 | ||
| Referring Expression Segmentation | RefCOCOg-test | GROUNDHOG | Overall IoU | 74.6 | # 3 | ||
| Referring Expression Segmentation | RefCOCOg-val | GROUNDHOG | Overall IoU | 74.1 | # 3 | ||
| Referring Expression Segmentation | RefCOCO testA | GROUNDHOG | Overall IoU | 79.9 | # 4 | ||
| Referring Expression Segmentation | RefCOCO+ testA | GROUNDHOG | Overall IoU | 75.0 | # 4 | ||
| Referring Expression Segmentation | RefCOCO testB | GROUNDHOG | Overall IoU | 75.7 | # 2 | ||
| Referring Expression Segmentation | RefCOCO+ test B | GROUNDHOG | Overall IoU | 64.9 | # 4 | ||
| Referring Expression Segmentation | RefCoCo val | GROUNDHOG | Overall IoU | 78.5 | # 5 | ||
| Referring Expression Segmentation | RefCOCO+ val | GROUNDHOG | Overall IoU | 70.5 | # 5 |
