Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
In this work, we introduce the Qwen-VL series, a set of large-scale vision-language models (LVLMs) designed to perceive and understand both texts and images. Starting from the Qwen-LM as a foundation, we endow it with visual capacity by the meticulously designed (i) visual receptor, (ii) input-output interface, (iii) 3-stage training pipeline, and (iv) multilingual multimodal cleaned corpus. Beyond the conventional image description and question-answering, we implement the grounding and text-reading ability of Qwen-VLs by aligning image-caption-box tuples. The resulting models, including Qwen-VL and Qwen-VL-Chat, set new records for generalist models under similar model scales on a broad range of visual-centric benchmarks (e.g., image captioning, question answering, visual grounding) and different settings (e.g., zero-shot, few-shot). Moreover, on real-world dialog benchmarks, our instruction-tuned Qwen-VL-Chat also demonstrates superiority compared to existing vision-language chatbots. Code, demo and models are available at https://github.com/QwenLM/Qwen-VL.
PDF AbstractCode
Spaces
MS COCO
Visual Question Answering
Visual Genome
Flickr30k
GQA
RefCOCO
OK-VQA
ScienceQA
NoCaps
VizWiz
CC12M
MM-Vet
SEED-Bench
AI2D
DVQA
InfiMM-Eval
GRIT
ViP-Bench
| Task | Dataset | Model | Metric Name | Metric Value | Global Rank | Uses Extra Training Data |
Benchmark |
|---|---|---|---|---|---|---|---|
| Chart Question Answering | ChartQA | Qwen-VL-Chat | 1:1 Accuracy | 66.3 | # 16 | ||
| Chart Question Answering | ChartQA | Qwen-VL | 1:1 Accuracy | 65.7 | # 18 | ||
| Visual Question Answering (VQA) | DocVQA test | Qwen-VL-Plus | ANLS | 0.9024 | # 4 | ||
| Visual Question Answering (VQA) | DocVQA test | Qwen-VL | ANLS | 0.651 | # 30 | ||
| Visual Question Answering (VQA) | DocVQA test | Qwen-VL-Chat | ANLS | 0.626 | # 32 | ||
| Visual Question Answering (VQA) | InfiMM-Eval | Qwen-VL-Chat | Overall score | 37.39 | # 3 | ||
| Deductive | 37.55 | # 3 | |||||
| Abductive | 44.39 | # 6 | |||||
| Analogical | 30.42 | # 2 | |||||
| Params | 16B | # 1 | |||||
| Visual Question Answering | MM-Vet | Qwen-VL-Max | GPT-4 score | 66.6±0.5 | # 3 | ||
| Visual Question Answering | MM-Vet | Qwen-VL-Plus | GPT-4 score | 61.1±0.2 | # 7 | ||
| Visual Question Answering | ViP-Bench | Qwen-VL-Chat (Coordinates) | GPT-4 score (bbox) | 45.3 | # 5 | ||
| Visual Question Answering | ViP-Bench | Qwen-VL-Chat (Visual Prompt) | GPT-4 score (bbox) | 39.2 | # 7 | ||
| GPT-4 score (human) | 41.7 | # 5 |
