TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK
Image Captioning	COCO Captions	BLIP-2 ViT-G OPT 2.7B (zero-shot)	BLEU-4	43.7	# 4
Image Captioning	COCO Captions	BLIP-2 ViT-G OPT 2.7B (zero-shot)	CIDER	145.8	# 6
Image Captioning	COCO Captions	BLIP-2 ViT-G OPT 6.7B (zero-shot)	BLEU-4	43.5	# 5
Image Captioning	COCO Captions	BLIP-2 ViT-G OPT 6.7B (zero-shot)	CIDER	145.2	# 8
Image Captioning	COCO Captions	BLIP-2 ViT-G FlanT5 XL (zero-shot)	BLEU-4	42.4	# 8
Image Captioning	COCO Captions	BLIP-2 ViT-G FlanT5 XL (zero-shot)	CIDER	144.5	# 9
Image-to-Text Retrieval	Flickr30k	BLIP-2 ViT-L (zero-shot, 1K test set)	Recall@1	96.9	# 5
Image-to-Text Retrieval	Flickr30k	BLIP-2 ViT-L (zero-shot, 1K test set)	Recall@5	100	# 1
Image-to-Text Retrieval	Flickr30k	BLIP-2 ViT-L (zero-shot, 1K test set)	Recall@10	100	# 1
Image Retrieval	Flickr30k	BLIP-2 ViT-G (zero-shot, 1K test set)	Recall@5	98.1	# 1
Image Retrieval	Flickr30k	BLIP-2 ViT-G (zero-shot, 1K test set)	Recall@10	98.9	# 1
Image Retrieval	Flickr30k	BLIP-2 ViT-G (zero-shot, 1K test set)	Recall@1	89.7	# 1
Image-to-Text Retrieval	Flickr30k	BLIP-2 ViT-G (zero-shot, 1K test set)	Recall@1	97.6	# 2
Image-to-Text Retrieval	Flickr30k	BLIP-2 ViT-G (zero-shot, 1K test set)	Recall@5	100	# 1
Image-to-Text Retrieval	Flickr30k	BLIP-2 ViT-G (zero-shot, 1K test set)	Recall@10	100	# 1
Image Retrieval	Flickr30k	BLIP-2 ViT-L (zero-shot, 1K test set)	Recall@5	97.6	# 2
Image Retrieval	Flickr30k	BLIP-2 ViT-L (zero-shot, 1K test set)	Recall@10	98.9	# 1
Image Retrieval	Flickr30k	BLIP-2 ViT-L (zero-shot, 1K test set)	Recall@1	88.6	# 2
Visual Question Answering (VQA)	GQA test-dev	BLIP-2 ViT-G FlanT5 XL (zero-shot)	Accuracy	44.2	# 9
Visual Question Answering (VQA)	GQA test-dev	BLIP-2 ViT-G FlanT5 XXL (zero-shot)	Accuracy	44.7	# 7
Visual Question Answering (VQA)	GQA test-dev	BLIP-2 ViT-L OPT 2.7B (zero-shot)	Accuracy	33.9	# 13
Visual Question Answering (VQA)	GQA test-dev	BLIP-2 ViT-G OPT 2.7B (zero-shot)	Accuracy	34.6	# 12
Visual Question Answering (VQA)	GQA test-dev	BLIP-2 ViT-G OPT 6.7B (zero-shot)	Accuracy	36.4	# 11
Visual Question Answering (VQA)	GQA test-dev	BLIP-2 ViT-L FlanT5 XL (zero-shot)	Accuracy	44.4	# 8
Visual Question Answering (VQA)	InfiMM-Eval	BLIP-2-OPT2.7B	Overall score	19.31	# 12
Visual Question Answering (VQA)	InfiMM-Eval	BLIP-2-OPT2.7B	Deductive	2.76	# 14
Visual Question Answering (VQA)	InfiMM-Eval	BLIP-2-OPT2.7B	Abductive	18.96	# 12
Visual Question Answering (VQA)	InfiMM-Eval	BLIP-2-OPT2.7B	Analogical	7.5	# 12
Visual Question Answering (VQA)	InfiMM-Eval	BLIP-2-OPT2.7B	Params	3B	# 1
Visual Question Answering (VQA)	InfoSeek	BLIP2	Accuracy	14.6	# 6
visual instruction following	LLaVA-Bench	BLIP-2	avg score	38.1	# 7
Visual Question Answering	MM-Vet	BLIP-2-12B	GPT-4 score	22.4±0.2	# 90
Visual Question Answering	MM-Vet	BLIP-2-12B	Params	12B	# 1
Image Retrieval	MS COCO	BLIP-2 ViT-G (fine-tuned)	Recall@10	92.6	# 3
Image Retrieval	MS COCO	BLIP-2 ViT-G (fine-tuned)	recall@1	68.3	# 1
Image Retrieval	MS COCO	BLIP-2 ViT-G (fine-tuned)	recall@5	87.7	# 2
Image-to-Text Retrieval	MS COCO	BLIP-2 ViT-L (fine-tuned)	Recall@10	98.0	# 4
Image-to-Text Retrieval	MS COCO	BLIP-2 ViT-L (fine-tuned)	Recall@1	83.5	# 3
Image-to-Text Retrieval	MS COCO	BLIP-2 ViT-L (fine-tuned)	Recall@5	96.0	# 3
Image-to-Text Retrieval	MS COCO	BLIP-2 ViT-G (fine-tuned)	Recall@10	98.5	# 2
Image-to-Text Retrieval	MS COCO	BLIP-2 ViT-G (fine-tuned)	Recall@1	85.4	# 1
Image-to-Text Retrieval	MS COCO	BLIP-2 ViT-G (fine-tuned)	Recall@5	97.0	# 1
Image Retrieval	MS COCO	BLIP-2 ViT-L (fine-tuned)	Recall@10	91.8	# 4
Image Retrieval	MS COCO	BLIP-2 ViT-L (fine-tuned)	recall@1	66.3	# 3
Image Retrieval	MS COCO	BLIP-2 ViT-L (fine-tuned)	recall@5	86.5	# 3
Image Captioning	nocaps-val-in-domain	BLIP-2 ViT-G OPT 6.7B (zero-shot)	CIDEr	123.7	# 1
Image Captioning	nocaps-val-in-domain	BLIP-2 ViT-G OPT 6.7B (zero-shot)	SPICE	15.8	# 2
Image Captioning	nocaps-val-in-domain	BLIP-2 ViT-G OPT 6.7B (zero-shot)	Pre-train (#images)	1.1B	# 1
Image Captioning	nocaps-val-in-domain	BLIP-2 ViT-G FlanT5 XL (zero-shot)	CIDEr	123.7	# 1
Image Captioning	nocaps-val-in-domain	BLIP-2 ViT-G FlanT5 XL (zero-shot)	SPICE	16.3	# 1
Image Captioning	nocaps-val-in-domain	BLIP-2 ViT-G FlanT5 XL (zero-shot)	Pre-train (#images)	1.1B	# 1
Image Captioning	nocaps-val-in-domain	BLIP-2 ViT-G OPT 2.7B (zero-shot)	CIDEr	123	# 3
Image Captioning	nocaps-val-in-domain	BLIP-2 ViT-G OPT 2.7B (zero-shot)	SPICE	15.8	# 2
Image Captioning	nocaps-val-in-domain	BLIP-2 ViT-G OPT 2.7B (zero-shot)	Pre-train (#images)	1.1B	# 1
Image Captioning	nocaps-val-near-domain	BLIP-2 ViT-G OPT 2.7B (zero-shot)	CIDEr	117.8	# 3
Image Captioning	nocaps-val-near-domain	BLIP-2 ViT-G OPT 2.7B (zero-shot)	SPICE	15.4	# 2
Image Captioning	nocaps-val-near-domain	BLIP-2 ViT-G OPT 2.7B (zero-shot)	Pre-train (#images)	1.1B	# 1
Image Captioning	nocaps-val-near-domain	BLIP-2 ViT-G FlanT5 XL (zero-shot)	CIDEr	120.2	# 1
Image Captioning	nocaps-val-near-domain	BLIP-2 ViT-G FlanT5 XL (zero-shot)	SPICE	15.9	# 1
Image Captioning	nocaps-val-near-domain	BLIP-2 ViT-G FlanT5 XL (zero-shot)	Pre-train (#images)	1.1B	# 1
Image Captioning	nocaps-val-near-domain	BLIP-2 ViT-G OPT 6.7B (zero-shot)	CIDEr	119.2	# 2
Image Captioning	nocaps-val-near-domain	BLIP-2 ViT-G OPT 6.7B (zero-shot)	SPICE	15.3	# 3
Image Captioning	nocaps-val-near-domain	BLIP-2 ViT-G OPT 6.7B (zero-shot)	Pre-train (#images)	1.1B	# 1
Image Captioning	nocaps-val-out-domain	BLIP-2 ViT-G OPT 6.7B (zero-shot)	CIDEr	124.4	# 2
Image Captioning	nocaps-val-out-domain	BLIP-2 ViT-G OPT 6.7B (zero-shot)	SPICE	14.8	# 3
Image Captioning	nocaps-val-out-domain	BLIP-2 ViT-G OPT 6.7B (zero-shot)	Pretrain (#images)	1.1B	# 1
Image Captioning	nocaps-val-out-domain	BLIP-2 ViT-G OPT 2.7B (zero-shot)	CIDEr	123.4	# 3
Image Captioning	nocaps-val-out-domain	BLIP-2 ViT-G OPT 2.7B (zero-shot)	SPICE	15.1	# 1
Image Captioning	nocaps-val-out-domain	BLIP-2 ViT-G OPT 2.7B (zero-shot)	Pretrain (#images)	1.1B	# 1
Image Captioning	nocaps-val-out-domain	BLIP-2 ViT-G FlanT5 XL (zero-shot)	CIDEr	124.8	# 1
Image Captioning	nocaps-val-out-domain	BLIP-2 ViT-G FlanT5 XL (zero-shot)	SPICE	15.1	# 1
Image Captioning	nocaps-val-out-domain	BLIP-2 ViT-G FlanT5 XL (zero-shot)	Pretrain (#images)	1.1B	# 1
Image Captioning	nocaps-val-overall	BLIP-2 ViT-G OPT 6.7B (zero-shot)	CIDEr	121.0	# 2
Image Captioning	nocaps-val-overall	BLIP-2 ViT-G OPT 6.7B (zero-shot)	SPICE	15.3	# 3
Image Captioning	nocaps-val-overall	BLIP-2 ViT-G OPT 6.7B (zero-shot)	Pretrain (#images)	1.1B	# 1
Image Captioning	nocaps-val-overall	BLIP-2 ViT-G FlanT5 XL (zero-shot)	CIDEr	121.6	# 1
Image Captioning	nocaps-val-overall	BLIP-2 ViT-G FlanT5 XL (zero-shot)	SPICE	15.8	# 1
Image Captioning	nocaps-val-overall	BLIP-2 ViT-G FlanT5 XL (zero-shot)	Pretrain (#images)	1.1B	# 1
Image Captioning	nocaps-val-overall	BLIP-2 ViT-G OPT 2.7B (zero-shot)	CIDEr	119.7	# 3
Image Captioning	nocaps-val-overall	BLIP-2 ViT-G OPT 2.7B (zero-shot)	SPICE	15.4	# 2
Image Captioning	nocaps-val-overall	BLIP-2 ViT-G OPT 2.7B (zero-shot)	Pretrain (#images)	1.1B	# 1
Visual Question Answering (VQA)	OK-VQA	BLIP-2 ViT-L OPT 2.7B (zero-shot)	Accuracy	30.2	# 32
Visual Question Answering (VQA)	OK-VQA	BLIP-2 ViT-G OPT 2.7B (zero-shot)	Accuracy	31.7	# 31
Visual Question Answering (VQA)	OK-VQA	BLIP-2 ViT-G OPT 6.7B (zero-shot)	Accuracy	36.4	# 29
Visual Question Answering (VQA)	OK-VQA	BLIP-2 ViT-L FlanT5 XL (zero-shot)	Accuracy	39.4	# 28
Visual Question Answering (VQA)	OK-VQA	BLIP-2 ViT-G FlanT5 XL (zero-shot)	Accuracy	40.7	# 27
Visual Question Answering (VQA)	OK-VQA	BLIP-2 ViT-G FlanT5 XXL (zero-shot)	Accuracy	45.9	# 22
Open Vocabulary Attribute Detection	OVAD-Box benchmark	BLIP 2 (pretrained)	mean average precision	25.5	# 2
Medical Visual Question Answering	PMC-VQA	BLIP-2	Accuracy	24.3	# 4
Generative Visual Question Answering	PMC-VQA	BLIP-2	BLEU-1	7.6	# 2
Visual Question Answering (VQA)	PMC-VQA	BLIP-2	Accuracy	24.3	# 4
Visual Question Answering	VQA v2 test-dev	BLIP-2 ViT-G OPT 2.7B (fine-tuned)	Accuracy	81.74	# 4
Visual Question Answering (VQA)	VQA v2 test-dev	BLIP-2 ViT-G FlanT5 XXL (zero-shot)	Accuracy	65	# 42
Visual Question Answering (VQA)	VQA v2 test-dev	BLIP-2 ViT-L OPT 2.7B (zero-shot)	Accuracy	49.7	# 54
Visual Question Answering	VQA v2 test-dev	BLIP-2 ViT-G OPT 6.7B (fine-tuned)	Accuracy	82.30	# 1
Visual Question Answering	VQA v2 test-dev	BLIP-2 ViT-G FlanT5 XL (fine-tuned)	Accuracy	81.66	# 5
Visual Question Answering (VQA)	VQA v2 test-dev	BLIP-2 ViT-G OPT 2.7B (zero-shot)	Accuracy	52.3	# 51
Visual Question Answering (VQA)	VQA v2 test-dev	BLIP-2 ViT-G OPT 6.7B (zero-shot)	Accuracy	52.6	# 50
Visual Question Answering (VQA)	VQA v2 test-dev	BLIP-2 ViT-L FlanT5 XL (zero-shot)	Accuracy	62.3	# 48
Visual Question Answering (VQA)	VQA v2 test-dev	BLIP-2 ViT-G FlanT5 XL (zero-shot)	Accuracy	63	# 47
Visual Question Answering (VQA)	VQA v2 val	BLIP-2 ViT-G OPT 2.7B (zero-shot)	Accuracy	53.5	# 6
Visual Question Answering (VQA)	VQA v2 val	BLIP-2 ViT-G FlanT5 XXL (zero-shot)	Accuracy	65.2	# 1
Visual Question Answering (VQA)	VQA v2 val	BLIP-2 ViT-G OPT 6.7B (zero-shot)	Accuracy	54.3	# 5
Visual Question Answering (VQA)	VQA v2 val	BLIP-2 ViT-G FlanT5 XL (zero-shot)	Accuracy	63.1	# 3
Visual Question Answering (VQA)	VQA v2 val	BLIP-2 ViT-L FlanT5 XL (zero-shot)	Accuracy	62.6	# 4
Visual Question Answering	VQA v2 val	BLIP-2 ViT-G FlanT5 XL (fine-tuned)	Accuracy	81.55	# 3
Visual Question Answering	VQA v2 val	BLIP-2 ViT-G OPT 2.7B (fine-tuned)	Accuracy	81.59	# 2
Visual Question Answering	VQA v2 val	BLIP-2 ViT-G OPT 6.7B (fine-tuned)	Accuracy	82.19	# 1
Visual Question Answering (VQA)	VQA v2 val	BLIP-2 ViT-L OPT 2.7B (zero-shot)	Accuracy	50.1	# 7

Badge	Markdown
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/blip-2-bootstrapping-language-image-pre/image-retrieval-on-flickr30k)](https://paperswithcode.com/sota/image-retrieval-on-flickr30k?p=blip-2-bootstrapping-language-image-pre)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/blip-2-bootstrapping-language-image-pre/image-retrieval-on-coco)](https://paperswithcode.com/sota/image-retrieval-on-coco?p=blip-2-bootstrapping-language-image-pre)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/blip-2-bootstrapping-language-image-pre/image-to-text-retrieval-on-coco)](https://paperswithcode.com/sota/image-to-text-retrieval-on-coco?p=blip-2-bootstrapping-language-image-pre)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/blip-2-bootstrapping-language-image-pre/image-captioning-on-nocaps-val-in-domain)](https://paperswithcode.com/sota/image-captioning-on-nocaps-val-in-domain?p=blip-2-bootstrapping-language-image-pre)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/blip-2-bootstrapping-language-image-pre/image-captioning-on-nocaps-val-near-domain)](https://paperswithcode.com/sota/image-captioning-on-nocaps-val-near-domain?p=blip-2-bootstrapping-language-image-pre)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/blip-2-bootstrapping-language-image-pre/image-captioning-on-nocaps-val-out-domain)](https://paperswithcode.com/sota/image-captioning-on-nocaps-val-out-domain?p=blip-2-bootstrapping-language-image-pre)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/blip-2-bootstrapping-language-image-pre/image-captioning-on-nocaps-val-overall)](https://paperswithcode.com/sota/image-captioning-on-nocaps-val-overall?p=blip-2-bootstrapping-language-image-pre)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/blip-2-bootstrapping-language-image-pre/visual-question-answering-on-vqa-v2-test-dev-1)](https://paperswithcode.com/sota/visual-question-answering-on-vqa-v2-test-dev-1?p=blip-2-bootstrapping-language-image-pre)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/blip-2-bootstrapping-language-image-pre/visual-question-answering-on-vqa-v2-val)](https://paperswithcode.com/sota/visual-question-answering-on-vqa-v2-val?p=blip-2-bootstrapping-language-image-pre)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/blip-2-bootstrapping-language-image-pre/visual-question-answering-on-vqa-v2-val-1)](https://paperswithcode.com/sota/visual-question-answering-on-vqa-v2-val-1?p=blip-2-bootstrapping-language-image-pre)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/blip-2-bootstrapping-language-image-pre/image-to-text-retrieval-on-flickr30k)](https://paperswithcode.com/sota/image-to-text-retrieval-on-flickr30k?p=blip-2-bootstrapping-language-image-pre)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/blip-2-bootstrapping-language-image-pre/open-vocabulary-attribute-detection-on-ovad-1)](https://paperswithcode.com/sota/open-vocabulary-attribute-detection-on-ovad-1?p=blip-2-bootstrapping-language-image-pre)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/blip-2-bootstrapping-language-image-pre/generative-visual-question-answering-on-pmc)](https://paperswithcode.com/sota/generative-visual-question-answering-on-pmc?p=blip-2-bootstrapping-language-image-pre)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/blip-2-bootstrapping-language-image-pre/image-captioning-on-coco-captions)](https://paperswithcode.com/sota/image-captioning-on-coco-captions?p=blip-2-bootstrapping-language-image-pre)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/blip-2-bootstrapping-language-image-pre/medical-visual-question-answering-on-pmc-vqa)](https://paperswithcode.com/sota/medical-visual-question-answering-on-pmc-vqa?p=blip-2-bootstrapping-language-image-pre)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/blip-2-bootstrapping-language-image-pre/visual-question-answering-vqa-on-pmc-vqa)](https://paperswithcode.com/sota/visual-question-answering-vqa-on-pmc-vqa?p=blip-2-bootstrapping-language-image-pre)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/blip-2-bootstrapping-language-image-pre/visual-question-answering-vqa-on-infoseek)](https://paperswithcode.com/sota/visual-question-answering-vqa-on-infoseek?p=blip-2-bootstrapping-language-image-pre)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/blip-2-bootstrapping-language-image-pre/visual-question-answering-on-gqa-test-dev)](https://paperswithcode.com/sota/visual-question-answering-on-gqa-test-dev?p=blip-2-bootstrapping-language-image-pre)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/blip-2-bootstrapping-language-image-pre/visual-instruction-following-on-llava-bench)](https://paperswithcode.com/sota/visual-instruction-following-on-llava-bench?p=blip-2-bootstrapping-language-image-pre)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/blip-2-bootstrapping-language-image-pre/visual-question-answering-vqa-on-core-mm)](https://paperswithcode.com/sota/visual-question-answering-vqa-on-core-mm?p=blip-2-bootstrapping-language-image-pre)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/blip-2-bootstrapping-language-image-pre/visual-question-answering-on-ok-vqa)](https://paperswithcode.com/sota/visual-question-answering-on-ok-vqa?p=blip-2-bootstrapping-language-image-pre)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/blip-2-bootstrapping-language-image-pre/visual-question-answering-on-vqa-v2-test-dev)](https://paperswithcode.com/sota/visual-question-answering-on-vqa-v2-test-dev?p=blip-2-bootstrapping-language-image-pre)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/blip-2-bootstrapping-language-image-pre/visual-question-answering-on-mm-vet)](https://paperswithcode.com/sota/visual-question-answering-on-mm-vet?p=blip-2-bootstrapping-language-image-pre)`

BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

30 Jan 2023 · Junnan Li, Dongxu Li, Silvio Savarese, Steven Hoi ·

The cost of vision-and-language pre-training has become increasingly prohibitive due to end-to-end training of large-scale models. This paper proposes BLIP-2, a generic and efficient pre-training strategy that bootstraps vision-language pre-training from off-the-shelf frozen pre-trained image encoders and frozen large language models. BLIP-2 bridges the modality gap with a lightweight Querying Transformer, which is pre-trained in two stages. The first stage bootstraps vision-language representation learning from a frozen image encoder. The second stage bootstraps vision-to-language generative learning from a frozen language model. BLIP-2 achieves state-of-the-art performance on various vision-language tasks, despite having significantly fewer trainable parameters than existing methods. For example, our model outperforms Flamingo80B by 8.7% on zero-shot VQAv2 with 54x fewer trainable parameters. We also demonstrate the model's emerging capabilities of zero-shot image-to-text generation that can follow natural language instructions.

PDF Abstract

Code

Add Remove Mark official

salesforce/lavis official

8,732

huggingface/transformers

125,118

thudm/visualglm-6b

3,937

baaivision/eva

1,968

junshutang/Make-It-3D

1,687

See all 12 implementations

Tasks

Add Remove

Generative Visual Question Answering

Image Captioning

Image Retrieval

Image-to-Text Retrieval

Language Modelling

Medical Visual Question Answering

Open Vocabulary Attribute Detection

Representation Learning

Text Generation

visual instruction following

Visual Question Answering

Visual Question Answering (VQA)

Visual Reasoning

Datasets

MS COCO

Visual Genome

Flickr30k

GQA

Visual Question Answering v2.0

OK-VQA

COCO Captions

NoCaps

MM-Vet LLaVA-Bench PMC-VQA

InfiMM-Eval

InfoSeek

OVAD benchmark

Results from the Paper

Edit

Ranked #1 on Image Retrieval on Flickr30k

Get a GitHub badge

Task	Dataset	Model	Metric Name	Metric Value	Global Rank	Benchmark
Image Captioning	COCO Captions	BLIP-2 ViT-G OPT 2.7B (zero-shot)	BLEU-4	43.7	# 4	Compare
Image Captioning	COCO Captions	BLIP-2 ViT-G OPT 2.7B (zero-shot)	CIDER	145.8	# 6	Compare
Image Captioning	COCO Captions	BLIP-2 ViT-G OPT 6.7B (zero-shot)	BLEU-4	43.5	# 5	Compare
Image Captioning	COCO Captions	BLIP-2 ViT-G OPT 6.7B (zero-shot)	CIDER	145.2	# 8	Compare
Image Captioning	COCO Captions	BLIP-2 ViT-G FlanT5 XL (zero-shot)	BLEU-4	42.4	# 8	Compare
Image Captioning	COCO Captions	BLIP-2 ViT-G FlanT5 XL (zero-shot)	CIDER	144.5	# 9	Compare
Image-to-Text Retrieval	Flickr30k	BLIP-2 ViT-L (zero-shot, 1K test set)	Recall@1	96.9	# 5	Compare
			Recall@5	100	# 1	Compare
			Recall@10	100	# 1	Compare
Image Retrieval	Flickr30k	BLIP-2 ViT-G (zero-shot, 1K test set)	Recall@5	98.1	# 1	Compare
			Recall@10	98.9	# 1	Compare
			Recall@1	89.7	# 1	Compare
Image-to-Text Retrieval	Flickr30k	BLIP-2 ViT-G (zero-shot, 1K test set)	Recall@1	97.6	# 2	Compare
			Recall@5	100	# 1	Compare
			Recall@10	100	# 1	Compare
Image Retrieval	Flickr30k	BLIP-2 ViT-L (zero-shot, 1K test set)	Recall@5	97.6	# 2	Compare
			Recall@10	98.9	# 1	Compare
			Recall@1	88.6	# 2	Compare
Visual Question Answering (VQA)	GQA test-dev	BLIP-2 ViT-G FlanT5 XL (zero-shot)	Accuracy	44.2	# 9	Compare
Visual Question Answering (VQA)	GQA test-dev	BLIP-2 ViT-G FlanT5 XXL (zero-shot)	Accuracy	44.7	# 7	Compare
Visual Question Answering (VQA)	GQA test-dev	BLIP-2 ViT-L OPT 2.7B (zero-shot)	Accuracy	33.9	# 13	Compare
Visual Question Answering (VQA)	GQA test-dev	BLIP-2 ViT-G OPT 2.7B (zero-shot)	Accuracy	34.6	# 12	Compare
Visual Question Answering (VQA)	GQA test-dev	BLIP-2 ViT-G OPT 6.7B (zero-shot)	Accuracy	36.4	# 11	Compare
Visual Question Answering (VQA)	GQA test-dev	BLIP-2 ViT-L FlanT5 XL (zero-shot)	Accuracy	44.4	# 8	Compare
Visual Question Answering (VQA)	InfiMM-Eval	BLIP-2-OPT2.7B	Overall score	19.31	# 12	Compare
			Deductive	2.76	# 14	Compare
			Abductive	18.96	# 12	Compare
			Analogical	7.5	# 12	Compare
			Params	3B	# 1	Compare
Visual Question Answering (VQA)	InfoSeek	BLIP2	Accuracy	14.6	# 6	Compare
visual instruction following	LLaVA-Bench	BLIP-2	avg score	38.1	# 7	Compare
Visual Question Answering	MM-Vet	BLIP-2-12B	GPT-4 score	22.4±0.2	# 90	Compare
Visual Question Answering	MM-Vet	BLIP-2-12B	Params	12B	# 1	Compare
Image Retrieval	MS COCO	BLIP-2 ViT-G (fine-tuned)	Recall@10	92.6	# 3	Compare
			recall@1	68.3	# 1	Compare
			recall@5	87.7	# 2	Compare
Image-to-Text Retrieval	MS COCO	BLIP-2 ViT-L (fine-tuned)	Recall@10	98.0	# 4	Compare
			Recall@1	83.5	# 3	Compare
			Recall@5	96.0	# 3	Compare
Image-to-Text Retrieval	MS COCO	BLIP-2 ViT-G (fine-tuned)	Recall@10	98.5	# 2	Compare
			Recall@1	85.4	# 1	Compare
			Recall@5	97.0	# 1	Compare
Image Retrieval	MS COCO	BLIP-2 ViT-L (fine-tuned)	Recall@10	91.8	# 4	Compare
			recall@1	66.3	# 3	Compare
			recall@5	86.5	# 3	Compare
Image Captioning	nocaps-val-in-domain	BLIP-2 ViT-G OPT 6.7B (zero-shot)	CIDEr	123.7	# 1	Compare
			SPICE	15.8	# 2	Compare
			Pre-train (#images)	1.1B	# 1	Compare
Image Captioning	nocaps-val-in-domain	BLIP-2 ViT-G FlanT5 XL (zero-shot)	CIDEr	123.7	# 1	Compare
			SPICE	16.3	# 1	Compare
			Pre-train (#images)	1.1B	# 1	Compare
Image Captioning	nocaps-val-in-domain	BLIP-2 ViT-G OPT 2.7B (zero-shot)	CIDEr	123	# 3	Compare
			SPICE	15.8	# 2	Compare
			Pre-train (#images)	1.1B	# 1	Compare
Image Captioning	nocaps-val-near-domain	BLIP-2 ViT-G OPT 2.7B (zero-shot)	CIDEr	117.8	# 3	Compare
			SPICE	15.4	# 2	Compare
			Pre-train (#images)	1.1B	# 1	Compare
Image Captioning	nocaps-val-near-domain	BLIP-2 ViT-G FlanT5 XL (zero-shot)	CIDEr	120.2	# 1	Compare
			SPICE	15.9	# 1	Compare
			Pre-train (#images)	1.1B	# 1	Compare
Image Captioning	nocaps-val-near-domain	BLIP-2 ViT-G OPT 6.7B (zero-shot)	CIDEr	119.2	# 2	Compare
			SPICE	15.3	# 3	Compare
			Pre-train (#images)	1.1B	# 1	Compare
Image Captioning	nocaps-val-out-domain	BLIP-2 ViT-G OPT 6.7B (zero-shot)	CIDEr	124.4	# 2	Compare
			SPICE	14.8	# 3	Compare
			Pretrain (#images)	1.1B	# 1	Compare
Image Captioning	nocaps-val-out-domain	BLIP-2 ViT-G OPT 2.7B (zero-shot)	CIDEr	123.4	# 3	Compare
			SPICE	15.1	# 1	Compare
			Pretrain (#images)	1.1B	# 1	Compare
Image Captioning	nocaps-val-out-domain	BLIP-2 ViT-G FlanT5 XL (zero-shot)	CIDEr	124.8	# 1	Compare
			SPICE	15.1	# 1	Compare
			Pretrain (#images)	1.1B	# 1	Compare
Image Captioning	nocaps-val-overall	BLIP-2 ViT-G OPT 6.7B (zero-shot)	CIDEr	121.0	# 2	Compare
			SPICE	15.3	# 3	Compare
			Pretrain (#images)	1.1B	# 1	Compare
Image Captioning	nocaps-val-overall	BLIP-2 ViT-G FlanT5 XL (zero-shot)	CIDEr	121.6	# 1	Compare
			SPICE	15.8	# 1	Compare
			Pretrain (#images)	1.1B	# 1	Compare
Image Captioning	nocaps-val-overall	BLIP-2 ViT-G OPT 2.7B (zero-shot)	CIDEr	119.7	# 3	Compare
			SPICE	15.4	# 2	Compare
			Pretrain (#images)	1.1B	# 1	Compare
Visual Question Answering (VQA)	OK-VQA	BLIP-2 ViT-L OPT 2.7B (zero-shot)	Accuracy	30.2	# 32	Compare
Visual Question Answering (VQA)	OK-VQA	BLIP-2 ViT-G OPT 2.7B (zero-shot)	Accuracy	31.7	# 31	Compare
Visual Question Answering (VQA)	OK-VQA	BLIP-2 ViT-G OPT 6.7B (zero-shot)	Accuracy	36.4	# 29	Compare
Visual Question Answering (VQA)	OK-VQA	BLIP-2 ViT-L FlanT5 XL (zero-shot)	Accuracy	39.4	# 28	Compare
Visual Question Answering (VQA)	OK-VQA	BLIP-2 ViT-G FlanT5 XL (zero-shot)	Accuracy	40.7	# 27	Compare
Visual Question Answering (VQA)	OK-VQA	BLIP-2 ViT-G FlanT5 XXL (zero-shot)	Accuracy	45.9	# 22	Compare
Open Vocabulary Attribute Detection	OVAD-Box benchmark	BLIP 2 (pretrained)	mean average precision	25.5	# 2	Compare
Medical Visual Question Answering	PMC-VQA	BLIP-2	Accuracy	24.3	# 4	Compare
Generative Visual Question Answering	PMC-VQA	BLIP-2	BLEU-1	7.6	# 2	Compare
Visual Question Answering (VQA)	PMC-VQA	BLIP-2	Accuracy	24.3	# 4	Compare
Visual Question Answering	VQA v2 test-dev	BLIP-2 ViT-G OPT 2.7B (fine-tuned)	Accuracy	81.74	# 4	Compare
Visual Question Answering (VQA)	VQA v2 test-dev	BLIP-2 ViT-G FlanT5 XXL (zero-shot)	Accuracy	65	# 42	Compare
Visual Question Answering (VQA)	VQA v2 test-dev	BLIP-2 ViT-L OPT 2.7B (zero-shot)	Accuracy	49.7	# 54	Compare
Visual Question Answering	VQA v2 test-dev	BLIP-2 ViT-G OPT 6.7B (fine-tuned)	Accuracy	82.30	# 1	Compare
Visual Question Answering	VQA v2 test-dev	BLIP-2 ViT-G FlanT5 XL (fine-tuned)	Accuracy	81.66	# 5	Compare
Visual Question Answering (VQA)	VQA v2 test-dev	BLIP-2 ViT-G OPT 2.7B (zero-shot)	Accuracy	52.3	# 51	Compare
Visual Question Answering (VQA)	VQA v2 test-dev	BLIP-2 ViT-G OPT 6.7B (zero-shot)	Accuracy	52.6	# 50	Compare
Visual Question Answering (VQA)	VQA v2 test-dev	BLIP-2 ViT-L FlanT5 XL (zero-shot)	Accuracy	62.3	# 48	Compare
Visual Question Answering (VQA)	VQA v2 test-dev	BLIP-2 ViT-G FlanT5 XL (zero-shot)	Accuracy	63	# 47	Compare
Visual Question Answering (VQA)	VQA v2 val	BLIP-2 ViT-G OPT 2.7B (zero-shot)	Accuracy	53.5	# 6	Compare
Visual Question Answering (VQA)	VQA v2 val	BLIP-2 ViT-G FlanT5 XXL (zero-shot)	Accuracy	65.2	# 1	Compare
Visual Question Answering (VQA)	VQA v2 val	BLIP-2 ViT-G OPT 6.7B (zero-shot)	Accuracy	54.3	# 5	Compare
Visual Question Answering (VQA)	VQA v2 val	BLIP-2 ViT-G FlanT5 XL (zero-shot)	Accuracy	63.1	# 3	Compare
Visual Question Answering (VQA)	VQA v2 val	BLIP-2 ViT-L FlanT5 XL (zero-shot)	Accuracy	62.6	# 4	Compare
Visual Question Answering	VQA v2 val	BLIP-2 ViT-G FlanT5 XL (fine-tuned)	Accuracy	81.55	# 3	Compare
Visual Question Answering	VQA v2 val	BLIP-2 ViT-G OPT 2.7B (fine-tuned)	Accuracy	81.59	# 2	Compare
Visual Question Answering	VQA v2 val	BLIP-2 ViT-G OPT 6.7B (fine-tuned)	Accuracy	82.19	# 1	Compare
Visual Question Answering (VQA)	VQA v2 val	BLIP-2 ViT-L OPT 2.7B (zero-shot)	Accuracy	50.1	# 7	Compare

Methods

Add Remove

Absolute Position Encodings • Adam • BPE • Dense Connections • Dropout • Label Smoothing • Layer Normalization • Linear Layer • Multi-Head Attention • Position-Wise Feed-Forward Layer • Residual Connection • Scaled Dot-Product Attention • Softmax • Transformer

Edit Social Preview

BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

Code Edit Add Remove Mark official

Tasks Edit Add Remove

Datasets Edit

Results from the Paper Edit

Methods Edit Add Remove

Code

Add Remove Mark official

Tasks

Add Remove

Datasets

Results from the Paper

Edit

Methods

Add Remove