TASK	DATASET	MODEL	METRIC NAME	METRIC VALUE	GLOBAL RANK
Image Captioning	nocaps entire	Microsoft Cognitive Services team	CIDEr	114.25	# 4
Image Captioning	nocaps entire	Microsoft Cognitive Services team	B1	85.62	# 3
Image Captioning	nocaps entire	Microsoft Cognitive Services team	B2	71.36	# 3
Image Captioning	nocaps entire	Microsoft Cognitive Services team	B3	53.62	# 3
Image Captioning	nocaps entire	Microsoft Cognitive Services team	B4	34.65	# 3
Image Captioning	nocaps entire	Microsoft Cognitive Services team	ROUGE-L	61.2	# 3
Image Captioning	nocaps entire	Microsoft Cognitive Services team	METEOR	31.27	# 3
Image Captioning	nocaps entire	Microsoft Cognitive Services team	SPICE	14.85	# 4
Image Captioning	nocaps in-domain	Microsoft Cognitive Services team	CIDEr	112.82	# 6
Image Captioning	nocaps in-domain	Microsoft Cognitive Services team	B1	86.33	# 5
Image Captioning	nocaps in-domain	Microsoft Cognitive Services team	B2	72.83	# 5
Image Captioning	nocaps in-domain	Microsoft Cognitive Services team	B3	55.94	# 5
Image Captioning	nocaps in-domain	Microsoft Cognitive Services team	B4	37.97	# 5
Image Captioning	nocaps in-domain	Microsoft Cognitive Services team	ROUGE-L	62.48	# 5
Image Captioning	nocaps in-domain	Microsoft Cognitive Services team	METEOR	32.7	# 5
Image Captioning	nocaps in-domain	Microsoft Cognitive Services team	SPICE	15.22	# 5
Image Captioning	nocaps near-domain	Microsoft Cognitive Services team	CIDEr	115.54	# 5
Image Captioning	nocaps near-domain	Microsoft Cognitive Services team	B1	86.48	# 5
Image Captioning	nocaps near-domain	Microsoft Cognitive Services team	B2	72.6	# 5
Image Captioning	nocaps near-domain	Microsoft Cognitive Services team	B3	55.26	# 5
Image Captioning	nocaps near-domain	Microsoft Cognitive Services team	B4	36.31	# 5
Image Captioning	nocaps near-domain	Microsoft Cognitive Services team	ROUGE-L	61.9	# 5
Image Captioning	nocaps near-domain	Microsoft Cognitive Services team	METEOR	31.8	# 5
Image Captioning	nocaps near-domain	Microsoft Cognitive Services team	SPICE	15.06	# 6
Image Captioning	nocaps out-of-domain	Microsoft Cognitive Services team	CIDEr	110.14	# 5
Image Captioning	nocaps out-of-domain	Microsoft Cognitive Services team	B1	81.73	# 5
Image Captioning	nocaps out-of-domain	Microsoft Cognitive Services team	B2	65.48	# 5
Image Captioning	nocaps out-of-domain	Microsoft Cognitive Services team	B3	45.58	# 5
Image Captioning	nocaps out-of-domain	Microsoft Cognitive Services team	B4	25.78	# 5
Image Captioning	nocaps out-of-domain	Microsoft Cognitive Services team	ROUGE-L	57.57	# 5
Image Captioning	nocaps out-of-domain	Microsoft Cognitive Services team	METEOR	28.17	# 5
Image Captioning	nocaps out-of-domain	Microsoft Cognitive Services team	SPICE	13.74	# 10
Image Captioning	nocaps-XD entire	Microsoft Cognitive Services team	CIDEr	100.12	# 5
Image Captioning	nocaps-XD entire	Microsoft Cognitive Services team	B1	82.27	# 5
Image Captioning	nocaps-XD entire	Microsoft Cognitive Services team	B2	66.04	# 5
Image Captioning	nocaps-XD entire	Microsoft Cognitive Services team	B3	47.48	# 5
Image Captioning	nocaps-XD entire	Microsoft Cognitive Services team	B4	28.95	# 5
Image Captioning	nocaps-XD entire	Microsoft Cognitive Services team	ROUGE-L	58.26	# 5
Image Captioning	nocaps-XD entire	Microsoft Cognitive Services team	METEOR	29.47	# 5
Image Captioning	nocaps-XD entire	Microsoft Cognitive Services team	SPICE	14.04	# 6
Image Captioning	nocaps-XD in-domain	Microsoft Cognitive Services team	CIDEr	100.62	# 4
Image Captioning	nocaps-XD in-domain	Microsoft Cognitive Services team	B1	82.94	# 4
Image Captioning	nocaps-XD in-domain	Microsoft Cognitive Services team	B2	67.56	# 4
Image Captioning	nocaps-XD in-domain	Microsoft Cognitive Services team	B3	49.66	# 4
Image Captioning	nocaps-XD in-domain	Microsoft Cognitive Services team	B4	32.07	# 4
Image Captioning	nocaps-XD in-domain	Microsoft Cognitive Services team	ROUGE-L	59.43	# 4
Image Captioning	nocaps-XD in-domain	Microsoft Cognitive Services team	METEOR	30.62	# 4
Image Captioning	nocaps-XD in-domain	Microsoft Cognitive Services team	SPICE	14.7	# 5
Image Captioning	nocaps-XD near-domain	Microsoft Cognitive Services team	CIDEr	101.2	# 4
Image Captioning	nocaps-XD near-domain	Microsoft Cognitive Services team	B1	82.88	# 4
Image Captioning	nocaps-XD near-domain	Microsoft Cognitive Services team	B2	67.01	# 4
Image Captioning	nocaps-XD near-domain	Microsoft Cognitive Services team	B3	48.73	# 4
Image Captioning	nocaps-XD near-domain	Microsoft Cognitive Services team	B4	30.21	# 4
Image Captioning	nocaps-XD near-domain	Microsoft Cognitive Services team	ROUGE-L	58.76	# 4
Image Captioning	nocaps-XD near-domain	Microsoft Cognitive Services team	METEOR	30.0	# 4
Image Captioning	nocaps-XD near-domain	Microsoft Cognitive Services team	SPICE	14.27	# 5
Image Captioning	nocaps-XD out-of-domain	Microsoft Cognitive Services team	CIDEr	95.5	# 3
Image Captioning	nocaps-XD out-of-domain	Microsoft Cognitive Services team	B1	79.44	# 4
Image Captioning	nocaps-XD out-of-domain	Microsoft Cognitive Services team	B2	61.15	# 3
Image Captioning	nocaps-XD out-of-domain	Microsoft Cognitive Services team	B3	41.03	# 3
Image Captioning	nocaps-XD out-of-domain	Microsoft Cognitive Services team	B4	21.79	# 3
Image Captioning	nocaps-XD out-of-domain	Microsoft Cognitive Services team	ROUGE-L	55.49	# 3
Image Captioning	nocaps-XD out-of-domain	Microsoft Cognitive Services team	METEOR	26.56	# 4
Image Captioning	nocaps-XD out-of-domain	Microsoft Cognitive Services team	SPICE	12.66	# 5

Badge	Markdown
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/vivo-surpassing-human-performance-in-novel/image-captioning-on-nocaps-xd-out-of-domain)](https://paperswithcode.com/sota/image-captioning-on-nocaps-xd-out-of-domain?p=vivo-surpassing-human-performance-in-novel)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/vivo-surpassing-human-performance-in-novel/image-captioning-on-nocaps-entire)](https://paperswithcode.com/sota/image-captioning-on-nocaps-entire?p=vivo-surpassing-human-performance-in-novel)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/vivo-surpassing-human-performance-in-novel/image-captioning-on-nocaps-xd-in-domain)](https://paperswithcode.com/sota/image-captioning-on-nocaps-xd-in-domain?p=vivo-surpassing-human-performance-in-novel)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/vivo-surpassing-human-performance-in-novel/image-captioning-on-nocaps-xd-near-domain)](https://paperswithcode.com/sota/image-captioning-on-nocaps-xd-near-domain?p=vivo-surpassing-human-performance-in-novel)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/vivo-surpassing-human-performance-in-novel/image-captioning-on-nocaps-near-domain)](https://paperswithcode.com/sota/image-captioning-on-nocaps-near-domain?p=vivo-surpassing-human-performance-in-novel)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/vivo-surpassing-human-performance-in-novel/image-captioning-on-nocaps-out-of-domain)](https://paperswithcode.com/sota/image-captioning-on-nocaps-out-of-domain?p=vivo-surpassing-human-performance-in-novel)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/vivo-surpassing-human-performance-in-novel/image-captioning-on-nocaps-xd-entire)](https://paperswithcode.com/sota/image-captioning-on-nocaps-xd-entire?p=vivo-surpassing-human-performance-in-novel)`
	`[![PWC](https://img.shields.io/endpoint.svg?url=https://paperswithcode.com/badge/vivo-surpassing-human-performance-in-novel/image-captioning-on-nocaps-in-domain)](https://paperswithcode.com/sota/image-captioning-on-nocaps-in-domain?p=vivo-surpassing-human-performance-in-novel)`

VIVO: Visual Vocabulary Pre-Training for Novel Object Captioning

28 Sep 2020 · Xiaowei Hu, Xi Yin, Kevin Lin, Lijuan Wang, Lei Zhang, Jianfeng Gao, Zicheng Liu ·

It is highly desirable yet challenging to generate image captions that can describe novel objects which are unseen in caption-labeled training data, a capability that is evaluated in the novel object captioning challenge (nocaps). In this challenge, no additional image-caption training data, other thanCOCO Captions, is allowed for model training. Thus, conventional Vision-Language Pre-training (VLP) methods cannot be applied. This paper presents VIsual VOcabulary pretraining (VIVO) that performs pre-training in the absence of caption annotations. By breaking the dependency of paired image-caption training data in VLP, VIVO can leverage large amounts of paired image-tag data to learn a visual vocabulary. This is done by pre-training a multi-layer Transformer model that learns to align image-level tags with their corresponding image region features. To address the unordered nature of image tags, VIVO uses a Hungarian matching loss with masked tag prediction to conduct pre-training. We validate the effectiveness of VIVO by fine-tuning the pre-trained model for image captioning. In addition, we perform an analysis of the visual-text alignment inferred by our model. The results show that our model can not only generate fluent image captions that describe novel objects, but also identify the locations of these objects. Our single model has achieved new state-of-the-art results on nocaps and surpassed the human CIDEr score.

PDF Abstract

Code

Add Remove Mark official

No code implementations yet. Submit your code now

Tasks

Add Remove

Image Captioning

Object

TAG

Datasets

COCO Captions

NoCaps

Results from the Paper

Edit

Ranked #3 on Image Captioning on nocaps-XD out-of-domain

Get a GitHub badge

Task	Dataset	Model	Metric Name	Metric Value	Global Rank	Benchmark
Image Captioning	nocaps entire	Microsoft Cognitive Services team	CIDEr	114.25	# 4	Compare
			B1	85.62	# 3	Compare
			B2	71.36	# 3	Compare
			B3	53.62	# 3	Compare
			B4	34.65	# 3	Compare
			ROUGE-L	61.2	# 3	Compare
			METEOR	31.27	# 3	Compare
			SPICE	14.85	# 4	Compare
Image Captioning	nocaps in-domain	Microsoft Cognitive Services team	CIDEr	112.82	# 6	Compare
			B1	86.33	# 5	Compare
			B2	72.83	# 5	Compare
			B3	55.94	# 5	Compare
			B4	37.97	# 5	Compare
			ROUGE-L	62.48	# 5	Compare
			METEOR	32.7	# 5	Compare
			SPICE	15.22	# 5	Compare
Image Captioning	nocaps near-domain	Microsoft Cognitive Services team	CIDEr	115.54	# 5	Compare
			B1	86.48	# 5	Compare
			B2	72.6	# 5	Compare
			B3	55.26	# 5	Compare
			B4	36.31	# 5	Compare
			ROUGE-L	61.9	# 5	Compare
			METEOR	31.8	# 5	Compare
			SPICE	15.06	# 6	Compare
Image Captioning	nocaps out-of-domain	Microsoft Cognitive Services team	CIDEr	110.14	# 5	Compare
			B1	81.73	# 5	Compare
			B2	65.48	# 5	Compare
			B3	45.58	# 5	Compare
			B4	25.78	# 5	Compare
			ROUGE-L	57.57	# 5	Compare
			METEOR	28.17	# 5	Compare
			SPICE	13.74	# 10	Compare
Image Captioning	nocaps-XD entire	Microsoft Cognitive Services team	CIDEr	100.12	# 5	Compare
			B1	82.27	# 5	Compare
			B2	66.04	# 5	Compare
			B3	47.48	# 5	Compare
			B4	28.95	# 5	Compare
			ROUGE-L	58.26	# 5	Compare
			METEOR	29.47	# 5	Compare
			SPICE	14.04	# 6	Compare
Image Captioning	nocaps-XD in-domain	Microsoft Cognitive Services team	CIDEr	100.62	# 4	Compare
			B1	82.94	# 4	Compare
			B2	67.56	# 4	Compare
			B3	49.66	# 4	Compare
			B4	32.07	# 4	Compare
			ROUGE-L	59.43	# 4	Compare
			METEOR	30.62	# 4	Compare
			SPICE	14.7	# 5	Compare
Image Captioning	nocaps-XD near-domain	Microsoft Cognitive Services team	CIDEr	101.2	# 4	Compare
			B1	82.88	# 4	Compare
			B2	67.01	# 4	Compare
			B3	48.73	# 4	Compare
			B4	30.21	# 4	Compare
			ROUGE-L	58.76	# 4	Compare
			METEOR	30.0	# 4	Compare
			SPICE	14.27	# 5	Compare
Image Captioning	nocaps-XD out-of-domain	Microsoft Cognitive Services team	CIDEr	95.5	# 3	Compare
			B1	79.44	# 4	Compare
			B2	61.15	# 3	Compare
			B3	41.03	# 3	Compare
			B4	21.79	# 3	Compare
			ROUGE-L	55.49	# 3	Compare
			METEOR	26.56	# 4	Compare
			SPICE	12.66	# 5	Compare

Methods

Add Remove

Absolute Position Encodings • Adam • BPE • Dense Connections • Dropout • Label Smoothing • Layer Normalization • Linear Layer • Multi-Head Attention • Position-Wise Feed-Forward Layer • Residual Connection • Scaled Dot-Product Attention • Softmax • Transformer

Edit Social Preview

VIVO: Visual Vocabulary Pre-Training for Novel Object Captioning

Code Edit Add Remove Mark official

Tasks Edit Add Remove

Datasets Edit

Results from the Paper Edit

Methods Edit Add Remove

Code

Add Remove Mark official

Tasks

Add Remove

Datasets

Results from the Paper

Edit

Methods

Add Remove