VisualBERT: A Simple and Performant Baseline for Vision and Language

9 Aug 2019  ·  Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, Kai-Wei Chang ·

We propose VisualBERT, a simple and flexible framework for modeling a broad range of vision-and-language tasks. VisualBERT consists of a stack of Transformer layers that implicitly align elements of an input text and regions in an associated input image with self-attention. We further propose two visually-grounded language model objectives for pre-training VisualBERT on image caption data. Experiments on four vision-and-language tasks including VQA, VCR, NLVR2, and Flickr30K show that VisualBERT outperforms or rivals with state-of-the-art models while being significantly simpler. Further analysis demonstrates that VisualBERT can ground elements of language to image regions without any explicit supervision and is even sensitive to syntactic relationships, tracking, for example, associations between verbs and image regions corresponding to their arguments.

PDF Abstract

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Phrase Grounding Flickr30k Entities Dev VisualBERT R@1 70.4 # 3
R@10 86.31 # 2
R@5 84.49 # 2
Phrase Grounding Flickr30k Entities Test VisualBERT R@1 71.33 # 10
R@10 86.51 # 4
R@5 84.98 # 4
Visual Reasoning NLVR VisualBERT Accuracy (Dev) 67.4% # 1
Accuracy (Test-P) 67% # 1
Accuracy (Test-U) 67.3% # 1
Visual Reasoning NLVR2 Dev VisualBERT Accuracy 66.7 # 14
Visual Question Answering (VQA) VCR (Q-A) dev VisualBERT Accuracy 70.8 # 3
Visual Question Answering (VQA) VCR (Q-AR) dev VisualBERT Accuracy 52.2 # 3
Visual Question Answering (VQA) VCR (QA-R) dev VisualBERT Accuracy 73.2 # 3
Visual Question Answering (VQA) VCR (Q-AR) test VisualBERT Accuracy 52.4 # 7
Visual Question Answering (VQA) VCR (QA-R) test VisualBERT Accuracy 73.2 # 8
Visual Question Answering (VQA) VCR (Q-A) test VisualBERT Accuracy 71.6 # 9
Visual Question Answering (VQA) VQA v2 test-dev VisualBERT Accuracy 70.8 # 27
Visual Question Answering (VQA) VQA v2 test-std VisualBERT overall 71 # 25

Methods