Texts

WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images

Introduced by Bitton-Guetta et al. in Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images

WHOOPS! Is a dataset and benchmark for visual commonsense. The dataset is comprised of purposefully commonsense-defying images created by designers using publicly-available image generation tools like Midjourney. It contains commonsense-defying image from a wide range of reasons, deviations from expected social norms and everyday knowledge.

Homepage

Benchmarks

Add a new result Link an existing benchmark

Task	Dataset Variant	Best Model
Explanation Generation	WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images	Ground-truth Caption -> GPT3
Image-to-Text Retrieval	WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images	BLIP2 FlanT5-XXL
Image Captioning	WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images	BLIP2 FlanT5-XXL
Visual Question Answering (VQA)	WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images	BLIP2 FlanT5-XXL