Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese

2 Nov 2022  ยท  An Yang, Junshu Pan, Junyang Lin, Rui Men, Yichang Zhang, Jingren Zhou, Chang Zhou ยท

The tremendous success of CLIP (Radford et al., 2021) has promoted the research and application of contrastive learning for vision-language pretraining. In this work, we construct a large-scale dataset of image-text pairs in Chinese, where most data are retrieved from publicly available datasets, and we pretrain Chinese CLIP models on the new dataset. We develop 5 Chinese CLIP models of multiple sizes, spanning from 77 to 958 million parameters. Furthermore, we propose a two-stage pretraining method, where the model is first trained with the image encoder frozen and then trained with all parameters being optimized, to achieve enhanced model performance. Our comprehensive experiments demonstrate that Chinese CLIP can achieve the state-of-the-art performance on MUGE, Flickr30K-CN, and COCO-CN in the setups of zero-shot learning and finetuning, and it is able to achieve competitive performance in zero-shot image classification based on the evaluation on the ELEVATER benchmark (Li et al., 2022). We have released our codes, models, and demos in https://github.com/OFA-Sys/Chinese-CLIP

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Zero-shot Image Retrieval COCO-CN CN-CLIP (ViT-H/14) R@1 69.2 # 4
R@5 89.9 # 5
R@10 96.1 # 5
Zero-shot Image Retrieval COCO-CN CN-CLIP (ViT-B/16) R@1 62.2 # 8
R@5 86.6 # 8
R@10 94.9 # 6
Zero-shot Image Retrieval COCO-CN CN-CLIP (ViT-L/14) R@1 64.0 # 7
R@5 89.2 # 7
R@10 94.4 # 8
Zero-shot Image Retrieval COCO-CN CN-CLIP (ViT-L/14@336px) R@1 64.7 # 6
R@5 89.6 # 6
R@10 94.6 # 7
Zero-shot Image Retrieval COCO-CN CN-CLIP (RN50) R@1 48.1 # 12
R@5 81.3 # 10
R@10 90.5 # 10
Zero-shot Text-to-Image Retrieval COCO-CN Chinese CLIP ViT-H/14 Recall@1 69.2 # 3
Recall@5 89.9 # 3
Recall@10 96.1 # 3
Image Retrieval COCO-CN CN-CLIP (ViT-B/16) R@1 77.0 # 5
R@5 97.1 # 1
R@10 99.0 # 3
Image Retrieval COCO-CN CN-CLIP (ViT-L/14) R@1 78.9 # 4
R@5 96.3 # 5
R@10 99.0 # 3
Image Retrieval COCO-CN CN-CLIP (ViT-L/14@336px) R@1 80.1 # 2
R@5 96.7 # 3
R@10 99.2 # 1
Image Retrieval COCO-CN CN-CLIP (ViT-H/14) R@1 81.5 # 1
R@5 96.9 # 2
R@10 99.1 # 2
Image Retrieval COCO-CN CN-CLIP (RN50) R@1 66.8 # 9
R@5 91.1 # 9
R@10 97.0 # 8
Zero-shot Text Retrieval Flickr30k-CN WukongViT-L/14 R@1 76.1 # 7
R@5 94.8 # 7
R@10 97.5 # 7
Zero-shot Image Retrieval Flickr30k-CN CN-CLIP (ViT-L/14) R@1 68.0 # 8
R@5 89.7 # 8
R@10 94.4 # 8
Zero-shot Image Retrieval Flickr30k-CN CN-CLIP (ViT-L/14@336px) R@1 69.0 # 7
R@5 90.7 # 6
R@10 95.4 # 6
Zero-shot Image Retrieval Flickr30k-CN CN-CLIP (ViT-H/14) R@1 71.2 # 5
R@5 91.4 # 5
R@10 95.5 # 4
Zero-shot Text Retrieval Flickr30k-CN CN-CLIPViT-L/14@336px R@1 83.3 # 3
R@5 97.2 # 4
R@10 98.5 # 5
Image Retrieval Flickr30k-CN CN-CLIP (ViT-L/14@336px) R@1 84.4 # 3
R@5 97.1 # 3
R@10 98.7 # 1
Image Retrieval Flickr30k-CN CN-CLIP (ViT-L/14) R@1 82.7 # 6
R@5 96.7 # 5
R@10 98.6 # 2
Image Retrieval Flickr30k-CN CN-CLIP (ViT-B/16) R@1 79.1 # 7
R@5 94.8 # 7
R@10 97.4 # 5
Image Retrieval Flickr30k-CN CN-CLIP (RN50) R@1 66.7 # 11
R@5 89.4 # 11
R@10 94.1 # 11
Zero-shot Image Retrieval Flickr30k-CN CN-CLIP (RN50) R@1 48.8 # 12
R@5 76.0 # 12
R@10 84.6 # 12
Zero-shot Image Retrieval Flickr30k-CN CN-CLIP (ViT-B/16) R@1 62.7 # 9
R@5 86.9 # 9
R@10 92.8 # 9
Zero-shot Text Retrieval Flickr30k-CN CN-CLIPViT-H/14 R@1 81.6 # 4
R@5 97.5 # 2
R@10 98.8 # 3
Image Retrieval Flickr30k-CN CN-CLIP (ViT-H/14) R@1 83.8 # 5
R@5 96.9 # 4
R@10 98.6 # 2
Zero-shot Text Retrieval Flickr30k-CN CN-CLIPViT-L/14 R@1 80.2 # 5
R@5 96.6 # 6
R@10 98.2 # 6
Zero-shot Text Retrieval Flickr30k-CN R2D2ViT-L/14 R@1 77.6 # 6
R@5 96.7 # 5
R@10 98.9 # 2
Zero-shot Image Retrieval MUGE Retrieval CN-CLIP (ViT-B/16) R@1 52.1 # 4
R@5 76.7 # 4
R@10 84.4 # 4
Mean Recall 71.1 # 4
Zero-shot Image Retrieval MUGE Retrieval CN-CLIP (ViT-H/14) R@1 63.0 # 1
R@5 84.1 # 1
R@10 89.2 # 1
Mean Recall 78.8 # 1
Image Retrieval MUGE Retrieval CN-CLIP (ViT-B/16) R@1 58.4 # 5
R@5 83.6 # 4
R@10 90.0 # 4
Mean Recall 77.4 # 5
Image Retrieval MUGE Retrieval CN-CLIP (RN50) R@1 48.6 # 7
R@5 75.1 # 7
R@10 84.0 # 7
Mean Recall 69.2 # 7
Zero-shot Image Retrieval MUGE Retrieval CN-CLIP (RN50) R@1 42.6 # 7
R@5 68.6 # 7
R@10 77.9 # 7
Mean Recall 63.0 # 7
Zero-shot Image Retrieval MUGE Retrieval CN-CLIP (ViT-L/14@336px) R@1 59.0 # 2
R@5 81.4 # 2
R@10 87.8 # 2
Mean Recall 76.1 # 2
Image Retrieval MUGE Retrieval CN-CLIP (ViT-H/14) R@1 68.9 # 1
R@5 88.7 # 1
R@10 93.1 # 1
Mean Recall 83.6 # 1
Image Retrieval MUGE Retrieval CN-CLIP (ViT-L/14@336px) R@1 65.3 # 2
R@5 86.7 # 2
R@10 92.1 # 2
Mean Recall 81.3 # 2
Image Retrieval MUGE Retrieval CN-CLIP (ViT-L/14) R@1 63.3 # 3
R@5 85.6 # 3
R@10 91.3 # 3
Mean Recall 80.1 # 3
Zero-shot Image Retrieval MUGE Retrieval CN-CLIP (ViT-L/14) R@1 56.3 # 3
R@5 79.8 # 3
R@10 86.2 # 3
Mean Recall 74.1 # 3

Methods