VLAP: Efficient Video-Language Alignment via Frame Prompting and Distilling for Video Question Answering
In this work, we propose an efficient Video-Language Alignment via Frame-Prompting and Distilling (VLAP) network. Our VLAP model addresses both efficient frame sampling and effective cross-modal alignment in a unified way. In our VLAP network, we design a new learnable question-aware Frame-Prompter together with a new cross-modal distillation (QFormer-Distiller) module. Pre-trained large image-language models have shown promising results on problems such as visual question answering. However, how to efficiently and effectively sample image frames when adapting pre-trained large image-language model to video-language alignment is still the major challenge. Compared with prior work, our VLAP model demonstrates the capability of selecting key frames with critical contents, thus improving the video-language alignment accuracy while reducing the inference latency (+3.3% on NExT-QA Temporal with 3.0X speed up). Overall, our VLAP network outperforms (e.g. +4.6% on STAR Interaction and +2.2% on STAR average with 3.0X speed up, ours 2-frames out-perform SeViLA 4-frames on VLEP with 4.2X speed up) the state-of-the-art methods on the video question-answering benchmarks.
PDF Abstract
Datasets
TVQA
NExT-QA
How2QA
STAR Benchmark
VLEP
| Task | Dataset | Model | Metric Name | Metric Value | Global Rank | Benchmark |
|---|---|---|---|---|---|---|
| Video Question Answering | NExT-QA | VLAP (3B) | Accuracy | 75.6 | # 1 | |
| Video Question Answering | NExT-QA | VLAP (3B 4 frames) | Accuracy | 74.4 | # 3 | |
| Video Question Answering | STAR Benchmark | VLAP (4 frames) | Average Accuracy | 67.1 | # 1 |
