Context-Guided Spatio-Temporal Video Grounding

3 Jan 2024  ·  Xin Gu, Heng Fan, Yan Huang, Tiejian Luo, Libo Zhang ·

Spatio-temporal video grounding (or STVG) task aims at locating a spatio-temporal tube for a specific instance given a text query. Despite advancements, current methods easily suffer the distractors or heavy object appearance variations in videos due to insufficient object information from the text, leading to degradation. Addressing this, we propose a novel framework, context-guided STVG (CG-STVG), which mines discriminative instance context for object in videos and applies it as a supplementary guidance for target localization. The key of CG-STVG lies in two specially designed modules, including instance context generation (ICG), which focuses on discovering visual context information (in both appearance and motion) of the instance, and instance context refinement (ICR), which aims to improve the instance context from ICG by eliminating irrelevant or even harmful information from the context. During grounding, ICG, together with ICR, are deployed at each decoding stage of a Transformer architecture for instance context learning. Particularly, instance context learned from one decoding stage is fed to the next stage, and leveraged as a guidance containing rich and discriminative object feature to enhance the target-awareness in decoding feature, which conversely benefits generating better new instance context for improving localization finally. Compared to existing methods, CG-STVG enjoys object information in text query and guidance from mined instance visual context for more accurate target localization. In our experiments on three benchmarks, including HCSTVG-v1/-v2 and VidSTG, CG-STVG sets new state-of-the-arts in m_tIoU and m_vIoU on all of them, showing its efficacy. The code will be released at https://github.com/HengLan/CGSTVG.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Benchmark
Spatio-Temporal Video Grounding HC-STVG1 CG-STVG m_vIoU 38.4 # 1
vIoU@0.3 61.5 # 1
vIoU@0.5 36.3 # 1
Spatio-Temporal Video Grounding HC-STVG2 CG-STVG Val m_vIoU 39.5 # 1
Val vIoU@0.3 64.5 # 2
Val vIoU@0.5 36.3 # 1
Spatio-Temporal Video Grounding VidSTG CG-STVG Declarative m_vIoU 34.0 # 1
Declarative vIoU@0.3 47.7 # 1
Declarative vIoU@0.5 33.1 # 1
Interrogative m_vIoU 29.0 # 1
Interrogative vIoU@0.3 40.5 # 1
Interrogative vIoU@0.5 27.5 # 1

Methods