Search Results for author:
Found 13 papers, 4 papers with code
Koala: Key frame-conditioned long video-LLM
Long video question answering is a challenging task that involves recognizing short-term activities and reasoning about their fine-grained relationships.
Socratis: Are large multimodal models emotionally aware?
We further see that current captioning metrics based on large vision-language models also fail to correlate with human preferences.
Multiscale Video Pretraining for Long-Term Activity Forecasting
To alleviate this issue, we propose Multiscale Video Pretraining (MVP), a novel self-supervised pretraining approach that learns robust representations for forecasting by learning to predict contextualized representations of future video clips over multiple timescales.
EgoAdapt: A multi-stream evaluation study of adaptation to real-world egocentric user video
We introduce an evaluation framework that directly exploits the user's data stream with new metrics to measure the adaptation gain over the population model, online generalization, and hindsight performance.
Language-Guided Audio-Visual Source Separation via Trimodal Consistency
We propose a self-supervised approach for learning to perform audio source separation in videos based on natural language queries, using only unlabeled video and audio pairs as training data.
NewsStories: Illustrating articles with visual summaries
Thus, we explore a novel setting where the goal is to learn a self-supervised visual-language representation that is robust to varying text length and the number of images.
Look at What I’m Doing: Self-Supervised Spatial Grounding of Narrations in Instructional Videos
Key to our approach is the ability to learn to spatially localize interactions with self-supervision on a large corpus of videos with accompanying transcribed narrations.
Look at What I'm Doing: Self-Supervised Spatial Grounding of Narrations in Instructional Videos
Key to our approach is the ability to learn to spatially localize interactions with self-supervision on a large corpus of videos with accompanying transcribed narrations.
Detecting Cross-Modal Inconsistency to Defend Against Neural Fake News
In addition to the valuable insights gleaned from our user study experiments, we provide a relatively effective approach based on detecting visual-semantic inconsistencies, which will serve as an effective first line of defense and a useful reference for future work in defending against machine-generated disinformation.
LoGAN: Latent Graph Co-Attention Network for Weakly-Supervised Video Moment Retrieval
However, while such approaches tend to focus on identifying relationships between elements of the video and language modalities, there is less emphasis on modeling relational context between video frames given the semantic context of the query.
wMAN: WEAKLY-SUPERVISED MOMENT ALIGNMENT NETWORK FOR TEXT-BASED VIDEO SEGMENT RETRIEVAL
Given a video and a sentence, the goal of weakly-supervised video moment retrieval is to locate the video segment which is described by the sentence without having access to temporal annotations during training.
Learning Similarity Conditions Without Explicit Supervision
Many real-world tasks require models to compare images along multiple similarity conditions (e. g. similarity in color, category or shape).
Language Features Matter: Effective Language Representations for Vision-Language Tasks
Shouldn't language and vision features be treated equally in vision-language (VL) tasks?
