Search Results for author:
Found 35 papers, 20 papers with code
FiGCLIP: Fine-Grained CLIP Adaptation via Densely Annotated Videos
We use the SRL and verb information to create rule-based detailed captions, making sure they capture most of the visual concepts.
Eye vs. AI: Human Gaze and Model Attention in Video Memorability
Understanding the factors that determine video memorability has important applications in areas such as educational technology and advertising.
Generalized Cross-domain Multi-label Few-shot Learning for Chest X-rays
Real-world application of chest X-ray abnormality classification requires dealing with several challenges: (i) limited training data; (ii) training and evaluation sets that are derived from different domains; and (iii) classes that appear during training may have partial overlap with classes of interest during evaluation.
How you feelin'? Learning Emotions and Mental States in Movie Scenes
Towards this goal, we formulate emotion understanding as predicting a diverse and multi-label set of emotions at the level of a movie scene and for each character.
GrapeQA: GRaph Augmentation and Pruning to Enhance Question-Answering
Commonsense question-answering (QA) methods combine the power of pre-trained Language Models (LM) with the reasoning provided by Knowledge Graphs (KG).
Ranked #8 on
Question Answering
on OpenBookQA
Test of Time: Instilling Video-Language Models with a Sense of Time
Our work serves as a first step towards probing and instilling a sense of time in existing video-language models without the need for data and compute-intense training from scratch.
Ranked #1 on
Video-Text Retrieval
on Test-of-Time
(using extra training data)
Sonus Texere! Automated Dense Soundtrack Construction for Books using Movie Adaptations
Reading, much like music listening, is an immersive experience that transports readers while taking them on an emotional journey.
Can we Adopt Self-supervised Pretraining for Chest X-Rays?
Chest radiograph (or Chest X-Ray, CXR) is a popular medical imaging modality that is used by radiologists across the world to diagnose heart or lung conditions.
Language Conditioned Spatial Relation Reasoning for 3D Object Grounding
In this work we propose a language-conditioned transformer model for grounding 3D objects and their spatial relations.
Unsupervised Audio-Visual Lecture Segmentation
We formulate lecture segmentation as an unsupervised task that leverages visual, textual, and OCR cues from the lecture, while clip representations are fine-tuned on a pretext self-supervised task of matching the narration with the temporally aligned visual content.
Grounded Video Situation Recognition
Recently, Video Situation Recognition (VidSitu) is framed as a task for structured prediction of multiple events, their relationships, and actions and various verb-role pairs attached to descriptive entities.
Instruction-driven history-aware policies for robotic manipulations
In human environments, robots are expected to accomplish a variety of manipulation tasks given simple natural language instructions.
Ranked #2 on
Robot Manipulation
on RLBench
(Succ. Rate (10 tasks, 100 demos/task) metric)
Learning from Unlabeled 3D Environments for Vision-and-Language Navigation
Our resulting HM3D-AutoVLN dataset is an order of magnitude larger than existing VLN datasets in terms of navigation environments and instructions.
Ranked #1 on
Visual Navigation
on SOON Test
Learning Object Manipulation Skills from Video via Approximate Differentiable Physics
We evaluate our approach on a 3D reconstruction task that consists of 54 video demonstrations sourced from 9 actions such as pull something from right to left or put something in front of something.
Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigation
To balance the complexity of large action space reasoning and fine-grained language grounding, we dynamically combine a fine-scale encoding over local observations and a coarse-scale encoding on a global map via graph transformers.
Ranked #4 on
Visual Navigation
on SOON Test
Feature Generation for Long-tail Classification
Oversampling instances of the tail classes attempts to solve this imbalance.
Ranked #1 on
Long-tail Learning
on mini-ImageNet-LT
Airbert: In-domain Pretraining for Vision-and-Language Navigation
Given the scarcity of domain-specific training data and the high diversity of image and language inputs, the generalization of VLN agents to unseen environments remains challenging.
Ranked #3 on
Vision and Language Navigation
on VLN Challenge
Learning Object Manipulation Skills via Approximate State Estimation from Real Videos
We evaluate our method on simple single- and two-object actions from the Something-Something dataset.
Deep Multimodal Feature Encoding for Video Ordering
True understanding of videos comes from a joint analysis of all its modalities: the video frames, the audio track, and any accompanying text such as closed captions.
Clustering based Contrastive Learning for Improving Face Representations
We demonstrate our method on the challenging task of learning representations for video face clustering.
Learning Interactions and Relationships between Movie Characters
Localizing the pair of interacting characters in video is a time-consuming process, instead, we train our model to learn from clip-level weak labels.
The Shmoop Corpus: A Dataset of Stories with Loosely Aligned Summaries
Understanding stories is a challenging reading comprehension problem for machines as it requires reading a large volume of text and following long-range dependencies.
Video Face Clustering with Unknown Number of Clusters
Understanding videos such as TV series and movies requires analyzing who the characters are and what they are doing.
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
In this work, we propose instead to learn such embeddings from video data with readily available natural language annotations in the form of automatically transcribed narrations.
Ranked #4 on
Temporal Action Localization
on CrossTask
Action Localization
Long Video Retrieval (Background Removed)
+3
Self-Supervised Learning of Face Representations for Video Face Clustering
In this paper, we address video face clustering using unsupervised methods.
Visual Reasoning by Progressive Module Networks
Thus, a module for a new task learns to query existing modules and composes their outputs in order to produce its own output.
Now You Shake Me: Towards Automatic 4D Cinema
We are interested in enabling automatic 4D cinema by parsing physical and special effects from untrimmed movies.
MovieGraphs: Towards Understanding Human-Centric Situations from Videos
Towards this goal, we introduce a novel dataset called MovieGraphs which provides detailed, graph-based annotations of social situations depicted in movie clips.
Situation Recognition with Graph Neural Networks
We address the problem of recognizing situations in images.
Ranked #9 on
Situation Recognition
on imSitu
Relaxed Earth Mover's Distances for Chain- and Tree-connected Spaces and their use as a Loss Function in Deep Learning
The Earth Mover's Distance (EMD) computes the optimal cost of transforming one distribution into another, given a known transport metric between them.
Recovering the Missing Link: Predicting Class-Attribute Associations for Unsupervised Zero-Shot Learning
In this work, we aim to carry out attribute-based zero-shot classification in an unsupervised manner.
MovieQA: Understanding Stories in Movies through Question-Answering
We introduce the MovieQA dataset which aims to evaluate automatic story comprehension from both video and text.
Book2Movie: Aligning Video Scenes With Book Chapters
Such an alignment facilitates finding differences between the adaptation and the original source, and also acts as a basis for deriving rich descriptions from the novel for the video clips.
StoryGraphs: Visualizing Character Interactions as a Timeline
We present a novel way to automatically summarize and represent the storyline of a TV episode by visualizing character interactions as a chart.
Semi-supervised Learning with Constraints for Person Identification in Multimedia Data
We address the problem of person identification in TV series.
