Search Results for author:
Found 87 papers, 49 papers with code
Learning Actionness via Long-range Temporal Order Verification
The annotation is particularly difficult for temporal action localization where large parts of the video present no action, or background.
SUGAR: Pre-training 3D Visual Representations for Robotics
SUGAR employs a versatile transformer-based model to jointly address five pre-training tasks, namely cross-modal knowledge distillation for semantic learning, masked point modeling to understand geometry structures, grasping pose synthesis for object affordance, 3D instance segmentation and referring expression grounding to analyze cluttered scenes.
GenHowTo: Learning to Generate Actions and State Transformations from Instructional Videos
We address the task of generating temporally consistent and physically plausible images of actions and object state transformations.
PolarNet: 3D Point Clouds for Language-Guided Robotic Manipulation
The ability for robots to comprehend and execute manipulation tasks based on natural language instructions is a long-term goal in robotics.
Ranked #5 on
Robot Manipulation
on RLBench
VidChapters-7M: Video Chapters at Scale
To address this issue, we present VidChapters-7M, a dataset of 817K user-chaptered videos including 7M chapters in total.
Object Goal Navigation with Recursive Implicit Maps
Object goal navigation aims to navigate an agent to locations of a given object category in unseen environments.
Robust Visual Sim-to-Real Transfer for Robotic Manipulation
While previous work mainly evaluates DR for disembodied tasks, such as pose estimation and object detection, here we systematically explore visual domain randomization methods and benchmark them on a rich set of challenging robotic manipulation tasks.
Learning Video-Conditioned Policies for Unseen Manipulation Tasks
To encourage generalization to new tasks, we avoid particular tasks during training and learn our policy from unlabelled robot trajectories and corresponding robot videos.
gSDF: Geometry-Driven Signed Distance Functions for 3D Hand-Object Reconstruction
In particular, we address reconstruction of hands and manipulated objects from monocular RGB images.
Ranked #5 on
hand-object pose
on DexYCB
Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning
In this work, we introduce Vid2Seq, a multi-modal single-stage dense event captioning model pretrained on narrated videos which are readily-available at scale.
Ranked #1 on
Dense Video Captioning
on ActivityNet Captions
(using extra training data)
Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluation
One of the major challenges of machine translation (MT) is ambiguity, which can in some cases be resolved by accompanying context such as images.
Image Compression with Product Quantized Masked Image Modeling
In this work, we attempt to bring these lines of research closer by revisiting vector quantization for image compression.
Multi-Task Learning of Object State Changes from Uncurated Videos
We aim to learn to temporally localize object state changes and the corresponding state-modifying actions by observing people interacting with objects in long uncurated web videos.
Language Conditioned Spatial Relation Reasoning for 3D Object Grounding
In this work we propose a language-conditioned transformer model for grounding 3D objects and their spatial relations.
Enforcing the consensus between Trajectory Optimization and Policy Learning for precise robot control
Reinforcement learning (RL) and trajectory optimization (TO) present strong complementary advantages.
Instruction-driven history-aware policies for robotic manipulations
In human environments, robots are expected to accomplish a variety of manipulation tasks given simple natural language instructions.
Ranked #2 on
Robot Manipulation
on RLBench
(Succ. Rate (10 tasks, 100 demos/task) metric)
Learning from Unlabeled 3D Environments for Vision-and-Language Navigation
Our resulting HM3D-AutoVLN dataset is an order of magnitude larger than existing VLN datasets in terms of navigation environments and instructions.
Ranked #1 on
Visual Navigation
on SOON Test
AlignSDF: Pose-Aligned Signed Distance Fields for Hand-Object Reconstruction
We show that such aligned SDFs better focus on reconstructing shape details and improve reconstruction accuracy both for hands and objects.
Ranked #9 on
hand-object pose
on DexYCB
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models
Manual annotation of question and answers for videos, however, is tedious and prohibits scalability.
Ranked #1 on
Zero-Shot Video Question Answer
on TVQA
Weakly-supervised segmentation of referring expressions
Visual grounding localizes regions (boxes or segments) in the image corresponding to given referring expressions.
Learning to Answer Visual Questions from Web Videos
We use our method to generate the WebVidVQA3M dataset from the WebVid dataset, i. e., videos with alt-text annotations, and show its benefits for training VideoQA models.
TubeDETR: Spatio-Temporal Video Grounding with Transformers
We consider the problem of localizing a spatio-temporal tube in a video corresponding to a given text query.
Ranked #2 on
Spatio-Temporal Video Grounding
on VidSTG
Language-Based Temporal Localization
Natural Language Visual Grounding
+5
Look for the Change: Learning Object States and State-Modifying Actions from Untrimmed Web Videos
In this paper, we seek to temporally localize object states (e. g. "empty" and "full" cup) together with the corresponding state-modifying actions ("pouring coffee") in long uncurated videos with minimal supervision.
Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigation
To balance the complexity of large action space reasoning and fine-grained language grounding, we dynamically combine a fine-scale encoding over local observations and a coarse-scale encoding on a global map via graph transformers.
Ranked #4 on
Visual Navigation
on SOON Test
Are Large-scale Datasets Necessary for Self-Supervised Pre-training?
Our study shows that denoising autoencoders, such as BEiT or a variant that we introduce in this paper, are more robust to the type and size of the pre-training data than popular self-supervised methods trained by comparing image embeddings. We obtain competitive performance compared to ImageNet pre-training on a variety of classification datasets, from different domains.
Estimating 3D Motion and Forces of Human-Object Interactions from Internet Videos
First, we introduce an approach to jointly estimate the motion and the actuation forces of the person on the manipulated object by modeling contacts and the dynamics of the interactions.
History Aware Multimodal Transformer for Vision-and-Language Navigation
Vision-and-language navigation (VLN) aims to build autonomous visual agents that follow instructions and navigate in real scenes.
Ranked #3 on
Vision and Language Navigation
on RxR
Differentiable Rendering with Perturbed Optimizers
Notably, images depend both on the properties of observed scenes and on the process of image formation.
Reconstructing and grounding narrated instructional videos in 3D
Contrary to the standard scenario of instance-level 3D reconstruction, where identical objects or scenes are present in all views, objects in different instructional videos may have large appearance variations given varying conditions and versions of the same product.
Airbert: In-domain Pretraining for Vision-and-Language Navigation
Given the scarcity of domain-specific training data and the high diversity of image and language inputs, the generalization of VLN agents to unseen environments remains challenging.
Ranked #3 on
Vision and Language Navigation
on VLN Challenge
Towards unconstrained joint hand-object reconstruction from RGB videos
Our work aims to obtain 3D reconstruction of hands and manipulated objects from monocular videos.
Ranked #5 on
hand-object pose
on HO-3D
Goal-Conditioned Reinforcement Learning with Imagined Subgoals
Goal-conditioned reinforcement learning endows an agent with a large variety of skills, but it often struggles to solve tasks that require more temporally extended reasoning.
XCiT: Cross-Covariance Image Transformers
We propose a "transposed" version of self-attention that operates across feature channels rather than tokens, where the interactions are based on the cross-covariance matrix between keys and queries.
Ranked #55 on
Instance Segmentation
on COCO minival
Segmenter: Transformer for Semantic Segmentation
In this paper we introduce Segmenter, a transformer model for semantic segmentation.
Ranked #15 on
Semantic Segmentation
on PASCAL Context
Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers
We also extend our method to the video domain, improving the state of the art on the VATEX dataset.
Training Vision Transformers for Image Retrieval
Transformers have shown outstanding results for natural language understanding and, more recently, for image classification.
Just Ask: Learning to Answer Questions from Millions of Narrated Videos
In this work, we propose to avoid manual annotation and generate a large-scale training dataset for video question answering making use of automatic cross-modal supervision.
Ranked #1 on
Video Question Answering
on VideoQA
Learning Object Manipulation Skills via Approximate State Estimation from Real Videos
We evaluate our method on simple single- and two-object actions from the Something-Something dataset.
Learning Obstacle Representations for Neural Motion Planning
Motion planning and obstacle avoidance is a key challenge in robotics applications.
Robotics
RareAct: A video dataset of unusual interactions
This paper introduces a manually annotated video dataset of unusual actions, namely RareAct, including actions such as "blend phone", "cut keyboard" and "microwave shoes".
The End-of-End-to-End: A Video Understanding Pentathlon Challenge (2020)
This report summarizes the results of the first edition of the challenge together with the findings of the participants.
Occlusion resistant learning of intuitive physics from videos
In this work we propose a probabilistic formulation of learning intuitive physics in 3D scenes with significant inter-object occlusions.
Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstruction
Modeling hand-object manipulations is essential for understanding how humans interact with their environment.
Ranked #9 on
hand-object pose
on HO-3D
Learning visual policies for building 3D shape categories
We then show the success of our visual policies for building arches from different primitives.
Learning Interactions and Relationships between Movie Characters
Localizing the pair of interacting characters in video is a time-consuming process, instead, we train our model to learn from clip-level weak labels.
Action Modifiers: Learning from Adverbs in Instructional Videos
We present a method to learn a representation for adverbs from instructional videos using weak supervision from the accompanying narrations.
End-to-End Learning of Visual Representations from Uncurated Instructional Videos
Annotating videos is cumbersome, expensive and not scalable.
Ranked #3 on
Action Recognition
on RareAct
Synthetic Humans for Action Recognition from Unseen Viewpoints
Although synthetic training data has been shown to be beneficial for tasks such as human pose estimation, its use for RGB human action recognition is relatively unexplored.
Learning to combine primitive skills: A step towards versatile robotic manipulation
Manipulation tasks such as preparing a meal or assembling furniture remain highly challenging for robotics and vision.
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
In this work, we propose instead to learn such embeddings from video data with readily available natural language annotations in the form of automatically transcribed narrations.
Ranked #4 on
Temporal Action Localization
on CrossTask
Action Localization
Long Video Retrieval (Background Removed)
+3
Monte-Carlo Tree Search for Efficient Visually Guided Rearrangement Planning
We address the problem of visually guided rearrangement planning with many movable objects, i. e., finding a sequence of actions to move a set of objects from an initial arrangement to a desired one, while relying on visual inputs coming from an RGB camera.
Deep Metric Learning Beyond Binary Supervision
Metric Learning for visual similarity has mostly adopted binary supervision indicating whether a pair of images are of the same class or not.
Learning joint reconstruction of hands and manipulated objects
Previous work has made significant progress towards reconstruction of hand poses and object shapes in isolation.
Ranked #7 on
hand-object pose
on DexYCB
Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Video
First, we introduce an approach to jointly estimate the motion and the actuation forces of the person on the manipulated object by modeling contacts and the dynamics of their interactions.
Cross-task weakly supervised learning from instructional videos
In this paper we investigate learning visual models for the steps of ordinary tasks using weak supervision via instructional narrations and an ordered list of steps instead of strong supervision via temporal annotations.
Ranked #5 on
Temporal Action Localization
on CrossTask
Learning to Augment Synthetic Images for Sim2Real Policy Transfer
Policies learned in simulators, however, do not transfer well to real scenes given the domain gap between real and synthetic data.
Detecting unseen visual relations using analogies
We seek to detect visual relations in images of the form of triplets t = (subject, predicate, object), such as "person riding dog", where training examples of the individual entities are available but their combinations are unseen at training.
Tube-CNN: Modeling temporal evolution of appearance for object detection in video
Our goal in this paper is to learn discriminative models for the temporal evolution of object appearance and to use such models for object detection.
MobileFace: 3D Face Reconstruction with Efficient CNN Regression
Estimation of facial shapes plays a central role for face transfer and animation.
Learning to Localize and Align Fine-Grained Actions to Sparse Instructions
Automatic generation of textual video descriptions that are time-aligned with video content is a long-standing goal in computer vision.
A flexible model for training action localization with varying levels of supervision
Our model is based on discriminative clustering and integrates different types of supervision as constraints on the optimization.
Modeling Spatio-Temporal Human Track Structure for Action Localization
In order to localize actions in time, we propose a recurrent localization network (RecLNet) designed to model the temporal structure of actions on the level of person tracks.
BodyNet: Volumetric Inference of 3D Human Body Shapes
Human shape estimation is an important task for video editing, animation and fashion industry.
Ranked #3 on
3D Human Pose Estimation
on Surreal
(using extra training data)
Learning a Text-Video Embedding from Incomplete and Heterogeneous Data
We evaluate our method on the task of video retrieval and report results for the MPII Movie Description and MSR-VTT datasets.
Ranked #33 on
Video Retrieval
on LSMDC
(using extra training data)
Weakly-supervised learning of visual relations
This paper introduces a novel approach for modeling visual relations between pairs of objects.
Learning from Video and Text via Large-Scale Discriminative Clustering
Discriminative clustering has been successfully applied to a number of weakly-supervised learning tasks.
Ranked #35 on
Video Retrieval
on LSMDC
Learnable pooling with Context Gating for video classification
In particular, we evaluate our method on the large-scale multi-modal Youtube-8M v2 dataset and outperform all other methods in the Youtube 8M Large-Scale Video Understanding challenge.
Joint Discovery of Object States and Manipulation Actions
We assume a consistent temporal order for the changes in object states and manipulation actions, and introduce new optimization techniques to learn model parameters without additional supervision.
Learning from Synthetic Humans
In this work we present SURREAL (Synthetic hUmans foR REAL tasks): a new large-scale dataset with synthetically-generated but realistic images of people rendered from 3D sequences of human motion capture data.
ContextLocNet: Context-Aware Deep Network Models for Weakly Supervised Localization
The additive model encourages the predicted object region to be supported by its surrounding context region.
Ranked #4 on
Weakly Supervised Object Detection
on Charades
Much Ado About Time: Exhaustive Annotation of Temporal Data
We conclude that the optimal strategy is to ask as many questions as possible in a HIT (up to 52 binary questions after watching a 30-second video clip in our experiments).
Thin-Slicing for Pose: Learning to Understand Pose Without Explicit Pose Estimation
We address the problem of learning a pose-aware, compact embedding that projects images with similar human poses to be placed close-by in the embedding space.
Instance-Level Video Segmentation From Object Tracks
We address the problem of segmenting multiple object instances in complex videos.
The THUMOS Challenge on Action Recognition for Videos "in the Wild"
Additionally, we include a comprehensive empirical study evaluating the differences in action recognition between trimmed and untrimmed videos, and how well methods trained on trimmed videos generalize to untrimmed videos.
Long-term Temporal Convolutions for Action Recognition
Typical human actions last several seconds and exhibit characteristic spatio-temporal structure.
Ranked #63 on
Action Recognition
on HMDB-51
Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding
Each video is annotated by multiple free-text descriptions, action labels, action intervals and classes of interacted objects.
Context-aware CNNs for person head detection
First, we leverage person-scene relations and propose a Global CNN model trained to predict positions and scales of heads directly from the full image.
Unsupervised Learning from Narrated Instruction Videos
Third, we experimentally demonstrate that the proposed method can automatically discover, in an unsupervised manner, the main steps to achieve the task and locate the steps in the input videos.
Ranked #7 on
Temporal Action Localization
on CrossTask
P-CNN: Pose-based CNN Features for Action Recognition
This work targets human action recognition in video.
Is Object Localization for Free? - Weakly-Supervised Learning With Convolutional Neural Networks
Successful visual object recognition methods typically rely on training datasets containing lots of richly annotated images.
Weakly-Supervised Alignment of Video With Text
Given vectorial features for both video and text, we propose to cast this task as a temporal assignment problem, with an implicit linear mapping between the two feature modalities.
Unsupervised Object Discovery and Tracking in Video Collections
This paper addresses the problem of automatically localizing dominant objects as spatio-temporal tubes in a noisy collection of videos with minimal or even no supervision.
On Pairwise Costs for Network Flow Multi-Object Tracking
Multi-object tracking has been recently approached with the min-cost network flow optimization techniques.
Weakly Supervised Action Labeling in Videos Under Ordering Constraints
We are given a set of video clips, each one annotated with an {\em ordered} list of actions, such as "walk" then "sit" then "answer phone" extracted from, for example, the associated text script.
Learning and Transferring Mid-Level Image Representations using Convolutional Neural Networks
We show that despite differences in image statistics and tasks in the two datasets, the transferred representation leads to significantly improved results for object and action classification, outperforming the current state of the art on Pascal VOC 2007 and 2012 datasets.
Efficient Feature Extraction, Encoding and Classification for Action Recognition
Local video features provide state-of-the-art performance for action recognition.
Learning person-object interactions for action recognition in still images
First, we replace the standard quantized local HOG/SIFT features with stronger discriminatively trained body part and object detectors.
