Search Results for author:
Found 190 papers, 76 papers with code
MoReVQA: Exploring Modular Reasoning Models for Video Question Answering
This paper addresses the task of video question answering (videoQA) via a decomposed multi-stage, modular reasoning framework.
Learning Correlation Structures for Vision Transformers
We introduce a new attention mechanism, dubbed structural self-attention (StructSA), that leverages rich correlation patterns naturally emerging in key-query interactions of attention.
Ranked #4 on
Action Recognition
on Diving-48
SUGAR: Pre-training 3D Visual Representations for Robotics
SUGAR employs a versatile transformer-based model to jointly address five pre-training tasks, namely cross-modal knowledge distillation for semantic learning, masked point modeling to understand geometry structures, grasping pose synthesis for object affordance, 3D instance segmentation and referring expression grounding to analyze cluttered scenes.
Streaming Dense Video Captioning
An ideal model for dense video captioning -- predicting captions localized temporally in a video -- should be able to handle long input videos, predict rich, detailed textual descriptions, and be able to produce outputs before processing the entire video.
A Generative Approach for Wikipedia-Scale Visual Entity Recognition
In this paper, we address web-scale visual entity recognition, specifically the task of mapping a given query image to one of the 6 million existing entities in Wikipedia.
SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Code
SceneCraft first models a scene graph as a blueprint, detailing the spatial relationships among assets in the scene.
Time-, Memory- and Parameter-Efficient Visual Adaptation
Here, we outperform a prior adaptor-based method which could only scale to a 1 billion parameter backbone, or fully-finetuning a smaller backbone, with the same GPU and less training time.
RAVEN: Rethinking Adversarial Video Generation with Efficient Tri-plane Networks
To capture these dependencies, our approach incorporates a hybrid explicit-implicit tri-plane representation inspired by 3D-aware generative frameworks developed for three-dimensional object representation and employs a singular latent code to model an entire video sequence.
Pixel Aligned Language Models
When taking locations as inputs, the model performs location-conditioned captioning, which generates captions for the indicated object or region.
Dense Optical Tracking: Connecting the Dots
Code, data, and videos showcasing the capabilities of our approach are available in the project webpage: https://16lemoing. github. io/dot .
PolarNet: 3D Point Clouds for Language-Guided Robotic Manipulation
The ability for robots to comprehend and execute manipulation tasks based on natural language instructions is a long-term goal in robotics.
Ranked #5 on
Robot Manipulation
on RLBench
VidChapters-7M: Video Chapters at Scale
To address this issue, we present VidChapters-7M, a dataset of 817K user-chaptered videos including 7M chapters in total.
CoVR: Learning Composed Video Retrieval from Web Video Captions
Most CoIR approaches require manually annotated datasets, comprising image-text-image triplets, where the text describes a modification from the query image to the target image.
Ranked #1 on
Composed Video Retrieval (CoVR)
on WebVid-CoVR
POCO: 3D Pose and Shape Estimation with Confidence
To address this, we develop POCO, a novel framework for training HPS regressors to estimate not only a 3D human body, but also their confidence, in a single feed-forward pass.
UnLoc: A Unified Framework for Video Localization Tasks
While large-scale image-text pretrained models such as CLIP have been used for multiple video-level tasks on trimmed videos, their use for temporal localization in untrimmed videos is still a relatively unexplored task.
Ranked #1 on
Action Segmentation
on COIN
Object Goal Navigation with Recursive Implicit Maps
Object goal navigation aims to navigate an agent to locations of a given object category in unseen environments.
Robust Visual Sim-to-Real Transfer for Robotic Manipulation
While previous work mainly evaluates DR for disembodied tasks, such as pose estimation and object detection, here we systematically explore visual domain randomization methods and benchmark them on a rich set of challenging robotic manipulation tasks.
Does Visual Pretraining Help End-to-End Reasoning?
A positive result would refute the common belief that explicit visual abstraction (e. g. object detection) is essential for compositional generalization on visual reasoning, and confirm the feasibility of a neural network "generalist" to solve visual recognition and reasoning tasks.
How can objects help action recognition?
In this paper, we investigate how we can use knowledge of objects to design better video models, namely to process fewer tokens and to improve recognition accuracy.
Retrieval-Enhanced Contrastive Vision-Text Models
Contrastive image-text models such as CLIP form the building blocks of many state-of-the-art systems.
Ranked #3 on
Fine-Grained Image Recognition
on OVEN
Waffling around for Performance: Visual Classification with Random Words and Broad Concepts
The visual classification performance of vision-language models such as CLIP has been shown to benefit from additional semantic knowledge from large language models (LLMs) such as GPT-3.
Modular Visual Question Answering via Code Generation
We present a framework that formulates visual question answering as modular code generation.
Learning Video-Conditioned Policies for Unseen Manipulation Tasks
To encourage generalization to new tasks, we avoid particular tasks during training and learn our policy from unlabelled robot trajectories and corresponding robot videos.
End-to-End Spatio-Temporal Action Localisation with Video Transformers
The most performant spatio-temporal action localisation models use external person proposals and complex external memory banks.
Ranked #1 on
Action Recognition
on AVA v2.1
(using extra training data)
gSDF: Geometry-Driven Signed Distance Functions for 3D Hand-Object Reconstruction
In particular, we address reconstruction of hands and manipulated objects from monocular RGB images.
Ranked #5 on
hand-object pose
on DexYCB
Verbs in Action: Improving verb understanding in video-language models
Understanding verbs is crucial to modelling how people and objects interact with each other and the environment through space and time.
Ranked #11 on
Zero-Shot Video Question Answer
on NExT-QA
Improving Image Recognition by Retrieving from Web-Scale Image-Text Data
Retrieval augmented models are becoming increasingly popular for computer vision tasks after their recent success in NLP problems.
Ranked #1 on
Image Classification
on WebVision-1000
(using extra training data)
Exposing and Mitigating Spurious Correlations for Cross-Modal Retrieval
In this work, we introduce ODmAP@k, an object decorrelation metric that measures a model's robustness to spurious correlations in the training data.
Bridging the Gap between Model Explanations in Partially Annotated Multi-label Classification
Based on these findings, we propose to boost the attribution scores of the model trained with partial labels to make its explanation resemble that of the model trained with full labels.
AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASR
(ii) We also introduce a simple curriculum scheme during training which we show is crucial to enable the model to jointly process audio and visual information effectively; and finally (iii) we show that our model achieves state of the art zero-shot results on three different AV-ASR benchmarks (How2, VisSpeech and Ego4D), while also crucially preserving decent performance on traditional audio-only speech recognition benchmarks (LibriSpeech).
Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning
In this work, we introduce Vid2Seq, a multi-modal single-stage dense event captioning model pretrained on narrated videos which are readily-available at scale.
Ranked #1 on
Dense Video Captioning
on ActivityNet Captions
(using extra training data)
Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluation
One of the major challenges of machine translation (MT) is ambiguity, which can in some cases be resolved by accompanying context such as images.
REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory
REVEAL consists of four key components: the memory, the encoder, the retriever and the generator.
Ranked #9 on
Visual Question Answering (VQA)
on OK-VQA
Audiovisual Masked Autoencoders
Can we leverage the audiovisual information already present in video to improve self-supervised representation learning?
Ranked #1 on
Audio Classification
on EPIC-KITCHENS-100
(using extra training data)
Location-Aware Self-Supervised Transformers for Semantic Segmentation
Pixel-level labels are particularly expensive to acquire.
WALDO: Future Video Synthesis using Object Layer Decomposition and Parametric Flow Prediction
This paper presents WALDO (WArping Layer-Decomposed Objects), a novel approach to the prediction of future video frames from past ones.
AVATAR submission to the Ego4D AV Transcription Challenge
In this report, we describe our submission to the Ego4D AudioVisual (AV) Speech Transcription Challenge 2022.
Language Conditioned Spatial Relation Reasoning for 3D Object Grounding
In this work we propose a language-conditioned transformer model for grounding 3D objects and their spatial relations.
Learning Reward Functions for Robotic Manipulation by Observing Humans
Unlike prior work on leveraging human videos to teach robots, our method, Human Offline Learned Distances (HOLD) requires neither a priori data from the robot environment, nor a set of task-specific human demonstrations, nor a predefined notion of correspondence across morphologies, yet it is able to accelerate training of several manipulation tasks on a simulated robot arm compared to using only a sparse reward obtained from task completion.
A Memory Transformer Network for Incremental Learning
We study class-incremental learning, a training setup in which new classes of data are observed over time for the model to learn from.
Enforcing the consensus between Trajectory Optimization and Policy Learning for precise robot control
Reinforcement learning (RL) and trajectory optimization (TO) present strong complementary advantages.
Instruction-driven history-aware policies for robotic manipulations
In human environments, robots are expected to accomplish a variety of manipulation tasks given simple natural language instructions.
Ranked #2 on
Robot Manipulation
on RLBench
(Succ. Rate (10 tasks, 100 demos/task) metric)
Learning from Unlabeled 3D Environments for Vision-and-Language Navigation
Our resulting HM3D-AutoVLN dataset is an order of magnitude larger than existing VLN datasets in terms of navigation environments and instructions.
Ranked #1 on
Visual Navigation
on SOON Test
TL;DW? Summarizing Instructional Videos with Task Relevance & Cross-Modal Saliency
In this work, we focus on summarizing instructional videos, an under-explored area of video summarization.
AlignSDF: Pose-Aligned Signed Distance Fields for Hand-Object Reconstruction
We show that such aligned SDFs better focus on reconstructing shape details and improve reconstruction accuracy both for hands and objects.
Ranked #9 on
hand-object pose
on DexYCB
Beyond Transfer Learning: Co-finetuning for Action Localisation
Transfer learning is the predominant paradigm for training deep networks on small target datasets.
M&M Mix: A Multimodal Multiview Transformer Ensemble
This report describes the approach behind our winning solution to the 2022 Epic-Kitchens Action Recognition Challenge.
Ranked #2 on
Action Recognition
on EPIC-KITCHENS-100
(using extra training data)
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models
Manual annotation of question and answers for videos, however, is tedious and prohibits scalability.
Ranked #1 on
Zero-Shot Video Question Answer
on TVQA
AVATAR: Unconstrained Audiovisual Speech Recognition
Audio-visual automatic speech recognition (AV-ASR) is an extension of ASR that incorporates visual cues, often from the movements of a speaker's mouth.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Learning to Answer Visual Questions from Web Videos
We use our method to generate the WebVidVQA3M dataset from the WebVid dataset, i. e., videos with alt-text annotations, and show its benefits for training VideoQA models.
Weakly-supervised segmentation of referring expressions
Visual grounding localizes regions (boxes or segments) in the image corresponding to given referring expressions.
Assembly Planning from Observations under Physical Constraints
This paper addresses the problem of copying an unknown assembly of primitives with known shape and appearance using information extracted from a single photograph by an off-the-shelf procedure for object detection and pose estimation.
Learning Audio-Video Modalities from Image Captions
To close this gap we propose a new video mining pipeline which involves transferring captions from image captioning datasets to video clips with no additional manual effort.
Ranked #6 on
Zero-shot Text to Audio Retrieval
on AudioCaps
TubeDETR: Spatio-Temporal Video Grounding with Transformers
We consider the problem of localizing a spatio-temporal tube in a video corresponding to a given text query.
Ranked #2 on
Spatio-Temporal Video Grounding
on VidSTG
Language-Based Temporal Localization
Natural Language Visual Grounding
+5
The Right Spin: Learning Object Motion from Rotation-Compensated Flow Fields
In this work, we argue that the coupling of camera rotation and camera translation can create complex motion fields that are difficult for a deep network to untangle directly.
Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigation
To balance the complexity of large action space reasoning and fine-grained language grounding, we dynamically combine a fine-scale encoding over local observations and a coarse-scale encoding on a global map via graph transformers.
Ranked #4 on
Visual Navigation
on SOON Test
Learning with Neighbor Consistency for Noisy Labels
Recent advances in deep learning have relied on large, labelled datasets to train high-capacity models.
End-to-end Generative Pretraining for Multimodal Video Captioning
Recent video and language pretraining frameworks lack the ability to generate sentences.
Ranked #13 on
Video Captioning
on MSR-VTT
(using extra training data)
Multiview Transformers for Video Recognition
Video understanding requires reasoning at multiple spatiotemporal resolutions -- from short fine-grained motions to events taking place over longer durations.
Ranked #5 on
Action Recognition
on EPIC-KITCHENS-100
(using extra training data)
Masking Modalities for Cross-modal Video Retrieval
Our proposal is to pre-train a video encoder using all the available video modalities as supervision, namely, appearance, sound, and transcribed speech.
History Aware Multimodal Transformer for Vision-and-Language Navigation
Vision-and-language navigation (VLN) aims to build autonomous visual agents that follow instructions and navigate in real scenes.
Ranked #3 on
Vision and Language Navigation
on RxR
Differentiable Rendering with Perturbed Optimizers
Notably, images depend both on the properties of observed scenes and on the process of image formation.
Variational Perturbations for Visual Feature Attribution
Explaining a complex black-box system in a post-hoc manner is important to understand its predictions.
Airbert: In-domain Pretraining for Vision-and-Language Navigation
Given the scarcity of domain-specific training data and the high diversity of image and language inputs, the generalization of VLN agents to unseen environments remains challenging.
Ranked #3 on
Vision and Language Navigation
on VLN Challenge
Towards unconstrained joint hand-object reconstruction from RGB videos
Our work aims to obtain 3D reconstruction of hands and manipulated objects from monocular videos.
Ranked #5 on
hand-object pose
on HO-3D
CCVS: Context-aware Controllable Video Synthesis
The prediction model is doubly autoregressive, in the latent space of an autoencoder for forecasting, and in image space for updating contextual information, which is also used to enforce spatio-temporal consistency through a learnable optical flow module.
Ranked #8 on
Video Generation
on BAIR Robot Pushing
Goal-Conditioned Reinforcement Learning with Imagined Subgoals
Goal-conditioned reinforcement learning endows an agent with a large variety of skills, but it often struggles to solve tasks that require more temporally extended reasoning.
Attention Bottlenecks for Multimodal Fusion
Humans perceive the world by concurrently processing and fusing high-dimensional inputs from multiple modalities such as vision and audio.
Ranked #2 on
Action Classification
on Kinetics-Sounds
HDMapGen: A Hierarchical Graph Generative Model of High Definition Maps
To address this issue, we introduce a new challenging task to generate HD maps.
Residual Reinforcement Learning from Demonstrations
Residual reinforcement learning (RL) has been proposed as a way to solve challenging robotic tasks by adapting control actions from a conventional feedback controller to maximize a reward signal.
Large-Scale Unsupervised Object Discovery
Extensive experiments on COCO and OpenImages show that, in the single-object discovery setting where a single prominent object is sought in each image, the proposed LOD (Large-scale Object Discovery) approach is on par with, or better than the state of the art for medium-scale datasets (up to 120K images), and over 37% better than the only other algorithms capable of scaling up to 1. 7M images.
Episodic Transformer for Vision-and-Language Navigation
We demonstrate that encoding the history with a transformer is critical to solve compositional tasks, and that pretraining and joint training with synthetic instructions further improve the performance.
Segmenter: Transformer for Semantic Segmentation
In this paper we introduce Segmenter, a transformer model for semantic segmentation.
Ranked #15 on
Semantic Segmentation
on PASCAL Context
Class-Balanced Distillation for Long-Tailed Visual Recognition
An effective and simple approach to long-tailed visual recognition is to learn feature representations and a classifier separately, with instance and class-balanced sampling, respectively.
Ranked #11 on
Long-tail Learning
on iNaturalist 2018
Local Metrics for Multi-Object Tracking
This paper introduces temporally local metrics for Multi-Object Tracking.
Composable Augmentation Encoding for Video Representation Learning
We focus on contrastive methods for self-supervised video representation learning.
Improving robustness against common corruptions with frequency biased models
CNNs perform remarkably well when the training and test distributions are i. i. d, but unseen image corruptions can cause a surprisingly large drop in performance.
Unified Graph Structured Models for Video Understanding
Accurate video understanding involves reasoning about the relationships between actors, objects and their environment, often over long temporal intervals.
ViViT: A Video Vision Transformer
We present pure-transformer based models for video classification, drawing upon the recent success of such models in image classification.
Ranked #8 on
Action Classification
on MiT
(Top 5 Accuracy metric, using extra
training data)
Learning Temporal Dynamics from Cycles in Narrated Video
Learning to model how the world changes as time elapses has proven a challenging problem for the computer vision community.
Image Matching with Scale Adjustment
In this paper we address the problem of matching two images with two different resolutions: a high-resolution image and a low-resolution one.
Look Before you Speak: Visually Contextualized Utterances
Leveraging recent advances in multimodal learning, our model consists of a novel co-attentional multimodal video transformer, and when trained on both textual and visual context, outperforms baselines that use textual inputs alone.
Just Ask: Learning to Answer Questions from Millions of Narrated Videos
In this work, we propose to avoid manual annotation and generate a large-scale training dataset for video question answering making use of automatic cross-modal supervision.
Ranked #1 on
Video Question Answering
on VideoQA
Learning Obstacle Representations for Neural Motion Planning
Motion planning and obstacle avoidance is a key challenge in robotics applications.
Robotics
TNT: Target-driveN Trajectory Prediction
Our key insight is that for prediction within a moderate time horizon, the future modes can be effectively captured by a set of target states.
The End-of-End-to-End: A Video Understanding Pentathlon Challenge (2020)
This report summarizes the results of the first edition of the challenge together with the findings of the participants.
Learning Video Representations from Textual Web Supervision
Based on this observation, we propose to use text as a method for learning video representations.
Multi-modal Transformer for Video Retrieval
In this paper, we present a multi-modal transformer to jointly encode the different modalities in video, which allows each of them to attend to the others.
Ranked #1 on
Zero-Shot Video Retrieval
on MSR-VTT
(text-to-video Mean Rank metric, using extra
training data)
Uncertainty-Aware Weakly Supervised Action Detection from Untrimmed Videos
Despite the recent advances in video classification, progress in spatio-temporal action recognition has lagged behind.
Unsupervised Learning of Video Representations via Dense Trajectory Clustering
This paper addresses the task of unsupervised learning of representations for action recognition in videos.
Consistency Guided Scene Flow Estimation
To handle inherent modeling error in the consistency loss (e. g. Lambertian assumptions) and for better generalization, we further introduce a learned, output refinement network, which takes the initial predictions, the loss, and the gradient as input, and efficiently predicts a correlated output update.
TAO: A Large-Scale Benchmark for Tracking Any Object
To this end, we ask annotators to label objects that move at any point in the video, and give names to them post factum.
What Makes for Good Views for Contrastive Learning?
Contrastive learning between multiple views of the data has recently achieved state of the art performance in the field of self-supervised representation learning.
Ranked #2 on
Contrastive Learning
on imagenet-1k
VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation
Behavior prediction in dynamic, multi-agent systems is an important problem in the context of self-driving cars, due to the complex representations and interactions of road components, including moving agents (e. g. pedestrians and vehicles) and road context information (e. g. lanes, traffic lights).
Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstruction
Modeling hand-object manipulations is essential for understanding how humans interact with their environment.
Ranked #9 on
hand-object pose
on HO-3D
Learning visual policies for building 3D shape categories
We then show the success of our visual policies for building arches from different primitives.
Memory-Efficient Incremental Learning Through Feature Adaptation
We assume that the model is updated incrementally for new classes as new data becomes available sequentially. This requires adapting the previously stored feature vectors to the updated feature space without having access to the corresponding original training images.
Speech2Action: Cross-modal Supervision for Action Recognition
We train a BERT-based Speech2Action classifier on over a thousand movie screenplays, to predict action labels from transcribed speech segments.
Selecting Relevant Features from a Multi-domain Representation for Few-shot Classification
Popular approaches for few-shot classification consist of first learning a generic data representation based on a large annotated dataset, before adapting the representation to new classes given only a few labeled samples.
Ranked #4 on
Few-Shot Image Classification
on Meta-Dataset Rank
Beyond the Camera: Neural Networks in World Coordinates
Eye movement and strategic placement of the visual field onto the retina, gives animals increased resolution of the scene and suppresses distracting information.
Radioactive data: tracing through training
The mark is robust to strong variations such as different architectures or optimization methods.
Optimized Generic Feature Learning for Few-shot Classification across Domains
To learn models or features that generalize across tasks and domains is one of the grand goals of machine learning.
Synthetic Humans for Action Recognition from Unseen Viewpoints
Although synthetic training data has been shown to be beneficial for tasks such as human pose estimation, its use for RGB human action recognition is relatively unexplored.
Learning to Track Any Object
Moreover, at test time the same network can be applied to detection and tracking, resulting in a unified approach for the two tasks.
Graph convolutional networks for learning with few clean and many noisy labels
In this work we consider the problem of learning a classifier from noisy labels when a few clean labeled examples are given.
White-box vs Black-box: Bayes Optimal Strategies for Membership Inference
Membership inference determines, given a sample and trained parameters of a machine learning model, whether the sample was part of the training set.
Learning to combine primitive skills: A step towards versatile robotic manipulation
Manipulation tasks such as preparing a meal or assembling furniture remain highly challenging for robotics and vision.
Moulding Humans: Non-parametric 3D Human Shape Estimation from Single Images
In this paper, we tackle the problem of 3D human shape estimation from single RGB images.
Self-supervised Learning with Geometric Constraints in Monocular Video: Connecting Flow, Depth, and Camera
We present GLNet, a self-supervised framework for learning depth, optical flow, camera pose and intrinsic parameters from monocular video - addressing the difficulty of acquiring realistic ground-truth for such tasks.
Learning Video Representations using Contrastive Bidirectional Transformer
This paper proposes a self-supervised learning approach for video features that results in significantly improved performance on downstream tasks (such as video classification, captioning and segmentation) compared to existing methods.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
A Study on Action Detection in the Wild
In this work we study the problem of action detection in a highly-imbalanced dataset.
Learning joint reconstruction of hands and manipulated objects
Previous work has made significant progress towards reconstruction of hand poses and object shapes in isolation.
Ranked #7 on
hand-object pose
on DexYCB
VideoBERT: A Joint Model for Video and Language Representation Learning
Self-supervised learning has become increasingly important to leverage the abundance of unlabeled data available on platforms like YouTube.
Ranked #1 on
Action Classification
on YouCook2
Diversity with Cooperation: Ensemble Methods for Few-Shot Classification
Few-shot classification consists of learning a predictive model that is able to effectively adapt to a new class, given only a few annotated samples.
Learning to Augment Synthetic Images for Sim2Real Policy Transfer
Policies learned in simulators, however, do not transfer well to real scenes given the domain gap between real and synthetic data.
AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detection
The dataset contains temporally labeled face tracks in video, where each face instance is labeled as speaking or not, and whether the speech is audible.
Audio-Visual Active Speaker Detection
speaker-diarization
+2
Adaptive Density Estimation for Generative Models
We show that our model significantly improves over existing hybrid models: offering GAN-like samples, IS and FID scores that are competitive with fully adversarial models, and improved likelihood scores.
Detecting unseen visual relations using analogies
We seek to detect visual relations in images of the form of triplets t = (subject, predicate, object), such as "person riding dog", where training examples of the individual entities are available but their combinations are unseen at training.
A Structured Model For Action Detection
A dominant paradigm for learning-based approaches in computer vision is training generic models, such as ResNet for image recognition, or I3D for video understanding, on large datasets and allowing them to discover the optimal representation for the problem at hand.
Modulated Policy Hierarchies
To achieve this, we study different modulation signals and exploration for hierarchical controllers.
Coverage and Quality Driven Training of Generative Image Models
First, we propose a model that extends variational autoencoders by using deterministic invertible transformation layers to map samples from the decoder to the image space.
Déjà Vu: an empirical evaluation of the memorization properties of ConvNets
Convolutional neural networks memorize part of their training data, which is why strategies such as data augmentation and drop-out are employed to mitigate overfitting.
On the Importance of Visual Context for Data Augmentation in Scene Understanding
In this work, we consider object detection, semantic and instance segmentation and augment the training images by blending objects in existing scenes, using instance segmentation annotations.
Actor-Centric Relation Network
A visualization of the learned relation features confirms that our approach is able to attend to the relevant relations for each action.
Ranked #15 on
Action Recognition
on AVA v2.1
End-to-End Incremental Learning
Although deep learning approaches have stood out in recent years due to their state-of-the-art results, they continue to suffer from catastrophic forgetting, a dramatic decrease in overall performance when training with new classes added incrementally.
Ranked #2 on
Incremental Learning
on ImageNet100 - 10 steps
(# M Params metric)
How good is my GAN?
Generative adversarial networks (GANs) are one of the most popular methods for generating images today.
Modeling Visual Context is Key to Augmenting Object Detection Datasets
For this approach to be successful, we show that modeling appropriately the visual context surrounding objects is crucial to place them in the right environment.
A flexible model for training action localization with varying levels of supervision
Our model is based on discriminative clustering and integrates different types of supervision as constraints on the optimization.
Modeling Spatio-Temporal Human Track Structure for Action Localization
In order to localize actions in time, we propose a recurrent localization network (RecLNet) designed to model the temporal structure of actions on the level of person tracks.
Spreading vectors for similarity search
Discretizing multi-dimensional data distributions is a fundamental step of modern indexing methods.
PoTion: Pose MoTion Representation for Action Recognition
We use the human joints as these keypoints and term our Pose moTion representation PoTion.
Ranked #1 on
Skeleton Based Action Recognition
on J-HMDB
Unsupervised Learning of Artistic Styles with Archetypal Style Analysis
In this paper, we introduce an unsupervised learning approach to automatically discover, summarize, and manipulate artistic styles from large collections of paintings.
Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videos
In this paper we describe the egocentric aspect of the dataset and present annotations for Charades-Ego with 68, 536 activity instances in 68. 8 hours of first and third-person video, making it one of the largest and most diverse egocentric datasets available.
Actor and Observer: Joint Modeling of First and Third-Person Videos
Several theories in cognitive neuroscience suggest that when people interact with the world, or simulate interactions, they do so from a first-person egocentric perspective, and seamlessly transfer knowledge between third-person (observer) and first-person (actor).
BodyNet: Volumetric Inference of 3D Human Body Shapes
Human shape estimation is an important task for video editing, animation and fashion industry.
Ranked #3 on
3D Human Pose Estimation
on Surreal
(using extra training data)
LCR-Net++: Multi-person 2D and 3D Pose Detection in Natural Images
We propose an end-to-end architecture for joint 2D and 3D human pose estimation in natural images.
3D Human Pose Estimation
3D Multi-Person Pose Estimation (absolute)
+1
Image-based Synthesis for Deep 3D Human Pose Estimation
Here, we propose a solution to generate a large set of photorealistic synthetic images of humans with 3D pose annotations.
Learning to Segment Moving Objects
We formulate this as a learning problem and design our framework with three cues: (i) independent object motion between a pair of frames, which complements object recognition, (ii) object appearance, which helps to correct errors in motion estimation, and (iii) temporal consistency, which imposes additional constraints on the segmentation.
Joint Learning of Object and Action Detectors
dog and cat jumping, enabling to detect actions of an object without training with these object-actions pairs.
Incremental Learning of Object Detectors without Catastrophic Forgetting
Despite their success for object detection, convolutional neural networks are ill-equipped for incremental learning, i. e., adapting the original model trained on a set of classes to additionally detect objects of new classes, in the absence of the initial training data.
BlitzNet: A Real-Time Deep Network for Scene Understanding
Real-time scene understanding has become crucial in many applications such as autonomous driving.
Ranked #2 on
Real-Time Object Detection
on PASCAL VOC 2007
Weakly-supervised learning of visual relations
This paper introduces a novel approach for modeling visual relations between pairs of objects.
Detecting Parts for Action Localization
In this paper, we propose a new framework for action localization that tracks people in videos and extracts full-body human tubes, i. e., spatio-temporal regions localizing actions, even in the case of occlusions or truncations.
Developing the Path Signature Methodology and its Application to Landmark-based Human Action Recognition
To this end, we regard the evolving landmark data as a high-dimensional path and apply non-linear path signature techniques to provide an expressive, robust, non-linear, and interpretable representation for the sequential events.
LCR-Net: Localization-Classification-Regression for Human Pose
We propose an end-to-end architecture for joint 2D and 3D human pose estimation in natural images.
Ranked #4 on
3D Multi-Person Pose Estimation (root-relative)
on MuPoTS-3D
(MPJPE metric)
3D Human Pose Estimation
3D Multi-Person Pose Estimation (absolute)
+4
AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions
The AVA dataset densely annotates 80 atomic visual actions in 430 15-minute video clips, where actions are localized in space and time, resulting in 1. 58M action labels with multiple labels per person occurring frequently.
Ranked #6 on
Action Detection
on UCF101-24
SCNet: Learning Semantic Correspondence
This paper addresses the problem of establishing semantic correspondences between images depicting different instances of the same object or scene category.
Action Tubelet Detector for Spatio-Temporal Action Localization
We propose the ACtion Tubelet detector (ACT-detector) that takes as input a sequence of frames and outputs tubelets, i. e., sequences of bounding boxes with associated scores.
Spatio-Temporal Action Localization
Temporal Action Localization
SfM-Net: Learning of Structure and Motion from Video
We propose SfM-Net, a geometry-aware neural network for motion estimation in videos that decomposes frame-to-frame pixel motion in terms of scene and object depth, camera motion and 3D object rotations and translations.
Learning Video Object Segmentation with Visual Memory
The module to build a "visual memory" in video, i. e., a joint representation of all the video frames, is realized with a convolutional recurrent unit learned from a small number of training video sequences.
Proposal Flow: Semantic Correspondences from Object Proposals
Finding image correspondences remains a challenging problem in the presence of intra-class variations and large changes in scene layout.
Learning from Synthetic Humans
In this work we present SURREAL (Synthetic hUmans foR REAL tasks): a new large-scale dataset with synthetically-generated but realistic images of people rendered from 3D sequences of human motion capture data.
Learning Motion Patterns in Videos
The problem of determining whether an object is in motion, irrespective of camera motion, is far from being solved.
Areas of Attention for Image Captioning
We propose "Areas of Attention", a novel attention-based model for automatic image captioning.
Multi-region two-stream R-CNN for action detection
We propose a multi-region two-stream R-CNN model for action detection in realistic videos.
Ranked #2 on
Action Detection
on UCF Sports
MoCap-guided Data Augmentation for 3D Pose Estimation in the Wild
Here, we propose a solution to generate a large set of photorealistic synthetic images of humans with 3D pose annotations.
Ranked #117 on
3D Human Pose Estimation
on Human3.6M
(PA-MPJPE metric)
Human Action Localization with Sparse Spatial Supervision
We introduce an approach for spatio-temporal human action localization using sparse spatial supervision.
Long-term Temporal Convolutions for Action Recognition
Typical human actions last several seconds and exhibit characteristic spatio-temporal structure.
Ranked #63 on
Action Recognition
on HMDB-51
Weakly-Supervised Semantic Segmentation using Motion Cues
We also demonstrate that the performance of M-CNN learned with 150 weak video annotations is on par with state-of-the-art weakly-supervised methods trained with thousands of images.
Image Segmentation
Weakly supervised Semantic Segmentation
+1
Convolutional Patch Representations for Image Retrieval: an Unsupervised Approach
Convolutional neural networks (CNNs) have recently received a lot of attention due to their ability to model local stationary structures in natural images in a multi-scale fashion, when learning all model parameters with supervision.
Local Convolutional Features With Unsupervised Training for Image Retrieval
Patch-level descriptors underlie several important computer vision tasks, such as stereo-matching or content-based image retrieval.
Proposal Flow
Finding image correspondences remains a challenging problem in the presence of intra-class variations and large changes in scene layout.~Semantic flow methods are designed to handle images depicting different instances of the same object or scene category.
Approximate Fisher Kernels of non-iid Image Models for Image Categorization
It has been experimentally observed that the performance of BoW and FV representations can be improved by employing discounting transformations such as power normalization.
Online Object Tracking with Proposal Selection
Tracking-by-detection approaches are some of the most successful object trackers in recent years.
Expanded Parts Model for Semantic Description of Humans in Still Images
We validate our method on three recent challenging datasets of human attributes and actions.
Beat-Event Detection in Action Movie Franchises
While important advances were recently made towards temporally localizing and recognizing specific human actions or activities in videos, efficient detection and classification of long video chunks belonging to semantically defined categories such as "pursuit" or "romance" remains challenging. We introduce a new dataset, Action Movie Franchises, consisting of a collection of Hollywood action movie franchises.
DeepMatching: Hierarchical Deformable Dense Matching
We introduce a novel matching algorithm, called DeepMatching, to compute dense correspondences between images.
Ranked #4 on
Dense Pixel Correspondence Estimation
on HPatches
Dense Pixel Correspondence Estimation
Optical Flow Estimation
P-CNN: Pose-based CNN Features for Action Recognition
This work targets human action recognition in video.
Circulant temporal encoding for video retrieval and temporal alignment
We address the problem of specific video event retrieval.
Learning to track for spatio-temporal action localization
We present experimental results for spatio-temporal localization on the UCF-Sports, J-HMDB and UCF-101 action localization datasets, where our approach outperforms the state of the art with a margin of 15%, 7% and 12% respectively in mAP.
Spatio-Temporal Action Localization
Temporal Action Localization
+1
Learning to Detect Motion Boundaries
We compare the results obtained with several state-of-the-art optical flow approaches and study the impact of the different cues used in the random forest. Furthermore, we introduce a new dataset, the YouTube Motion Boundaries dataset (YMB), that comprises 60 sequences taken from real-world videos with manually annotated motion boundaries.
Weakly-Supervised Alignment of Video With Text
Given vectorial features for both video and text, we propose to cast this task as a temporal assignment problem, with an implicit linear mapping between the two feature modalities.
Unsupervised Object Discovery and Tracking in Video Collections
This paper addresses the problem of automatically localizing dominant objects as spatio-temporal tubes in a noisy collection of videos with minimal or even no supervision.
A robust and efficient video representation for action recognition
We also use the homography to cancel out camera motion from the optical flow.
Label-Embedding for Image Classification
Attributes act as intermediate representations that enable parameter sharing between classes, a must when training data is scarce.
Ranked #7 on
Multi-label zero-shot learning
on Open Images V4
Weakly Supervised Object Localization with Multi-fold Multiple Instance Learning
In this case, the supervised information is restricted to binary labels that indicate the absence/presence of object instances in the image, without their locations.
Unsupervised Object Discovery and Localization in the Wild: Part-based Matching with Bottom-up Region Proposals
This paper addresses unsupervised discovery and localization of dominant objects from a noisy image collection with multiple object classes.
EpicFlow: Edge-Preserving Interpolation of Correspondences for Optical Flow
We propose a novel approach for optical flow estimation , targeted at large displacements with significant oc-clusions.
Analysing domain shift factors between videos and images for object detection
Object detection is one of the most important challenges in computer vision.
Weakly Supervised Action Labeling in Videos Under Ordering Constraints
We are given a set of video clips, each one annotated with an {\em ordered} list of actions, such as "walk" then "sit" then "answer phone" extracted from, for example, the associated text script.
Convolutional Kernel Networks
An important goal in visual recognition is to devise image representations that are invariant to particular transformations.
Ranked #23 on
Image Classification
on MNIST
Transformation Pursuit for Image Classification
We propose a principled algorithm Image Transformation Pursuit (ITP) for the automatic selection of a compact set of transformations.
Multi-fold MIL Training for Weakly Supervised Object Localization
In this case, the supervised information is restricted to binary labels that indicate the absence/presence of object instances in the image, without their locations.
Mixing Body-Part Sequences for Human Pose Estimation
In this paper, we present a method for estimating articulated human poses in videos.
Efficient Action Localization with Approximately Normalized Fisher Vectors
Transformation of the FV by power and L2 normalizations has shown to significantly improve its performance, and led to state-of-the-art results for a range of image and video classification and retrieval tasks.
Expanded Parts Model for Human Attribute and Action Recognition in Still Images
We propose a new model for recognizing human attributes (e. g. wearing a suit, sitting, short hair) and actions (e. g. running, riding a horse) in still images.
Label-Embedding for Attribute-Based Classification
The label embedding framework offers other advantages such as the ability to leverage alternative sources of information in addition to attributes (e. g. class hierarchies) or to transition smoothly from zero-shot learning to learning with large quantities of data.
Event Retrieval in Large Video Collections with Circulant Temporal Encoding
Furthermore, we extend product quantization to complex vectors in order to compress our descriptors, and to compare them in the compressed domain.
