Search Results for author:
Found 73 papers, 13 papers with code
Video ReCap: Recursive Captioning of Hour-Long Videos
We utilize a curriculum learning training scheme to learn the hierarchical structure of videos, starting from clip-level captions describing atomic actions, then focusing on segment-level descriptions, and concluding with generating summaries for hour-long videos.
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
We present Ego-Exo4D, a diverse, large-scale multimodal multiview video dataset and benchmark challenge.
Multiscale Video Pretraining for Long-Term Activity Forecasting
To alleviate this issue, we propose Multiscale Video Pretraining (MVP), a novel self-supervised pretraining approach that learns robust representations for forecasting by learning to predict contextualized representations of future video clips over multiple timescales.
Learning to Ground Instructional Articles in Videos through Narrations
To deal with the scarcity of labeled data at scale, we source the step descriptions from a language knowledge base (wikiHow) containing instructional articles for a large variety of procedural tasks.
Open-world Instance Segmentation: Top-down Learning with Bottom-up Supervision
Many top-down architectures for instance segmentation achieve significant success when trained and tested on pre-defined closed-world taxonomy.
MINOTAUR: Multi-task Video Grounding From Multimodal Queries
Video understanding tasks take many forms, from action detection to visual query localization and spatio-temporal grounding of sentences.
Egocentric Video Task Translation @ Ego4D Challenge 2022
With no modification to the baseline architectures, our proposed approach achieves competitive performance on two Ego4D challenges, ranking the 1st in the talking to me challenge and the 3rd in the PNR keyframe localization challenge.
HierVL: Learning Hierarchical Video-Language Embeddings
Video-language embeddings are a promising avenue for injecting semantics into visual representations, but existing methods capture only short-term associations between seconds-long video clips and their accompanying text.
Ranked #3 on
Action Recognition
on Charades-Ego
What You Say Is What You Show: Visual Narration Detection in Instructional Videos
Narrated ''how-to'' videos have emerged as a promising data source for a wide range of learning problems, from learning visual representations to training robot policies.
Ego-Only: Egocentric Action Detection without Exocentric Transferring
We find that this renders exocentric transferring unnecessary by showing remarkably strong results achieved by this simple Ego-Only approach on three established egocentric video datasets: Ego4D, EPIC-Kitchens-100, and Charades-Ego.
Relational Space-Time Query in Long-Form Videos
In this paper, we propose to study these problems in a joint framework for long video understanding.
Egocentric Video Task Translation
Different video understanding tasks are typically treated in isolation, and even with distinct types of curated data (e. g., classifying sports in one dataset, tracking animals in another).
HistoPerm: A Permutation-Based View Generation Approach for Improving Histopathologic Feature Representation Learning
These findings suggest that HistoPerm can be a valuable tool for improving representation learning of histopathology features when access to labeled data is limited and can lead to whole-slide classification results that are comparable to or superior to fully-supervised methods.
Deformable Video Transformer
Video transformers have recently emerged as an effective alternative to convolutional networks for action classification.
Calibrating Histopathology Image Classifiers using Label Smoothing
Moreover, we find that using model confidence as a proxy for annotator agreement also improves calibration and accuracy, suggesting that datasets without multiple annotators can still benefit from our proposed label smoothing methods via our proposed confidence-aware label smoothing methods.
Learning To Recognize Procedural Activities with Distant Supervision
In this paper we consider the problem of classifying fine-grained, multi-step activities (e. g., cooking different recipes, making disparate home improvements, creating various forms of arts and crafts) from long videos spanning up to several minutes.
Ranked #3 on
Video Classification
on Breakfast
Label Hallucination for Few-Shot Classification
At the same time, training a simple linear classifier on top of "frozen" features learned from the large labeled dataset fails to adapt the model to the properties of the novel classes, effectively inducing underfitting.
Ego4D: Around the World in 3,000 Hours of Egocentric Video
We introduce Ego4D, a massive-scale egocentric video dataset and benchmark suite.
Long-Short Temporal Contrastive Learning of Video Transformers
Our approach, named Long-Short Temporal Contrastive Learning (LSTCL), enables video transformers to learn an effective clip-level representation by predicting temporal context captured from a longer temporal extent.
Beyond Short Clips: End-to-End Video-Level Learning with Collaborative Memories
The standard way of training video models entails sampling at each iteration a single clip from a video and optimizing the clip prediction with respect to the video-level label.
A Multi-View Approach To Audio-Visual Speaker Verification
Although speaker verification has conventionally been an audio-only task, some practical applications provide both audio and visual streams of input.
Is Space-Time Attention All You Need for Video Understanding?
We present a convolution-free approach to video classification built exclusively on self-attention over space and time.
Ranked #1 on
Video Question Answering
on Howto100M-QA
A Petri Dish for Histopathology Image Analysis
With the rise of deep learning, there has been increased interest in using neural networks for histopathology image analysis, a field that investigates the properties of biopsy or resected specimens traditionally manually examined under a microscope by pathologists.
VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs
We present \textsc{Vx2Text}, a framework for text generation from multimodal inputs consisting of video plus text, speech, or audio.
Slot Machines: Discovering Winning Combinations of Random Weights in Neural Networks
By selecting a weight among a fixed set of random values for each individual connection, our method uncovers combinations of random weights that match the performance of traditionally-trained networks of the same capacity.
Resolution-Based Distillation for Efficient Histology Image Classification
Our results show that a combination of KD and self-supervision allows the student model to approach, and in some cases, surpass the classification accuracy of the teacher, while being much more efficient.
Learn like a Pathologist: Curriculum Learning by Annotator Agreement for Histopathology Image Classification
Based on the nature of histopathology images, a range of difficulty inherently exists among examples, and, since medical datasets are often labeled by multiple annotators, annotator agreement can be used as a natural proxy for the difficulty of a given example.
COBE: Contextualized Object Embeddings from Narrated Instructional Video
A fully-supervised approach to recognizing object states and their contexts in the real-world is unfortunately marred by the long-tailed, open-ended distribution of the data, which would effectively require massive amounts of annotations to capture the appearance of objects in all their different forms.
Generalized Few-Shot Video Classification with Video Retrieval and Feature Generation
Few-shot learning aims to recognize novel classes from a few examples.
Video Understanding as Machine Translation
With the advent of large-scale multimodal video datasets, especially sequences with audio or transcribed speech, there has been a growing interest in self-supervised learning of video representations.
Stein Variational Inference for Discrete Distributions
In addition, such transform can be straightforwardly employed in gradient-free kernelized Stein discrepancy to perform goodness-of-fit (GOF) test on discrete distributions.
Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation
We introduce a method for simultaneously classifying, segmenting and tracking object instances in a video sequence.
Listen to Look: Action Recognition by Previewing Audio
In the face of the video data deluge, today's expensive clip-level classifiers are increasingly impractical.
Ranked #8 on
Action Recognition
on ActivityNet
STAR-Caps: Capsule Networks with Straight-Through Attentive Routing
Capsule networks have been shown to be powerful models for image classification, thanks to their ability to represent and capture viewpoint variations of an object.
Self-Supervised Learning by Cross-Modal Audio-Video Clustering
To the best of our knowledge, XDC is the first self-supervised learning method that outperforms large-scale fully-supervised pretraining for action recognition on the same architecture.
Only Time Can Tell: Discovering Temporal Data for Temporal Modeling
However, in current video datasets it has been observed that action classes can often be recognized without any temporal information from a single frame of video.
UniDual: A Unified Model for Image and Video Understanding
It consists of a shared 2D spatial convolution followed by two parallel point-wise convolutional layers, one devoted to images and the other one used for videos.
Video Modeling with Correlation Networks
Motion is a salient cue to recognize actions in video.
Ranked #108 on
Action Classification
on Kinetics-400
Learning Temporal Pose Estimation from Sparsely-Labeled Videos
To reduce the need for dense annotations, we propose a PoseWarper network that leverages training videos with sparse annotations (every k frames) to learn to perform dense temporal pose propagation and estimation.
Ranked #2 on
Multi-Person Pose Estimation
on PoseTrack2018
(using extra training data)
Attentive Action and Context Factorization
This is a challenging task due to the subtlety of human actions in video and the co-occurrence of contextual elements.
SCSampler: Sampling Salient Clips from Video for Efficient Action Recognition
We demonstrate that the computational cost of action recognition on untrimmed videos can be dramatically reduced by invoking recognition only on these most salient clips.
Ranked #1 on
Action Recognition
on miniSports
Video Classification with Channel-Separated Convolutional Networks
It is natural to ask: 1) if group convolution can help to alleviate the high computational cost of video classification networks; 2) what factors matter the most in 3D group convolutional networks; and 3) what are good computation/accuracy trade-offs with 3D group convolutional networks.
Ranked #1 on
Action Recognition
on Sports-1M
DistInit: Learning Video Representations Without a Single Labeled Video
In this work, we propose an alternative approach to learning video representations that require no semantically labeled videos and instead leverages the years of effort in collecting and labeling large and clean still-image datasets.
Ranked #72 on
Action Recognition
on HMDB-51
(using extra training data)
Learning Discriminative Motion Features Through Detection
Our network learns to spatially sample features from Frame B in order to maximize pose detection accuracy in Frame A.
Scenes-Objects-Actions: A Multi-Task, Multi-Label Video Dataset
The videos retrieved by the search engines are then veried for correctness by human annotators.
MaskConnect: Connectivity Learning by Gradient Descent
Such simple connectivity rules are unlikely to yield the optimal architecture for the given problem.
Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization
There is a natural correlation between the visual and auditive elements of a video.
Ranked #7 on
Self-Supervised Audio Classification
on ESC-50
What Makes a Video a Video: Analyzing Temporal Information in Video Understanding Models and Datasets
The ability to capture temporal information has been critical to the development of video understanding models.
Object Detection in Video with Spatiotemporal Sampling Networks
We propose a Spatiotemporal Sampling Network (STSN) that uses deformable convolutions across time for object detection in videos.
Detect-and-Track: Efficient Pose Estimation in Videos
This paper addresses the problem of estimating and tracking human body keypoints in complex, multi-person video.
Ranked #8 on
Pose Tracking
on PoseTrack2017
(using extra training data)
HACS: Human Action Clips and Segments Dataset for Recognition and Temporal Localization
This paper presents a new large-scale dataset for recognition and temporal localization of human actions collected from Web videos.
Ranked #10 on
Temporal Action Localization
on HACS
A Closer Look at Spatiotemporal Convolutions for Action Recognition
In this paper we discuss several forms of spatiotemporal convolutions for video analysis and study their effects on action recognition.
Ranked #3 on
Action Recognition
on Sports-1M
Connectivity Learning in Multi-Branch Networks
While much of the work in the design of convolutional networks over the last five years has revolved around the empirical investigation of the importance of depth, filter sizes, and number of feature channels, recent studies have shown that branching, i. e., splitting the computation along parallel but distinct threads and then aggregating their outputs, represents a new promising dimension for significant improvements in performance.
Learning to Inpaint for Image Compression
We study the design of deep architectures for lossy image compression.
BranchConnect: Large-Scale Visual Recognition with Learned Branch Connections
We introduce an architecture for large-scale image categorization that enables the end-to-end learning of separate visual features for the different classes to distinguish.
Deep-Learning for Classification of Colorectal Polyps on Whole-Slide Images
In this work, we built an automatic image-understanding method that can accurately classify different types of colorectal polyps in whole-slide histology images to help pathologists with histopathological characterization and diagnosis of colorectal polyps.
VideoMCC: a New Benchmark for Video Comprehension
Language has been exploited to sidestep the problem of defining video categories, by formulating video understanding as the task of captioning or description.
Multiple Hypothesis Colorization
In the experiments we show that our proposed method outperforms traditional JPEG color coding by a large margin, producing colors that are nearly indistinguishable from the ground truth at the storage cost of just a few hundred bytes for high-resolution pictures!
Local Perturb-and-MAP for Structured Prediction
In this work, we present a new Local Perturb-and-MAP (locPMAP) framework that replaces the global optimization with a local optimization by exploiting our observed connection between locPMAP and the pseudolikelihood of the original CRF model.
Convolutional Random Walk Networks for Semantic Image Segmentation
It combines these two objectives via a novel random walk layer that enforces consistent spatial grouping in the deep layers of the network.
Network of Experts for Large-Scale Image Categorization
The training of our "network of experts" is completely end-to-end: the partition of categories into disjoint subsets is learned simultaneously with the parameters of the network trunk and the experts are trained jointly by minimizing a single learning objective over all classes.
Recurrent Mixture Density Network for Spatiotemporal Visual Attention
In this work, we propose a spatiotemporal attentional model that learns where to look in a video directly from human fixation data.
Deep End2End Voxel2Voxel Prediction
Over the last few years deep learning methods have emerged as one of the most prominent approaches for video analysis.
Semantic Segmentation with Boundary Neural Fields
To overcome these problems, we introduce a Boundary Neural Field (BNF), which is a global energy model integrating FCN predictions with boundary cues.
High-for-Low and Low-for-High: Efficient Boundary Detection from Deep Object Features and its Applications to High-Level Vision
We can view this process as a "Low-for-High" scheme, where low-level boundaries aid high-level vision tasks.
Coupled Depth Learning
Crucially, the depth basis and the regression function are {\bf coupled} and jointly optimized by our learning scheme.
DeepEdge: A Multi-Scale Bifurcated Deep Network for Top-Down Contour Detection
This section of the network is applied to four different scales of the image input.
Learning Spatiotemporal Features with 3D Convolutional Networks
We propose a simple, yet effective approach for spatiotemporal feature learning using deep 3-dimensional convolutional networks (3D ConvNets) trained on a large scale supervised video dataset.
Ranked #8 on
Action Recognition
on Sports-1M
Action Recognition In Videos
Dynamic Facial Expression Recognition
Self-taught Object Localization with Deep Networks
This paper introduces self-taught object localization, a novel approach that leverages deep convolutional networks trained for whole-image recognition to localize objects in images without additional human supervision, i. e., without using any ground-truth bounding boxes for training.
EXMOVES: Classifier-based Features for Scalable Action Recognition
We show the generality of our approach by building our mid-level descriptors from two different low-level feature representations.
Leveraging Structure from Motion to Learn Discriminative Codebooks for Scalable Landmark Classification
In this paper we propose a new technique for learning a discriminative codebook for local feature descriptors, specifically designed for scalable landmark classification.
PiCoDes: Learning a Compact Code for Novel-Category Recognition
In contrast to previous approaches to learn compact codes, we optimize explicitly for (an upper bound on) classification performance.
Exploiting weakly-labeled Web images to improve object classification: a domain adaptation approach
In this paper we investigate and compare methods that learn image classifiers by combining very few manually annotated examples (e. g., 1-10 images per class) and a large number of weakly-labeled Web photos retrieved using keyword-based image search.
