Search Results for author:
Found 55 papers, 37 papers with code
LeGrad: An Explainability Method for Vision Transformers via Feature Formation Sensitivity
Vision Transformers (ViTs), with their ability to model long-range dependencies through self-attention mechanisms, have become a standard architecture in computer vision.
Uncertainty Quantification via Stable Distribution Propagation
We propose a new approach for propagating stable probability distributions through neural networks.
Grounding Everything: Emerging Localization Properties in Vision-Language Transformers
To leverage those capabilities, we propose a Grounding Everything Module (GEM) that generalizes the idea of value-value attention introduced by CLIPSurgery to a self-self attention path.
Learning Human Action Recognition Representations Without Real Humans
To this end, we present, for the first time, a benchmark that leverages real-world videos with humans removed and synthetic data containing virtual humans to pre-train a model.
HowToCaption: Prompting LLMs to Transform Video Annotations at Scale
Specifically, we prompt an LLM to create plausible video descriptions based on ASR narrations of the video for a large-scale instructional video dataset.
In-Style: Bridging Text and Uncurated Videos with Style Transfer for Text-Video Retrieval
Large-scale noisy web image-text datasets have been proven to be efficient for learning robust vision-language models.
Preserving Modality Structure Improves Multi-Modal Learning
Self-supervised learning on large-scale multi-modal datasets allows learning semantically meaningful embeddings in a joint multi-modal representation space without relying on human annotations.
Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languages
Recent models such as XLS-R and Whisper have made multilingual speech technologies more accessible by pre-training on audio from around 100 spoken languages each.
ISAAC Newton: Input-based Approximate Curvature for Newton's Method
We present ISAAC (Input-baSed ApproximAte Curvature), a novel method that conditions the gradient using selected second-order information and has an asymptotically vanishing computational overhead, assuming a batch size smaller than the number of neurons.
Learning Situation Hyper-Graphs for Video Question Answering
The proposed method is trained in an end-to-end manner and optimized by a VQA loss with the cross-entropy function and a Hungarian matching loss for the situation graph prediction.
Ranked #6 on
Video Question Answering
on AGQA 2.0 balanced
(Average Accuracy metric)
WEAR: An Outdoor Sports Dataset for Wearable and Egocentric Activity Recognition
Though research has shown the complementarity of camera- and inertial-based data, datasets which offer both egocentric video and inertial-based sensor data remain scarce.
Egocentric Activity Recognition
Human Activity Recognition
+2
What, when, and where? -- Self-Supervised Spatio-Temporal Grounding in Untrimmed Multi-Action Videos from Narrated Instructions
Spatio-temporal grounding describes the task of localizing events in space and time, e. g., in video data, based on verbal descriptions only.
Temperature Schedules for Self-Supervised Contrastive Methods on Long-Tail Data
Such a schedule results in a constant `task switching' between an emphasis on instance discrimination and group-wise discrimination and thereby ensures that the model learns both group-wise features, as well as instance-specific details.
MAtch, eXpand and Improve: Unsupervised Finetuning for Zero-Shot Action Recognition with Language Knowledge
We adapt a VL model for zero-shot and few-shot action recognition using a collection of unlabeled videos and an unpaired action dictionary.
Ranked #3 on
Zero-Shot Action Recognition
on Kinetics
TAEC: Unsupervised Action Segmentation with Temporal-Aware Embedding and Clustering
Temporal action segmentation in untrimmed videos has gained increased attention recently.
Learning by Sorting: Self-supervised Learning with Group Ordering Constraints
Contrastive learning has become an important tool in learning representations from unlabeled data mainly relying on the idea of minimizing distance between positive data pairs, e. g., views from the same images, and maximizing distance between negative data pairs, e. g., views from different images.
Video Test-Time Adaptation for Action Recognition
Our proposed method demonstrates a substantial performance gain over existing test-time adaptation approaches in both evaluations of a single distribution shift and the challenging case of random distribution shifts.
Deep Differentiable Logic Gate Networks
Recently, research has increasingly focused on developing efficient neural network architectures.
C2KD: Cross-Lingual Cross-Modal Knowledge Distillation for Multilingual Text-Video Retrieval
Inspired by the fact that English text-video retrieval outperforms other languages, we train a student model using input text in different languages to match the cross-modal predictions from teacher models using input text in English.
Contrastive Audio-Visual Masked Autoencoder
In this paper, we first extend the recent Masked Auto-Encoder (MAE) model from a single modality to audio-visual multi-modalities.
Ranked #1 on
Audio Tagging
on AudioSet
(using extra training data)
VL-Taboo: An Analysis of Attribute-based Zero-shot Capabilities of Vision-Language Models
We follow up with the analysis of the attribute-based zero-shot learning capabilities of these models, evaluating how well this classical zero-shot notion emerges from large-scale webly supervision.
Augmentation Learning for Semi-Supervised Classification
As the accuracy for ImageNet and similar datasets increased over time, the performance on tasks beyond the classification of natural images is yet to be explored.
Weakly Supervised Grounding for VQA in Vision-Language Transformers
Transformers for visual-language representation learning have been getting a lot of interest and shown tremendous performance on visual question answering (VQA) and grounding.
Differentiable Top-k Classification Learning
In this work, we relax this assumption and optimize the model for multiple k simultaneously instead of using a single k. Leveraging recent advances in differentiable sorting and ranking, we propose a differentiable top-k cross-entropy classification loss.
Ranked #58 on
Image Classification
on ImageNet
CycDA: Unsupervised Cycle Domain Adaptation from Image to Video
To address these challenges, we propose Cycle Domain Adaptation (CycDA), a cycle-based approach for unsupervised image-to-video domain adaptation by leveraging the joint spatial information in images and videos on the one hand and, on the other hand, training an independent spatio-temporal model to bridge the modality gap.
Monotonic Differentiable Sorting Networks
We introduce a family of sigmoid functions and prove that they produce differentiable sorting networks that are monotonic.
Everything at Once - Multi-Modal Fusion Transformer for Video Retrieval
In this work, we present a multi-modal, modality agnostic fusion transformer that learns to exchange information between multiple modalities, such as video, audio, and text, and integrate them into a fused representation in a joined multi-modal embedding space.
Everything at Once -- Multi-modal Fusion Transformer for Video Retrieval
Multi-modal learning from video data has seen increased attention recently as it allows to train semantically meaningful embeddings without human annotation enabling tasks like zero-shot retrieval and classification.
Unsupervised Domain Generalization by Learning a Bridge Across Domains
The ability to generalize learned representations across significantly different visual domains, such as between real photos, clipart, paintings, and sketches, is a fundamental capacity of the human visual system.
Routing with Self-Attention for Multimodal Capsule Networks
We present a new multimodal capsule network that allows us to leverage the strength of capsules in the context of a multimodal learning framework on large amounts of video data.
Cascaded Multilingual Audio-Visual Learning from Videos
In this paper, we explore self-supervised audio-visual models that learn from instructional videos.
Style Agnostic 3D Reconstruction via Adversarial Style Transfer
Recently introduced differentiable renderers can be leveraged to learn the 3D geometry of objects from 2D images, but those approaches require additional supervision to enable the renderer to produce an output that can be compared to the input image.
Learning with Algorithmic Supervision via Continuous Relaxations
The integration of algorithmic components into neural architectures has gained increased attention recently, as it allows training neural networks with new forms of supervision such as ordering constraints or silhouettes instead of using ground truth labels.
Propagating Distributions through Neural Networks
We propose a new approach to propagating probability distributions through neural networks.
A Sampling-Free Approximation of Gaussian Variational Auto-Encoders
We propose a sampling-free approximate formulation of Gaussian variational auto-encoders.
Generalized and Incremental Few-Shot Learning by Explicit Learning and Calibration without Forgetting
Both generalized and incremental few-shot learning have to deal with three major challenges: learning novel classes from only few samples per class, preventing catastrophic forgetting of base classes, and classifier calibration across novel and base classes.
Found a Reason for me? Weakly-supervised Grounded Visual Question Answering using Capsules
In this paper, we focus on a more relaxed setting: the grounding of relevant visual entities in a weakly supervised manner by training on the VQA task alone.
Differentiable Sorting Networks for Scalable Sorting and Ranking Supervision
Sorting and ranking supervision is a method for training neural networks end-to-end based on ordering constraints.
Unsupervised Discriminative Embedding for Sub-Action Learning in Complex Activities
This paper proposes a novel approach for unsupervised sub-action learning in complex activities.
Ranked #28 on
Action Segmentation
on Breakfast
Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videos
Multimodal self-supervised learning is getting more and more attention as it allows not only to train large networks without human supervision but also to search and retrieve data across various modalities.
Detector-Free Weakly Supervised Grounding by Separation
In this work, we focus on the task of Detector-Free WSG (DF-WSG) to solve WSG without relying on a pre-trained detector.
Ranked #1 on
Phrase Grounding
on Visual Genome
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos
Further, we propose a tri-modal model that jointly processes raw audio, video, and text captions from videos to learn a multi-modal semantic embedding space useful for text-video retrieval.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
Joint Visual-Temporal Embedding for Unsupervised Learning of Actions in Untrimmed Sequences
Understanding the structure of complex activities in untrimmed videos is a challenging task in the area of action recognition.
More Is Less: Learning Efficient Video Representations by Big-Little Network and Depthwise Temporal Aggregation
Current state-of-the-art models for video action recognition are mostly based on expensive 3D ConvNets.
Ranked #89 on
Action Recognition
on Something-Something V2
(using extra training data)
A Hybrid RNN-HMM Approach for Weakly Supervised Temporal Action Segmentation
Action recognition has become a rapidly developing research field within the last decade.
Mining YouTube - A dataset for learning fine-grained action concepts from webly supervised video data
Action recognition is so far mainly focusing on the problem of classification of hand selected preclipped actions and reaching impressive results in this field.
Unsupervised learning of action classes with continuous temporal embedding
The task of temporally detecting and segmenting actions in untrimmed videos has seen an increased attention recently.
NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning
Video learning is an important task in computer vision and has experienced increasing interest over the recent years.
Recurrent Residual Learning for Action Recognition
In this work, we propose a novel recurrent ConvNet architecture called recurrent residual networks to address the task of action recognition.
Action Sets: Weakly Supervised Action Segmentation without Ordering Constraints
Action detection and temporal segmentation of actions in videos are topics of increasing interest.
Weakly Supervised Action Learning with RNN based Fine-to-coarse Modeling
We present an approach for weakly supervised learning of human actions.
Weakly supervised learning of actions from transcripts
Our system is based on the idea that, given a sequence of input data and a transcript, i. e. a list of the order the actions occur in the video, it is possible to infer the actions within the video stream, and thus, learn the related action models without the need for any frame-based annotation.
An end-to-end generative framework for video segmentation and recognition
We describe an end-to-end generative approach for the segmentation and recognition of human activities.
Cooking in the kitchen: Recognizing and Segmenting Human Activities in Videos
Through extensive system evaluations, we demonstrate that combining compact video representations based on Fisher Vectors with HMM-based modeling yields very significant gains in accuracy and when properly trained with sufficient training samples, structured temporal models outperform unstructured bag-of-word types of models by a large margin on the tested performance metric.
The Language of Actions: Recovering the Syntax and Semantics of Goal-Directed Human Activities
This paper describes a framework for modeling human activities as temporally structured processes.
