Search Results for author:
Found 17 papers, 11 papers with code
Audio-Synchronized Visual Animation
We hope our established benchmark can open new avenues for controllable visual generation.
Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modeling
Thus, to address the computational complexity, we propose an alternative procedure that factorizes the local representations before representing audio-visual interactions.
Why Is Prompt Tuning for Vision-Language Models Robust to Noisy Labels?
Vision-language models such as CLIP learn a generic text-image embedding from large-scale training data.
A Unified Audio-Visual Learning Framework for Localization, Separation, and Recognition
The ability to accurately recognize, localize and separate sound sources is fundamental to any audio-visual perception task.
Learning State-Aware Visual Representations from Audible Interactions
However, learning representations from videos can be challenging.
Ranked #1 on
Long Term Action Anticipation
on Ego4D
(ED@20 Noun metric)
A Closer Look at Weakly-Supervised Audio-Visual Source Localization
We also propose a new approach for visual sound source localization that addresses both these problems.
The Challenges of Continuous Self-Supervised Learning
As a result, SSL holds the promise to learn representations from data in-the-wild, i. e., without the need for finite and static datasets.
Localizing Visual Sounds the Easy Way
Unsupervised audio-visual source localization aims at localizing visible sound sources in a video without relying on ground-truth localization for training.
Robust Audio-Visual Instance Discrimination
Second, since self-supervised contrastive learning relies on random sampling of negative instances, instances that are semantically similar to the base instance can be used as faulty negatives.
Learning Representations from Audio-Visual Spatial Alignment
To learn from these spatial cues, we tasked a network to perform contrastive audio-visual spatial alignment of 360{\deg} video and spatial audio.
Deep Hashing with Hash-Consistent Large Margin Proxy Embeddings
The use of a fixed set of proxies (weights of the CNN classification layer) is proposed to eliminate this ambiguity, and a procedure to design proxy sets that are nearly optimal for both classification and hashing is introduced.
Solving Long-tailed Recognition with Deep Realistic Taxonomic Classifier
Motivated by this, a deep realistic taxonomic classifier (Deep-RTC) is proposed as a new solution to the long-tail problem, combining realism with hierarchical predictions.
Audio-Visual Instance Discrimination with Cross-Modal Agreement
Our method uses contrastive learning for cross-modal discrimination of video from audio and vice-versa.
Ranked #3 on
Self-Supervised Audio Classification
on ESC-50
NetTailor: Tuning the Architecture, Not Just the Weights
Under the standard paradigm of network fine-tuning, an entirely new CNN is learned per task, and the final network size is independent of task complexity.
Self-Supervised Generation of Spatial Audio for 360° Video
We introduce an approach to convert mono audio recorded by a 360° video camera into spatial audio, a representation of the distribution of sound over the full viewing sphere.
Self-Supervised Generation of Spatial Audio for 360 Video
Using our approach, we show that it is possible to infer the spatial location of sound sources based only on 360 video and a mono audio track.
Semantically Consistent Regularization for Zero-Shot Recognition
The role of semantics in zero-shot learning is considered.
