Search Results for author:
Found 51 papers, 33 papers with code
T-VSL: Text-Guided Visual Sound Source Localization in Mixtures
Visual sound source localization poses a significant challenge in identifying the semantic region of each sounding source within a video.
Robust Active Speaker Detection in Noisy Environments
Experiments demonstrate that non-speech audio noises significantly impact ASD models, and our proposed approach improves ASD performance in noisy environments.
Text-to-Audio Generation Synchronized with Videos
Extensive evaluations on the AudioCaps and T2AV-Bench demonstrate that our T2AV sets a new standard for video-aligned TTA generation in ensuring visual alignment and temporal consistency.
OSCaR: Object State Captioning and State Change Representation
To address these challenges, in this paper, we introduce the Object State Captioning and State Change Representation (OSCaR) dataset and benchmark.
Efficiently Leveraging Linguistic Priors for Scene Text Spotting
This paper proposes a method that leverages linguistic knowledge from a large text corpus to replace the traditional one-hot encoding used in auto-regressive scene text spotting and recognition models.
DREAM-Talk: Diffusion-based Realistic Emotional Audio-driven Method for Single Image Talking Face Generation
The generation of emotional talking faces from a single portrait image remains a significant challenge.
LAVSS: Location-Guided Audio-Visual Spatial Audio Separation
LAVSS is inspired by the correlation between spatial audio and visual location.
Separating Invisible Sounds Toward Universal Audiovisual Scene-Aware Sound Separation
The audio-visual sound separation field assumes visible sources in videos, but this excludes invisible sounds beyond the camera's view.
Neural Acoustic Context Field: Rendering Realistic Room Impulse Response With Neural Fields
Room impulse response (RIR), which measures the sound propagation within an environment, is critical for synthesizing high-fidelity audio for a given environment.
CMRxRecon: An open cardiac MRI dataset for the competition of accelerated image reconstruction
However, a limitation of CMR is its slow imaging speed, which causes patient discomfort and introduces artifacts in the images.
Class-Incremental Grouping Network for Continual Audio-Visual Learning
Our CIGN leverages learnable audio-visual class tokens and audio-visual grouping to continually aggregate class-aware features.
SignDiff: Learning Diffusion Models for American Sign Language Production
The field of Sign Language Production (SLP) lacked a large-scale, pre-trained model based on deep learning for continuous American Sign Language (ASL) production in the past decade.
DiffI2I: Efficient Diffusion Model for Image-to-Image Translation
Compared to traditional DMs, the compact IPR enables DiffI2I to obtain more accurate outcomes and employ a lighter denoising network and fewer iterations.
Audio-Visual Class-Incremental Learning
We demonstrate that joint audio-visual modeling can improve class-incremental learning, but current methods fail to preserve semantic similarity between audio and visual features as incremental step grows.
DAVIS: High-Quality Audio-Visual Separation with Generative Diffusion Models
We propose DAVIS, a Diffusion model-based Audio-VIusal Separation framework that solves the audio-visual sound source separation task through a generative manner.
Dual Arbitrary Scale Super-Resolution for Multi-Contrast MRI
Limited by imaging systems, the reconstruction of Magnetic Resonance Imaging (MRI) images from partial measurement is essential to medical imaging research.
Unveiling Cross Modality Bias in Visual Question Answering: A Causal View with Possible Worlds VQA
In this paper, we first model a confounding effect that causes language and vision bias simultaneously, then propose a counterfactual inference to remove the influence of this effect.
EgoVSR: Towards High-Quality Egocentric Video Super-Resolution
We explicitly tackle motion blurs in egocentric videos using a Dual Branch Deblur Network (DB$^2$Net) in the VSR framework.
DiffAVA: Personalized Text-to-Audio Generation with Visual Alignment
In this work, we propose a novel and personalized text-to-sound generation approach with visual alignment based on latent diffusion models, namely DiffAVA, that can simply fine-tune lightweight visual-text alignment modules with frozen modality-specific encoders to update visual-aligned text embeddings as the condition.
AV-SAM: Segment Anything Model Meets Audio-Visual Localization and Segmentation
In this work, we propose a simple yet effective audio-visual localization and segmentation framework based on the Segment Anything Model, namely AV-SAM, that can generate sounding object masks corresponding to the audio.
Audio-Visual Grouping Network for Sound Localization from Mixtures
Sound source localization is a typical and challenging task that predicts the location of sound sources in a video.
Egocentric Audio-Visual Object Localization
In this paper, we explore the challenging egocentric audio-visual object localization task and observe that 1) egomotion commonly exists in first-person recordings, even within a short duration; 2) The out-of-view sound components can be created while wearers shift their attention.
DiffIR: Efficient Diffusion Model for Image Restoration
Diffusion model (DM) has achieved SOTA performance by modeling the image synthesis process into a sequential application of a denoising network.
Knowledge Distillation based Degradation Estimation for Blind Super-Resolution
It consists of a knowledge distillation based implicit degradation estimator network (KD-IDE) and an efficient SR network.
Basic Binary Convolution Unit for Binarized Image Restoration Network
In this study, we reconsider components in binary convolution, such as residual connection, BatchNorm, activation function, and structure, for IR tasks.
Learning in Audio-visual Context: A Review, Analysis, and New Perspective
A comprehensive survey that can systematically organize and analyze studies of the audio-visual field is expected.
Meta-Learning based Degradation Representation for Blind Super-Resolution
The most of CNN based super-resolution (SR) methods assume that the degradation is known (\eg, bicubic).
Structured Sparsity Learning for Efficient Video Super-Resolution
In SSL, we design pruning schemes for several key components in VSR models, including residual blocks, recurrent networks, and upsampling networks.
Transformer-empowered Multi-scale Contextual Matching and Aggregation for Multi-contrast MRI Super-resolution
However, existing methods still have two shortcomings: (1) they neglect that the multi-contrast features at different scales contain different anatomical details and hence lack effective mechanisms to match and fuse these features for better reconstruction; and (2) they are still deficient in capturing long-range dependencies, which are essential for the regions with complicated anatomical structures.
Learning to Answer Questions in Dynamic Audio-Visual Scenarios
In this paper, we focus on the Audio-Visual Question Answering (AVQA) task, which aims to answer questions regarding different visual objects, sounds, and their associations in videos.
Ranked #5 on
Audio-visual Question Answering
on MUSIC-AVQA
Learning Spatio-Temporal Downsampling for Effective Video Upscaling
Consequently, the inverse task of upscaling a low-resolution, low frame-rate video in space and time becomes a challenging ill-posed problem due to information loss and aliasing artifacts.
STDAN: Deformable Attention Network for Space-Time Video Super-Resolution
Second, we put forward a spatial-temporal deformable feature aggregation (STDFA) module, in which spatial and temporal contexts in dynamic video frames are adaptively captured and aggregated to enhance SR reconstruction.
Coarse-to-Fine Embedded PatchMatch and Multi-Scale Dynamic Aggregation for Reference-based Super-Resolution
To improve matching efficiency, we design a novel Embedded PatchMacth scheme with random samples propagation, which involves end-to-end training with asymptotic linear computational cost to the input size.
Efficient Non-Local Contrastive Attention for Image Super-Resolution
To demonstrate the effectiveness of ENLCA, we build an architecture called Efficient Non-Local Contrastive Network (ENLCN) by adding a few of our modules in a simple backbone.
Space-Time Memory Network for Sounding Object Localization in Videos
Leveraging temporal synchronization and association within sight and sound is an essential step towards robust localization of sounding objects.
Zooming SlowMo: An Efficient One-Stage Framework for Space-Time Video Super-Resolution
A na\"ive method is to decompose it into two sub-tasks: video frame interpolation (VFI) and video super-resolution (VSR).
Space-time Video Super-resolution
Video Frame Interpolation
+1
Can audio-visual integration strengthen robustness under multimodal attacks?
In this paper, we propose to make a systematic study on machines multisensory perception under attacks.
Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separation
There are rich synchronized audio and visual events in our daily life.
Video Matting via Consistency-Regularized Graph Neural Networks
In this paper, we propose to enhance the temporal coherence by Consistency-Regularized Graph Neural Networks (CRGNN) with the aid of a synthesized video matting dataset.
Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing
In this paper, we introduce a new problem, named audio-visual video parsing, which aims to parse a video into temporal event segments and label them as either audible, visible, or both.
TDAN: Temporally-Deformable Alignment Network for Video Super-Resolution
Video super-resolution (VSR) aims to restore a photo-realistic high-resolution (HR) video frame from both its corresponding low-resolution (LR) frame (reference frame) and multiple neighboring frames (supporting frames).
Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution
Rather than synthesizing missing LR video frames as VFI networks do, we firstly temporally interpolate LR frame features in missing LR video frames capturing local temporal contexts by the proposed feature temporal interpolation network.
Ranked #4 on
Video Frame Interpolation
on Vid4 - 4x upscaling
Space-time Video Super-resolution
Video Frame Interpolation
+1
LCSCNet: Linear Compressing Based Skip-Connecting Network for Image Super-Resolution
In this paper, we develop a concise but efficient network architecture called linear compressing based skip-connecting network (LCSCNet) for image super-resolution.
Ranked #14 on
Image Super-Resolution
on Set14 - 3x upscaling
CFSNet: Toward a Controllable Feature Space for Image Restoration
Deep learning methods have witnessed the great progress in image restoration with specific metrics (e. g., PSNR, SSIM).
Residual Dense Network for Image Restoration
We fully exploit the hierarchical features from all the convolutional layers.
TDAN: Temporally Deformable Alignment Network for Video Super-Resolution
Video super-resolution (VSR) aims to restore a photo-realistic high-resolution (HR) video frame from both its corresponding low-resolution (LR) frame (reference frame) and multiple neighboring frames (supporting frames).
An Attempt towards Interpretable Audio-Visual Video Captioning
To achieve this, we propose a multimodal convolutional neural network-based audio-visual video captioning framework and introduce a modality-aware module for exploring modality selection during sentence generation.
Deep Learning for Single Image Super-Resolution: A Brief Review
Single image super-resolution (SISR) is a notoriously challenging ill-posed problem, which aims to obtain a high-resolution (HR) output from one of its low-resolution (LR) versions.
Audio-Visual Event Localization in Unconstrained Videos
In this paper, we introduce a novel problem of audio-visual event localization in unconstrained videos.
Residual Dense Network for Image Super-Resolution
In this paper, we propose a novel residual dense network (RDN) to address this problem in image SR. We fully exploit the hierarchical features from all the convolutional layers.
Ranked #3 on
Color Image Denoising
on CBSD68 sigma50
