Search Results for author:
Found 125 papers, 44 papers with code
Learning with Noisy Foundation Models
Foundation models are usually pre-trained on large-scale datasets and then adapted to downstream tasks through tuning.
$\text{R}^2$-Bench: Benchmarking the Robustness of Referring Perception Models under Perturbations
Referring perception, which aims at grounding visual objects with multimodal referring guidance, is essential for bridging the gap between humans, who provide instructions, and the environment where intelligent systems perceive.
AutoPRM: Automating Procedural Supervision for Multi-Step Reasoning via Controllable Question Decomposition
Recent advancements in large language models (LLMs) have shown promise in multi-step reasoning tasks, yet their reliance on extensive manual labeling to provide procedural feedback remains a significant impediment.
Evaluating and Improving Continual Learning in Spoken Language Understanding
In this work, we propose an evaluation methodology that provides a unified evaluation on stability, plasticity, and generalizability in continual learning.
Customizable Perturbation Synthesis for Robust SLAM Benchmarking
To this end, we propose a novel, customizable pipeline for noisy data synthesis, aimed at assessing the resilience of multi-modal SLAM models against various perturbations.
On Catastrophic Inheritance of Large Foundation Models
Large foundation models (LFMs) are claiming incredible performances.
A General Framework for Learning from Weak Supervision
Weakly supervised learning generally faces challenges in applicability to various scenarios with diverse weak supervision and in scalability due to the complexity of existing algorithms, thereby hindering the practical deployment.
PAM: Prompting Audio-Language Models for Audio Quality Assessment
Here, we exploit this capability and introduce PAM, a no-reference metric for assessing audio quality for different audio processing tasks.
AugSumm: towards generalizable speech summarization using synthetic labels from large language model
We tackle this challenge by proposing AugSumm, a method to leverage large language models (LLMs) as a proxy for human annotators to generate augmented summaries for training and evaluation.
FALCON: Fairness Learning via Contrastive Attention Approach to Continual Semantic Scene Understanding in Open World
In particular, we first introduce a new Fairness Contrastive Clustering loss to address the problems of catastrophic forgetting and fairness.
Token Prediction as Implicit Classification to Identify LLM-Generated Text
This paper introduces a novel approach for identifying the possible large language models (LLMs) involved in text generation.
Pairwise Similarity Learning is SimPLE
In this paper, we focus on a general yet important learning problem, pairwise similarity learning (PSL).
Psychoacoustic Challenges Of Speech Enhancement On VoIP Platforms
Within the ambit of VoIP (Voice over Internet Protocol) telecommunications, the complexities introduced by acoustic transformations merit rigorous analysis.
Privacy-oriented manipulation of speaker representations
Speaker embeddings are ubiquitous, with applications ranging from speaker recognition and diarization to speech synthesis and voice anonymisation.
Continual Contrastive Spoken Language Understanding
In this paper, we investigate the problem of learning sequence-to-sequence models for spoken language understanding in a class-incremental learning (CIL) setting and we propose COCONUT, a CIL method that relies on the combination of experience replay and contrastive learning.
Prompting Audios Using Acoustic Properties For Emotion Representation
In this work, we address the challenge of automatically generating these prompts and training a model to better learn emotion representations from audio and prompt pairs.
LoFT: Local Proxy Fine-tuning For Improving Transferability Of Adversarial Attacks Against Large Language Model
We hypothesize that for attacks to be transferrable, it is sufficient if the proxy can approximate the target model in the neighborhood of the harmful query.
uSee: Unified Speech Enhancement and Editing with Conditional Diffusion Models
Speech enhancement aims to improve the quality of speech signals in terms of quality and intelligibility, and speech editing refers to the process of editing the speech according to specific user needs.
Completing Visual Objects via Bridging Generation and Segmentation
This paper presents a novel approach to object completion, with the primary goal of reconstructing a complete object from its partially visible components.
Evaluating Speech Synthesis by Training Recognizers on Synthetic Speech
In this paper, we propose an evaluation technique involving the training of an ASR model on synthetic speech and assessing its performance on real speech.
Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks
This paper aims to understand the nature of noise in pre-training datasets and to mitigate its impact on downstream tasks.
Towards Robust Audiovisual Segmentation in Complex Environments with Quantization-based Semantic Decomposition
We propose a semantic decomposition method based on product quantization, where the multi-source semantics can be decomposed and represented by several disentangled and noise-suppressed single-source semantics.
Importance of negative sampling in weak label learning
Weak-label learning is a challenging task that requires learning from data "bags" containing positive and negative instances, but only the bag labels are known.
Training Audio Captioning Models without Audio
During inference, the text encoder is replaced with the pretrained CLAP audio encoder.
Fixed Inter-Neuron Covariability Induces Adversarial Robustness
The vulnerability to adversarial perturbations is a major flaw of Deep Neural Networks (DNNs) that raises question about their reliability when in real-world scenarios.
Rethinking Voice-Face Correlation: A Geometry View
Previous works on voice-face matching and voice-guided face synthesis demonstrate strong correlations between voice and face, but mainly rely on coarse semantic cues such as gender, age, and emotion.
The Hidden Dance of Phonemes and Visage: Unveiling the Enigmatic Link between Phonemes and Facial Features
This work unveils the enigmatic link between phonemes and facial features.
BASS: Block-wise Adaptation for Speech Summarization
End-to-end speech summarization has been shown to improve performance over cascade baselines.
UTOPIA: Unconstrained Tracking Objects without Preliminary Examination via Cross-Domain Adaptation
Then, a new cross-domain MOT adaptation from existing datasets is proposed without any pre-defined human knowledge in understanding and modeling objects.
PaintSeg: Training-free Segmentation via Painting
The paper introduces PaintSeg, a new unsupervised method for segmenting objects without any training.
Imprecise Label Learning: A Unified Framework for Learning with Various Imprecise Label Configurations
In this paper, we introduce imprecise label learning (ILL), a framework for the unification of learning with various imprecise label configurations.
Ranked #1 on
Learning with noisy labels
on mini WebVision 1.0
GPT-Sentinel: Distinguishing Human and ChatGPT Generated Content
This paper presents a novel approach for detecting ChatGPT-generated vs. human-written text using language models.
FREDOM: Fairness Domain Adaptation Approach to Semantic Scene Understanding
Although Domain Adaptation in Semantic Scene Segmentation has shown impressive improvement in recent years, the fairness concerns in the domain adaptation have yet to be well defined and addressed.
Ranked #5 on
Domain Adaptation
on SYNTHIA-to-Cityscapes
Improving Perceptual Quality, Intelligibility, and Acoustics on VoIP Platforms
In this paper, we present a method for fine-tuning models trained on the Deep Noise Suppression (DNS) 2020 Challenge to improve their performance on Voice over Internet Protocol (VoIP) applications.
Approach to Learning Generalized Audio Representation Through Batch Embedding Covariance Regularization and Constant-Q Transforms
Preliminary results show (1) the proposed BECR can incur a more dispersed embedding on the test set, (2) BECR improves the PaSST model without extra computation complexity, and (3) STFT preprocessing outperforms CQT in all tasks we tested.
Synergy between human and machine approaches to sound/scene recognition and processing: An overview of ICASSP special session
Machine Listening, as usually formalized, attempts to perform a task that is, from our perspective, fundamentally human-performable, and performed by humans.
PAAPLoss: A Phonetic-Aligned Acoustic Parameter Loss for Speech Enhancement
We can add this criterion as an auxiliary loss to any model that produces speech, to optimize speech outputs to match the values of clean speech in these features.
TAPLoss: A Temporal Acoustic Parameter Loss for Speech Enhancement
We propose an objective for perceptual quality based on temporal acoustic parameters.
SoftMatch: Addressing the Quantity-Quality Trade-off in Semi-supervised Learning
The critical challenge of Semi-Supervised Learning (SSL) is how to effectively leverage the limited labeled data and massive unlabeled data to improve the model's generalization performance.
Understanding Political Polarisation using Language Models: A dataset and method
To understand the polarization we begin by showing results from some classical language models in Word2Vec and Doc2Vec.
Robust Referring Video Object Segmentation with Cyclic Structural Consensus
Our model achieves state-of-the-art performance on R-VOS benchmarks, Ref-DAVIS17 and Ref-Youtube-VOS, and also our RRYTVOS dataset.
VLTinT: Visual-Linguistic Transformer-in-Transformer for Coherent Video Paragraph Captioning
Video paragraph captioning aims to generate a multi-sentence description of an untrimmed video with several temporal event locations in coherent storytelling.
Ranked #2 on
Video Captioning
on ActivityNet Captions
Panoramic Video Salient Object Detection with Ambisonic Audio Guidance
In this paper, we aim to tackle the video salient object detection problem for panoramic videos, with their corresponding ambisonic audios.
An Embarrassingly Simple Baseline for Imbalanced Semi-Supervised Learning
While standard SSL assumes uniform data distribution, we consider a more realistic and challenging setting called imbalanced SSL, where imbalanced class distributions occur in both labeled and unlabeled data.
Describing emotions with acoustic property prompts for speech emotion recognition
We investigate how the model can learn to associate the audio with the descriptions, resulting in performance improvement of Speech Emotion Recognition and Speech Audio Retrieval.
Unifying the Discrete and Continuous Emotion labels for Speech Emotion Recognition
Accordingly, models that have been proposed for emotion detection use one or the other of these label types.
XNOR-FORMER: Learning Accurate Approximations in Long Speech Transformers
Transformers are among the state of the art for many tasks in speech, vision, and natural language processing, among others.
Privacy-preserving Automatic Speaker Diarization
Automatic Speaker Diarization (ASD) is an enabling technology with numerous applications, which deals with recordings of multiple speakers, raising special concerns in terms of privacy.
There is more than one kind of robustness: Fooling Whisper with adversarial examples
Whisper is a recent Automatic Speech Recognition (ASR) model displaying impressive robustness to both out-of-distribution inputs and random noise.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
AOE-Net: Entities Interactions Modeling with Adaptive Attention Mechanism for Temporal Action Proposals Generation
PMR module represents each video snippet by a visual-linguistic feature, in which main actors and surrounding environment are represented by visual information, whereas relevant objects are depicted by linguistic features through an image-text model.
Watch What You Pretrain For: Targeted, Transferable Adversarial Examples on Self-Supervised Speech Recognition models
To exploit ASR models in real-world, black-box settings, an adversary can leverage the transferability property, i. e. that an adversarial sample produced for a proxy ASR can also fool a different remote ASR.
USB: A Unified Semi-supervised Learning Benchmark for Classification
We further provide the pre-trained versions of the state-of-the-art neural models for CV tasks to make the cost affordable for further tuning.
Online Video Instance Segmentation via Robust Context Fusion
We propose a robust context fusion network to tackle VIS in an online fashion, which predicts instance segmentation frame-by-frame with a few preceding frames.
How many perturbations break this model? Evaluating robustness beyond adversarial accuracy
Finally, with sparsity we can measure increases in robustness that do not affect accuracy: we show for example that data augmentation can by itself increase adversarial robustness, without using adversarial training.
Towards Robust Referring Video Object Segmentation with Cyclic Relational Consensus
Referring Video Object Segmentation (R-VOS) is a challenging task that aims to segment an object in a video based on a linguistic expression.
Ranked #11 on
Referring Video Object Segmentation
on Refer-YouTube-VOS
Referring Expression Segmentation
Referring Video Object Segmentation
+2
Improving Speech Enhancement through Fine-Grained Speech Characteristics
We first identify key acoustic parameters that have been found to correlate well with voice quality (e. g. jitter, shimmer, and spectral flux) and then propose objective functions which are aimed at reducing the difference between clean speech and enhanced speech with respect to these features.
Self-supervision and Learnable STRFs for Age, Emotion, and Country Prediction
This work presents a multitask approach to the simultaneous estimation of age, country of origin, and emotion given vocal burst audio for the 2022 ICML Expressive Vocalizations Challenge ExVo-MultiTask track.
Towards End-to-End Private Automatic Speaker Recognition
This poses two important issues: first, knowledge of the speaker embedding extraction model may create security and robustness liabilities for the authentication system, as this knowledge might help attackers in crafting adversarial examples able to mislead the system; second, from the point of view of a service provider the speaker embedding extraction model is arguably one of the most valuable components in the system and, as such, disclosing it would be highly undesirable.
Bear the Query in Mind: Visual Grounding with Query-conditioned Convolution
With our proposed QCM, the downstream fusion module receives visual features that are more discriminative and focused on the desired object described in the expression, leading to more accurate predictions.
FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning
Semi-supervised Learning (SSL) has witnessed great success owing to the impressive performances brought by various methods based on pseudo labeling and consistency regularization.
On the pragmatism of using binary classifiers over data intensive neural network classifiers for detection of COVID-19 from voice
Lately, there has been a global effort by multiple research groups to detect COVID-19 from voice.
Recent improvements of ASR models in the face of adversarial attacks
Like many other tasks involving neural networks, Speech Recognition models are vulnerable to adversarial attacks.
Point3D: tracking actions as moving points with 3D CNNs
Our Point3D consists of a Point Head for action localization and a 3D Head for action classification.
HEAR: Holistic Evaluation of Audio Representations
The aim of the HEAR benchmark is to develop a general-purpose audio representation that provides a strong basis for learning in a wide variety of tasks and scenarios.
Ontological Learning from Weak Labels
We find that the baseline Siamese does not perform better by incorporating ontology information in the weak and multi-label scenario, but that the GCN does capture the ontology knowledge better for weak, multi-labeled data.
Sequential Randomized Smoothing for Adversarially Robust Speech Recognition
We apply adaptive versions of state-of-the-art attacks, such as the Imperceptible ASR attack, to our model, and show that our strongest defense is robust to all attacks that use inaudible noise, and can only be broken with very high distortion.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Self-Supervised 3D Face Reconstruction via Conditional Estimation
We present a conditional estimation (CEST) framework to learn 3D facial parameters from 2D single-view images by self-supervised training from videos.
Ranked #15 on
3D Face Reconstruction
on REALY
SphereFace Revived: Unifying Hyperspherical Face Recognition
As one of the earliest works in hyperspherical face recognition, SphereFace explicitly proposed to learn face embeddings with large inter-class angular margin.
The Right to Talk: An Audio-Visual Transformer Approach
Therefore, this work introduces a new Audio-Visual Transformer approach to the problem of localization and highlighting the main speaker in both audio and visual channels of a multi-speaker conversation video in the wild.
SphereFace2: Binary Classification is All You Need for Deep Face Recognition
In this paper, we start by identifying the discrepancy between training and evaluation in the existing multi-class classification framework and then discuss the potential limitations caused by the "competitive" nature of softmax normalization.
Controlled AutoEncoders to Generate Faces from Voices
With this in perspective, we propose a framework to morph a target face in response to a given voice in a way that facial features are implicitly guided by learned voice-face correlation in this paper.
Improving weakly supervised sound event detection with self-supervised auxiliary tasks
To that extent, we propose a shared encoder architecture with sound event detection as a primary task and an additional secondary decoder for a self-supervised auxiliary task.
Training image classifiers using Semi-Weak Label Data
Thus, this paper introduces a novel semi-weak label learning paradigm as a middle ground to mitigate the problem.
Constant Random Perturbations Provide Adversarial Robustness with Minimal Effect on Accuracy
This paper proposes an attack-independent (non-adversarial training) technique for improving adversarial robustness of neural network models, with minimal loss of standard accuracy.
Contrast and Order Representations for Video Self-Supervised Learning
To address this, we present a contrast-and-order representation (CORP) framework for learning self-supervised video representations that can automatically capture both the appearance information within each frame and temporal information across different frames.
Is normalization indispensable for training deep neural network?
In this paper, we study what would happen when normalization layers are removed from the network, and show how to train deep neural networks without normalization layers and without performance degradation.
FoolHD: Fooling speaker identification by Highly imperceptible adversarial Disturbances
Speaker identification models are vulnerable to carefully designed adversarial perturbations of their input signals that induce misclassification.
Masked Proxy Loss For Text-Independent Speaker Verification
We further propose Multinomial Masked Proxy (MMP) loss to leverage the hardness of speaker pairs.
Multi-Task Learning for Interpretable Weakly Labelled Sound Event Detection
Weakly Labelled learning has garnered lot of attention in recent years due to its potential to scale Sound Event Detection (SED) and is formulated as Multiple Instance Learning (MIL) problem.
Exploiting Non-Linear Redundancy for Neural Model Compression
Deploying deep learning models, comprising of non-linear combination of millions, even billions, of parameters is challenging given the memory, power and compute constraints of the real world.
Automatic In-the-wild Dataset Annotation with Deep Generalized Multiple Instance Learning
The automation of the diagnosis and monitoring of speech affecting diseases in real life situations, such as Depression or Parkinson{'}s disease, depends on the existence of rich and large datasets that resemble real life conditions, such as those collected from in-the-wild multimedia repositories like YouTube.
Face Reconstruction from Voice using Generative Adversarial Networks
The network learns to generate faces from voices by matching the identities of generated faces to those of the speakers, on a training set.
The phonetic bases of vocal expressed emotion: natural versus acted
Our tests show significant differences in the manner and choice of phonemes in acted and natural speech, concluding moderate to low validity and value in using acted speech databases for emotion classification tasks.
Detecting gender differences in perception of emotion in crowdsourced data
While we limit ourselves to a single modality (i. e. speech), our framework is applicable to studies of emotion perception from all such loosely annotated data in general.
Non-Determinism in Neural Networks for Adversarial Robustness
Recent breakthroughs in the field of deep learning have led to advancements in a broad spectrum of tasks in computer vision, audio processing, natural language processing and other areas.
Reconstructing faces from voices
Voice profiling aims at inferring various human parameters from their speech, e. g. gender, age, etc.
Nonlinear Semi-Parametric Models for Survival Analysis
Semi-parametric survival analysis methods like the Cox Proportional Hazards (CPH) regression (Cox, 1972) are a popular approach for survival analysis.
Hierarchical Routing Mixture of Experts
Addressing these problems, we propose a binary tree-structured hierarchical routing mixture of experts (HRME) model that has classifiers as non-leaf node experts and simple regression models as leaf node experts.
Hide and Speak: Towards Deep Neural Networks for Speech Steganography
Steganography is the science of hiding a secret message within an ordinary public message, which is referred to as Carrier.
Learning Sound Events From Webly Labeled Data
In the last couple of years, weakly labeled learning for sound events has turned out to be an exciting approach for audio event detection.
Higher-order Network for Action Recognition
Capturing spatiotemporal dynamics is an essential topic in video recognition.
Neural Regression Trees
Regression-via-Classification (RvC) is the process of converting a regression problem to a classification one.
Neural Regression Tree
Regression-via-Classification (RvC) is the process of converting a regression problem to a classification one.
Disjoint Mapping Network for Cross-modal Matching of Voices and Faces
We propose a novel framework, called Disjoint Mapping Network (DIMNet), for cross-modal biometric matching, in particular of voices and faces.
Optimal Strategies for Matching and Retrieval Problems by Comparing Covariates
In many retrieval problems, where we must retrieve one or more entries from a gallery in response to a probe, it is common practice to learn to do by directly comparing the probe and gallery entries to one another.
A Closer Look at Weak Label Learning for Audio Events
In this work, we first describe a CNN based approach for weakly supervised training of audio events.
Voice Impersonation using Generative Adversarial Networks
In voice impersonation, the resultant voice must convincingly convey the impression of having been naturally produced by the target speaker, mimicking not only the pitch and other perceivable signal qualities, but also the style of the target speaker.
Sound Audio and Speech Processing
Framework for evaluation of sound event detection in web videos
The framework crawls videos using search queries corresponding to 78 sound event labels drawn from three datasets.
Be Careful What You Backpropagate: A Case For Linear Output Activations & Gradient Boosting
To this end, we demonstrate faster convergence and better performance on diverse classification tasks: image classification using CIFAR-10 and ImageNet, and semantic segmentation using PASCAL VOC 2012.
Deep CNN Framework for Audio Event Recognition using Weakly Labeled Web Data
We propose that learning algorithms that can exploit weak labels offer an effective method to learn from web data.
SphereFace: Deep Hypersphere Embedding for Face Recognition
This paper addresses deep face recognition (FR) problem under open-set protocol, where ideal face features are expected to have smaller maximal intra-class distance than minimal inter-class distance under a suitably chosen metric space.
Ranked #1 on
Face Verification
on CK+
On the Origin of Deep Learning
This paper is a review of the evolutionary history of deep learning models.
The Incredible Shrinking Neural Network: New Perspectives on Learning Representations Through The Lens of Pruning
How much can pruning algorithms teach us about the fundamentals of learning representations in neural networks?
Audio Event and Scene Recognition: A Unified Approach using Strongly and Weakly Labeled Data
In this paper we propose a novel learning framework called Supervised and Weakly Supervised Learning where the goal is to learn simultaneously from weakly and strongly labeled data.
Discovering Sound Concepts and Acoustic Relations In Text
In this paper we describe approaches for discovering acoustic concepts and relations in text.
An Approach for Self-Training Audio Event Detectors Using Web Data
The audio event detectors are trained on the labeled audio and ran on the unlabeled audio downloaded from YouTube.
Features and Kernels for Audio Event Recognition
One of the most important problems in audio event detection research is absence of benchmark results for comparison with any proposed method.
Sound Multimedia
AudioPairBank: Towards A Large-Scale Tag-Pair-Based Audio Content Analysis
One contribution is the previously unavailable documentation of the challenges and implications of collecting audio recordings with these type of labels.
Classifier Risk Estimation under Limited Labeling Resources
In this paper we propose strategies for estimating performance of a classifier when labels cannot be obtained for the whole test set.
Weakly Supervised Scalable Audio Content Analysis
Audio Event Detection is an important task for content analysis of multimedia data.
Audio Event Detection using Weakly Labeled Data
This helps in obtaining a complete description of the recording and is notable since temporal information was never known in the first place in weakly labeled data.
Content-based Video Indexing and Retrieval Using Corr-LDA
Existing video indexing and retrieval methods on popular web-based multimedia sharing websites are based on user-provided sparse tagging.
Environmental Noise Embeddings for Robust Speech Recognition
We propose a novel deep neural network architecture for speech recognition that explicitly employs knowledge of the background environmental noise within a deep neural network acoustic model.
Handcrafted Local Features are Convolutional Neural Networks
We approach this problem by first showing that local handcrafted features and Convolutional Neural Networks (CNNs) share the same convolution-pooling network structure.
A Survey: Time Travel in Deep Learning Space: An Introduction to Deep Learning Models and How Deep Learning Models Evolved from the Initial Ideas
Further, we will also look into the development history of modelling time series data with neural networks.
Privacy-Preserving Multi-Document Summarization
State-of-the-art extractive multi-document summarization systems are usually designed without any concern about privacy issues, meaning that all documents are open to third parties.
Plagiarism Detection in Polyphonic Music using Monaural Signal Separation
Given the large number of new musical tracks released each year, automated approaches to plagiarism detection are essential to help us track potential violations of copyright.
Unsupervised Fusion Weight Learning in Multiple Classifier Systems
We also introduce a novel metric for ranking instances based on an index which depends upon the rank of weighted scores of test points among the weighted scores of training points.
Beyond Gaussian Pyramid: Multi-skip Feature Stacking for Action Recognition
MIFS compensates for information lost from using differential operators by recapturing information at coarse scales.
Unsupervised Structure Discovery for Semantic Analysis of Audio
Approaches to audio classification and retrieval tasks largely rely on detection-based discriminative models.
Learning Model-Based Sparsity via Projected Gradient Descent
As an example we elaborate on application of the main results to estimation in Generalized Linear Model.
Multiparty Differential Privacy via Aggregation of Locally Trained Classifiers
As increasing amounts of sensitive personal information finds its way into data repositories, it is important to develop analysis mechanisms that can derive aggregate information from these repositories without revealing information about individual data instances.
A Sparse Non-Parametric Approach for Single Channel Separation of Known Sounds
In this paper we present an algorithm for separating mixed sounds from a monophonic recording.
Sparse Overcomplete Latent Variable Decomposition of Counts Data
An important problem in many fields is the analysis of counts data to extract meaningful latent components.
