Search Results for author:
Found 64 papers, 32 papers with code
GenTranslate: Large Language Models are Generative Multilingual Speech and Machine Translators
Leveraging the rich linguistic knowledge and strong reasoning abilities of LLMs, our new paradigm can integrate the rich information in N-best candidates to generate a higher-quality translation result.
It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition
Recent studies have successfully shown that large language models (LLMs) can be successfully used for generative error correction (GER) on top of the automatic speech recognition (ASR) output.
Audio-Visual Speech Recognition
Automatic Speech Recognition
+3
Large Language Models are Efficient Learners of Noise-Robust Speech Recognition
To this end, we propose to extract a language-space noise embedding from the N-best list to represent the noise conditions of source speech, which can promote the denoising process in GER.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+6
Investigating Training Strategies and Model Robustness of Low-Rank Adaptation for Language Modeling in Speech Recognition
The use of low-rank adaptation (LoRA) with frozen pretrained language models (PLMs) has become increasing popular as a mainstream, resource-efficient modeling approach for memory-constrained hardware.
Paralinguistics-Enhanced Large Language Modeling of Spoken Dialogue
Specifically, our framework serializes tasks in the order of current paralinguistic attribute prediction, response paralinguistic attribute prediction, and response text generation with autoregressive conditioning.
Multimodal Attention Merging for Improved Speech Recognition and Audio Event Classification
In cases where some data/compute is available, we present Learnable-MAM, a data-driven approach to merging attention matrices, resulting in a further 2. 90% relative reduction in WER for ASR and 18. 42% relative reduction in AEC compared to fine-tuning.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Conditional Modeling Based Automatic Video Summarization
The aim of video summarization is to shorten videos automatically while retaining the key information necessary to convey the overall story.
Generative error correction for code-switching speech recognition using large language models
In this work, we propose to leverage large language models (LLMs) and lists of hypotheses generated by an ASR to address the CS problem.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition
We introduce a new cross-modal fusion technique designed for generative error correction in automatic speech recognition (ASR).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models
We make our results publicly accessible for reproducible pipelines with released pre-trained models, thus providing a new evaluation paradigm for ASR error correction with LLMs.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Prompting
We explore the ability of large language models (LLMs) to act as speech recognition post-processors that perform rescoring and error correction.
Ranked #3 on
Speech Recognition
on WSJ eval92
(using extra training data)
Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognition
We propose a neural language modeling system based on low-rank adaptation (LoRA) for speech recognition output rescoring.
Can Whisper perform speech-based in-context learning?
Language-level adaptation experiments using Chinese dialects showed that when applying SICL to isolated word ASR, consistent and considerable relative WER reductions can be achieved using Whisper models of any size on two dialects, which is on average 32. 3%.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Causal Video Summarizer for Video Exploration
Multi-modal video summarization has a video input and a text-based query input.
How to Estimate Model Transferability of Pre-Trained Speech Models?
In this work, we introduce a "score-based assessment" framework for estimating the transferability of pre-trained speech models (PSMs) for fine-tuning target tasks.
A Multi-dimensional Deep Structured State Space Approach to Speech Enhancement Using Small-footprint Models
We propose a multi-dimensional structured state space (S4) approach to speech enhancement.
A Neural State-Space Model Approach to Efficient Speech Separation
In this work, we introduce S4M, a new efficient speech separation framework based on neural state-space models (SSM).
Differentially Private Adapters for Parameter Efficient Acoustic Modeling
Evaluated on the open-access Multilingual Spoken Words (MLSW) dataset, our solution reduces the number of trainable parameters by 97. 5% using the RAs with only a 4% performance drop with respect to fine-tuning the cross-lingual speech classifier while preserving DP guarantees.
Parameter-Efficient Learning for Text-to-Speech Accent Adaptation
This paper presents a parameter-efficient learning (PEL) to develop a low-resource accent adaptation for text-to-speech (TTS).
A Parameter-Efficient Learning Approach to Arabic Dialect Identification with Pre-Trained General-Purpose Speech Model
In this work, we explore Parameter-Efficient-Learning (PEL) techniques to repurpose a General-Purpose-Speech (GSM) model for Arabic dialect identification (ADI).
Pre-training Tensor-Train Networks Facilitates Machine Learning with Variational Quantum Circuits
Variational quantum circuit (VQC) is a promising approach for implementing quantum neural networks on noisy intermediate-scale quantum (NISQ) devices.
Causalainer: Causal Explainer for Automatic Video Summarization
In this work, a Causal Explainer, dubbed Causalainer, is proposed to address this issue.
From English to More Languages: Parameter-Efficient Model Reprogramming for Cross-Lingual Speech Recognition
In this work, we propose a new parameter-efficient learning framework based on neural model reprogramming for cross-lingual speech recognition, which can \textbf{re-purpose} well-trained English automatic speech recognition (ASR) models to recognize the other languages.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Certified Robustness of Quantum Classifiers against Adversarial Examples through Quantum Noise
Recently, quantum classifiers have been found to be vulnerable to adversarial attacks, in which quantum classifiers are deceived by imperceptible noises, leading to misclassification.
A Quantum Kernel Learning Approach to Acoustic Modeling for Spoken Command Recognition
We propose a quantum kernel learning (QKL) framework to address the inherent data sparsity issues often encountered in training large-scare acoustic models in low-resource scenarios.
Low-Resource Music Genre Classification with Cross-Modal Neural Model Reprogramming
In this work, we introduce a novel method for leveraging pre-trained models for low-resource (music) classification based on the concept of Neural Model Reprogramming (NMR).
Inference and Denoise: Causal Inference-based Neural Speech Enhancement
This study addresses the speech enhancement (SE) task within the causal inference paradigm by modeling the noise presence as an intervention.
An Ensemble Teacher-Student Learning Approach with Poisson Sub-sampling to Differential Privacy Preserving Speech Recognition
We propose an ensemble learning framework with Poisson sub-sampling to effectively train a collection of teacher models to issue some differential privacy (DP) guarantee for training data.
An Experimental Study on Private Aggregation of Teacher Ensemble Learning for End-to-End Speech Recognition
We evaluate three end-to-end deep models, including LAS, hybrid CTC/attention, and RNN transducer, on the open-source LibriSpeech and TIMIT corpora.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Theoretical Error Performance Analysis for Variational Quantum Circuit Based Functional Regression
In this work, we first put forth an end-to-end quantum neural network, TTN-VQC, which consists of a quantum tensor network based on a tensor-train network (TTN) for dimensionality reduction and a VQC for functional regression.
Treatment Learning Causal Transformer for Noisy Image Classification
In this work, we incorporate this binary information of "existence of noise" as treatment into image classification tasks to improve prediction accuracy by jointly estimating their treatment effects.
Exploiting Low-Rank Tensor-Train Deep Neural Networks Based on Riemannian Gradient Descent With Illustrations of Speech Processing
This work focuses on designing low complexity hybrid tensor networks by considering trade-offs between the model complexity and practical performance.
A study on joint modeling and data augmentation of multi-modalities for audio-visual scene classification
In this paper, we propose two techniques, namely joint modeling and data augmentation, to improve system performances for audio-visual scene classification (AVSC).
Mitigating Closed-model Adversarial Examples with Bayesian Neural Modeling for Enhanced End-to-End Speech Recognition
In this work, we aim to enhance the system robustness of end-to-end automatic speech recognition (ASR) against adversarially-noisy speech examples.
A Study of Designing Compact Audio-Visual Wake Word Spotting System Based on Iterative Fine-Tuning in Neural Network Pruning
Audio-only-based wake word spotting (WWS) is challenging under noisy conditions due to environmental interference in signal transmission.
When BERT Meets Quantum Temporal Convolution Learning for Text Classification in Heterogeneous Computing
Our experiments on intent classification show that our proposed BERT-QTC model attains competitive experimental results in the Snips and ATIS spoken language datasets.
Pessimistic Model Selection for Offline Deep Reinforcement Learning
Deep Reinforcement Learning (DRL) has demonstrated great potentials in solving sequential decision making problems in many applications.
A Variational Bayesian Approach to Learning Latent Variables for Acoustic Knowledge Transfer
We propose a variational Bayesian (VB) approach to learning distributions of latent variables in deep neural network (DNN) models for cross-domain knowledge transfer, to address acoustic mismatches between training and testing conditions.
Neural Model Reprogramming with Similarity Based Mapping for Low-Resource Spoken Command Recognition
In this study, we propose a novel adversarial reprogramming (AR) approach for low-resource spoken command recognition (SCR), and build an AR-SCR system.
QTN-VQC: An End-to-End Learning framework for Quantum Neural Networks
The advent of noisy intermediate-scale quantum (NISQ) computers raises a crucial challenge to design quantum neural networks for fully quantum learning tasks.
A Lottery Ticket Hypothesis Framework for Low-Complexity Device-Robust Neural Acoustic Scene Classification
We propose a novel neural model compression strategy combining data augmentation, knowledge transfer, pruning, and quantization for device-robust acoustic scene classification (ASC).
Voice2Series: Reprogramming Acoustic Models for Time Series Classification
Learning to classify time series with limited data is a practical yet challenging problem.
Longer Version for "Deep Context-Encoding Network for Retinal Image Captioning"
Automatically generating medical reports for retinal images is one of the promising ways to help ophthalmologists reduce their workload and improve work efficiency.
PATE-AAE: Incorporating Adversarial Autoencoder into Private Aggregation of Teacher Ensembles for Spoken Command Classification
We propose using an adversarial autoencoder (AAE) to replace generative adversarial network (GAN) in the private aggregation of teacher ensembles (PATE), a solution for ensuring differential privacy in speech applications.
Ranked #3 on
Keyword Spotting
on Google Speech Commands
(10-keyword Speech Commands dataset metric)
Training a Resilient Q-Network against Observational Interference
Deep reinforcement learning (DRL) has demonstrated impressive performance in various gaming simulators and real-world applications.
Multi-task Language Modeling for Improving Speech Recognition of Rare Words
We show that our rescoring model trained with these additional tasks outperforms the baseline rescoring model, trained with only the language modeling task, by 1. 4% on a general test and by 2. 6% on a rare word test set in terms of word-error-rate relative (WERR).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
A Two-Stage Approach to Device-Robust Acoustic Scene Classification
To improve device robustness, a highly desirable key feature of a competitive data-driven acoustic scene classification (ASC) system, a novel two-stage system based on fully convolutional neural networks (CNNs) is proposed.
Ranked #1 on
Acoustic Scene Classification
on TAU Urban Acoustic Scenes 2019
(using extra training data)
DeepOpht: Medical Report Generation for Retinal Images via Deep Models and Visual Explanation
To train and validate the effectiveness of our DNN-based module, we propose a large-scale retinal disease image dataset.
Decentralizing Feature Extraction with Quantum Convolutional Neural Network for Automatic Speech Recognition
Testing on the Google Speech Commands Dataset, the proposed QCNN encoder attains a competitive accuracy of 95. 12% in a decentralized model, which is better than the previous architectures using centralized RNN models with convolutional features.
Ranked #1 on
Keyword Spotting
on Google Speech Commands
(10-keyword Speech Commands dataset metric)
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Exploring Deep Hybrid Tensor-to-Vector Network Architectures for Regression Based Speech Enhancement
Finally, our experiments of multi-channel speech enhancement on a simulated noisy WSJ0 corpus demonstrate that our proposed hybrid CNN-TT architecture achieves better results than both DNN and CNN models in terms of better-enhanced speech qualities and smaller parameter sizes.
Wavelet Channel Attention Module with a Fusion Network for Single Image Deraining
Wavelet transform and the inverse wavelet transform are substituted for down-sampling and up-sampling so feature maps from the wavelet transform and convolutions contain different frequencies and scales.
Device-Robust Acoustic Scene Classification Based on Two-Stage Categorization and Data Augmentation
On Task 1b development data set, we achieve an accuracy of 96. 7\% with a model size smaller than 500KB.
Characterizing Speech Adversarial Examples Using Self-Attention U-Net Enhancement
Recent studies have highlighted adversarial examples as ubiquitous threats to the deep neural network (DNN) based speech recognition systems.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Y-net: Multi-scale feature aggregation network with wavelet structure similarity loss function for single image dehazing
Extensive experimental results demonstrate that the proposed Y-net with the W-SSIM loss function restores high-quality clear images and outperforms state-of-the-art algorithms.
Enhanced Adversarial Strategically-Timed Attacks against Deep Reinforcement Learning
Recent deep neural networks based techniques, especially those equipped with the ability of self-adaptation in the system level such as deep reinforcement learning (DRL), are shown to possess many advantages of optimizing robot learning systems (e. g., autonomous navigation and continuous robot arm control.)
Tensor-to-Vector Regression for Multi-channel Speech Enhancement based on Tensor-Train Network
Finally, in 8-channel conditions, a PESQ of 3. 12 is achieved using 20 million parameters for TTN, whereas a DNN with 68 million parameters can only attain a PESQ of 3. 06.
Evolving Neural Networks through a Reverse Encoding Tree
Using RET, two types of approaches -- NEAT with Binary search encoding (Bi-NEAT) and NEAT with Golden-Section search encoding (GS-NEAT) -- have been designed to solve problems in benchmark continuous learning environments such as logic gates, Cartpole, and Lunar Lander, and tested against classical NEAT and FS-NEAT as baselines.
Submodular Rank Aggregation on Score-based Permutations for Distributed Automatic Speech Recognition
Distributed automatic speech recognition (ASR) requires to aggregate outputs of distributed deep neural network (DNN)-based models.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Interpretable Self-Attention Temporal Reasoning for Driving Behavior Understanding
Performing driving behaviors based on causal reasoning is essential to ensure driving safety.
Reinforcement Learning based Interconnection Routing for Adaptive Traffic Optimization
In this work, we demonstrate the promise of applying reinforcement learning (RL) to optimize NoC runtime performance.
Variational Quantum Circuits for Deep Reinforcement Learning
To the best of our knowledge, this work is the first proof-of-principle demonstration of variational quantum circuits to approximate the deep $Q$-value function for decision-making and policy-selection reinforcement learning with experience replay and target network.
Synthesizing New Retinal Symptom Images by Multiple Generative Models
Age-Related Macular Degeneration (AMD) is an asymptomatic retinal disease which may result in loss of vision.
When Causal Intervention Meets Adversarial Examples and Image Masking for Deep Neural Networks
To study the intervention effects on pixel-level features for causal reasoning, we introduce pixel-wise masking and adversarial perturbation.
Controllability, Multiplexing, and Transfer Learning in Networks using Evolutionary Learning
Next, (2) the learned networks are technically controllable as only a small number of driver nodes are required to move the system to a new state.
