Search Results for author:
Found 43 papers, 19 papers with code
CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations
CoVoMix is capable of first converting dialogue text into multiple streams of discrete tokens, with each token stream representing semantic information for individual talkers.
WavLLM: Towards Robust and Adaptive Speech Large Language Model
In this work, we introduce WavLLM, a robust and adaptive speech large language model with dual encoders, and a prompt-aware LoRA weight adapter, optimized by a two-stage curriculum learning approach.
Boosting Large Language Model for Speech Synthesis: An Empirical Study
In this paper, we conduct a comprehensive empirical exploration of boosting LLMs with the ability to generate speech, by combining pre-trained LLM LLaMA/OPT and text-to-speech synthesis model VALL-E. We compare three integration methods between LLMs and speech synthesis models, including directly fine-tuned LLMs, superposed layers of LLMs and VALL-E, and coupled LLMs and VALL-E using LLMs as a powerful text encoder.
Diffusion Conditional Expectation Model for Efficient and Robust Target Speech Extraction
Additionally, we introduce Regenerate-DCEM (R-DCEM) that can regenerate and optimize speech quality based on pre-processed speech from a discriminative model.
On decoder-only architecture for speech-to-text and large language model integration
Large language models (LLMs) have achieved remarkable success in the field of natural language processing, enabling better human-computer interaction using natural language.
VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation
Recent research shows a big convergence in model architecture, training objectives, and inference methods across various tasks for different modalities.
ComSL: A Composite Speech-Language Model for End-to-End Speech-to-Text Translation
Joint speech-language training is challenging due to the large demand for training data and GPU consumption, as well as the modality gap between speech and language.
Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling
We propose a cross-lingual neural codec language model, VALL-E X, for cross-lingual speech synthesis.
Building High-accuracy Multilingual ASR with Gated Language Experts and Curriculum Training
We propose gated language experts and curriculum training to enhance multilingual transformer transducer models without requiring language identification (LID) input from users during inference.
Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
In addition, we find Vall-E could preserve the speaker's emotion and acoustic environment of the acoustic prompt in synthesis.
VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning
Although speech is a simple and effective way for humans to communicate with the outside world, a more realistic speech interaction contains multimodal information, e. g., vision, text.
LAMASSU: Streaming Language-Agnostic Multilingual Speech Recognition and Translation Using Neural Transducers
In this paper, we propose LAMASSU, a streaming language-agnostic multilingual speech recognition and translation model using neural transducers.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech Translation
However, direct S2ST suffers from the data scarcity problem because the corpora from speech of the source language to speech of the target language are very rare.
Robust Data2vec: Noise-robust Speech Representation Learning for ASR by Combining Regression and Improved Contrastive Learning
Self-supervised pre-training methods based on contrastive learning or regression tasks can utilize more unlabeled data to improve the performance of automatic speech recognition (ASR).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training
The rapid development of single-modal pre-training has prompted researchers to pay more attention to cross-modal pre-training methods.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Data
In this paper, we propose a cross-modal Speech and Language Model (SpeechLM) to explicitly align speech and text pre-training with a pre-defined unified discrete representation.
The YiTrans End-to-End Speech Translation System for IWSLT 2022 Offline Shared Task
The YiTrans system is built on large-scale pre-trained encoder-decoder models.
Pre-Training Transformer Decoder for End-to-End ASR Model with Unpaired Speech Data
In this way, the decoder learns to reconstruct original speech information with codes before learning to generate correct text.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERT
LightHuBERT outperforms the original HuBERT on ASR and five SUPERB tasks with the HuBERT size, achieves comparable performance to the teacher model in most tasks with a reduction of 29% parameters, and obtains a $3. 5\times$ compression ratio in three SUPERB tasks, e. g., automatic speaker verification, keyword spotting, and intent classification, with a slight accuracy loss.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+6
A Roadmap for Big Model
With the rapid development of deep learning, training Big Models (BMs) for multiple downstream tasks becomes a popular paradigm.
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
Self-supervised learning (SSL) achieves great success in speech recognition, while limited exploration has been attempted for other speech processing tasks.
SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing
Motivated by the success of T5 (Text-To-Text Transfer Transformer) in pre-trained natural language processing models, we propose a unified-modal SpeechT5 framework that explores the encoder-decoder pre-training for self-supervised speech/text representation learning.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+7
Multi-View Self-Attention Based Transformer for Speaker Recognition
In this work, we propose a novel multi-view self-attention mechanism and present an empirical study of different Transformer variants with or without the proposed attention mechanism for speaker recognition.
Jointly Learning to Repair Code and Generate Commit Message
We propose a novel task of jointly repairing program codes and generating commit messages.
SemFace: Pre-training Encoder and Decoder with a Semantic Interface for Neural Machine Translation
While pre-training techniques are working very well in natural language processing, how to pre-train a decoder and effectively use it for neural machine translation (NMT) still remains a tricky issue.
A Configurable Multilingual Model is All You Need to Recognize All Languages
Particularly, a single CMM can be deployed to any user scenario where the users can pre-select any combination of languages.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Benchmark datasets have a significant impact on accelerating research in programming language tasks.
Ranked #1 on
Cloze Test
on CodeXGLUE - CT-maxmin
CodeBLEU: a Method for Automatic Evaluation of Code Synthesis
Evaluation metrics play a vital role in the growth of an area as it defines the standard of distinguishing between good and bad models.
GraphCodeBERT: Pre-training Code Representations with Data Flow
Instead of taking syntactic-level structure of code like abstract syntax tree (AST), we use data flow in the pre-training stage, which is a semantic-level structure of code that encodes the relation of "where-the-value-comes-from" between variables.
Ranked #3 on
Type prediction
on ManyTypes4TypeScript
Improving Autoregressive NMT with Non-Autoregressive Model
In this work, we propose a novel Encoder-NAD-AD framework for NMT, aiming at boosting AT with global information produced by NAT model.
CASIA's System for IWSLT 2020 Open Domain Translation
This paper describes the CASIA{'}s system for the IWSLT 2020 open domain translation task.
Synchronous Speech Recognition and Speech-to-Text Translation with Interactive Decoding
Speech-to-text translation (ST), which translates source language speech into target language text, has attracted intensive attention in recent years.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Synchronously Generating Two Languages with Interactive Decoding
In this paper, we introduce a novel interactive approach to translate a source language into two different languages simultaneously and interactively.
A Compact and Language-Sensitive Multilingual Translation Method
We verify our methods on various translation scenarios, including one-to-many, many-to-many and zero-shot.
Sequence Generation: From Both Sides to the Middle
The encoder-decoder framework has achieved promising process for many sequence generation tasks, such as neural machine translation and text summarization.
Synchronous Bidirectional Neural Machine Translation
In this paper, we introduce a synchronous bidirectional neural machine translation (SB-NMT) that predicts its outputs using left-to-right and right-to-left decoding simultaneously and interactively, in order to leverage both of the history and future information at the same time.
Ranked #28 on
Machine Translation
on WMT2014 English-German
Synchronous Bidirectional Inference for Neural Sequence Generation
In this work, we propose a synchronous bidirectional inference model to generate outputs using both left-to-right and right-to-left decoding simultaneously and interactively.
Language-Independent Representor for Neural Machine Translation
Current Neural Machine Translation (NMT) employs a language-specific encoder to represent the source sentence and adopts a language-specific decoder to generate target translation.
Word, Subword or Character? An Empirical Study of Granularity in Chinese-English NMT
Our experiments show that subword model performs best for Chinese-to-English translation with the vocabulary which is not so big while hybrid word-character model is most suitable for English-to-Chinese translation.
Look-ahead Attention for Generation in Neural Machine Translation
The attention model has become a standard component in neural machine translation (NMT) and it guides translation process by selectively focusing on parts of the source sentence when predicting each target word.
Neural System Combination for Machine Translation
Neural machine translation (NMT) becomes a new approach to machine translation and generates much more fluent results compared to statistical machine translation (SMT).
