Search Results for author:
Found 23 papers, 9 papers with code
Speech Data Augmentation for Improving Phoneme Transcriptions of Aphasic Speech Using Wav2Vec 2.0 for the PSST Challenge
As part of the PSST challenge, we explore how data augmentations, data sources, and model size affect phoneme transcription accuracy on speech produced by individuals with aphasia.
Fake it to make it: Using synthetic data to remedy the data shortage in joint multimodal speech-and-gesture synthesis
Specifically, we use unimodal synthesis models trained on large datasets to create multimodal (but synthetic) parallel training data, and then pre-train a joint synthesis model on that material.
Unified speech and gesture synthesis using flow matching
As text-to-speech technologies achieve remarkable naturalness in read-aloud tasks, there is growing interest in multimodal synthesis of verbal and non-verbal communicative behaviour, such as spontaneous speech and associated body gestures.
Ranked #1 on
Motion Synthesis
on Trinity Speech-Gesture Dataset
Diffusion-Based Co-Speech Gesture Generation Using Joint Text and Audio Representation
The output of the CSMP module is used as a conditioning signal in the diffusion-based gesture synthesis model in order to achieve semantically-aware co-speech gesture generation.
Matcha-TTS: A fast TTS architecture with conditional flow matching
We introduce Matcha-TTS, a new encoder-decoder architecture for speedy TTS acoustic modelling, trained using optimal-transport conditional flow matching (OT-CFM).
Ranked #1 on
Text-To-Speech Synthesis
on LJSpeech
(MOS metric)
Diff-TTSG: Denoising probabilistic integrated speech and gesture synthesis
With read-aloud speech synthesis achieving high naturalness scores, there is a growing research interest in synthesising spontaneous speech.
Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models
Diffusion models have experienced a surge of interest as highly expressive yet efficiently trainable probabilistic models.
OverFlow: Putting flows on top of neural transducers for better TTS
Neural HMMs are a type of neural transducer recently proposed for sequence-to-sequence modelling in text-to-speech.
Ranked #11 on
Text-To-Speech Synthesis
on LJSpeech
(using extra training data)
Neural HMMs are all you need (for high-quality attention-free TTS)
Neural sequence-to-sequence TTS has achieved significantly better output quality than statistical speech synthesis using HMMs.
Ranked #3 on
Speech Synthesis
on LJSpeech
Integrated Speech and Gesture Synthesis
Text-to-speech and co-speech gesture synthesis have until now been treated as separate areas by two different research communities, and applications merely stack the two technologies using a simple system-level pipeline.
Transflower: probabilistic autoregressive dance generation with multimodal attention
First, we present a novel probabilistic autoregressive architecture that models the distribution over future poses with a normalizing flow conditioned on previous poses as well as music context, using a multimodal transformer encoder.
Generating coherent spontaneous speech and gesture from text
In contrast to previous approaches for joint speech-and-gesture generation, we generate full-body gestures from speech synthesis trained on recordings of spontaneous speech from the same person as the motion-capture data.
Let's Face It: Probabilistic Multi-modal Interlocutor-aware Generation of Facial Gestures in Dyadic Settings
Our contributions are: a) a method for feature extraction from multi-party video and speech recordings, resulting in a representation that allows for independent control and manipulation of expression and speech articulation in a 3D avatar; b) an extension to MoGlow, a recent motion-synthesis method based on normalizing flows, to also take multi-modal signals from the interlocutor as input and subsequently output interlocutor-aware facial gestures; and c) a subjective evaluation assessing the use and relative importance of the input modalities.
Style-Controllable Speech-Driven Gesture Synthesis Using Normalising Flows
In interactive scenarios, systems for generating natural animations on the fly are key to achieving believable and relatable characters.
MoGlow: Probabilistic and controllable motion synthesis using normalising flows
Data-driven modelling and synthesis of motion is an active research area with applications that include animation, games, and social robotics.
A Neural Network Approach to Missing Marker Reconstruction in Human Motion Capture
Optical motion capture systems have become a widely used technology in various fields, such as augmented reality, robotics, movie production, etc.
Self-Supervised Vision-Based Detection of the Active Speaker as Support for Socially-Aware Language Acquisition
Active speaker detection is a fundamental prerequisite for any artificial cognitive system attempting to acquire language in social settings.
Machine Learning and Social Robotics for Detecting Early Signs of Dementia
This paper presents the EACare project, an ambitious multi-disciplinary collaboration with the aim to develop an embodied system, capable of carrying out neuropsychological tests to detect early signs of dementia, e. g., due to Alzheimer's disease.
A Multi-party Multi-modal Dataset for Focus of Visual Attention in Human-human and Human-robot Interaction
This papers describes a data collection setup and a newly recorded dataset.
3rd party observer gaze as a continuous measure of dialogue flow
We present an attempt at using 3rd party observer gaze to get a measure of how appropriate each segment in a dialogue is for a speaker change.
