Search Results for author:
Found 104 papers, 24 papers with code
IMSurReal Too: IMS in the Surface Realization Shared Task 2020
We introduce the IMS contribution to the Surface Realization Shared Task 2020.
Findings of the AmericasNLP 2021 Shared Task on Open Machine Translation for Indigenous Languages of the Americas
This paper presents the results of the 2021 Shared Task on Open Machine Translation for Indigenous Languages of the Americas.
IMS’ Systems for the IWSLT 2021 Low-Resource Speech Translation Task
This paper describes the submission to the IWSLT 2021 Low-Resource Speech Translation Shared Task by IMS team.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
“It’s our fault!”: Insights Into Users’ Understanding and Interaction With an Explanatory Collaborative Dialog System
We conclude that creating shared mental models between users and AI systems is important to achieving successful dialogs.
»textklang« – Towards a Multi-Modal Exploration Platform for German Poetry
We present the steps taken towards an exploration platform for a multi-modal corpus of German lyric poetry from the Romantic era developed in the project »textklang«.
A Two-stage Model for Slot Filling in Low-resource Settings: Domain-agnostic Non-slot Reduction and Pretrained Contextual Embeddings
In this paper, we propose a novel two-stage model architecture that can be trained with only a few in-domain hand-labeled examples.
“It seemed like an annoying woman”: On the Perception and Ethical Considerations of Affective Language in Text-Based Conversational Agents
Based on statistical and qualitative analysis of the responses, we found language style played an important role in how human-like participants perceived a dialog agent as well as how likable.
Teaching a Multilingual Large Language Model to Understand Multilingual Speech via Multi-Instructional Training
Our zero-shot evaluation results confirm the robustness of our approach across multiple tasks, including speech translation and multilingual spoken language understanding, thereby opening new avenues for applying LLMs in the speech domain.
Intrinsic Subgraph Generation for Interpretable Graph based Visual Question Answering
In this work, we introduce an interpretable approach for graph-based VQA and demonstrate competitive performance on the GQA dataset.
Towards a Zero-Data, Controllable, Adaptive Dialog System
Conversational Tree Search (V\"ath et al., 2023) is a recent approach to controllable dialog systems, where domain experts shape the behavior of a Reinforcement Learning agent through a dialog tree.
Explaining Pre-Trained Language Models with Attribution Scores: An Analysis in Low-Resource Settings
Attribution scores indicate the importance of different input parts and can, thus, explain model behaviour.
The IMS Toucan System for the Blizzard Challenge 2023
For our contribution to the Blizzard Challenge 2023, we improved on the system we submitted to the Blizzard Challenge 2021.
Controllable Generation of Artificial Speaker Embeddings through Discovery of Principal Directions
Customizing voice and speaking style in a speech synthesis system with intuitive and fine-grained controls is challenging, given that little data with appropriate labels is available.
Data Augmentation Techniques for Machine Translation of Code-Switched Texts: A Comparative Study
Linguistic theories and random lexical replacement prove to be effective in the lack of CSW parallel data, where both approaches achieve similar results.
Leveraging Multilingual Self-Supervised Pretrained Models for Sequence-to-Sequence End-to-End Spoken Language Understanding
A number of methods have been proposed for End-to-End Spoken Language Understanding (E2E-SLU) using pretrained models, however their evaluation often lacks multilingual setup and tasks that require prediction of lexical fillers, such as slot filling.
Neural Machine Translation for the Indigenous Languages of the Americas: An Introduction
Neural models have drastically advanced state of the art for machine translation (MT) between high-resource languages.
Ethical Considerations for Machine Translation of Indigenous Languages: Giving a Voice to the Speakers
In recent years machine translation has become very successful for high-resource language pairs.
Neighboring Words Affect Human Interpretation of Saliency Explanations
Word-level saliency explanations ("heat maps over words") are often used to communicate feature-attribution in text-based models.
Modeling Speaker-Listener Interaction for Backchannel Prediction
We present our latest findings on backchannel modeling novelly motivated by the canonical use of the minimal responses Yeah and Uh-huh in English and their correspondent tokens in German, and the effect of encoding the speaker-listener interaction.
Oh, Jeez! or Uh-huh? A Listener-aware Backchannel Predictor on ASR Transcriptions
This paper presents our latest investigation on modeling backchannel in conversations.
Conversational Tree Search: A New Hybrid Dialog Task
Conversational interfaces provide a flexible and easy way for users to seek information that may otherwise be difficult or inconvenient to obtain.
ArzEn-ST: A Three-way Speech Translation Corpus for Code-Switched Egyptian Arabic - English
We present our work on collecting ArzEn-ST, a code-switched Egyptian Arabic - English Speech Translation Corpus.
Low-Resource Multilingual and Zero-Shot Multispeaker TTS
While neural methods for text-to-speech (TTS) have shown great advances in modeling multiple speakers, even in zero-shot settings, the amount of data needed for those approaches is generally not feasible for the vast majority of the world's over 6, 000 spoken languages.
Combining Contrastive and Non-Contrastive Losses for Fine-Tuning Pretrained Models in Speech Analysis
This pretrained model is then finetuned to a specific task.
Improving Semi-supervised End-to-end Automatic Speech Recognition using CycleGAN and Inter-domain Losses
In this paper, we exploit the advantages from both inter-domain loss and CycleGAN to achieve better shared representation of unpaired speech and text inputs and thus improve the speech-to-text mapping.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Challenges in Explanation Quality Evaluation
This approach assumes that explanations which reach higher proxy scores will also provide a greater benefit to human users.
Anonymizing Speech with Generative Adversarial Networks to Preserve Speaker Privacy
In order to protect the privacy of speech data, speaker anonymization aims for hiding the identity of a speaker by changing the voice in speech recordings.
Exploring Segmentation Approaches for Neural Machine Translation of Code-Switched Egyptian Arabic-English Text
For extreme low-resource scenarios, a combination of frequency and morphology-based segmentations is shown to perform the best.
The Who in Code-Switching: A Case Study for Predicting Egyptian Arabic-English Code-Switching Levels based on Character Profiles
Given that the factors giving rise to CS vary from one country to the other, as well as from one person to the other, CS is found to be a speaker-dependant behaviour, where the frequency by which the foreign language is embedded differs across speakers.
PoeticTTS -- Controllable Poetry Reading for Literary Studies
Speech synthesis for poetry is challenging due to specific intonation patterns inherent to poetic speech.
Speaker Anonymization with Phonetic Intermediate Representations
In this work, we propose a speaker anonymization pipeline that leverages high quality automatic speech recognition and synthesis systems to generate speech conditioned on phonetic transcriptions and anonymized speaker embeddings.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Exact Prosody Cloning in Zero-Shot Multispeaker Text-to-Speech
The cloning of a speaker's voice using an untranscribed reference sample is one of the great advances of modern neural text-to-speech (TTS) methods.
Investigating Lexical Replacements for Arabic-English Code-Switched Data Augmentation
Results show that using a predictive model results in more natural CS sentences compared to the random approach, as reported in human judgements.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
Meta Learning for Natural Language Processing: A Survey
Deep learning has been the mainstream technique in natural language processing (NLP) area.
BPE vs. Morphological Segmentation: A Case Study on Machine Translation of Four Polysynthetic Languages
Morphologically-rich polysynthetic languages present a challenge for NLP systems due to data sparsity, and a common strategy to handle this issue is to apply subword segmentation.
Language-Agnostic Meta-Learning for Low-Resource Text-to-Speech with Articulatory Features
While neural text-to-speech systems perform remarkably well in high-resource scenarios, they cannot be applied to the majority of the over 6, 000 spoken languages in the world due to a lack of appropriate training data.
Human Interpretation of Saliency-based Explanation Over Text
In this work, we focus on this question through a study of saliency-based explanations over textual data.
Investigation of Densely Connected Convolutional Networks with Domain Adversarial Learning for Noise Robust Speech Recognition
We investigate densely connected convolutional networks (DenseNets) and their extension with domain adversarial training for noise robust speech recognition.
Integrating Knowledge in End-to-End Automatic Speech Recognition for Mandarin-English Code-Switching
Code-Switching (CS) is a common linguistic phenomenon in multilingual communities that consists of switching between languages while speaking.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Predicting User Code-Switching Level from Sociological and Psychological Profiles
Multilingual speakers tend to alternate between languages within a conversation, a phenomenon referred to as "code-switching" (CS).
Improving Code-switching Language Modeling with Artificially Generated Texts using Cycle-consistent Adversarial Networks
This paper presents our latest effort on improving Code-switching language models that suffer from data scarcity.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Improving Speech Recognition on Noisy Speech via Speech Enhancement with Multi-Discriminators CycleGAN
This paper presents our latest investigations on improving automatic speech recognition for noisy speech via speech enhancement.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
ESPnet-SLU: Advancing Spoken Language Understanding through ESPnet
However, there are few open source toolkits that can be used to generate reproducible results on different Spoken Language Understanding (SLU) benchmarks.
Beyond Accuracy: A Consolidated Tool for Visual Question Answering Benchmarking
On the way towards general Visual Question Answering (VQA) systems that are able to answer arbitrary questions, the need arises for evaluation beyond single-metric leaderboards for specific datasets.
Does External Knowledge Help Explainable Natural Language Inference? Automatic Evaluation vs. Human Ratings
For this, we investigate different sources of external knowledge and evaluate the performance of our models on in-domain data as well as on special transfer datasets that are designed to assess fine-grained reasoning capabilities.
Investigations on Speech Recognition Systems for Low-Resource Dialectal Arabic-English Code-Switching Speech
In this paper, we present our work on code-switched Egyptian Arabic-English automatic speech recognition (ASR).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Meta Learning and Its Applications to Natural Language Processing
Meta-learning is one of the most important new techniques in machine learning in recent years.
Thought Flow Nets: From Single Predictions to Trains of Model Thought
In addition, we conduct a qualitative analysis of thought flow correction patterns and explore how thought flow predictions affect human users within a crowdsourcing study.
IMS' Systems for the IWSLT 2021 Low-Resource Speech Translation Task
This paper describes the submission to the IWSLT 2021 Low-Resource Speech Translation Shared Task by IMS team.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languages
Continued pretraining offers improvements, with an average accuracy of 44. 05%.
Few-shot Learning for Slot Tagging with Attentive Relational Network
Metric-based learning is a well-known family of methods for few-shot learning, especially in computer vision.
Investigations on Audiovisual Emotion Recognition in Noisy Conditions
In this paper we explore audiovisual emotion recognition under noisy acoustic conditions with a focus on speech features.
Meta-Learning for improving rare word recognition in end-to-end ASR
We propose a new method of generating meaningful embeddings for speech, changes to four commonly used meta learning approaches to enable them to perform keyword spotting in continuous signals and an approach of combining their outcomes into an end-to-end automatic speech recognition system to improve rare word recognition.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Fine-tuning BERT for Low-Resource Natural Language Understanding via Active Learning
Recently, leveraging pre-trained Transformer based language models in down stream, task specific models has advanced state of the art results in natural language understanding tasks.
Interpreting Attention Models with Human Visual Attention in Machine Reading Comprehension
We compare state of the art networks based on long short-term memory (LSTM), convolutional neural models (CNN) and XLNet Transformer architectures.
F1 is Not Enough! Models and Evaluation Towards User-Centered Explainable Question Answering
The user study shows that our models increase the ability of the users to judge the correctness of the system and that scores like F1 are not enough to estimate the usefulness of a model in a practical setting with human users.
Pretrained Semantic Speech Embeddings for End-to-End Spoken Language Understanding via Cross-Modal Teacher-Student Learning
Spoken language understanding is typically based on pipeline architectures including speech recognition and natural language understanding steps.
Ensemble Self-Training for Low-Resource Languages: Grapheme-to-Phoneme Conversion and Morphological Inflection
We present an iterative data augmentation framework, which trains and searches for an optimal ensemble and simultaneously annotates new training data in a self-training style.
Fast and Accurate Non-Projective Dependency Tree Linearization
We propose a graph-based method to tackle the dependency tree linearization task.
ADVISER: A Toolkit for Developing Multi-modal, Multi-domain and Socially-engaged Conversational Agents
We present ADVISER - an open-source, multi-domain dialog system toolkit that enables the development of multi-modal (incorporating speech, text and vision), socially-engaged (e. g. emotion recognition, engagement level prediction and backchanneling) conversational agents.
ArzEn: A Speech Corpus for Code-switched Egyptian Arabic-English
In this paper, we first discuss the CS phenomenon in Egypt and the factors that gave rise to the current language.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Cairo Student Code-Switch (CSCS) Corpus: An Annotated Egyptian Arabic-English Corpus
Code-switching has become a prevalent phenomenon across many communities.
IMSurReal: IMS at the Surface Realization Shared Task 2019
We introduce the IMS contribution to the Surface Realization Shared Task 2019.
Head-First Linearization with Tree-Structured Representation
We present a dependency tree linearization model with two novel components: (1) a tree-structured encoder based on bidirectional Tree-LSTM that propagates information first bottom-up then top-down, which allows each token to access information from the entire tree; and (2) a linguistically motivated head-first decoder that emphasizes the central role of the head and linearizes the subtree by incrementally attaching the dependents on both sides of the head.
Code-switching Language Modeling With Bilingual Word Embeddings: A Case Study for Egyptian Arabic-English
Code-switching (CS) is a widespread phenomenon among bilingual and multilingual societies.
To Combine or Not To Combine? A Rainbow Deep Reinforcement Learning Agent for Dialog Policies
In this paper, we explore state-of-the-art deep reinforcement learning methods for dialog policy training such as prioritized experience replay, double deep Q-Networks, dueling network architectures and distributional learning.
IMS-Speech: A Speech to Text Tool
We present the IMS-Speech, a web based tool for German and English speech transcription aiming to facilitate research in various disciplines which require accesses to lexical information in spoken language materials.
Ranked #4 on
Speech Recognition
on TUDA
(using extra training data)
End-to-End Multi-Speaker Speech Recognition using Speaker Embeddings and Transfer Learning
This paper presents our latest investigation on end-to-end automatic speech recognition (ASR) for overlapped speech.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Learning the Dyck Language with Attention-based Seq2Seq Models
The generalized Dyck language has been used to analyze the ability of Recurrent Neural Networks (RNNs) to learn context-free grammars (CFGs).
ADVISER: A Dialog System Framework for Education \& Research
In this paper, we present ADVISER - an open source dialog system framework for education and research purposes.
Context-aware Neural-based Dialog Act Classification on Automatically Generated Transcriptions
This paper presents our latest investigations on dialog act (DA) classification on automatically generated transcriptions.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
Approximate Dynamic Oracle for Dependency Parsing with Reinforcement Learning
We present a general approach with reinforcement learning (RL) to approximate dynamic oracles for transition systems where exact dynamic oracles are difficult to derive.
Sequence-to-Sequence Models for Data-to-Text Natural Language Generation: Word- vs. Character-based Processing and Output Diversity
We present a comparison of word-based and character-based sequence-to-sequence models for data-to-text natural language generation, which generate natural language descriptions for structured inputs.
Comparing Attention-based Convolutional and Recurrent Neural Networks: Success and Limitations in Machine Reading Comprehension
We propose a machine reading comprehension model based on the compare-aggregate framework with two-staged attention that achieves state-of-the-art results on the MovieQA question answering dataset.
Densely Connected Convolutional Networks for Speech Recognition
This paper presents our latest investigation on Densely Connected Convolutional Networks (DenseNets) for acoustic modelling (AM) in automatic speech recognition.
Unsupervised Domain Adaptation by Adversarial Learning for Robust Speech Recognition
In this paper, we investigate the use of adversarial learning for unsupervised adaptation to unseen recording conditions, more specifically, single microphone far-field speech.
Low-Resource Text Classification using Domain-Adversarial Learning
Deep learning techniques have recently shown to be successful in many natural language processing tasks forming state-of-the-art systems.
Addressing Low-Resource Scenarios with Character-aware Embeddings
Most modern approaches to computing word embeddings assume the availability of text corpora with billions of words.
Effects of Word Embeddings on Neural Network-based Pitch Accent Detection
Pitch accent detection often makes use of both acoustic and lexical features based on the fact that pitch accents tend to correlate with certain words.
Investigations on End-to-End Audiovisual Fusion
Audiovisual speech recognition (AVSR) is a method to alleviate the adverse effect of noise in the acoustic signal.
Introducing two Vietnamese Datasets for Evaluating Semantic Models of (Dis-)Similarity and Relatedness
We present two novel datasets for the low-resource language Vietnamese to assess models of semantic similarity: ViCon comprises pairs of synonyms and antonyms across word classes, thus offering data to distinguish between similarity and dissimilarity.
Lexico-acoustic Neural-based Models for Dialog Act Classification
We propose a neural model that processes both lexical and acoustic features for classification.
Cross-lingual and Multilingual Speech Emotion Recognition on English and French
Research on multilingual speech emotion recognition faces the problem that most available speech corpora differ from each other in important ways, such as annotation methods or interaction scenarios.
Syntactic and Semantic Features For Code-Switching Factored Language Models
The experimental results reveal that Brown word clusters, part-of-speech tags and open-class words are the most effective at reducing the perplexity of factored language models on the Mandarin-English Code-Switching corpus SEAME.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Enriching ASR Lattices with POS Tags for Dependency Parsing
Parsing speech requires a richer representation than 1-best or n-best hypotheses, e. g. lattices.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+5
Neural-based Context Representation Learning for Dialog Act Classification
We explore context representation learning methods in neural-based models for dialog act classification.
Improving coreference resolution with automatically predicted prosodic information
Adding manually annotated prosodic information, specifically pitch accents and phrasing, to the typical text-based feature set for coreference resolution has previously been shown to have a positive effect on German data.
Hierarchical Embeddings for Hypernymy Detection and Directionality
We present a novel neural model HyperVec to learn hierarchical embeddings for hypernymy detection and directionality.
Encoding Word Confusion Networks with Recurrent Neural Networks for Dialog State Tracking
This paper presents our novel method to encode word confusion networks, which can represent a rich hypothesis space of automatic speech recognition systems, via recurrent neural networks.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
A General-Purpose Tagger with Convolutional Neural Networks
We present a general-purpose tagger based on convolutional neural networks (CNN), used for both composing word vectors and encoding context information.
Prosodic Event Recognition using Convolutional Neural Networks with Context Information
This paper demonstrates the potential of convolutional neural networks (CNN) for detecting and classifying prosodic events on words, specifically pitch accents and phrase boundary tones, from frame-based acoustic features.
Attentive Convolutional Neural Network based Speech Emotion Recognition: A Study on the Impact of Input Features, Signal Length, and Acted Speech
Speech emotion recognition is an important and challenging task in the realm of human-computer interaction.
Character Composition Model with Convolutional Neural Networks for Dependency Parsing on Morphologically Rich Languages
We present a transition-based dependency parser that uses a convolutional neural network to compose word representations from characters.
Distinguishing Antonyms and Synonyms in a Pattern-based Neural Network
Distinguishing between antonyms and synonyms is a key task to achieve high performance in NLP systems.
Challenges of Computational Processing of Code-Switching
This paper addresses challenges of Natural Language Processing (NLP) on non-canonical multilingual data in which two or more languages are mixed.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+7
Neural-based Noise Filtering from Word Embeddings
Word embeddings have been demonstrated to benefit NLP tasks impressively.
Sequential Convolutional Neural Networks for Slot Filling in Spoken Language Understanding
We investigate the usage of convolutional neural networks (CNNs) for the slot filling task in spoken language understanding.
Integrating Distributional Lexical Contrast into Word Embeddings for Antonym-Synonym Distinction
We propose a novel vector representation that integrates lexical contrast into distributional vectors and strengthens the most salient features for determining degrees of word similarity.
Combining Recurrent and Convolutional Neural Networks for Relation Classification
This paper investigates two different neural architectures for the task of relation classification: convolutional neural networks and recurrent neural networks.
