Search Results for author:
Found 20 papers, 8 papers with code
D-Flow: Differentiating through Flows for Controlled Generation
Taming the generation outcome of state of the art Diffusion and Flow-Matching (FM) models without having to re-train a task-specific model unlocks a powerful tool for solving inverse problems, conditional generation, and controlled generation in general.
SpiRit-LM: Interleaved Spoken and Written Language Model
We introduce SPIRIT-LM, a foundation multimodal language model that freely mixes text and speech.
Masked Audio Generation using a Single Non-Autoregressive Transformer
We introduce MAGNeT, a masked generative sequence modeling method that operates directly over several streams of audio tokens.
Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation
The proposed method is based on a lightweight adaptor network, which learns to map the audio-based representation to the input representation expected by the text-to-video generation model.
Code Llama: Open Foundation Models for Code
We release Code Llama, a family of large language models for code based on Llama 2 providing state-of-the-art performance among open models, infilling capabilities, support for large input contexts, and zero-shot instruction following ability for programming tasks.
Ranked #27 on
Code Generation
on MBPP
EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis
Recent work has shown that it is possible to resynthesize high-quality speech based, not on text, but on low bitrate discrete units that have been learned in a self-supervised fashion and can therefore capture expressive aspects of speech that are hard to transcribe (prosody, voice styles, non-verbal vocalization).
Simple and Controllable Music Generation
We tackle the task of conditional music generation.
Ranked #4 on
Text-to-Music Generation
on MusicCaps
Textually Pretrained Speech Language Models
In this work, we propose TWIST, a method for training SpeechLMs using a warm-start from a pretrained textual language models.
AudioToken: Adaptation of Text-Conditioned Diffusion Models for Audio-to-Image Generation
In this paper, we propose a novel method utilizing latent diffusion models trained for text-to-image-generation to generate images conditioned on audio recordings.
Layer Collaboration in the Forward-Forward Algorithm
We show that the current version of the forward-forward algorithm is suboptimal when considering information flow in the network, resulting in a lack of collaboration between layers of the network.
On the Importance of Gradient Norm in PAC-Bayesian Bounds
Generalization bounds which assess the difference between the true risk and the empirical risk, have been studied extensively.
Augmentation Invariant Discrete Representation for Generative Spoken Language Modeling
This work focuses on improving the robustness of discrete input representations for generative spoken language modeling.
A Functional Information Perspective on Model Interpretation
This work suggests a theoretical framework for model interpretability by measuring the contribution of relevant features to the functional entropy of the network with respect to the input.
Towards a Common Speech Analysis Engine
Beyond that, a common engine should be capable of supporting distributed training with client in-house private data.
Speech Emotion Recognition using Self-Supervised Features
Self-supervised pre-trained features have consistently delivered state-of-art results in the field of natural language processing (NLP); however, their merits in the field of speech emotion recognition (SER) still need further investigation.
Speaker Normalization for Self-supervised Speech Emotion Recognition
Large speech emotion recognition datasets are hard to obtain, and small datasets may contain biases.
Ranked #1 on
Speech Emotion Recognition
on IEMOCAP
(AUC metric)
Latent Space Explanation by Intervention
The success of deep neural nets heavily relies on their ability to encode complex relations between their input and their output.
Perceptual Score: What Data Modalities Does Your Model Perceive?
To study and quantify this concern, we introduce the perceptual score, a metric that assesses the degree to which a model relies on the different subsets of the input features, i. e., modalities.
Are VQA Systems RAD? Measuring Robustness to Augmented Data with Focused Interventions
Deep learning algorithms have shown promising results in visual question answering (VQA) tasks, but a more careful look reveals that they often do not understand the rich signal they are being fed with.
Removing Bias in Multi-modal Classifiers: Regularization by Maximizing Functional Entropies
However, regularization with the functional entropy is challenging.
Ranked #3 on
Visual Question Answering (VQA)
on VQA-CP
