Search Results for author:
Found 16 papers, 14 papers with code
Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation
The proposed method is based on a lightweight adaptor network, which learns to map the audio-based representation to the input representation expected by the text-to-video generation model.
AudioToken: Adaptation of Text-Conditioned Diffusion Models for Audio-to-Image Generation
In this paper, we propose a novel method utilizing latent diffusion models trained for text-to-image-generation to generate images conditioned on audio recordings.
Discriminative Class Tokens for Text-to-Image Diffusion Models
This approach has two disadvantages: (i) supervised datasets are generally small compared to large-scale scraped text-image datasets on which text-to-image models are trained, affecting the quality and diversity of the generated images, or (ii) the input is a hard-coded label, as opposed to free-form text, limiting the control over the generated images.
Describing Sets of Images with Textual-PCA
We seek to semantically describe a set of images, capturing both the attributes of single images and the variations within the set.
Zero-Shot Video Captioning with Evolving Pseudo-Tokens
We introduce a zero-shot video captioning method that employs two frozen networks: the GPT-2 language model and the CLIP image-text matching model.
Optimizing Relevance Maps of Vision Transformers Improves Robustness
It has been observed that visual classification models often rely mostly on the image background, neglecting the foreground, which hurts their robustness to distribution changes.
Ranked #1 on
Out-of-Distribution Generalization
on ImageNet-W
Latent Space Explanation by Intervention
The success of deep neural nets heavily relies on their ability to encode complex relations between their input and their output.
ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic
While such models can provide a powerful score for matching and subsequent zero-shot tasks, they are not capable of generating caption given an image.
Perceptual Score: What Data Modalities Does Your Model Perceive?
To study and quantify this concern, we introduce the perceptual score, a metric that assesses the degree to which a model relies on the different subsets of the input features, i. e., modalities.
Video and Text Matching with Conditioned Embeddings
Traditionally video and text matching is done by learning a shared embedding space and the encoding of one modality is independent of the other.
Ordered Attention for Coherent Visual Storytelling
OIA models interactions between the sentence-corresponding image and important regions in other images of the sequence.
Ensemble of MRR and NDCG models for Visual Dialog
However, the NDCG metric favors the usually applicable uncertain answers such as `I don't know.
Ranked #1 on
Visual Dialog
on VisDial v1.0 test-std
Removing Bias in Multi-modal Classifiers: Regularization by Maximizing Functional Entropies
However, regularization with the functional entropy is challenging.
Ranked #3 on
Visual Question Answering (VQA)
on VQA-CP
A Simple Baseline for Audio-Visual Scene-Aware Dialog
The recently proposed audio-visual scene-aware dialog task paves the way to a more data-driven way of learning virtual assistants, smart speakers and car navigation systems.
Ranked #1 on
Scene-Aware Dialogue
on AVSD
Factor Graph Attention
We address this issue and develop a general attention mechanism for visual dialog which operates on any number of data utilities.
Ranked #1 on
Visual Dialog
on VisDial v0.9 val
High-Order Attention Models for Visual Question Answering
The quest for algorithms that enable cognitive abilities is an important part of machine learning.
