Multimodal Deep Learning
66 papers with code • 1 benchmarks • 17 datasets
Multimodal deep learning is a type of deep learning that combines information from multiple modalities, such as text, image, audio, and video, to make more accurate and comprehensive predictions. It involves training deep neural networks on data that includes multiple types of information and using the network to make predictions based on this combined data.
One of the key challenges in multimodal deep learning is how to effectively combine information from multiple modalities. This can be done using a variety of techniques, such as fusing the features extracted from each modality, or using attention mechanisms to weight the contribution of each modality based on its importance for the task at hand.
Multimodal deep learning has many applications, including image captioning, speech recognition, natural language processing, and autonomous vehicles. By combining information from multiple modalities, multimodal deep learning can improve the accuracy and robustness of models, enabling them to perform better in real-world scenarios where multiple types of information are present.
Benchmarks
These leaderboards are used to track progress in Multimodal Deep Learning
| Trend | Dataset | Best Model | Paper | Code | Compare |
|---|---|---|---|---|---|
|
Datasets
CUB-200-2011
ScienceQA
MVK
OLIVES Dataset
QUILT-1M
Multimodal PISA
Uncorrelated Corrupted Dataset
REBUS
Latest papers
Formalizing Multimedia Recommendation through Multimodal Deep Learning
sisinflab/formal-multimod-rec
•
•
Recommender systems (RSs) offer personalized navigation experiences on online platforms, but recommendation remains a challenging task, particularly in specific scenarios and domains.
Multimodal Foundation Models For Echocardiogram Interpretation
Multimodal deep learning foundation models can learn the relationship between images and text.
On the Adversarial Robustness of Multi-Modal Foundation Models
chs20/robustvlm
•
•
In this paper we show that imperceivable attacks on images in order to change the caption output of a multi-modal foundation model can be used by malicious content providers to harm honest users e. g. by guiding them to malicious websites or broadcast fake information.
MultiZoo & MultiBench: A Standardized Toolkit for Multimodal Deep Learning
Learning multimodal representations involves integrating information from multiple heterogeneous sources of data.
Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning
claws-lab/multimodal-robustness-xmai
•
•
The robustness of multimodal deep learning models to realistic changes in the input text is critical for their applicability to important tasks such as text-to-image retrieval and cross-modal entailment.
Towards Balanced Active Learning for Multimodal Classification
Our studies demonstrate that the proposed method achieves more balanced multimodal learning by avoiding greedy sample selection from the dominant modality.
Multimodal Neural Databases
giovannitra/multimodalneuraldatabases
•
•
The rise in loosely-structured data available through text, images, and other modalities has called for new ways of querying them.
Building Multimodal AI Chatbots
Therefore, this work proposes a complete chatbot system using two multimodal deep learning models: an image retriever that understands texts and a response generator that understands images.
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention
We present LLaMA-Adapter, a lightweight adaption method to efficiently fine-tune LLaMA into an instruction-following model.
Zorro: the masked multimodal transformer
Attention-based models are appealing for multimodal processing because inputs from multiple modalities can be concatenated and fed to a single backbone network - thus requiring very little fusion engineering.
