Search Results for author:
Found 13 papers, 5 papers with code
LVLM-Intrepret: An Interpretability Tool for Large Vision-Language Models
In this work, we present a novel interactive application aimed towards understanding the internal mechanisms of large vision-language models.
LLaVA-Gemma: Accelerating Multimodal Foundation Models with a Compact Language Model
We train a suite of multimodal foundation models (MMFM) using the popular LLaVA framework with the recently released Gemma family of large language models (LLMs).
LDM3D-VR: Latent Diffusion Model for 3D VR
Our models are fine-tuned from existing pretrained models on datasets containing panoramic/high-resolution RGB images, depth maps and captions.
ManagerTower: Aggregating the Insights of Uni-Modal Experts for Vision-Language Representation Learning
With only 4M VLP data, ManagerTower achieves superior performances on various downstream VL tasks, especially 79. 15% accuracy on VQAv2 Test-Std, 86. 56% IR@1 and 95. 64% TR@1 on Flickr30K.
LDM3D: Latent Diffusion Model for 3D
This research paper proposes a Latent Diffusion Model for 3D (LDM3D) that generates both image and depth map data from a given text prompt, allowing users to generate RGBD images from text prompts.
MuMUR : Multilingual Multimodal Universal Retrieval
In this paper, we propose a framework MuMUR, that utilizes knowledge transfer from a multilingual model to boost the performance of multi-modal (image and video) retrieval.
VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformers
Breakthroughs in transformer-based models have revolutionized not only the NLP field, but also vision and multimodal systems.
CALM: Contrastive Aligned Audio-Language Multirate and Multimodal Representations
By aligning audio representations to pretrained language representations and utilizing contrastive information between acoustic inputs, CALM is able to bootstrap audio embedding competitive with existing audio representation models in only a few hours of training time.
KD-VLP: Improving End-to-End Vision-and-Language Pretraining with Object Knowledge Distillation
Self-supervised vision-and-language pretraining (VLP) aims to learn transferable multi-modal representations from large-scale image-text data and to achieve strong performances on a broad scope of vision-language tasks after finetuning.
Multimodal Embeddings from Language Models
Word embeddings such as ELMo have recently been shown to model word semantics with greater efficacy through contextualized learning on large-scale language corpora, resulting in significant improvement in state of the art across many natural language tasks.
Behavior Gated Language Models
In this work we derive motivation from psycholinguistics and propose the addition of behavioral information into the context of language modeling.
Predicting Behavior in Cancer-Afflicted Patient and Spouse Interactions using Speech and Language
Cancer impacts the quality of life of those diagnosed as well as their spouse caregivers, in addition to potentially influencing their day-to-day behaviors.
Unsupervised Online Multitask Learning of Behavioral Sentence Embeddings
Unsupervised learning has been an attractive method for easily deriving meaningful data representations from vast amounts of unlabeled data.
