Search Results for author:
Found 12 papers, 5 papers with code
FGAIF: Aligning Large Vision-Language Models with Fine-grained AI Feedback
To handle these limitations, we propose an innovative method to align modalities in LVLMs through Fine-Grained Artificial Intelligence Feedback (FGAIF), which mainly consists of three steps: AI-based Feedback Collection, Fine-grained Reward Model Training, and Reinforcement Learning with Fine-grained Reward.
Fine-grained and Explainable Factuality Evaluation for Multimodal Summarization
To evaluate the factuality of multimodal summarization models, we propose two fine-grained and explainable evaluation frameworks (FALLACIOUS) for different application scenarios, i. e. reference-based factuality evaluation framework and reference-free factuality evaluation framework.
Sentiment-enhanced Graph-based Sarcasm Explanation in Dialogue
Although existing studies have achieved great success based on the generative pretrained language model BART, they overlook exploiting the sentiments residing in the utterance, video and audio, which are vital clues for sarcasm explanation.
Debiasing Multimodal Sarcasm Detection with Contrastive Learning
Moreover, we propose a novel debiasing multimodal sarcasm detection framework with contrastive learning, which aims to mitigate the harmful effect of biased textual factors for robust OOD generalization.
VK-G2T: Vision and Context Knowledge enhanced Gloss2Text
Existing sign language translation methods follow a two-stage pipeline: first converting the sign language video to a gloss sequence (i. e. Sign2Gloss) and then translating the generated gloss sequence into a spoken language sentence (i. e. Gloss2Text).
FAITHSCORE: Evaluating Hallucinations in Large Vision-Language Models
We introduce FAITHSCORE (Faithfulness to Atomic Image Facts Score), a reference-free and fine-grained evaluation metric that measures the faithfulness of the generated free-form answers from large vision-language models (LVLMs).
Knowledge-enhanced Memory Model for Emotional Support Conversation
The prevalence of mental disorders has become a significant issue, leading to the increased focus on Emotional Support Conversation as an effective supplement for mental health support.
General Debiasing for Multimodal Sentiment Analysis
To this end, we propose a general debiasing framework based on Inverse Probability Weighting (IPW), which adaptively assigns small weights to the samples with larger bias (i. e., the severer spurious correlations).
Multi-source Semantic Graph-based Multimodal Sarcasm Explanation Generation
Multimodal Sarcasm Explanation (MuSE) is a new yet challenging task, which aims to generate a natural language sentence for a multimodal social post (an image as well as its caption) to explain why it contains sarcasm.
Stylized Data-to-Text Generation: A Case Study in the E-Commerce Domain
This task is non-trivial, due to three challenges: the logic of the generated text, unstructured style reference, and biased training samples.
Counterfactual Reasoning for Out-of-distribution Multimodal Sentiment Analysis
Inspired by this, we devise a model-agnostic counterfactual framework for multimodal sentiment analysis, which captures the direct effect of textual modality via an extra text model and estimates the indirect one by a multimodal model.
Multimodal Dialog Systems with Dual Knowledge-enhanced Generative Pretrained Language Model
To address these limitations, we propose a novel dual knowledge-enhanced generative pretrained language model for multimodal task-oriented dialog systems (DKMD), consisting of three key components: dual knowledge selection, dual knowledge-enhanced context learning, and knowledge-enhanced response generation.
