Search Results for author:
Found 120 papers, 68 papers with code
MMCoQA: Conversational Question Answering over Text, Tables, and Images
In this paper, we hence define a novel research task, i. e., multimodal conversational question answering (MMCoQA), aiming to answer users’ questions with multimodal knowledge sources via multi-turn conversations.
MMGRec: Multimodal Generative Recommendation with Transformer Model
Multimodal recommendation aims to recommend user-preferred candidates based on her/his historically interacted items and associated multimodal information.
Dynamic in Static: Hybrid Visual Correspondence for Self-Supervised Video Object Segmentation
To achieve this objective, we present a unified self-supervised approach to learn visual representations of static-dynamic feature similarity.
FecTek: Enhancing Term Weight in Lexicon-Based Retrieval with Feature Context and Term-level Knowledge
To effectively enrich the feature context representations of term weight, the Feature Context Module (FCM) is introduced, which leverages the power of BERT's representation to determine dynamic weights for each element in the embedding.
Cluster-based Graph Collaborative Filtering
This model performs high-order graph convolution on cluster-specific graphs, which are constructed by capturing the multiple interests of users and identifying the common interests among them.
LLMvsSmall Model? Large Language Model Based Text Augmentation Enhanced Personality Detection Model
Furthermore, we utilize the LLM to enrich the information of personality labels for enhancing the detection performance.
Discriminative Probing and Tuning for Text-to-Image Generation
We present a discriminative adapter built on T2I models to probe their discriminative abilities on two representative tasks and leverage discriminative fine-tuning to improve their text-image alignment.
WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains More
Large Language Models (LLMs) face significant deployment challenges due to their substantial memory requirements and the computational demands of auto-regressive text generation process.
Interactive Garment Recommendation with User in the Loop
Recommending fashion items often leverages rich user profiles and makes targeted suggestions based on past history and previous purchases.
Generative Cross-Modal Retrieval: Memorizing Images in Multimodal Language Models for Retrieval and Beyond
Building upon this capability, we propose to enable multimodal large language models (MLLMs) to memorize and recall images within their parameters.
Distillation Enhanced Generative Retrieval
Generative retrieval is a promising new paradigm in text retrieval that generates identifier strings of relevant passages as the retrieval target.
Sentiment-enhanced Graph-based Sarcasm Explanation in Dialogue
Although existing studies have achieved great success based on the generative pretrained language model BART, they overlook exploiting the sentiments residing in the utterance, video and audio, which are vital clues for sarcasm explanation.
GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding
Speculative decoding is a relatively new decoding framework that leverages small and efficient draft models to reduce the latency of LLMs.
Diffusion Facial Forgery Detection
In particular, this dataset leverages 30, 000 carefully collected textual and visual prompts, ensuring the synthesis of images with both high fidelity and semantic consistency.
Spatial Structure Constraints for Weakly Supervised Semantic Segmentation
In this paper, we propose spatial structure constraints (SSC) for weakly supervised semantic segmentation to alleviate the unwanted object over-activation of attention expansion.
Enhancing Emotional Generation Capability of Large Language Models via Emotional Chain-of-Thought
In this paper, we propose the Emotional Chain-of-Thought (ECoT), a plug-and-play prompting method that enhances the performance of LLMs on various emotional generation tasks by aligning with human emotional intelligence guidelines.
Understanding Before Recommendation: Semantic Aspect-Aware Review Exploitation via Large Language Models
Recommendation systems harness user-item interactions like clicks and reviews to learn their representations.
Attribute-driven Disentangled Representation Learning for Multimodal Recommendation
To obtain robust and independent representations for each factor associated with a specific attribute, we first disentangle the representations of features both within and across different modalities.
VK-G2T: Vision and Context Knowledge enhanced Gloss2Text
Existing sign language translation methods follow a two-stage pipeline: first converting the sign language video to a gloss sequence (i. e. Sign2Gloss) and then translating the generated gloss sequence into a spoken language sentence (i. e. Gloss2Text).
Unsupervised Temporal Action Localization via Self-paced Incremental Learning
Thereafter, we design two (constant- and variable- speed) incremental instance learning strategies for easy-to-hard model training, thus ensuring the reliability of these video pseudolabels and further improving overall localization performance.
GaussianAvatar: Towards Realistic Human Avatar Modeling from a Single Video via Animatable 3D Gaussians
We present GaussianAvatar, an efficient approach to creating realistic human avatars with dynamic 3D appearances from a single video.
GPS-Gaussian: Generalizable Pixel-wise 3D Gaussian Splatting for Real-time Human Novel View Synthesis
We present a new approach, termed GPS-Gaussian, for synthesizing novel views of a character in a real-time manner.
RTQ: Rethinking Video-language Understanding Based on Image-text Model
Recent advancements in video-language understanding have been established on the foundation of image-text models, resulting in promising outcomes due to the shared knowledge between images and videos.
Ranked #9 on
Video Retrieval
on MSR-VTT-1kA
Generating Human-Centric Visual Cues for Human-Object Interaction Detection via Large Vision-Language Models
To capitalize on these rich Human-Centric Visual Cues, we propose a novel approach named HCVC for HOI detection.
LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge
1) Progressive incorporation of fine-grained spatial-aware visual knowledge.
An Empirical Study of Frame Selection for Text-to-Video Retrieval
In this paper, we make the first empirical study of frame selection for TVR.
UNK-VQA: A Dataset and a Probe into the Abstention Ability of Multi-modal Large Models
Teaching Visual Question Answering (VQA) models to refrain from answering unanswerable questions is necessary for building a trustworthy AI system.
Uncovering Hidden Connections: Iterative Tracking and Reasoning for Video-grounded Dialog
In response to this gap, we present an iterative tracking and reasoning strategy that amalgamates a textual encoder, a visual encoder, and a generator.
ELIP: Efficient Language-Image Pre-training with Fewer Vision Tokens
Learning a versatile language-image model is computationally prohibitive under a limited computing budget.
Detecting and Grounding Multi-Modal Media Manipulation and Beyond
HAMMER performs 1) manipulation-aware contrastive learning between two uni-modal encoders as shallow manipulation reasoning, and 2) modality-aware cross-attention by multi-modal aggregator as deep manipulation reasoning.
Building Emotional Support Chatbots in the Era of LLMs
The integration of emotional support into various conversational scenarios presents profound societal benefits, such as social interactions, mental health counseling, and customer service.
Temporal Sentence Grounding in Streaming Videos
The goal of TSGSV is to evaluate the relevance between a video stream and a given sentence query.
LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generation
Afterward, we propose a fine-grained object-interaction diffusion method to synthesize high-faithfulness images conditioned on the prompt and the automatically generated layout.
Semantic-Guided Feature Distillation for Multimodal Recommendation
The teacher model first extracts rich modality features from the generic modality feature by considering both the semantic information of items and the complementary information of multiple modalities.
StyleEDL: Style-Guided High-order Attention Network for Image Emotion Distribution Learning
Emotion distribution learning has gained increasing attention with the tendency to express emotions through images.
Sample Less, Learn More: Efficient Action Recognition via Frame Feature Restoration
Training an effective video action recognition model poses significant computational challenges, particularly under limited resource budgets.
Towards Generalizable Deepfake Detection by Primary Region Regularization
The existing deepfake detection methods have reached a bottleneck in generalizing to unseen forgeries and manipulation approaches.
General Debiasing for Multimodal Sentiment Analysis
To this end, we propose a general debiasing framework based on Inverse Probability Weighting (IPW), which adaptively assigns small weights to the samples with larger bias (i. e., the severer spurious correlations).
LightGT: A Light Graph Transformer for Multimedia Recommendation
Considering its challenges in effectiveness and efficiency, we propose a novel Transformer-based recommendation model, termed as Light Graph Transformer model (LightGT).
Ranked #1 on
Multi-Media Recommendation
on Kwai
(Recall@10 metric)
Multi-source Semantic Graph-based Multimodal Sarcasm Explanation Generation
Multimodal Sarcasm Explanation (MuSE) is a new yet challenging task, which aims to generate a natural language sentence for a multimodal social post (an image as well as its caption) to explain why it contains sarcasm.
A Survey on Video Moment Localization
Video moment localization, also known as video moment retrieval, aiming to search a target segment within a video described by a given natural language query.
Learning Geometric Transformation for Point Cloud Completion
It exploits the repetitive geometric structures in common 3D objects to recover the complete shapes, which contains three sub-networks: geometric patch network, structure transformation network, and detail refinement network.
Ranked #4 on
Point Cloud Completion
on ShapeNet
DeepFake-Adapter: Dual-Level Adapter for DeepFake Detection
Unlike existing deepfake detection methods merely focusing on low-level forgery patterns, the forgery detection process of our model can be regularized by generalizable high-level semantics from a pre-trained ViT and adapted by global and local low-level forgeries of deepfake data.
RaSa: Relation and Sensitivity Aware Representation Learning for Text-based Person Search
RA offsets the overfitting risk by introducing a novel positive relation detection task (i. e., learning to distinguish strong and weak positive pairs).
Ranked #2 on
Text based Person Retrieval
on RSTPReid
Text-based Person Search without Parallel Image-Text Data
Text-based person search (TBPS) aims to retrieve the images of the target person from a large image gallery based on a given natural language description.
Dual Semantic Knowledge Composed Multimodal Dialog Systems
Thereafter, considering that the attribute knowledge and relation knowledge can benefit the responding to different levels of questions, we design a multi-level knowledge composition module in MDS-S2 to obtain the latent composed response representation.
Stylized Data-to-Text Generation: A Case Study in the E-Commerce Domain
This task is non-trivial, due to three challenges: the logic of the generated text, unstructured style reference, and biased training samples.
Learnable Pillar-based Re-ranking for Image-Text Retrieval
Image-text retrieval aims to bridge the modality gap and retrieve cross-modal content based on semantic similarities.
ChatLLM Network: More brains, More intelligence
Dialogue-based language models mark a huge milestone in the field of artificial intelligence, by their impressive ability to interact with users, as well as a series of challenging tasks prompted by customized instructions.
Rethinking Context Aggregation in Natural Image Matting
For natural image matting, context information plays a crucial role in estimating alpha mattes especially when it is challenging to distinguish foreground from its background.
Ranked #1 on
Image Matting
on Composition-1K
Learning Reliable Representations for Incomplete Multi-View Partial Multi-Label Classification
The application of multi-view contrastive learning has further facilitated this process, however, the existing multi-view contrastive learning methods crudely separate the so-called negative pair, which largely results in the separation of samples belonging to the same category or similar ones.
Micro-video Tagging via Jointly Modeling Social Influence and Tag Relation
Existing methods mostly focus on analyzing video content, neglecting users' social influence and tag relation.
Efficient Image-Text Retrieval via Keyword-Guided Pre-Screening
The proposed framework equipped with only two embedding layers achieves $O(1)$ querying time complexity, while improving the retrieval efficiency and keeping its performance, when applied prior to the common image-text retrieval methods.
Deep Learning and Medical Imaging for COVID-19 Diagnosis: A Comprehensive Survey
In this survey, we investigate the main contributions of deep learning applications using medical images in fighting against COVID-19 from the aspects of image classification, lesion localization, and severity quantification, and review different deep learning architectures and some image preprocessing techniques for achieving a preciser diagnosis.
Learning to Agree on Vision Attention for Visual Commonsense Reasoning
Visual Commonsense Reasoning (VCR) remains a significant yet challenging research problem in the realm of visual reasoning.
HS-GCN: Hamming Spatial Graph Convolutional Networks for Recommendation
Towards this end, existing methods tend to code users by modeling their Hamming similarities with the items they historically interact with, which are termed as the first-order similarities in this work.
CNVid-3.5M: Build, Filter, and Pre-Train the Large-Scale Public Chinese Video-Text Dataset
Though there are certain methods studying the Chinese video-text pre-training, they pre-train their models on private datasets whose videos and text are unavailable.
CHMATCH: Contrastive Hierarchical Matching and Robust Adaptive Threshold Boosted Semi-Supervised Learning
In the meantime, we make full use of the structured information in the hierarchical labels to learn an accurate affinity graph for contrastive learning.
Multi-queue Momentum Contrast for Microvideo-Product Retrieval
The booming development and huge market of micro-videos bring new e-commerce channels for merchants.
Causal Inference for Knowledge Graph based Recommendation
Knowledge Graph (KG), as a side-information, tends to be utilized to supplement the collaborative filtering (CF) based recommendation model.
Multimodal Matching-aware Co-attention Networks with Mutual Knowledge Distillation for Fake News Detection
Additionally, based on the designed image-text matching-aware co-attention mechanism, we propose to build two co-attention networks respectively centered on text and image for mutual knowledge distillation to improve fake news detection.
A Survey of Knowledge Enhanced Pre-trained Language Models
We introduce appropriate taxonomies respectively for Natural Language Understanding (NLU) and Natural Language Generation (NLG) to highlight these two main tasks of NLP.
Privacy-Preserving Synthetic Data Generation for Recommendation Systems
At the item level, a synthetic data generation module is proposed to generate a synthetic item corresponding to the selected item based on the user's preferences.
Deep Convolutional Pooling Transformer for Deepfake Detection
Most existing deep learning methods mainly focus on local features and relations within the face image using convolutional neural networks as a backbone.
Visual Perturbation-aware Collaborative Learning for Overcoming the Language Prior Problem
Several studies have recently pointed that existing Visual Question Answering (VQA) models heavily suffer from the language prior problem, which refers to capturing superficial statistical correlations between the question type and the answer whereas ignoring the image contents.
Counterfactual Reasoning for Out-of-distribution Multimodal Sentiment Analysis
Inspired by this, we devise a model-agnostic counterfactual framework for multimodal sentiment analysis, which captures the direct effect of textual modality via an extra text model and estimates the indirect one by a multimodal model.
Semantic-aware Modular Capsule Routing for Visual Question Answering
Visual Question Answering (VQA) is fundamentally compositional in nature, and many questions are simply answered by decomposing them into modular sub-problems.
Multimodal Dialog Systems with Dual Knowledge-enhanced Generative Pretrained Language Model
To address these limitations, we propose a novel dual knowledge-enhanced generative pretrained language model for multimodal task-oriented dialog systems (DKMD), consisting of three key components: dual knowledge selection, dual knowledge-enhanced context learning, and knowledge-enhanced response generation.
Lipschitz Continuity Retained Binary Neural Network
Relying on the premise that the performance of a binary neural network can be largely restored with eliminated quantization error between full-precision weight vectors and their corresponding binary vectors, existing works of network binarization frequently adopt the idea of model robustness to reach the aforementioned objective.
Network Binarization via Contrastive Learning
Neural network binarization accelerates deep models by quantizing their weights and activations into 1-bit.
A Unified End-to-End Retriever-Reader Framework for Knowledge-based VQA
On the other hand, pertaining to the implicit knowledge, the multi-modal implicit knowledge for knowledge-based VQA still remains largely unexplored.
Image-text Retrieval: A Survey on Recent Research and Development
On top of this, the efficiency-focused study on the ITR system is introduced as the third perspective.
Stacked Hybrid-Attention and Group Collaborative Learning for Unbiased Scene Graph Generation
Scene Graph Generation, which generally follows a regular encoder-decoder pipeline, aims to first encode the visual contents within the given image and then parse them into a compact summary graph.
Ranked #1 on
Unbiased Scene Graph Generation
on Visual Genome
(mR@20 metric)
Disentangled Multimodal Representation Learning for Recommendation
However, existing methods ignore the fact that different modalities contribute differently towards a user's preference on various factors of an item.
Voice-Face Homogeneity Tells Deepfake
To this end, a voice-face matching method is devised to measure the matching degree of these two.
MERIt: Meta-Path Guided Contrastive Learning for Logical Reasoning
Logical reasoning is of vital importance to natural language understanding.
Ranked #3 on
Reading Comprehension
on ReClor
On Modality Bias Recognition and Reduction
From the results on four datasets regarding the above three tasks, our method yields remarkable performance improvements compared with the baselines, demonstrating its superiority on reducing the modality bias problem.
Joint Answering and Explanation for Visual Commonsense Reasoning
Given that our framework is model-agnostic, we apply it to the existing popular baselines and validate its effectiveness on the benchmark dataset.
Win the Lottery Ticket via Fourier Analysis: Frequencies Guided Network Pruning
Extensive experiments on CIFAR-10 and CIFAR-100 demonstrate the superiority of our novel Fourier analysis based MBP compared to other traditional MBP algorithms.
Learning Robust Recommender from Noisy Implicit Feedback
Inspired by this observation, we propose a new training strategy named Adaptive Denoising Training (ADT), which adaptively prunes the noisy interactions by two paradigms (i. e., Truncated Loss and Reweighted Loss).
Hierarchical Deep Residual Reasoning for Temporal Moment Localization
Temporal Moment Localization (TML) in untrimmed videos is a challenging task in the field of multimedia, which aims at localizing the start and end points of the activity in the video, described by a sentence query.
Multi-Modal Interaction Graph Convolutional Network for Temporal Language Localization in Videos
Towards this end, in this work, we propose a Multi-modal Interaction Graph Convolutional Network (MIGCN), which jointly explores the complex intra-modal relations and inter-modal interactions residing in the video and sentence query to facilitate the understanding and semantic correspondence capture of the video and sentence query.
Contrastive Mutual Information Maximization for Binary Neural Networks
Neural network binarization accelerates deep models by quantizing their weights and activations into 1-bit.
Lipschitz Continuity Guided Knowledge Distillation
Knowledge distillation has become one of the most important model compression techniques by distilling knowledge from larger teacher networks to smaller student ones.
When Product Search Meets Collaborative Filtering: A Hierarchical Heterogeneous Graph Neural Network Approach
In addition, we empirically prove that collaborative filtering and semantic matching are complementary to each other in product search performance enhancement.
Contrastive Learning for Cold-Start Recommendation
It aims to maximize the mutual dependencies between item content and collaborative signals.
Dynamic Modality Interaction Modeling for Image-Text Retrieval
To address these issues, we develop a novel modality interaction modeling network based upon the routing mechanism, which is the first unified and dynamic multimodal interaction framework towards image-text retrieval.
Review Polarity-wise Recommender
To be specific, in this model, positive and negative reviews are separately gathered and utilized to model the user-preferred and user-rejected aspects, respectively.
REPT: Bridging Language Models and Machine Reading Comprehension via Retrieval-Based Pre-training
Pre-trained Language Models (PLMs) have achieved great success on Machine Reading Comprehension (MRC) over the past few years.
AdaVQA: Overcoming Language Priors with Adapted Margin Cosine Loss
Experimental results demonstrate that our adapted margin cosine loss can greatly enhance the baseline models with an absolute performance gain of 15\% on average, strongly verifying the potential of tackling the language prior problem in VQA from the angle of the answer feature space learning.
A Graph-guided Multi-round Retrieval Method for Conversational Open-domain Question Answering
Moreover, in order to collect more complementary information in the historical context, we also propose to incorporate the multi-round relevance feedback technique to explore the impact of the retrieval context on current question understanding.
Conversational Question Answering
Open-Domain Question Answering
+1
Graph Contrastive Clustering
On the other hand, a novel graph-based contrastive learning strategy is proposed to learn more compact clustering assignments.
Feature-level Attentive ICF for Recommendation
To demonstrate the effectiveness of our method, we design a light attention neural network to integrate both item-level and feature-level attention for neural ICF models.
Interest-aware Message-Passing GCN for Recommendation
To form the subgraphs, we design an unsupervised subgraph generation module, which can effectively identify users with common interests by exploiting both user feature and graph structure.
Answer Questions with Right Image Regions: A Visual Attention Regularization Approach
However, recent studies have pointed out that the highlighted image regions from the visual attention are often irrelevant to the given question and answer, leading to model confusion for correct visual reasoning.
Incremental Knowledge Based Question Answering
In the past years, Knowledge-Based Question Answering (KBQA), which aims to answer natural language questions using facts in a knowledge base, has been well developed.
Market2Dish: Health-aware Food Recommendation
With the rising incidence of some diseases, such as obesity and diabetes, a healthy diet is arousing increasing attention.
Loss re-scaling VQA: Revisiting the LanguagePrior Problem from a Class-imbalance View
Concretely, we design a novel interpretation scheme whereby the loss of mis-predicted frequent and sparse answers of the same question type is distinctly exhibited during the late training phase.
Enhancing Factorization Machines with Generalized Metric Learning
Traditional FMs adopt the inner product to model the second-order interactions between different attributes, which are represented via feature vectors.
Denoising Implicit Feedback for Recommendation
In this work, we explore the central theme of denoising implicit feedback for recommender training.
A^2-GCN: An Attribute-aware Attentive GCN Model for Recommendation
Considering the fact that for different users, the attributes of an item have different influence on their preference for this item, we design a novel attention mechanism to filter the message passed from an item to a target user by considering the attribute information.
Improving Distantly-Supervised Relation Extraction with Joint Label Embedding
Distantly-supervised relation extraction has proven to be effective to find relational facts from texts.
MMGCN: Multi-modal Graph Convolution Network for Personalized Recommendation of Micro-video
Existing works on multimedia recommendation largely exploit multi-modal contents to enrich item representations, while less effort is made to leverage information interchange between users and items to enhance user representations and further capture user's fine-grained preferences on different modalities.
Ranked #1 on
Multi-Media Recommendation
on MovieLens 10M
Personalized Hashtag Recommendation for Micro-videos
The hashtags, that a user provides to a post (e. g., a micro-video), are the ones which in her mind can well describe the post content where she is interested in.
User Diverse Preference Modeling by Multimodal Attentive Metric Learning
To tackle this problem, in this paper, we propose a novel Multimodal Attentive Metric Learning (MAML) method to model user diverse preferences for various items.
Quantifying and Alleviating the Language Prior Problem in Visual Question Answering
Benefiting from the advancement of computer vision, natural language processing and information retrieval techniques, visual question answering (VQA), which aims to answer questions about an image or a video, has received lots of attentions over the past few years.
Explicit Interaction Model towards Text Classification
To address this problem, we introduce the interaction mechanism to incorporate word-level matching signals into the text classification task.
Ranked #4 on
Text Classification
on Yahoo! Answers
Learning to Ask Questions in Open-domain Conversational Systems with Typed Decoders
Asking good questions in large-scale, open-domain conversational systems is quite significant yet rather untouched.
Discrete Factorization Machines for Fast Feature-based Recommendation
In this paper, we develop a generic feature-based recommendation model, called Discrete Factorization Machine (DFM), for fast and accurate recommendation.
Neural Compatibility Modeling with Attentive Knowledge Distillation
Nevertheless, existing studies overlook the rich valuable knowledge (rules) accumulated in fashion domain, especially the rules regarding clothing matching.
Neural Collaborative Filtering
When it comes to model the key factor in collaborative filtering -- the interaction between user and item features, they still resorted to matrix factorization and applied an inner product on the latent features of users and items.
Laplacian-Steered Neural Style Transfer
However in the traditional optimization objective, low-level features of the content image are absent, and the low-level features of the style image dominate the low-level detail structures of the new image.
Item Silk Road: Recommending Items from Information Domains to Social Users
In this work, we address the problem of cross-domain social recommendation, i. e., recommending relevant items of information domains to potential users of social networks.
Ranked #2 on
Recommendation Systems
on WeChat
Supervised Deep Hashing for Hierarchical Labeled Data
Specifically, we define a novel similarity formula for hierarchical labeled data by weighting each layer, and design a deep convolutional neural network to obtain a hash code for each data point.
Simple to Complex Cross-modal Learning to Rank
The heterogeneity-gap between different modalities brings a significant challenge to multimedia information retrieval.
SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioning
Existing visual attention models are generally spatial, i. e., the attention is modeled as spatial probabilities that re-weight the last conv-layer feature map of a CNN encoding an input image.
Action2Activity: Recognizing Complex Activities from Sensor Data
As compared to simple actions, activities are much more complex, but semantically consistent with a human's real life.
