Search Results for author:
Found 75 papers, 36 papers with code
Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback
The rapidly developing Large Vision Language Models (LVLMs) have shown notable capabilities on a range of multi-modal tasks, but still face the hallucination phenomena where the generated texts do not align with the given contexts, significantly restricting the usages of LVLMs.
EVA: Zero-shot Accurate Attributes and Multi-Object Video Editing
We find that the crux of the issue stems from the imprecise distribution of attention weights across designated regions, including inaccurate text-to-attribute control and attention leakage.
Knowledge-Enhanced Dual-stream Zero-shot Composed Image Retrieval
The pseudo-word tokens generated in this stream are explicitly aligned with fine-grained semantics in the text embedding space.
Ghost Sentence: A Tool for Everyday Users to Copyright Data from Large Language Models
These concealed passphrases in user documents, referred to as \textit{ghost sentences}, once they are identified in the generated content of LLMs, users can be sure that their data is used for training.
CapHuman: Capture Your Moments in Parallel Universes
Moreover, we introduce the 3D facial prior to equip our model with control over the human head in a flexible and 3D-consistent manner.
DGL: Dynamic Global-Local Prompt Tuning for Text-Video Retrieval
(2) Equipping the visual and text encoder with separated prompts failed to mitigate the visual-text modality gap.
AntEval: Evaluation of Social Interaction Competencies in LLM-Driven Agents
To address this challenge, we introduce the Multi-Agent Interaction Evaluation Framework (AntEval), encompassing a novel interaction framework and evaluation methods.
FlowZero: Zero-Shot Text-to-Video Synthesis with LLM-Driven Dynamic Scene Syntax
Text-to-video (T2V) generation is a rapidly growing research area that aims to translate the scenes, objects, and actions within complex video text into a sequence of coherent visual frames.
Text Augmented Spatial-aware Zero-shot Referring Image Segmentation
This task aims to identify the instance mask that is most related to a referring expression without training on pixel-level annotations.
Combating Label Noise With A General Surrogate Model For Sample Selection
Sample selection is an effective way to deal with label noise.
IcoCap: Improving Video Captioning by Compounding Images
Video captioning is a more challenging task compared to image captioning, primarily due to differences in content density.
Ranked #5 on
Video Captioning
on VATEX
(using extra training data)
DiverseMotion: Towards Diverse Human Motion Generation via Discrete Diffusion
We present DiverseMotion, a new approach for synthesizing high-quality human motions conditioned on textual descriptions while preserving motion diversity. Despite the recent significant process in text-based human motion generation, existing methods often prioritize fitting training motions at the expense of action diversity.
Ranked #3 on
Motion Synthesis
on HumanML3D
(using extra training data)
Tachikuma: Understading Complex Interactions with Multi-Character and Novel Objects by Large Language Models
MOE challenges models to understand characters' intentions and accurately determine their actions within intricate contexts involving multi-character and novel object interactions.
Co-Learning Meets Stitch-Up for Noisy Multi-label Visual Recognition
In real-world scenarios, collected and annotated data often exhibit the characteristics of multiple classes and long-tailed distribution.
Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models
Given a single test sample, the VLM is forced to maximize the CLIP reward between the input and sampled results from the VLM output distribution.
Whitening-based Contrastive Learning of Sentence Embeddings
Consequently, using multiple positive samples with enhanced diversity further improves contrastive learning due to better alignment.
CLIP4STR: A Simple Baseline for Scene Text Recognition with Pre-trained Vision-Language Model
With such merits, we transform CLIP into a scene text reader and introduce CLIP4STR, a simple yet effective STR method built upon image and text encoders of CLIP.
Ranked #1 on
Scene Text Recognition
on WOST
(using extra training data)
Gloss-Free End-to-End Sign Language Translation
In this paper, we tackle the problem of sign language translation (SLT) without gloss annotations.
Efficient Multimodal Fusion via Interactive Prompting
Large-scale pre-training has brought unimodal fields such as computer vision and natural language processing to a new era.
DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Training
This decoder is both data-efficient and computation-efficient: 1) it only requires the text data for training, easing the burden on the collection of paired data.
Variational Cross-Graph Reasoning and Adaptive Structured Semantics Learning for Compositional Temporal Grounding
To systematically benchmark the compositional generalizability of temporal grounding models, we introduce a new Compositional Temporal Grounding task and construct two new dataset splits, i. e., Charades-CG and ActivityNet-CG.
Temporal Perceiving Video-Language Pre-training
Specifically, text-video localization consists of moment retrieval, which predicts start and end boundaries in videos given the text description, and text localization which matches the subset of texts with the video features.
PointListNet: Deep Learning on 3D Point Lists
Deep neural networks on regular 1D lists (e. g., natural languages) and irregular 3D sets (e. g., point clouds) have made tremendous achievements.
Discriminative Radial Domain Adaptation
Domain adaptation methods reduce domain shift typically by learning domain-invariant features.
MAAL: Multimodality-Aware Autoencoder-Based Affordance Learning for 3D Articulated Objects
Experimental results and visualizations, based on a large-scale dataset PartNet-Mobility, show the effectiveness of MAAL in learning multi-modal data and solving the 3D articulated object affordance problem.
MIST: Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answering
To build Video Question Answering (VideoQA) systems capable of assisting humans in daily activities, seeking answers from long-form videos with diverse and complex events is a must.
Ranked #2 on
Video Question Answering
on AGQA 2.0 balanced
Penalizing the Hard Example But Not Too Much: A Strong Baseline for Fine-Grained Visual Classification
Second, we instantiate the loss function and provide a strong baseline for FGVC, where the performance of a naive backbone can be boosted and be comparable with recent methods.
Ranked #27 on
Fine-Grained Image Classification
on CUB-200-2011
Fine-Grained Image Classification
Fine-Grained Visual Recognition
Slimmable Networks for Contrastive Self-supervised Learning
In this work, we present a one-stage solution to obtain pre-trained small models without the need for extra teachers, namely, slimmable networks for contrastive self-supervised learning (\emph{SlimCLR}).
AFE-CNN: 3D Skeleton-based Action Recognition with Action Feature Enhancement
Existing 3D skeleton-based action recognition approaches reach impressive performance by encoding handcrafted action features to image format and decoding by CNNs.
Fine-Grained Semantically Aligned Vision-Language Pre-Training
Large-scale vision-language pre-training has shown impressive advances in a wide range of downstream tasks.
Dilated Context Integrated Network with Cross-Modal Consensus for Temporal Emotion Localization in Videos
In this paper, we introduce a new task, named Temporal Emotion Localization in videos~(TEL), which aims to detect human emotions and localize their corresponding temporal boundaries in untrimmed videos with aligned subtitles.
PoseGU: 3D Human Pose Estimation with Novel Human Pose Generator and Unbiased Learning
3D pose estimation has recently gained substantial interests in computer vision domain.
Ranked #35 on
3D Human Pose Estimation
on MPI-INF-3DHP
CenterCLIP: Token Clustering for Efficient Text-Video Retrieval
In this paper, to reduce the number of redundant video tokens, we design a multi-segment token clustering algorithm to find the most representative tokens and drop the non-essential ones.
Ranked #11 on
Video Retrieval
on MSVD
(using extra training data)
Unified Transformer Tracker for Object Tracking
Although UniTrack \cite{wang2021different} demonstrates that a shared appearance model with multiple heads can be used to tackle individual tracking tasks, it fails to exploit the large-scale tracking datasets for training and performs poorly on single object tracking.
Compositional Temporal Grounding with Structured Variational Cross-Graph Correspondence Learning
To systematically measure the compositional generalizability of temporal grounding models, we introduce a new Compositional Temporal Grounding task and construct two new dataset splits, i. e., Charades-CG and ActivityNet-CG.
Complex Video Action Reasoning via Learnable Markov Logic Network
The main contributions of this work are two-fold: 1) Different from existing black-box models, the proposed model simultaneously implements the localization of temporal boundaries and the recognition of action categories by grounding the logical rules of MLN in videos.
SEEG: Semantic Energized Co-Speech Gesture Generation
Talking gesture generation is a practical yet challenging task which aims to synthesize gestures in line with speech.
Ranked #6 on
Gesture Generation
on TED Gesture Dataset
A Simple Episodic Linear Probe Improves Visual Recognition in the Wild
In this paper, we propose an episodic linear probing (ELP) classifier to reflect the generalization of visual representations in an online manner.
Ranked #13 on
Fine-Grained Image Classification
on CUB-200-2011
Vector-Decomposed Disentanglement for Domain-Invariant Object Detection
Secondly, domain-specific representations are introduced as the differences between the input and domain-invariant representations.
Adaptive Hierarchical Graph Reasoning with Semantic Coherence for Video-and-Language Inference
Secondly, we introduce semantic coherence learning to explicitly encourage the semantic coherence of the adaptive hierarchical graph network from three hierarchies.
Less is More: Sparse Sampling for Dense Reaction Predictions
Obtaining viewer responses from videos can be useful for creators and streaming platforms to analyze the video performance and improve the future user experience.
OR-Net: Pointwise Relational Inference for Data Completion under Partial Observation
Specifically, we expect to approximate the real joint distribution over the partial observation and latent variables, thus infer the unseen targets respectively.
Faster Meta Update Strategy for Noise-Robust Deep Learning
It has been shown that deep neural networks are prone to overfitting on biased training data.
T2VLAD: Global-Local Sequence Alignment for Text-Video Retrieval
Moreover, a global alignment method is proposed to provide a global cross-modal measurement that is complementary to the local perspective.
Universal-Prototype Enhancing for Few-Shot Object Detection
Thus, we develop a new framework of few-shot object detection with universal prototypes ({FSOD}^{up}) that owns the merit of feature generalization towards novel objects.
Ranked #23 on
Few-Shot Object Detection
on MS-COCO (10-shot)
Learning to Anticipate Egocentric Actions by Imagination
We further improve ImagineRNN by residual anticipation, i. e., changing its target to predicting the feature difference of adjacent frames instead of the frame content.
Asynchronous Modeling: A Dual-phase Perspective for Long-Tailed Recognition
Motivated by such phenomenon, we propose to disentangle the distinctive effects of data-rich and data-poor gradient and asynchronously train a model via a dual-phase learning process.
Interactive Prototype Learning for Egocentric Action Recognition
To avoid these additional costs, we propose an end-to-end Interactive Prototype Learning (IPL) framework to learn better active object representations by leveraging the motion cues from the actor.
A Multi-Mode Modulator for Multi-Domain Few-Shot Classification
Most existing few-shot classification methods only consider generalization on one dataset (i. e., single-domain), failing to transfer across various seen and unseen domains.
Feature-Robust Optimal Transport for High-Dimensional Data
To show the effectiveness of FROT, we propose using the FROT algorithm for the layer selection problem in deep neural networks for semantic correspondence.
SemGloVe: Semantic Co-occurrences for GloVe from BERT
In this paper, we propose SemGloVe, which distills semantic co-occurrences from BERT into static GloVe word embeddings.
ActBERT: Learning Global-Local Video-Text Representations
In this paper, we introduce ActBERT for self-supervised learning of joint video-text representations from unlabeled data.
Ranked #8 on
Action Segmentation
on COIN
Inflated Episodic Memory With Region Self-Attention for Long-Tailed Visual Recognition
It is beneficial to incorporate more discriminative features to improve generalization on tail classes.
Ranked #16 on
Long-tail Learning
on Places-LT
Feature Robust Optimal Transport for High-dimensional Data
To show the effectiveness of FROT, we propose using the FROT algorithm for the layer selection problem in deep neural networks for semantic correspondence.
OpenMix: Reviving Known Knowledge for Discovering Novel Visual Categories in An Open World
In this paper, we tackle the problem of discovering new classes in unlabeled visual data given labeled data from disjoint classes.
Motion-Excited Sampler: Video Adversarial Attack with Sparked Prior
Most of previous work on adversarial attack mainly focus on image models, while the vulnerability of video models is less explored.
SF-Net: Single-Frame Supervision for Temporal Action Localization
To obtain the single-frame supervision, the annotators are asked to identify only a single frame within the temporal window of an action.
Ranked #5 on
Weakly Supervised Action Localization
on BEOID
Symbiotic Attention with Privileged Information for Egocentric Action Recognition
Due to the large action vocabulary in egocentric video datasets, recent studies usually utilize a two-branch structure for action recognition, ie, one branch for verb classification and the other branch for noun classification.
Ranked #4 on
Egocentric Activity Recognition
on EGTEA
Connective Cognition Network for Directional Visual Commonsense Reasoning
Inspired by this idea, towards VCR, we propose a connective cognition network (CCN) to dynamically reorganize the visual neuron connectivity that is contextualized by the meaning of questions and answers.
Instance-Invariant Domain Adaptive Object Detection via Progressive Disentanglement
Most state-of-the-art methods of object detection suffer from poor generalization ability when the training and test data are from different domains, e. g., with different styles.
Gated Channel Transformation for Visual Recognition
This lightweight layer incorporates a simple l2 normalization, enabling our transformation unit applicable to operator-level without much increase of additional parameters.
Learning to Transfer Learn: Reinforcement Learning-Based Selection for Adaptive Transfer Learning
We propose a novel adaptive transfer learning framework, learning to transfer learn (L2TL), to improve performance on a target dataset by careful extraction of the related information from a source dataset.
Baidu-UTS Submission to the EPIC-Kitchens Action Recognition Challenge 2019
In this report, we present the Baidu-UTS submission to the EPIC-Kitchens Action Recognition Challenge in CVPR 2019.
FASTER Recurrent Networks for Efficient Video Classification
FASTER aims to leverage the redundancy between neighboring clips and reduce the computational cost by learning to aggregate the predictions from models of different complexities.
Ranked #26 on
Action Recognition
on UCF101
Cubic LSTMs for Video Prediction
Predicting future frames in videos has become a promising direction of research for both computer vision and robot learning communities.
Sim-Real Joint Reinforcement Transfer for 3D Indoor Navigation
Specifically, our method employs an adversarial feature adaptation model for visual representation transfer and a policy mimic strategy for policy behavior imitation.
Filter Pruning by Switching to Neighboring CNNs with Good Attributes
In addition, when evaluating the filter importance, only the magnitude information of the filters is considered.
Auto-ReID: Searching for a Part-aware ConvNet for Person Re-Identification
We propose to automatically search for a CNN architecture that is specifically suitable for the reID task.
Ranked #9 on
Person Re-Identification
on CUHK03 detected
Compound Memory Networks for Few-shot Video Classification
In this paper, we propose a new memory network structure for few-shot video classification by making the following contributions.
Attentive Sequence to Sequence Translation for Localizing Clips of Interest by Natural Language Descriptions
We validate the effectiveness of our ASST on two large-scale datasets.
Decoupled Novel Object Captioner
Thus, the sequence model can be decoupled from the novel object descriptions.
UTS submission to Google YouTube-8M Challenge 2017
In this paper, we present our solution to Google YouTube-8M Video Classification Challenge 2017.
Few-Shot Object Recognition from Machine-Labeled Web Images
Then, we demonstrate that with our model, machine-labeled image annotations are very effective and abundant resources to perform object recognition on novel categories.
Bidirectional Multirate Reconstruction for Temporal Modeling in Videos
This learning process makes the learned model more capable of dealing with motion speed variance.
Uncovering Temporal Context for Video Question and Answering
In this work, we introduce Video Question Answering in temporal domain to infer the past, describe the present and predict the future.
