Search Results for author:
Found 35 papers, 15 papers with code
Debiasing Event Understanding for Visual Commonsense Tasks
We study event understanding as a critical step towards visual commonsense tasks. Meanwhile, we argue that current object-based event understanding is purely likelihood-based, leading to incorrect event prediction, due to biased correlation between events and objects. We propose to mitigate such biases with do-calculus, proposed in causality research, but overcoming its limited robustness, by an optimized aggregation with association-based prediction. We show the effectiveness of our approach, intrinsically by comparing our generated events with ground-truth event annotation, and extrinsically by downstream commonsense tasks.
One-Shot Sensitivity-Aware Mixed Sparsity Pruning for Large Language Models
Various Large Language Models~(LLMs) from the Generative Pretrained Transformer(GPT) family have achieved outstanding performances in a wide range of text generation tasks.
ViCo: Engaging Video Comment Generation with Human Preference Rewards
Experiments on ViCo-20k show that the comments generated by our ViCo model exhibit the best performance in terms of both quantitative and qualitative results, particularly when engagement is considered.
Improving Diversity in Zero-Shot GAN Adaptation with Semantic Variations
Due to the data absence, the textual description of the target domain and the vision-language models, e. g., CLIP, are utilized to effectively guide the generator.
Revisiting Latent Space of GAN Inversion for Real Image Editing
The exploration of the latent space in StyleGANs and GAN inversion exemplify impressive real-world image editing, yet the trade-off between reconstruction quality and editing quality remains an open problem.
SINC: Self-Supervised In-Context Learning for Vision-Language Tasks
To this end, we raise a question: ``How can we enable in-context learning without relying on the intrinsic in-context ability of large language models?".
Transferring Foundation Models for Generalizable Robotic Manipulation
In this paper, we propose a novel paradigm that effectively leverages language-reasoning segmentation mask generated by internet-scale foundation models, to condition robot manipulation tasks.
Balancing Reconstruction and Editing Quality of GAN Inversion for Real Image Editing with StyleGAN Prior Latent Space
The exploration of the latent space in StyleGANs and GAN inversion exemplify impressive real-world image editing, yet the trade-off between reconstruction quality and editing quality remains an open problem.
AlphaBlock: Embodied Finetuning for Vision-Language Reasoning in Robot Manipulation
We propose a novel framework for learning high-level cognitive capabilities in robot manipulation tasks, such as making a smiley face using building blocks.
Whisper-KDQ: A Lightweight Whisper via Guided Knowledge Distillation and Quantization for Efficient ASR
Due to the rapid development of computing hardware resources and the dramatic growth of data, pre-trained models in speech recognition, such as Whisper, have significantly improved the performance of speech recognition tasks.
Long-Form Video-Language Pre-Training with Multimodal Temporal Contrastive Learning
Large-scale video-language pre-training has shown significant improvement in video-language understanding tasks.
Ranked #2 on
Video Retrieval
on QuerYD
(using extra training data)
CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignment
and 2) how to mitigate the impact of these factors?
Ranked #2 on
Video Retrieval
on MSR-VTT-1kA
(using extra training data)
AI Illustrator: Translating Raw Descriptions into Images by Prompt-based Cross-Modal Generation
To address this issue, we propose a Prompt-based Cross-Modal Generation Framework (PCM-Frame) to leverage two powerful pre-trained models, including CLIP and StyleGAN.
Language-Guided Face Animation by Recurrent StyleGAN-based Generator
Specifically, we propose a recurrent motion generator to extract a series of semantic and motion information from the language and feed it along with visual information to a pre-trained StyleGAN to generate high-quality frames.
Exploring Anchor-based Detection for Ego4D Natural Language Query
In this paper we provide the technique report of Ego4D natural language query challenge in CVPR 2022.
Searching the Search Space of Vision Transformer
Vision Transformer has shown great visual representation power in substantial vision tasks such as recognition and detection, and thus been attracting fast-growing efforts on manually designing more effective architectures.
Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptions
To enable VL pre-training, we jointly optimize the HD-VILA model by a hybrid Transformer that learns rich spatiotemporal features, and a multimodal Transformer that enforces interactions of the learned video features with diversified texts.
Ranked #16 on
Video Retrieval
on MSR-VTT
Unifying Multimodal Transformer for Bi-directional Image and Text Generation
We adopt Transformer as our unified architecture for its strong performance and task-agnostic design.
A Picture is Worth a Thousand Words: A Unified System for Diverse Captions and Rich Images Generation
In this work, we demonstrate such an AI creation system to produce both diverse captions and rich images.
Learning Fine-Grained Motion Embedding for Landscape Animation
To tackle this problem, we propose a model named FGLA to generate high-quality and realistic videos by learning Fine-Grained motion embedding for Landscape Animation.
Reference-based Defect Detection Network
To solve the partial visual confusion issue, we propose to leverage the carried context information of context reference, which is the concentric bigger box of each region proposal, to perform more accurate region classification and regression.
Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training
To tackle this, we propose a fully Transformer visual embedding for VLP to better learn visual relation and further promote inter-modal alignment.
Probing Inter-modality: Visual Parsing with Self-Attention for Vision-and-Language Pre-training
To tackle this, we propose a fully Transformer visual embedding for VLP to better learn visual relation and further promote inter-modal alignment.
Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning
As region-based visual features usually represent parts of an image, it is challenging for existing vision-language models to fully understand the semantics from paired natural languages.
Ranked #5 on
Visual Entailment
on SNLI-VE val
Interpretable Machine Learning for COVID-19: An Empirical Study on Severity Prediction Task
The black-box nature of machine learning models hinders the deployment of some high-accuracy models in medical diagnosis.
Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
We aim to build a more accurate and thorough connection between image pixels and language semantics directly from image and sentence pairs instead of using region-based image features as the most recent vision and language tasks.
Neural Storyboard Artist: Visualizing Stories with Coherent Image Sequences
A storyboard is a sequence of images to illustrate a story containing multiple sentences, which has been a key process to create different story products.
Learning Rich Image Region Representation for Visual Question Answering
We propose to boost VQA by leveraging more powerful feature extractors by improving the representation ability of both visual and text features and the ensemble of models.
Gastroscopic Panoramic View: Application to Automatic Polyps Detection under Gastroscopy
The main contributions of this paper are: firstly, a gastroscopic panorama reconstruction method is developed.
SMP Challenge: An Overview of Social Media Prediction Challenge 2019
In the SMP Challenge at ACM Multimedia 2019, we introduce a novel prediction task Temporal Popularity Prediction, which focuses on predicting future interaction or attractiveness (in terms of clicks, views or likes etc.)
WSOD^2: Learning Bottom-up and Top-down Objectness Distillation for Weakly-supervised Object Detection
We study on weakly-supervised object detection (WSOD) which plays a vital role in relieving human involvement from object-level annotations.
WSOD2: Learning Bottom-up and Top-down Objectness Distillation forWeakly-supervised Object Detection
We study on weakly-supervised object detection (WSOD)which plays a vital role in relieving human involvement fromobject-level annotations.
Activitynet 2019 Task 3: Exploring Contexts for Dense Captioning Events in Videos
The overall system achieves the state-of-the-art performance on the dense-captioning events in video task with 9. 91 METEOR score on the challenge testing set.
Beyond Narrative Description: Generating Poetry from Images by Multi-Adversarial Training
Extensive experiments are conducted with 8K images, among which 1. 5K image are randomly picked for evaluation.
