Search Results for author:
Found 60 papers, 35 papers with code
Mining Inter-Video Proposal Relations for Video Object Detection
Recent studies have shown that, context aggregating information from proposals in different frames can clearly enhance the performance of video object detection.
Ranked #11 on
Video Object Detection
on ImageNet VID
EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World
Along with the videos we record high-quality gaze data and provide detailed multimodal annotations, formulating a playground for modeling the human ability to bridge asynchronous procedural actions from different viewpoints.
InternVideo2: Scaling Video Foundation Models for Multimodal Video Understanding
We introduce InternVideo2, a new video foundation model (ViFM) that achieves the state-of-the-art performance in action recognition, video-text tasks, and video-centric dialogue.
Ranked #1 on
Audio Classification
on ESC-50
(using extra training data)
VideoMamba: State Space Model for Efficient Video Understanding
Addressing the dual challenges of local redundancy and global dependencies in video understanding, this work innovatively adapts the Mamba to the video domain.
Percept, Chat, and then Adapt: Multimodal Knowledge Transfer of Foundation Models for Open-World Video Recognition
Finally, we blend external multimodal knowledge in Adapt stage, by inserting multimodal knowledge adaptation modules into networks.
From GPT-4 to Gemini and Beyond: Assessing the Landscape of MLLMs on Generalizability, Trustworthiness and Causality through Four Modalities
Multi-modal Large Language Models (MLLMs) have shown impressive abilities in generating reasonable responses with respect to multi-modal contents.
Vlogger: Make Your Dream A Vlog
More importantly, Vlogger can generate over 5-minute vlogs from open-world descriptions, without loss of video coherence on script and actor.
M-BEV: Masked BEV Perception for Robust Autonomous Driving
It mimics various missing cases by randomly masking features of different camera views, then leverages the original features of these views as self-supervision, and reconstructs the masked ones with the distinct spatio-temporal context across views.
MoVQA: A Benchmark of Versatile Question-Answering for Long-Form Movie Understanding
While several long-form VideoQA datasets have been introduced, the length of both videos used to curate questions and sub-clips of clues leveraged to answer those questions have not yet reached the criteria for genuine long-form video understanding.
MVBench: A Comprehensive Multi-modal Video Understanding Benchmark
With the rapid development of Multi-modal Large Language Models (MLLMs), a number of diagnostic benchmarks have recently emerged to evaluate the comprehension capabilities of these models.
Ranked #1 on
Zero-Shot Video Question Answer
on STAR Benchmark
SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction
The goal is to generate high-quality long videos with smooth and creative transitions between scenes and varying lengths of shot-level videos.
Harvest Video Foundation Models via Efficient Post-Pretraining
Building video-language foundation models is costly and difficult due to the redundant nature of video data and the lack of high-quality video-language datasets.
InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation
Specifically, we utilize a multi-scale approach to generate video-related descriptions.
VideoLLM: Modeling Video Sequence with Large Language Models
Building upon this insight, we propose a novel framework called VideoLLM that leverages the sequence reasoning capabilities of pre-trained LLMs from natural language processing (NLP) for video sequence understanding.
VideoChat: Chat-Centric Video Understanding
In this paper, we initiate an attempt of developing an end-to-end chat-centric video understanding system, coined as VideoChat.
Ranked #6 on
Video Question Answering
on MVBench
Video-based Generative Performance Benchmarking (Consistency)
Video-based Generative Performance Benchmarking (Contextual Understanding)
+5
InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Language
Different from existing interactive systems that rely on pure language, by incorporating pointing instructions, the proposed iGPT significantly improves the efficiency of communication between users and chatbots, as well as the accuracy of chatbots in vision-centric tasks, especially in complicated visual scenarios where the number of objects is greater than 2.
VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking
Finally, we successfully train a video ViT model with a billion parameters, which achieves a new state-of-the-art performance on the datasets of Kinetics (90. 0% on K400 and 89. 9% on K600) and Something-Something (68. 7% on V1 and 77. 0% on V2).
Ranked #1 on
Self-Supervised Action Recognition
on UCF101
(using extra training data)
Unmasked Teacher: Towards Training-Efficient Video Foundation Models
Previous VFMs rely on Image Foundation Models (IFMs), which face challenges in transferring to the video domain.
Ranked #1 on
Video Retrieval
on SSv2-template retrieval
(using extra training data)
Parameter is Not All You Need: Starting from Non-Parametric Networks for 3D Point Cloud Analysis
We present a Non-parametric Network for 3D point cloud analysis, Point-NN, which consists of purely non-learnable components: farthest point sampling (FPS), k-nearest neighbors (k-NN), and pooling operations, with trigonometric functions.
Ranked #1 on
Training-free 3D Part Segmentation
on ShapeNet-Part
3D Point Cloud Classification
Training-free 3D Part Segmentation
+1
UniFormerV2: Unlocking the Potential of Image ViTs for Video Understanding
The prolific performances of Vision Transformers (ViTs) in image tasks have prompted research into adapting the image ViTs for video tasks.
Starting From Non-Parametric Networks for 3D Point Cloud Analysis
We present a Non-parametric Network for 3D point cloud analysis, Point-NN, which consists of purely non-learnable components: farthest point sampling (FPS), k-nearest neighbors (k-NN), and pooling operations, with trigonometric functions.
HTML: Hybrid Temporal-scale Multimodal Learning Framework for Referring Video Object Segmentation
To tackle this problem, we propose a concise Hybrid Temporal-scale Multimodal Learning (HTML) framework, which can effectively align lingual and visual features to discover core object semantics in the video, by learning multimodal interaction hierarchically from different temporal scales.
Ranked #6 on
Referring Video Object Segmentation
on Refer-YouTube-VOS
(using extra training data)
MM-3DScene: 3D Scene Understanding by Customizing Masked Modeling with Informative-Preserved Reconstruction and Self-Distilled Consistency
Besides, such scenes with progressive masking ratios can also serve to self-distill their intrinsic spatial consistency, requiring to learn the consistent representations from unmasked areas.
InternVideo: General Video Foundation Models via Generative and Discriminative Learning
Specifically, InternVideo efficiently explores masked video modeling and video-language contrastive learning as the pretraining objectives, and selectively coordinates video representations of these two complementary frameworks in a learnable manner to boost various video applications.
Ranked #1 on
Action Recognition
on Something-Something V1
(using extra training data)
UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer
UniFormer has successfully alleviated this issue, by unifying convolution and self-attention as a relation aggregator in the transformer format.
InternVideo-Ego4D: A Pack of Champion Solutions to Ego4D Challenges
In this report, we present our champion solutions to five tracks at Ego4D challenge.
Ranked #1 on
State Change Object Detection
on Ego4D
VideoPipe 2022 Challenge: Real-World Video Understanding for Urban Pipe Inspection
For this reason, we propose to advance research areas of video understanding, with a shift from traditional action recognition to industrial anomaly analysis.
Low-Resolution Action Recognition for Tiny Actions Challenge
Second, these activities are naturally distributed in a long-tailed way.
Action Recognition With Motion Diversification and Dynamic Selection
To address this issue, we introduce a Motion Diversification and Selection (MoDS) module to generate diversified spatio-temporal motion features and then select the suitable motion representation dynamically for categorizing the input video.
Ranked #18 on
Action Recognition
on Something-Something V1
CP3: Unifying Point Cloud Completion by Pretrain-Prompt-Predict Paradigm
Inspired by prompting approaches from NLP, we creatively reinterpret point cloud generation and refinement as the prompting and predicting stages, respectively.
Cross Domain Object Detection by Target-Perceived Dual Branch Distillation
It can adaptively enhance source detector to perceive objects in a target image, by leveraging target proposal contexts from iterative cross-attention.
Target-Relevant Knowledge Preservation for Multi-Source Domain Adaptive Object Detection
Specifically, TRKP adopts the teacher-student framework, where the multi-head teacher network is built to extract knowledge from labeled source domains and guide the student network to learn detectors in unlabeled target domain.
Dual-AI: Dual-path Actor Interaction Learning for Group Activity Recognition
Learning spatial-temporal relation among multiple actors is crucial for group activity recognition.
UniFormer: Unifying Convolution and Self-attention for Visual Recognition
Different from the typical transformer blocks, the relation aggregators in our UniFormer block are equipped with local and global token affinity respectively in shallow and deep layers, allowing to tackle both redundancy and dependency for efficient and effective representation learning.
Ranked #151 on
Image Classification
on ImageNet
CP-Net: Contour-Perturbed Reconstruction Network for Self-Supervised Point Cloud Learning
To fill this gap, we propose a generic Contour-Perturbed Reconstruction Network (CP-Net), which can effectively guide self-supervised reconstruction to learn semantic content in the point cloud, and thus promote discriminative power of point cloud representation.
UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning
For Something-Something V1 and V2, our UniFormer achieves new state-of-the-art performances of 60. 9% and 71. 2% top-1 accuracy respectively.
Multi-View Partial (MVP) Point Cloud Challenge 2021 on Completion and Registration: Methods and Results
Based on the MVP dataset, this paper reports methods and results in the Multi-View Partial Point Cloud Challenge 2021 on Completion and Registration.
Self-slimmed Vision Transformer
Specifically, we first design a novel Token Slimming Module (TSM), which can boost the inference efficiency of ViTs by dynamic token aggregation.
MorphMLP: An Efficient MLP-Like Backbone for Spatial-Temporal Representation Learning
With such multi-dimension and multi-scale factorization, our MorphMLP block can achieve a great accuracy-computation balance.
Ranked #38 on
Action Recognition
on Something-Something V2
(using extra training data)
UniFormer: Unified Transformer for Efficient Spatial-Temporal Representation Learning
For Something-Something V1 and V2, our UniFormer achieves new state-of-the-art performances of 60. 8% and 71. 4% top-1 accuracy respectively.
Ranked #8 on
Action Recognition
on Something-Something V1
Self-Slimming Vision Transformer
Specifically, we first design a novel Token Slimming Module (TSM), which can boost the inference efficiency of ViTs by dynamic token aggregation.
Learning Dynamical Human-Joint Affinity for 3D Pose Estimation in Videos
Graph Convolution Network (GCN) has been successfully used for 3D human pose estimation in videos.
Ranked #10 on
3D Human Pose Estimation
on HumanEva-I
Digging into Uncertainty in Self-supervised Multi-view Stereo
Specially, the limitations can be categorized into two types: ambiguious supervision in foreground and invalid supervision in background.
TE-YOLOF: Tiny and efficient YOLOF for blood cell detection
In this work, an object detector based on YOLOF has been proposed to detect blood cell objects such as red blood cells, white blood cells and platelets.
CT-Net: Channel Tensorization Network for Video Classification
It can learn to exploit spatial, temporal and channel attention in a high-dimensional manner, to improve the cooperative power of all the feature dimensions in our CT-Module.
Ranked #18 on
Action Recognition
on Something-Something V1
FineAction: A Fine-Grained Video Dataset for Temporal Action Localization
Temporal action localization (TAL) is an important and challenging problem in video understanding.
PC-HMR: Pose Calibration for 3D Human Mesh Recovery from 2D Images/Videos
By coupling advanced 3D pose estimators and HMR in a serial or parallel manner, these two frameworks can effectively correct human mesh with guidance of a concise pose calibration module.
Ranked #4 on
3D Human Pose Estimation
on Surreal
Automatic Preference Based Multi-objective Evolutionary Algorithm on Vehicle Fleet Maintenance Scheduling Optimization
A preference based multi-objective evolutionary algorithm is proposed for generating solutions in an automatically detected knee point region.
SmallBigNet: Integrating Core and Contextual Views for Video Classification
Our SmallBig network outperforms a number of recent state-of-the-art approaches, in terms of accuracy and/or efficiency.
Improving Many-Objective Evolutionary Algorithms by Means of Edge-Rotated Cones
Given a point in $m$-dimensional objective space, any $\varepsilon$-ball of a point can be partitioned into the incomparable, the dominated and dominating region.
A Tailored NSGA-III Instantiation for Flexible Job Shop Scheduling
A customized multi-objective evolutionary algorithm (MOEA) is proposed for the multi-objective flexible job shop scheduling problem (FJSP).
Context-Transformer: Tackling Object Confusion for Few-Shot Detection
Few-shot object detection is a challenging but realistic scenario, where only a few annotated training images are available for training detectors.
Learning Attentive Pairwise Interaction for Fine-Grained Classification
These distinct gate vectors inherit mutual context on semantic differences, which allow API-Net to attentively capture contrastive clues by pairwise interaction between two images.
Ranked #12 on
Fine-Grained Image Classification
on Stanford Dogs
Progressive Object Transfer Detection
Inspired by this procedure of learning to detect, we propose a novel Progressive Object Transfer Detection (POTD) framework.
Clustering Bioactive Molecules in 3D Chemical Space with Unsupervised Deep Learning
We ex-plored the superior learning capability of deep autoencoders for unsupervised clustering of 1. 39 mil-lion bioactive molecules into band-clusters in a 3-dimensional latent chemical space.
Temporal Hallucinating for Action Recognition With Few Still Images
To mimic this capacity, we propose a novel Hybrid Video Memory (HVM) machine, which can hallucinate temporal features of still images from video memory, in order to boost action recognition with few still images.
LSTD: A Low-Shot Transfer Detector for Object Detection
Second, we introduce a novel regularized transfer learning framework for low-shot detection, where the transfer knowledge (TK) and background depression (BD) regularizations are proposed to leverage object knowledge respectively from source and target domains, in order to further enhance fine-tuning with a few target images.
Ranked #22 on
Few-Shot Object Detection
on MS-COCO (30-shot)
RPAN: An End-to-End Recurrent Pose-Attention Network for Action Recognition in Videos
Firstly, unlike previous works on pose-related action recognition, our RPAN is an end-to-end recurrent network which can exploit important spatial-temporal evolutions of human pose to assist action recognition in a unified framework.
Ranked #5 on
Skeleton Based Action Recognition
on J-HMDB
Weakly Supervised PatchNets: Describing and Aggregating Local Patches for Scene Recognition
In this paper, we propose a hybrid representation, which leverages the discriminative capacity of CNNs and the simplicity of descriptor encoding schema for image recognition, with a focus on scene recognition.
A Marginalized Particle Gaussian Process Regression
We present a novel marginalized particle Gaussian process (MPGP) regression, which provides a fast, accurate online Bayesian filtering framework to model the latent function.
