Search Results for author:
Found 229 papers, 115 papers with code
Combating Missing Modalities in Egocentric Videos at Test Time
Understanding videos that contain multiple modalities is crucial, especially in egocentric videos, where combining various sensory inputs significantly improves tasks like action recognition and moment localization.
Continual Learning on a Diet: Learning from Sparsely Labeled Streams Under Constrained Computation
DietCL meticulously allocates computational budget for both types of data.
SoccerNet Game State Reconstruction: End-to-End Athlete Tracking and Identification on a Minimap
This tracking and identification process is crucial for reconstructing the game state, defined by the athletes' positions and identities on a 2D top-view of the pitch, (i. e. a minimap).
Ranked #1 on
Game State Reconstruction
on SoccerNet-GSR
X-VARS: Introducing Explainability in Football Refereeing with Multi-Modal Large Language Model
The rapid advancement of artificial intelligence has led to significant improvements in automated decision-making.
DATENeRF: Depth-Aware Text-based Editing of NeRFs
However, extending these techniques to edit scenes in Neural Radiance Fields (NeRF) is complex, as editing individual 2D frames can result in inconsistencies across multiple views.
Towards Automated Movie Trailer Generation
Movie trailers are an essential tool for promoting films and attracting audiences.
Privacy-preserving Optics for Enhancing Protection in Face De-identification
Specifically, our approach first learns an optical encoder along with a regression model to obtain a face heatmap while hiding the face identity from the source image.
Efficient Image Pre-Training with Siamese Cropped Masked Autoencoders
In particular, SiamMAE recently introduced a Siamese network, training a shared-weight encoder from two frames of a video with a high asymmetric masking ratio (95%).
On Pretraining Data Diversity for Self-Supervised Learning
We explore the impact of training with more diverse datasets, characterized by the number of unique samples, on the performance of self-supervised learning (SSL) under a fixed computational budget.
GenView: Enhancing View Quality with Pretrained Generative Model for Self-Supervised Learning
To tackle these challenges, we present GenView, a controllable framework that augments the diversity of positive views leveraging the power of pretrained generative models while preserving semantics.
GES: Generalized Exponential Splatting for Efficient Radiance Field Rendering
With the aid of a frequency-modulated loss, GES achieves competitive performance in novel-view synthesis benchmarks while requiring less than half the memory storage of Gaussian Splatting and increasing the rendering speed by up to 39%.
Can Large Language Model Agents Simulate Human Trust Behaviors?
In addition, we probe into the biases in agent trust and the differences in agent trust towards agents and humans.
SPAD : Spatially Aware Multiview Diffusers
We present SPAD, a novel approach for creating consistent multi-view images from text prompts or single images.
SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training?
We present SynthCLIP, a novel framework for training CLIP models with entirely synthetic text-image pairs, significantly departing from previous methods relying on real data.
AToM: Amortized Text-to-Mesh using 2D Diffusion
We introduce Amortized Text-to-Mesh (AToM), a feed-forward text-to-mesh framework optimized across multiple text prompts simultaneously.
Exploring Missing Modality in Multimodal Egocentric Datasets
Multimodal video understanding is crucial for analyzing egocentric videos, where integrating multiple sensory signals significantly enhances action recognition and moment localization.
RAP-SAM: Towards Real-Time All-Purpose Segment Anything
Segment Anything Model (SAM) is one remarkable model that can achieve generalized segmentation.
Dr$^2$Net: Dynamic Reversible Dual-Residual Networks for Memory-Efficient Finetuning
We use two coefficients on either type of residual connections respectively, and introduce a dynamic training strategy that seamlessly transitions the pretrained model to a reversible network with much higher numerical precision.
Adaptive Guidance: Training-free Acceleration of Conditional Diffusion Models
Our findings provide insights into the efficiency of the conditional denoising process that contribute to more practical and swift deployment of text-conditioned diffusion models.
Artificial intelligence optical hardware empowers high-resolution hyperspectral video understanding at 1.2 Tb/s
The technology platform combines artificial intelligence hardware, processing information optically, with state-of-the-art machine vision networks, resulting in a data processing speed of 1. 2 Tb/s with hundreds of frequency bands and megapixel spatial resolution at video rates.
Behind the Magic, MERLIM: Multi-modal Evaluation Benchmark for Large Image-Language Models
This paper introduces a Multi-modal Evaluation Benchmark named MERLIM, a scalable test-bed to assess the performance of IT-LVLMs on fundamental computer vision tasks.
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
We present Ego-Exo4D, a diverse, large-scale multimodal multiview video dataset and benchmark challenge.
End-to-End Temporal Action Detection with 1B Parameters Across 1000 Frames
In this paper, we reduce the memory consumption for end-to-end training, and manage to scale up the TAD backbone to 1 billion parameters and the input video to 1, 536 frames, leading to significant detection performance.
Ranked #1 on
Temporal Action Localization
on EPIC-KITCHENS-100
SplitNeRF: Split Sum Approximation Neural Field for Joint Geometry, Illumination, and Material Estimation
We present a novel approach for digitizing real-world objects by estimating their geometry, material properties, and environmental lighting from a set of posed images with fixed lighting.
From Categories to Classifier: Name-Only Continual Learning by Exploring the Web
Continual Learning (CL) often relies on the availability of extensive annotated datasets, an assumption that is unrealistically time-consuming and costly in practice.
Towards Demystifying the Generalization Behaviors When Neural Collapse Emerges
While there have been extensive studies on optimization characteristics showing the global optimality of neural collapse, little research has been done on the generalization behaviors during the occurrence of NC.
Automatic Animation of Hair Blowing in Still Portrait Photos
Considering the complexity of hair structure, we innovatively treat hair wisp extraction as an instance segmentation problem, where a hair wisp is referred to as an instance.
SoccerNet 2023 Challenges Results
More information on the tasks, challenges, and leaderboards are available on https://www. soccer-net. org.
Learning Semantic Segmentation with Query Points Supervision on Aerial Images
In this work, we present a weakly supervised learning algorithm to train semantic segmentation algorithms that only rely on query point annotations instead of full mask labels.
Learning to Read Analog Gauges from Synthetic Data
We propose a two-stage CNN pipeline that identifies the key structural components of an analog gauge and outputs an angular reading.
ShadowNet for Data-Centric Quantum System Learning
Understanding the dynamics of large quantum systems is hindered by the curse of dimensionality.
Learning to Identify Critical States for Reinforcement Learning from Videos
Recent work on deep reinforcement learning (DRL) has pointed out that algorithmic information about good policies can be extracted from offline data which lack explicit information about executed actions.
Deformable Mixer Transformer with Gating for Multi-Task Learning of Dense Prediction
In this work, we present a novel MTL model by combining both merits of deformable CNN and query-based Transformer with shared gating for multi-task learning of dense prediction.
Neural Collapse Terminus: A Unified Solution for Class Incremental Learning and Its Variants
Beyond the normal case, long-tail class incremental learning and few-shot class incremental learning are also proposed to consider the data imbalance and data scarcity, respectively, which are common in real-world implementations and further exacerbate the well-known problem of catastrophic forgetting.
Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors
We present Magic123, a two-stage coarse-to-fine approach for high-quality, textured 3D meshes generation from a single unposed image in the wild using both2D and 3D priors.
Towards Open Vocabulary Learning: A Survey
To our knowledge, this is the first comprehensive literature review of open vocabulary learning.
Enhancing Neural Rendering Methods with Image Augmentations
We found that introducing image augmentations during training presents challenges such as geometric and photometric inconsistencies for learning NRMs from images.
Dynamically Masked Discriminator for Generative Adversarial Networks
By treating the generated data in training as a stream, we propose to detect whether the discriminator slows down the learning of new knowledge in generated data.
Exploring Open-Vocabulary Semantic Segmentation without Human Labels
Although acquired extensive knowledge of visual concepts, it is non-trivial to exploit knowledge from these VL models to the task of semantic segmentation, as they are usually trained at an image level.
Just a Glimpse: Rethinking Temporal Information for Video Continual Learning
Studying continual learning in the video domain poses even more challenges, as video data contains a large number of frames, which places a higher burden on the replay memory.
Mindstorms in Natural Language-Based Societies of Mind
What should be the social structure of an NLSOM?
How To Not Train Your Dragon: Training-free Embodied Object Goal Navigation with Semantic Frontiers
Our method propagates semantics on the scene graphs based on language priors and scene statistics to introduce semantic knowledge to the geometric frontiers.
Rapid Adaptation in Online Continual Learning: Are We Evaluating It Right?
We revisit the common practice of evaluating adaptation of Online Continual Learning (OCL) algorithms through the metric of online accuracy, which measures the accuracy of the model on the immediate next few samples.
Large-capacity and Flexible Video Steganography via Invertible Neural Network
For large-capacity, we present a reversible pipeline to perform multiple videos hiding and recovering through a single invertible neural network (INN).
LLM as A Robotic Brain: Unifying Egocentric Memory and Control
In this paper, we propose a novel and generalizable framework called LLM-Brain: using Large-scale Language Model as a robotic brain to unify egocentric memory and control.
Revisiting Test Time Adaptation under Online Evaluation
To address this issue, we propose a more realistic evaluation protocol for TTA methods, where data is received in an online fashion from a constant-speed data stream, thereby accounting for the method's adaptation speed.
SoccerNet-Caption: Dense Video Captioning for Soccer Broadcasts Commentaries
By providing broadcasters with a tool to summarize the content of their video with the same level of engagement as a live game, our method could help satisfy the needs of the numerous fans who follow their team but cannot necessarily watch the live game.
VARS: Video Assistant Referee System for Automated Soccer Decision Making from Multiple Views
The Video Assistant Referee (VAR) has revolutionized association football, enabling referees to review incidents on the pitch, make informed decisions, and ensure fairness.
Towards Active Learning for Action Spotting in Association Football Videos
In this paper, we propose an active learning framework that selects the most informative video samples to be annotated next, thus drastically reducing the annotation effort and accelerating the training of action spotting models to reach the highest accuracy at a faster pace.
Improving Visual Question Answering Models through Robustness Analysis and In-Context Learning with a Chain of Basic Questions
This work proposes a new method that utilizes semantically related questions, referred to as basic questions, acting as noise to evaluate the robustness of VQA models.
Boundary-Denoising for Video Activity Localization
To alleviate the boundary ambiguity, we propose to study the video activity localization problem from a denoising perspective.
Ranked #1 on
Video Grounding
on MAD
Online Distillation with Continual Learning for Cyclic Domain Shifts
In this paper, we propose a solution to this issue by leveraging the power of continual learning methods to reduce the impact of domain shifts.
CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society
Our contributions include introducing a novel communicative agent framework, offering a scalable approach for studying the cooperative behaviors and capabilities of multi-agent systems, and open-sourcing our library to support research on communicative agents and beyond: https://github. com/camel-ai/camel.
Don't FREAK Out: A Frequency-Inspired Approach to Detecting Backdoor Poisoned Samples in DNNs
In this paper we investigate the frequency sensitivity of Deep Neural Networks (DNNs) when presented with clean samples versus poisoned samples.
Computationally Budgeted Continual Learning: What Does Matter?
Our conclusions are consistent in a different number of stream time steps, e. g., 20 to 200, and under several computational budgets.
A Unified Continual Learning Framework with General Parameter-Efficient Tuning
1) Learning: the pre-trained model adapts to the new task by tuning an online PET module, along with our adaptation speed calibration to align different PET modules, 2) Accumulation: the task-specific knowledge learned by the online PET module is accumulated into an offline PET module through momentum update, 3) Ensemble: During inference, we respectively construct two experts with online/offline PET modules (which are favored by the novel/historical tasks) for prediction ensemble.
FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model
In this work, we propose a training-Free conditional Diffusion Model (FreeDoM) used for various conditions.
Re-ReND: Real-time Rendering of NeRFs across Devices
Re-ReND is designed to achieve real-time performance by converting the NeRF into a representation that can be efficiently processed by standard graphics pipelines.
Improving GAN Training via Feature Space Shrinkage
The evaluation results demonstrate that our AdaptiveMix can facilitate the training of GANs and effectively improve the image quality of generated samples.
Localizing Moments in Long Video Via Multimodal Guidance
In this paper, we propose a method for improving the performance of natural language grounding in long videos by identifying and pruning out non-describable windows.
Ranked #1 on
Natural Language Moment Retrieval
on MAD
Natural Language Moment Retrieval
Natural Language Visual Grounding
+2
Real-Time Evaluation in Online Continual Learning: A New Hope
We show that a simple baseline outperforms state-of-the-art CL methods under this evaluation, questioning the applicability of existing methods in realistic settings.
Look, Listen, and Attack: Backdoor Attacks Against Video Action Recognition
Although backdoor attacks have been extensively studied in the image domain, there are very few works that explore such attacks in the video domain, and they tend to conclude that image backdoor attacks are less effective in the video domain.
NewsNet: A Novel Dataset for Hierarchical Temporal Segmentation
Temporal video segmentation is the get-to-go automatic video analysis, which decomposes a long-form video into smaller components for the following-up understanding tasks.
AdaptiveMix: Improving GAN Training via Feature Space Shrinkage
The evaluation results demonstrate that our AdaptiveMix can facilitate the training of GANs and effectively improve the image quality of generated samples.
Exploring Open-Vocabulary Semantic Segmentation from CLIP Vision Encoder Distillation Only
Semantic segmentation is a crucial task in computer vision that involves segmenting images into semantically meaningful regions at the pixel level.
Re2TAL: Rewiring Pretrained Video Backbones for Reversible Temporal Action Localization
Temporal action localization (TAL) requires long-form reasoning to predict actions of various durations and complex content.
MVTN: Learning Multi-View Transformations for 3D Understanding
Multi-view projection techniques have shown themselves to be highly effective in achieving top-performing results in the recognition of 3D shapes.
SPARF: Large-Scale Learning of 3D Sparse Radiance Fields from Few Input Images
SuRFNet employs partial SRFs from few/one images and a specialized SRF loss to learn to generate high-quality sparse voxel radiance fields that can be rendered from novel views.
EgoLoc: Revisiting 3D Object Localization from Egocentric Videos with Visual Queries
Yet, we point out that the low number of camera poses caused by camera re-localization from previous VQ3D methods severally hinders their overall success rate.
PIVOT: Prompting for Video Continual Learning
In this paper, we address the problem of continual learning for video data.
SimCS: Simulation for Domain Incremental Online Continual Segmentation
This work explores the problem of Online Domain-Incremental Continual Segmentation (ODICS), where the model is continually trained over batches of densely labeled images from different domains, with limited computation and no information about the task boundaries.
On Robust Learning from Noisy Labels: A Permutation Layer Approach
We provide two variants of PermLL in this paper: one applies the permutation layer to the model's prediction, while the other applies it directly to the given noisy label.
Multi-Modal Few-Shot Temporal Action Detection
In this work, we introduce a new multi-modality few-shot (MMFS) TAD problem, which can be considered as a marriage of FS-TAD and ZS-TAD by leveraging few-shot support videos and new class names jointly.
Re^2TAL: Rewiring Pretrained Video Backbones for Reversible Temporal Action Localization
Temporal action localization (TAL) requires long-form reasoning to predict actions of various durations and complex content.
SegNeRF: 3D Part Segmentation with Neural Radiance Fields
The predicted semantic fields allow SegNeRF to achieve an average mIoU of $\textbf{30. 30%}$ for 2D novel view segmentation, and $\textbf{37. 46%}$ for 3D part segmentation, boasting competitive performance against point-based methods by using only a few posed images.
Diffusion-Based Scene Graph to Image Generation with Masked Contrastive Pre-Training
Generating images from graph-structured inputs, such as scene graphs, is uniquely challenging due to the difficulty of aligning nodes and connections in graphs with objects and their relations in images.
Where is my Wallet? Modeling Object Proposal Sets for Egocentric Visual Query Localization
Our experiments show the proposed adaptations improve egocentric query detection, leading to a better visual query localization system in both 2D and 3D configurations.
Estimating more camera poses for ego-centric videos is essential for VQ3D
Visual queries 3D localization (VQ3D) is a task in the Ego4D Episodic Memory Benchmark.
Decoupled Mixup for Generalized Visual Recognition
Since the observation is that noise-prone regions such as textural and clutter backgrounds are adverse to the generalization ability of CNN models during training, we enhance features from discriminative regions and suppress noise-prone ones when combining an image pair.
SoccerNet 2022 Challenges Results
The SoccerNet 2022 challenges were the second annual video understanding challenges organized by the SoccerNet team.
Generalizability of Adversarial Robustness Under Distribution Shifts
Recent progress in empirical and certified robustness promises to deliver reliable and deployable Deep Neural Networks (DNNs).
Combating Mode Collapse in GANs via Manifold Entropy Estimation
In this paper, we propose a novel training pipeline to address the mode collapse issue of GANs.
Pix4Point: Image Pretrained Standard Transformers for 3D Point Cloud Understanding
Pretrained on a large number of widely available images, significant gains of PViT are observed in the tasks of 3D point cloud classification, part segmentation, and semantic segmentation on ScanObjectNN, ShapeNetPart, and S3DIS, respectively.
Negative Frames Matter in Egocentric Visual Query 2D Localization
The repeated gradient computation of the same object lead to an inefficient training; (2) The false positive rate is high on background frames.
Egocentric Video-Language Pretraining @ Ego4D Challenge 2022
In this report, we propose a video-language pretraining (VLP) based solution \cite{kevin2022egovlp} for four Ego4D challenge tasks, including Natural Language Query (NLQ), Moment Query (MQ), Object State Change Classification (OSCC), and PNR Localization (PNR).
PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategies
In this work, we revisit the classical PointNet++ through a systematic study of model training and scaling strategies, and offer two major contributions.
Ranked #3 on
3D Semantic Segmentation
on OpenTrench3D
Certified Robustness in Federated Learning
However, as in the single-node supervised learning setup, models trained in federated learning suffer from vulnerability to imperceptible input transformations known as adversarial attacks, questioning their deployment in security-related applications.
Egocentric Video-Language Pretraining
Video-Language Pretraining (VLP), which aims to learn transferable representation to advance a wide range of video-text downstream tasks, has recently received increasing attention.
ETAD: Training Action Detection End to End on a Laptop
We propose to sequentially forward the snippet frame through the video encoder, and backward only a small necessary portion of gradients to update the encoder.
UnrealNAS: Can We Search Neural Architectures with Unreal Data?
Neural architecture search (NAS) has shown great success in the automatic design of deep neural networks (DNNs).
Contrastive Language-Action Pre-training for Temporal Localization
Long-form video understanding requires designing approaches that are able to temporally localize activities or language.
Contrastive Learning
Few Shot Temporal Action Localization
+3
SoccerNet-Tracking: Multiple Object Tracking Dataset and Benchmark in Soccer Videos
Tracking objects in soccer videos is extremely important to gather both player and team statistics, whether it is to estimate the total distance run, the ball possession or the team formation.
3DeformRS: Certifying Spatial Deformations on Point Clouds
In this work, we propose 3DeformRS, a method to certify the robustness of point cloud Deep Neural Networks (DNNs) against real-world deformations.
When NAS Meets Trees: An Efficient Algorithm for Neural Architecture Search
TNAS performs a modified bi-level Breadth-First Search in the proposed trees to discover a high-performance architecture.
Real-time Hyperspectral Imaging in Hardware via Trained Metasurface Encoders
Hyperspectral imaging has attracted significant attention to identify spectral signatures for image classification and automated pattern recognition in computer vision.
End-to-End Active Speaker Detection
Recent advances in the Active Speaker Detection (ASD) problem build upon a two-stage process: feature extraction and spatio-temporal context aggregation.
Ranked #4 on
Audio-Visual Active Speaker Detection
on AVA-ActiveSpeaker
(using extra training data)
R-DFCIL: Relation-Guided Representation Learning for Data-Free Class Incremental Learning
After RRL, the classification head is refined with global class-balanced classification loss to address the data imbalance issue as well as learn the decision boundaries between new and previous classes.
Learning Scene Flow in 3D Point Clouds with Noisy Pseudo Labels
To provide an alternative, we propose a novel approach that utilizes monocular RGB images and point clouds to generate pseudo scene flow labels for training scene flow networks.
SegTAD: Precise Temporal Action Detection via Semantic Segmentation
To address these issues and precisely model temporal action detection, we formulate the task of temporal action detection in a novel perspective of semantic segmentation.
OWL (Observe, Watch, Listen): Audiovisual Temporal Context for Localizing Actions in Egocentric Videos
We validate our approach in two large-scale datasets, EPIC-Kitchens, and HOMAGE.
Towards Assessing and Characterizing the Semantic Robustness of Face Recognition
We propose a methodology for assessing and characterizing the robustness of FRMs against semantic perturbations to their input.
On the Robustness of Quality Measures for GANs
Finally, we show that the FID can be robustified by simply replacing the standard Inception with a robust Inception.
vCLIMB: A Novel Video Class Incremental Learning Benchmark
We perform in-depth evaluations of existing CL methods in vCLIMB, and observe two unique challenges in video data.
Spatio-temporal Relation Modeling for Few-shot Action Recognition
Experiments are performed on four few-shot action recognition benchmarks: Kinetics, SSv2, HMDB51 and UCF101.
Ranked #1 on
Few Shot Action Recognition
on UCF101
(using extra training data)
MAD: A Scalable Dataset for Language Grounding in Videos from Movie Audio Descriptions
The recent and increasing interest in video-language research has driven the development of large-scale datasets that enable data-intensive machine learning techniques.
Ranked #2 on
Natural Language Moment Retrieval
on MAD
Low-Fidelity Video Encoder Optimization for Temporal Action Localization
This results in a task discrepancy problem for the video encoder – trained for action classification, but used for TAL.
Ranked #9 on
Temporal Action Localization
on HACS
Voint Cloud: Multi-View Point Cloud Representation for 3D Understanding
To this end, we introduce the concept of the multi-view point cloud (Voint cloud), representing each 3D point as a set of features extracted from several view-points.
ASSANet: An Anisotropic Separable Set Abstraction for Efficient Point Cloud Representation Learning
We then introduce a new Anisotropic Reduction function into our Separable SA module and propose an Anisotropic Separable SA (ASSA) module that substantially increases the network's accuracy.
Ranked #33 on
3D Part Segmentation
on ShapeNet-Part
Ego4D: Around the World in 3,000 Hours of Egocentric Video
We introduce Ego4D, a massive-scale egocentric video dataset and benchmark suite.
Relation-aware Video Reading Comprehension for Temporal Language Grounding
Temporal language grounding in videos aims to localize the temporal span relevant to the given query sentence.
MovieCuts: A New Dataset and Benchmark for Cut Type Recognition
Advances in automatic Cut-type recognition can unleash new experiences in the video editing industry, such as movie analysis for education, video re-editing, virtual cinematography, machine-assisted trailer generation, machine-assisted video editing, among others.
Check Your Other Door! Creating Backdoor Attacks in the Frequency Domain
Deep Neural Networks (DNNs) are ubiquitous and span a variety of applications ranging from image classification to real-time object detection.
Learning to Cut by Watching Movies
Video content creation keeps growing at an incredible pace; yet, creating engaging stories remains challenging and requires non-trivial video editing expertise.
Enhancing Adversarial Robustness via Test-time Transformation Ensembling
Deep learning models are prone to being fooled by imperceptible perturbations known as adversarial attacks.
ANCER: Anisotropic Certification via Sample-wise Volume Maximization
Randomized smoothing has recently emerged as an effective tool that enables certification of deep neural network classifiers at scale.
DeformRS: Certifying Input Deformations with Randomized Smoothing
Deep neural networks are vulnerable to input deformations in the form of vector fields of pixel displacements and to other parameterized geometric deformations e. g. translations, rotations, etc.
Training Graph Neural Networks with 1000 Layers
Deep graph neural networks (GNNs) have achieved excellent results on various tasks on increasingly large graph datasets with millions of nodes and edges.
Ranked #1 on
Node Property Prediction
on ogbn-proteins
APES: Audiovisual Person Search in Untrimmed Video
To showcase the potential of our new dataset, we propose an audiovisual baseline and benchmark for person retrieval.
SCTN: Sparse Convolution-Transformer Network for Scene Flow Estimation
We propose a novel scene flow estimation approach to capture and infer 3D motions from point clouds.
Camera Calibration and Player Localization in SoccerNet-v2 and Investigation of their Representations for Action Spotting
Specifically, we distill a powerful commercial calibration tool in a recent neural network architecture on the large-scale SoccerNet dataset, composed of untrimmed broadcast videos of 500 soccer games.
Temporally-Aware Feature Pooling for Action Spotting in Soccer Broadcasts
In this paper, we focus our analysis on action spotting in soccer broadcast, which consists in temporally localizing the main actions in a soccer game.
Ranked #7 on
Action Spotting
on SoccerNet-v2
(Average-mAP metric)
Low-Fidelity End-to-End Video Encoder Pre-training for Temporal Action Localization
This results in a task discrepancy problem for the video encoder -- trained for action classification, but used for TAL.
Combating Adversaries with Anti-Adversaries
Deep neural networks are vulnerable to small input perturbations known as adversarial attacks.
AniGAN: Style-Guided Generative Adversarial Networks for Unsupervised Anime Face Generation
Specifically, a new generator architecture is proposed to simultaneously transfer color/texture styles and transform local facial shapes into anime-like counterparts based on the style of a reference anime-face, while preserving the global structure of the source photo-face.
MAAS: Multi-modal Assignation for Active Speaker Detection
Active speaker detection requires a solid integration of multi-modal cues.
On the Decision Boundaries of Neural Networks. A Tropical Geometry Perspective
This work tackles the problem of characterizing and understanding the decision boundaries of neural networks with piecewise linear non-linearity activations.
High Quality Disparity Remapping With Two-Stage Warping
In the second stage, we derive another warping model to refine warping results in less important regions by eliminating serious distortions in shape, disparity and 3D structure.
Vocal Bursts Intensity Prediction
Vocal Bursts Valence Prediction
DeeperGCN: Training Deeper GCNs with Generalized Aggregation Functions
We add our generalized aggregation into a deep GCN framework and show it achieves state-of-the-art results in six benchmarks from OGB.
SALA: Soft Assignment Local Aggregation for Parameter Efficient 3D Semantic Segmentation
Since it is learnable, this mapping is allowed to be different per layer instead of being applied uniformly throughout the depth of the network.
Data-Dependent Randomized Smoothing
In this work, we revisit Gaussian randomized smoothing and show that the variance of the Gaussian distribution can be optimized at each input so as to maximize the certification radius for the construction of the smooth classifier.
Video Self-Stitching Graph Network for Temporal Action Localization
We have two key components in VSGN: video self-stitching (VSS) and cross-scale graph pyramid network (xGPN).
Ranked #16 on
Temporal Action Localization
on ActivityNet-1.3
SoccerNet-v2: A Dataset and Benchmarks for Holistic Understanding of Broadcast Soccer Videos
In this work, we propose SoccerNet-v2, a novel large-scale corpus of manual annotations for the SoccerNet video dataset, along with open challenges to encourage more research in soccer understanding and broadcast production.
Ranked #1 on
Camera shot segmentation
on SoccerNet-v2
MVTN: Multi-View Transformation Network for 3D Shape Recognition
MVTN exhibits clear performance gains in the tasks of 3D shape classification and 3D shape retrieval without the need for extra training supervision.
Ranked #1 on
3D Object Retrieval
on ModelNet40
TSP: Temporally-Sensitive Pretraining of Video Encoders for Localization Tasks
Extensive experiments show that using features trained with our novel pretraining strategy significantly improves the performance of recent state-of-the-art methods on three tasks: Temporal Action Localization, Action Proposal Generation, and Dense Video Captioning.
Boundary-sensitive Pre-training for Temporal Localization in Videos
However, most existing models developed for these tasks are pre-trained on general video action classification tasks.
Ranked #23 on
Temporal Action Localization
on ActivityNet-1.3
VLG-Net: Video-Language Graph Matching Network for Video Grounding
Grounding language queries in videos aims at identifying the time interval (or moment) semantically relevant to a language query.
Ranked #1 on
Natural Language Moment Retrieval
on DiDeMo
Robust Optimization as Data Augmentation for Large-scale Graphs
Data augmentation helps neural networks generalize better by enlarging the training set, but it remains an open question how to effectively augment graph data to enhance the performance of GNNs (Graph Neural Networks).
Ranked #1 on
Graph Property Prediction
on ogbg-ppa
LC-NAS: Latency Constrained Neural Architecture Search for Point Cloud Networks
In this paper, we introduce a new NAS framework, dubbed LC-NAS, where we search for point cloud architectures that are constrained to a target latency.
The End-of-End-to-End: A Video Understanding Pentathlon Challenge (2020)
This report summarizes the results of the first edition of the challenge together with the findings of the participants.
Learning Heat Diffusion for Network Alignment
EDNA uses the diffusion signal as a proxy for computing node similarities between networks.
Network Moments: Extensions and Sparse-Smooth Attacks
This has incited research on the reaction of DNNs to noisy input, namely developing adversarial input attacks and strategies that lead to robust DNNs to these attacks.
DeeperGCN: All You Need to Train Deeper GCNs
Graph Convolutional Networks (GCNs) have been drawing significant attention with the power of representation learning on graphs.
Ranked #1 on
Node Property Prediction
on ogbn-proteins
Rethinking Clustering for Robustness
This paper studies how encouraging semantically-aligned features during deep neural network training can increase network robustness.
Active Speakers in Context
Current methods for active speak er detection focus on modeling short-term audiovisual information from a single speaker.
Adaptive Learning of the Optimal Batch Size of SGD
Recent advances in the theoretical understanding of SGD led to a formula for the optimal batch size minimizing the number of effective data passes, i. e., the number of iterations times the batch size.
On the Decision Boundaries of Neural Networks: A Tropical Geometry Perspective
Our main finding is that the decision boundaries are a subset of a tropical hypersurface, which is intimately related to a polytope formed by the convex hull of two zonotopes.
RGB-based Semantic Segmentation Using Self-Supervised Depth Pre-Training
These labels, denoted by HN-labels, represent different height and normal patches, which allow mining of local semantic information that is useful in the task of semantic RGB segmentation.
Ranked #101 on
Semantic Segmentation
on NYU Depth v2
Analytical Moment Regularizer for Training Robust Networks
In this work, we propose a new training regularizer that aims to minimize the probabilistic expected training loss of a DNN subject to a generic Gaussian input.
Gabor Layers Enhance Network Robustness
We revisit the benefits of merging classical vision concepts with deep learning models.
A Context-Aware Loss Function for Action Spotting in Soccer Videos
We benchmark our loss on a large dataset of soccer videos, SoccerNet, and achieve an improvement of 12. 8% over the baseline.
Ranked #3 on
Action Spotting
on SoccerNet
AdvPC: Transferable Adversarial Perturbations on 3D Point Clouds
Our proposed attack increases the attack success rate by up to 40% for those transferred to unseen networks (transferability), while maintaining a high success rate on the attacked network.
SGAS: Sequential Greedy Architecture Search
Architecture design has become a crucial component of successful deep learning.
Ranked #4 on
Node Classification
on PPI
PU-GCN: Point Cloud Upsampling using Graph Convolutional Networks
We combine Inception DenseGCN with NodeShuffle into a new point upsampling pipeline called PU-GCN.
Assessing the Robustness of Visual Question Answering Models
In this work, we propose a new method that uses semantically related questions, dubbed basic questions, acting as noise to evaluate the robustness of VQA models.
Self-Supervised Learning by Cross-Modal Audio-Video Clustering
To the best of our knowledge, XDC is the first self-supervised learning method that outperforms large-scale fully-supervised pretraining for action recognition on the same architecture.
PointRGCN: Graph Convolution Networks for 3D Vehicles Detection Refinement
We integrate residual GCNs in a two-stage 3D object detection pipeline, where 3D object proposals are refined using a novel graph representation.
Ranked #14 on
3D Object Detection
on KITTI Cars Hard
G-TAD: Sub-Graph Localization for Temporal Action Detection
In this work, we propose a graph convolutional network (GCN) model to adaptively incorporate multi-level semantic context into video features and cast temporal action detection as a sub-graph localization problem.
Ranked #5 on
Temporal Action Localization
on EPIC-KITCHENS-100
DeepGCNs: Making GCNs Go as Deep as CNNs
This work transfers concepts such as residual/dense connections and dilated convolutions from CNNs to GCNs in order to successfully train very deep GCNs.
Ranked #5 on
3D Semantic Segmentation
on PartNet
Expected Tight Bounds for Robust Deep Neural Network Training
al. that bounded input intervals can be inexpensively propagated from layer to layer through deep networks.
On the Decision Boundaries of Deep Neural Networks: A Tropical Geometry Perspective
We use tropical geometry, a new development in the area of algebraic geometry, to provide a characterization of the decision boundaries of a simple neural network of the form (Affine, ReLU, Affine).
Finding Moments in Video Collections Using Natural Language
We evaluate our approach on two recently proposed datasets for temporal localization of moments in video with natural language (DiDeMo and Charades-STA) extended to our video corpus moment retrieval setting.
Constrained Clustering: General Pairwise and Cardinality Constraints
Extensive experiments on both synthetic and real data demonstrate when: (1) utilizing a single category of constraint, the proposed model is superior to or competitive with SOTA constrained clustering models, and (2) utilizing both categories of constraints jointly, the proposed model shows better performance than the case of the single category.
3D Instance Segmentation via Multi-Task Metric Learning
The second goal is to learn instance information by densely estimating directional information of the instance's center of mass for each voxel.
Ranked #2 on
3D Semantic Instance Segmentation
on ScanNetV2
Expected Tight Bounds for Robust Training
In this paper, we closely examine the bounds of a block of layers composed in the form of Affine-ReLU-Affine.
MAP Inference via L2-Sphere Linear Program Reformulation
Thus, LS-LP is equivalent to the original MAP inference problem.
Rethinking Learning-based Demosaicing, Denoising, and Super-Resolution Pipeline
In this work, we comprehensively study the effects of pipelines on the mixture problem of learning-based DN, DM, and SR, in both sequential and joint solutions.
Deep Layers as Stochastic Solvers
In particular, we show that a forward pass through a standard dropout layer followed by a linear layer and a non-linear activation is equivalent to optimizing a convex optimization objective with a single iteration of a $\tau$-nice Proximal Stochastic Gradient method.
Analytical Moment Regularizer for Gaussian Robust Networks
Despite the impressive performance of deep neural networks (DNNs) on numerous vision tasks, they still exhibit yet-to-understand uncouth behaviours.
Learning a Controller Fusion Network by Online Trajectory Filtering for Vision-based UAV Racing
A common approach is to learn an end-to-end policy that directly predicts controls from raw images by imitating an expert.
IAN: Combining Generative Adversarial Networks for Imaginative Face Generation
Generative Adversarial Networks (GANs) have gained momentum for their ability to model image distributions.
MAIN: Multi-Attention Instance Network for Video Segmentation
Instance-level video segmentation requires a solid integration of spatial and temporal information.
BAOD: Budget-Aware Object Detection
We adopt a hybrid supervised learning framework to train the object detector from both these types of annotation.
ThumbNet: One Thumbnail Image Contains All You Need for Recognition
Although deep convolutional neural networks (CNNs) have achieved great success in computer vision tasks, its real-world application is still impeded by its voracious demand of computational resources.
Towards Analyzing Semantic Robustness of Deep Neural Networks
Despite the impressive performance of Deep Neural Networks (DNNs) on various vision tasks, they still exhibit erroneous high sensitivity toward semantic primitives (e. g. object pose).
DeepGCNs: Can GCNs Go as Deep as CNNs?
Finally, we use these new concepts to build a very deep 56-layer GCN, and show how it significantly boosts performance (+3. 7% mIoU over state-of-the-art) in the task of point cloud semantic segmentation.
RefineLoc: Iterative Refinement for Weakly-Supervised Action Localization
RefineLoc shows competitive results with the state-of-the-art in weakly-supervised temporal localization.
Temporal Localization
Weakly Supervised Action Localization
+2
MortonNet: Self-Supervised Learning of Local Features in 3D Point Clouds
In fact, we show how Morton features can be used to significantly improve performance (+3% for 2 popular semantic segmentation algorithms) in the task of semantic segmentation of point clouds on the challenging and large-scale S3DIS dataset.
Efficient Bird Eye View Proposals for 3D Siamese Tracking
Successively, we refine our selection of 3D object candidates by exploiting the similarity capability of a 3D Siamese network.
Leveraging Shape Completion for 3D Siamese Tracking
We design a Siamese tracker that encodes model and candidate shapes into a compact latent representation.
SADA: Semantic Adversarial Diagnostic Attacks for Autonomous Applications
In contrast, we present a general framework for adversarial attacks on trained agents, which covers semantic perturbations to the environment of the agent performing the task as well as pixel-level attacks.
SOD-MTGAN: Small Object Detection via Multi-Task Generative Adversarial Network
In the MTGAN, the generator is a super-resolution network, which can up-sample small blurred images into fine-scale ones and recover detailed information for more accurate detection.
Face Super-resolution Guided by Facial Component Heatmaps
State-of-the-art face super-resolution methods use deep convolutional neural networks to learn a mapping between low-resolution (LR) facial patterns and their corresponding high-resolution (HR) counterparts by exploring local information.
The ActivityNet Large-Scale Activity Recognition Challenge 2018 Summary
The guest tasks focused on complementary aspects of the activity recognition problem at large scale and involved three challenging and recently compiled datasets: the Kinetics-600 dataset from Google DeepMind, the AVA dataset from Berkeley and Google, and the Moments in Time dataset from MIT and IBM Research.
Diagnosing Error in Temporal Action Detectors
Despite the recent progress in video understanding and the continuous rate of improvement in temporal action localization throughout the years, it is still unclear how far (or close?)
Finding Tiny Faces in the Wild With Generative Adversarial Network
In this paper, we proposed an algorithm to directly generate a clear high-resolution face from a blurry small one by adopting a generative adversarial network (GAN).
W2F: A Weakly-Supervised to Fully-Supervised Framework for Object Detection
Finally, we use these pseudo ground-truths to train a fully-supervised detector.
Analytic Expressions for Probabilistic Moments of PL-DNN With Gaussian Input
Moreover, we show how these expressions can be used to systematically construct targeted and non-targeted adversarial attacks.
Driving Policy Transfer via Modularity and Abstraction
Simulation can help end-to-end driving systems by providing a cheap, safe, and diverse training environment.
SoccerNet: A Scalable Dataset for Action Spotting in Soccer Videos
A total of 6, 637 temporal annotations are automatically parsed from online match reports at a one minute resolution for three main classes of events (Goal, Yellow/Red Card, and Substitution).
Ranked #6 on
Action Spotting
on SoccerNet
Supervised Convolutional Sparse Coding
Convolutional Sparse Coding (CSC) is a well-established image representation model especially suited for image restoration tasks.
Guess Where? Actor-Supervision for Spatiotemporal Action Localization
Second, we propose an actor-based attention mechanism that enables the localization of the actions from action class labels and actor proposals and is end-to-end trainable.
Multi-label Learning with Missing Labels using Mixed Dependency Graphs
This work focuses on the problem of multi-label learning with missing labels (MLML), which aims to label each test instance with multiple class labels given training instances that have an incomplete/partial set of these labels.
Tagging like Humans: Diverse and Distinct Image Annotation
In D2IA, we generate a relevant and distinct tag subset, in which the tags are relevant to the image contents and semantically distinct to each other, using sequential sampling from a determinantal point process (DPP) model.
TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild
In this work, we present TrackingNet, the first large-scale dataset and benchmark for object tracking in the wild.
OIL: Observational Imitation Learning
Recent work has explored the problem of autonomous navigation by imitating a teacher and learning an end-to-end policy, which directly predicts controls from raw images.
Contextual Multi-Scale Region Convolutional 3D Network for Activity Detection
In this paper, we propose the contextual multi-scale region convolutional 3D network (CMS-RC3D) for activity detection.
A Novel Framework for Robustness Analysis of Visual QA Models
In VQA, adversarial attacks can target the image and/or the proposed main question and yet there is a lack of proper analysis of the later.
ActivityNet Challenge 2017 Summary
The ActivityNet Large Scale Activity Recognition Challenge 2017 Summary: results and challenge participants papers.
2D-Driven 3D Object Detection in RGB-D Images
We then use the 3D information to orient, place, and score bounding boxes around objects.
Ranked #4 on
Object Detection In Indoor Scenes
on SUN RGB-D
Constrained Convolutional Sparse Coding for Parametric Based Reconstruction of Line Drawings
The process of drawing a line drawing can be approximated as the sparse spatial localization of a number of typical basic strokes, which in turn can be cast as a non-standard CSC model that considers the line drawing formation process from parametric curves.
