Search Results for author:
Found 64 papers, 22 papers with code
Du²Net: Learning Depth Estimation from Dual-Cameras and Dual-Pixels
Computational stereo has reached a high level of accuracy, but degrades in the presence of occlusions, repeated textures, and correspondence errors along edges.
GeneAvatar: Generic Expression-Aware Volumetric Head Avatar Editing from a Single Image
Recently, we have witnessed the explosive growth of various volumetric representations in modeling animatable head avatars.
CHOSEN: Contrastive Hypothesis Selection for Multi-View Depth Refinement
We propose CHOSEN, a simple yet flexible, robust and effective multi-view depth refinement framework.
Efficient 3D Implicit Head Avatar with Mesh-anchored Hash Table Blendshapes
To address these challenges, we propose a novel fast 3D neural implicit head avatar model that achieves real-time rendering while maintaining fine-grained controllability and high rendering quality.
T-Pixel2Mesh: Combining Global and Local Transformer for 3D Mesh Generation from a Single Image
Pixel2Mesh (P2M) is a classical approach for reconstructing 3D shapes from a single color image through coarse-to-fine mesh deformation.
Pushing Auto-regressive Models for 3D Shape Generation at Capacity and Scalability
In this paper, we extend auto-regressive models to 3D domains, and seek a stronger ability of 3D shape generation by improving auto-regressive models at capacity and scalability simultaneously.
One2Avatar: Generative Implicit Head Avatar For Few-shot User Adaptation
Traditional methods for constructing high-quality, personalized head avatars from monocular videos demand extensive face captures and training time, posing a significant challenge for scalability.
GO-NeRF: Generating Virtual Objects in Neural Radiance Fields
To this end, we propose a new method, GO-NeRF, capable of utilizing scene context for high-quality and harmonious 3D object generation within an existing NeRF.
InstructPipe: Building Visual Programming Pipelines with Human Instructions
Our user study (N=16) showed that InstructPipe empowers novice users to streamline their workflow in creating desired ML pipelines, reduce their learning curve, and spark innovative ideas with open-ended commands.
MVDD: Multi-View Depth Diffusion Models
State-of-the-art results from extensive experiments demonstrate MVDD's excellent ability in 3D shape generation, depth completion, and its potential as a 3D prior for downstream tasks.
Gaussian3Diff: 3D Gaussian Diffusion for 3D Full Head Synthesis and Editing
We present a novel framework for generating photorealistic 3D human head and subsequently manipulating and reposing them with remarkable flexibility.
Multi-Modal Neural Radiance Field for Monocular Dense SLAM with a Light-Weight ToF Sensor
Specifically, we propose a multi-modal implicit scene representation that supports rendering both the signals from the RGB camera and light-weight ToF sensor which drives the optimization by comparing with the raw sensor inputs.
Self-supervised Learning of Implicit Shape Representation with Dense Correspondence for Deformable Objects
In this paper, we propose a novel self-supervised approach to learn neural implicit shape representation for deformable objects, which can represent shapes with a template shape and dense correspondence in 3D.
Novel-view Synthesis and Pose Estimation for Hand-Object Interaction from Sparse Views
Hand-object interaction understanding and the barely addressed novel view synthesis are highly desired in the immersive communication, whereas it is challenging due to the high deformation of hand and heavy occlusions between hand and object.
Spectral Graphormer: Spectral Graph-based Transformer for Egocentric Two-Hand Reconstruction using Multi-View Color Images
We propose a novel transformer-based framework that reconstructs two high fidelity hands from multi-view RGB images.
Hybrid Neural Rendering for Large-Scale Scenes with Motion Blur
Rendering novel view images is highly desirable for many applications.
Grad-PU: Arbitrary-Scale Point Cloud Upsampling via Gradient Descent with Learned Distance Functions
Most existing point cloud upsampling methods have roughly three steps: feature extraction, feature expansion and 3D coordinate prediction.
Learning Personalized High Quality Volumetric Head Avatars from Monocular RGB Videos
The learnt avatar is driven by a parametric face model to achieve user-controlled facial expressions and head poses.
Learning Versatile 3D Shape Generation with Improved AR Models
Auto-Regressive (AR) models have achieved impressive results in 2D image generation by modeling joint distributions in the grid space.
SINE: Semantic-driven Image-based NeRF Editing with Prior-guided Editing Field
Despite the great success in 2D editing using user-friendly tools, such as Photoshop, semantic strokes, or even text prompts, similar capabilities in 3D areas are still limited, either relying on 3D modeling skills or allowing editing within only a few categories.
Learning Versatile 3D Shape Generation with Improved Auto-regressive Models
Auto-Regressive (AR) models have achieved impressive results in 2D image generation by modeling joint distributions in the grid space.
DELTAR: Depth Estimation from a Light-weight ToF Sensor and RGB Image
Light-weight time-of-flight (ToF) depth sensors are small, cheap, low-energy and have been massively deployed on mobile devices for the purposes like autofocus, obstacle detection, etc.
LoRD: Local 4D Implicit Representation for High-Fidelity Dynamic Human Modeling
Recent progress in 4D implicit representation focuses on globally controlling the shape and motion with low dimensional latent vectors, which is prone to missing surface details and accumulating tracking error.
PRIF: Primary Ray-based Implicit Function
We introduce a new implicit shape representation called Primary Ray-based Implicit Function (PRIF).
NeuMesh: Learning Disentangled Neural Mesh-based Implicit Field for Geometry and Texture Editing
Very recently neural implicit rendering techniques have been rapidly evolved and shown great advantages in novel view synthesis and 3D scene reconstruction.
Neural Rendering in a Room: Amodal 3D Understanding and Free-Viewpoint Rendering for the Closed Scene Composed of Pre-Captured Objects
We, as human beings, can understand and picture a familiar scene from arbitrary viewpoints given a single image, whereas this is still a grand challenge for computers.
Density-preserving Deep Point Cloud Compression
To address this, we propose a novel deep point cloud compression method that preserves local density information.
Pixel2Mesh++: 3D Mesh Generation and Refinement from Multi-View Images
We study the problem of shape generation in 3D mesh representation from a small number of color images with or without camera poses.
Efficient Virtual View Selection for 3D Hand Pose Estimation
3D hand pose estimation from single depth is a fundamental problem in computer vision, and has wide applications. However, the existing methods still can not achieve satisfactory hand pose estimation results due to view variation and occlusion of human hand.
Ranked #1 on
Hand Pose Estimation
on ICVL Hands
H4D: Human 4D Modeling by Learning Neural Compositional Representation
A simple yet effective linear motion model is proposed to provide a rough and regularized motion estimation, followed by per-frame compensation for pose and geometry details with the residual encoded in the auxiliary code.
OmniSyn: Synthesizing 360 Videos with Wide-baseline Panoramas
Immersive maps such as Google Street View and Bing Streetside provide true-to-life views with a massive collection of panoramas.
VoLux-GAN: A Generative Model for 3D Face Synthesis with HDRI Relighting
We propose VoLux-GAN, a generative framework to synthesize 3D-aware faces with convincing relighting.
Multiresolution Deep Implicit Functions for 3D Shape Representation
To the best of our knowledge, MDIF is the first deep implicit function model that can at the same time (1) represent different levels of detail and allow progressive decoding; (2) support both encoder-decoder inference and decoder-only latent optimization, and fulfill multiple applications; (3) perform detailed decoder-only shape completion.
Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering
In this paper, we present a novel neural scene rendering system, which learns an object-compositional neural radiance field and produces realistic rendering with editing capability for a clustered and real-world scene.
DeepPanoContext: Panoramic 3D Scene Understanding with Holistic Scene Context Graph and Relation-based Optimization
Panorama images have a much larger field-of-view thus naturally encode enriched scene context information compared to standard perspective images, which however is not well exploited in the previous scene understanding methods.
Deep Hybrid Self-Prior for Full 3D Mesh Generation
We present a deep learning pipeline that leverages network self-prior to recover a full 3D model consisting of both a triangular mesh and a texture map from the colored 3D point cloud.
Spatially-Varying Outdoor Lighting Estimation from Intrinsics
We train a deep neural network to regress intrinsic cues with physically-based constraints and use them to conduct global and local lightings estimation.
HumanGPS: Geodesic PreServing Feature for Dense Human Correspondences
In this paper, we address the problem of building dense correspondences between human images under arbitrary camera viewpoints and body poses.
Learning Compositional Representation for 4D Captures with Neural ODE
To model the motion, a neural Ordinary Differential Equation (ODE) is trained to update the initial state conditioned on the learned motion code, and a decoder takes the shape code and the updated state code to reconstruct the 3D model at each time stamp.
Holistic 3D Scene Understanding from a Single Image with Implicit Representation
We not only propose an image-based local structured implicit network to improve the object shape estimation, but also refine the 3D object pose and scene layout via a novel implicit scene graph neural network that exploits the implicit local object features.
Ranked #1 on
Monocular 3D Object Detection
on SUN RGB-D
(using extra training data)
Interacting Two-Hand 3D Pose and Shape Reconstruction From Single Color Image
In this paper, we propose a novel deep learning framework to reconstruct 3D hand poses and shapes of two interacting hands from a single color image.
Ranked #6 on
3D Interacting Hand Pose Estimation
on InterHand2.6M
GeoLayout: Geometry Driven Room Layout Estimation Based on Depth Maps of Planes
The task of room layout estimation is to locate the wall-floor, wall-ceiling, and wall-wall boundaries.
HITNet: Hierarchical Iterative Tile Refinement Network for Real-time Stereo Matching
Contrary to many recent neural network approaches that operate on a full cost volume and rely on 3D convolutions, our approach does not explicitly build a volume and instead relies on a fast multi-resolution initialization step, differentiable 2D geometric propagation and warping mechanisms to infer disparity hypotheses.
Ranked #1 on
Stereo Depth Estimation
on KITTI2015
Deep Implicit Volume Compression
We describe a novel approach for compressing truncated signed distance fields (TSDF) stored in 3D voxel grids, and their corresponding textures.
Du$^2$Net: Learning Depth Estimation from Dual-Cameras and Dual-Pixels
Computational stereo has reached a high level of accuracy, but degrades in the presence of occlusions, repeated textures, and correspondence errors along edges.
Neural Pose Transfer by Spatially Adaptive Instance Normalization
Pose transfer has been studied for decades, in which the pose of a source mesh is applied to a target mesh.
DeepSFM: Structure From Motion Via Deep Bundle Adjustment
The explicit constraints on both depth (structure) and pose (motion), when combined with the learning components, bring the merit from both traditional BA and emerging deep learning technology.
Neural Point Cloud Rendering via Multi-Plane Projection
The input to the network is the raw point cloud of a scene and the output are image or image sequences from a novel view or along a novel camera trajectory.
DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracing
We propose a differentiable sphere tracing algorithm to bridge the gap between inverse graphics methods and the recently proposed deep learning based implicit signed distance function.
Pixel2Mesh++: Multi-View 3D Mesh Generation via Deformation
We study the problem of shape generation in 3D mesh representation from a few color images with known camera poses.
DeepLiDAR: Deep Surface Normal Guided Depth Prediction for Outdoor Scene from Sparse LiDAR Data and Single Color Image
In this paper, we propose a deep learning architecture that produces accurate dense depth for the outdoor scene from a single color image and a sparse depth.
ActiveStereoNet: End-to-End Self-Supervised Learning for Active Stereo Systems
In this paper we present ActiveStereoNet, the first deep learning solution for active stereo systems.
Multi-level Semantic Feature Augmentation for One-shot Learning
In semantic space, we search for related concepts, which are then projected back into the image feature spaces by the decoder portion of the TriNet.
Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images
We propose an end-to-end deep learning architecture that produces a 3D shape in triangular mesh from a single color image.
Ranked #3 on
3D Object Reconstruction
on Data3D−R2N2
(Avg F1 metric)
Deep Depth Completion of a Single RGB-D Image
The goal of our work is to complete the depth channel of an RGB-D image.
VOCABULARY-INFORMED VISUAL FEATURE AUGMENTATION FOR ONE-SHOT LEARNING
Besides directly augmenting image features, we transform the image features to semantic space using the encoder and perform the data augmentation.
Matterport3D: Learning from RGB-D Data in Indoor Environments
Access to large, diverse RGB-D datasets is critical for training RGB-D scene understanding algorithms.
Hand3D: Hand Pose Estimation using 3D Neural Network
We propose a novel 3D neural network architecture for 3D hand pose estimation from a single depth image.
Physically-Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networks
One of the bottlenecks in training for better representations is the amount of available per-pixel ground truth data that is required for core scene understanding tasks such as semantic segmentation, normal prediction, and object edge detection.
Joint Hand Detection and Rotation Estimation by Using CNN
Hand detection is essential for many hand related tasks, e. g. parsing hand pose, understanding gesture, which are extremely useful for robotics and human-computer interaction.
DeepContext: Context-Encoding Neural Pathways for 3D Holistic Scene Understanding
In particular, 3D context has been shown to be an extremely important cue for scene understanding - yet very little research has been done on integrating context information with deep models.
LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
While there has been remarkable progress in the performance of visual recognition algorithms, the state-of-the-art models tend to be exceptionally data-hungry.
TurkerGaze: Crowdsourcing Saliency with Webcam based Eye Tracking
Traditional eye tracking requires specialized hardware, which means collecting gaze data from many observers is expensive, tedious and slow.
FrameBreak: Dramatic Image Extrapolation by Guided Shift-Maps
We analyze the self-similarity of the guide image to generate a set of allowable local transformations and apply them to the input image.
