Search Results for author:
Found 57 papers, 13 papers with code
RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusion
We introduce RealmDreamer, a technique for generation of general forward-facing 3D scenes from text descriptions.
Lift3D: Zero-Shot Lifting of Any 2D Vision Model to 3D
In recent years, there has been an explosion of 2D vision models for numerous tasks such as semantic segmentation, style transfer or scene editing, enabled by large-scale 2D image datasets.
What You See is What You GAN: Rendering Every Pixel for High-Fidelity Geometry in 3D GANs
3D-aware Generative Adversarial Networks (GANs) have shown remarkable progress in learning to generate multi-view-consistent images and 3D geometries of scenes from collections of 2D images via neural volume rendering.
OpenIllumination: A Multi-Illumination Dataset for Inverse Rendering Evaluation on Real Objects
We introduce OpenIllumination, a real-world dataset containing over 108K images of 64 objects with diverse materials, captured under 72 camera views and a large number of different illuminations.
A Theory of Topological Derivatives for Inverse Rendering of Geometry
We introduce a theoretical framework for differentiable surface evolution that allows discrete topology changes through the use of topological derivatives for variational optimization of image functionals.
NeRFs: The Search for the Best 3D Representation
Neural Radiance Fields or NeRFs have become the representation of choice for problems in view synthesis or image-based rendering, as well as in many other applications across computer graphics and vision, and beyond.
Neural Free-Viewpoint Relighting for Glossy Indirect Illumination
In this paper, we demonstrate a hybrid neural-wavelet PRT solution to high-frequency indirect illumination, including glossy reflection, for relighting with changing view.
PVP: Personalized Video Prior for Editable Dynamic Portraits using StyleGAN
In this work, our goal is to take as input a monocular video of a face, and create an editable dynamic portrait able to handle extreme head poses.
Real-Time Radiance Fields for Single-Image Portrait View Synthesis
We present a one-shot method to infer and render a photorealistic 3D representation from a single unposed image (e. g., face portrait) in real-time.
Factorized Inverse Path Tracing for Efficient and Accurate Material-Lighting Estimation
Inverse path tracing has recently been applied to joint material and lighting estimation, given geometry and multi-view HDR observations of an indoor scene.
NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusion
Novel view synthesis from a single image requires inferring occluded regions of objects and scenes whilst simultaneously maintaining semantic and physical consistency with the input.
Vision Transformer for NeRF-Based View Synthesis from a Single Input Image
Existing approaches condition on local image features to reconstruct a 3D object, but often render blurry predictions at viewpoints that are far away from the source view.
Physically-Based Editing of Indoor Scene Lighting from a Single Image
We tackle this problem using two novel components: 1) a holistic scene reconstruction method that estimates scene reflectance and parametric 3D lighting, and 2) a neural rendering framework that re-renders the scene from our predictions.
A Level Set Theory for Neural Implicit Evolution under Explicit Flows
Our method uses the flow field to deform parametric implicit surfaces by extending the classical theory of level sets.
Learning Neural Transmittance for Efficient Rendering of Reflectance Fields
However, it remains challenging and time-consuming to render such representations under complex lighting such as environment maps, which requires individual ray marching towards each single light to calculate the transmittance at every sampled point.
View Synthesis of Dynamic Scenes based on Deep 3D Mask Volume
Image view synthesis has seen great success in reconstructing photorealistic visuals, thanks to deep learning and various novel representations.
NeLF: Neural Light-transport Field for Portrait View Synthesis and Relighting
Our system is trained on a large number of synthetic models, and can generalize to different synthetic and real portraits under various lighting conditions.
OpenRooms: An Open Framework for Photorealistic Indoor Scene Datasets
Finally, we demonstrate that our framework may also be integrated with physics engines, to create virtual robotics environments with unique ground truth such as friction coefficients and correspondence to real scenes.
Modulated Periodic Activations for Generalizable Local Functional Representations
Our approach produces generalizable functional representations of images, videos and shapes, and achieves higher reconstruction quality than prior works that are optimized for a single signal.
NeuMIP: Multi-Resolution Neural Materials
We also introduce neural offsets, a novel method which allows rendering materials with intricate parallax effects without any tessellation.
Light Stage Super-Resolution: Continuous High-Frequency Relighting
The light stage has been widely used in computer graphics for the past two decades, primarily to enable the relighting of human faces.
Photon-Driven Neural Path Guiding
To fully make use of our deep neural network, we partition the scene space into an adaptive hierarchical grid, in which we apply our network to reconstruct high-quality sampling distributions for any local region in the scene.
Real-Time Selfie Video Stabilization
Our method is fully automatic and produces visually and quantitatively better results than previous real-time general video stabilization methods.
Neural Light Transport for Relighting and View Synthesis
In particular, we show how to fuse previously seen observations of illuminants and views to synthesize a new image of the same scene under a desired lighting condition from a chosen viewpoint.
Neural Reflectance Fields for Appearance Acquisition
We combine this representation with a physically-based differentiable ray marching framework that can render images from a neural reflectance field under any viewpoint and light.
Deep Multi Depth Panoramas for View Synthesis
We propose a learning-based approach for novel view synthesis for multi-camera 360$^{\circ}$ panorama capture rigs.
OpenRooms: An End-to-End Open Framework for Photorealistic Indoor Scene Datasets
Finally, we demonstrate that our framework may also be integrated with physics engines, to create virtual robotics environments with unique ground truth such as friction coefficients and correspondence to real scenes.
Deep Reflectance Volumes: Relightable Reconstructions from Multi-View Photometric Images
We also show that our learned reflectance volumes are editable, allowing for modifying the materials of the captured scenes.
Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
We show that passing input points through a simple Fourier feature mapping enables a multilayer perceptron (MLP) to learn high-frequency functions in low-dimensional problem domains.
Deep Photon Mapping
This network is easy to incorporate in many previous photon mapping methods (by simply swapping the kernel density estimator) and can produce high-quality reconstructions of complex global illumination effects like caustics with an order of magnitude fewer photons compared to previous photon mapping methods.
Deep CG2Real: Synthetic-to-Real Translation via Image Disentanglement
We present a method to improve the visual realism of low-quality, synthetic images, e. g. OpenGL renderings.
Deep 3D Capture: Geometry and Reflectance from Sparse Multi-View Images
We introduce a novel learning-based method to reconstruct the high-quality geometry and complex, spatially-varying BRDF of an arbitrary object from a sparse set of only six images captured by wide-baseline cameras under collocated point lighting.
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location $(x, y, z)$ and viewing direction $(\theta, \phi)$) and whose output is the volume density and view-dependent emitted radiance at that spatial location.
Ranked #3 on
Generalizable Novel View Synthesis
on NERDS 360
Generalizable Novel View Synthesis
Low-Dose X-Ray Ct Reconstruction
+2
Deep Stereo using Adaptive Thin Volume Representation with Uncertainty Awareness
In contrast, we propose adaptive thin volumes (ATVs); in an ATV, the depth hypothesis of each plane is spatially varying, which adapts to the uncertainties of previous per-pixel depth predictions.
Ranked #13 on
3D Reconstruction
on DTU
Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Image
Our inverse rendering network incorporates physical insights -- including a spatially-varying spherical Gaussian lighting representation, a differentiable rendering layer to model scene appearance, a cascade structure to iteratively refine the predictions and a bilateral solver for refinement -- allowing us to jointly reason about shape, lighting, and reflectance.
Single Image Portrait Relighting
Lighting plays a central role in conveying the essence and depth of the subject in a portrait photograph.
Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines
We present a practical and robust deep learning solution for capturing and rendering novel views of complex real world scenes for virtual exploration.
Pushing the Boundaries of View Extrapolation with Multiplane Images
We present a theoretical analysis showing how the range of views that can be rendered from an MPI increases linearly with the MPI disparity sampling frequency, as well as a novel MPI prediction procedure that theoretically enables view extrapolations of up to $4\times$ the lateral viewpoint movement allowed by prior work.
Fast and Full-Resolution Light Field Deblurring using a Deep Neural Network
Restoring a sharp light field image from its blurry input has become essential due to the increasing popularity of parallax-based image processing.
Deep Hybrid Real and Synthetic Training for Intrinsic Decomposition
Experimental results show that our approach produces better results than the state-of-the-art DL and non-DL methods on various synthetic and real datasets both visually and numerically.
Image to Image Translation for Domain Adaptation
This is achieved by adding extra networks and losses that help regularize the features extracted by the backbone encoder network.
Depth and Image Restoration From Light Field in a Scattering Medium
First, we present a new single image restoration algorithm which removes backscatter and attenuation from images better than existing methods, and apply it to each view in the light field.
Linear Differential Constraints for Photo-polarimetric Height Estimation
From a numerical point of view, we use a least-squares formulation of the discrete version of the problem.
Learning to Synthesize a 4D RGBD Light Field from a Single Image
We present a machine learning algorithm that takes as input a 2D RGB image and synthesizes a 4D RGBD light field (color and depth of the scene in each ray direction).
Robust Energy Minimization for BRDF-Invariant Shape From Light Fields
On the other hand, recent works have explored PDE invariants for shape recovery with complex BRDFs, but they have not been incorporated into robust numerical optimization frameworks.
Light Field Video Capture Using a Learning-Based Hybrid Imaging System
Given a 3 fps light field sequence and a standard 30 fps 2D video, our system can then generate a full light field video at 30 fps.
Light Field Blind Motion Deblurring
We study the problem of deblurring light fields of general 3D scenes captured under 3D camera motion and present both theoretical and practical contributions.
Learning-Based View Synthesis for Light Field Cameras
Specifically, we propose a novel learning-based approach to synthesize new views from a sparse set of input views.
A 4D Light-Field Dataset and CNN Architectures for Material Recognition
We introduce a new light-field dataset of materials, and take advantage of the recent success of deep learning to perform material recognition on the 4D light-field.
SVBRDF-Invariant Shape and Reflectance Estimation From Light-Field Cameras
Light-field cameras have recently emerged as a powerful tool for one-shot passive 3D shape capture.
Depth From Semi-Calibrated Stereo and Defocus
In this work, we propose a multi-camera system where we combine a main high-quality camera with two low-res auxiliary cameras.
Photometric Stereo in a Scattering Medium
Next, we model the blur due to scattering of light from the object.
Oriented Light-Field Windows for Scene Flow
2D spatial image windows are used for comparing pixel values in computer vision applications such as correspondence for optical flow and 3D reconstruction, bilateral filtering, and image segmentation.
Occlusion-Aware Depth Estimation Using Light-Field Cameras
In this paper, we develop a depth estimation algorithm that treats occlusion explicitly; the method also enables identification of occlusion edges, which may be useful in other applications.
Depth From Shading, Defocus, and Correspondence Using Light-Field Angular Coherence
Using shading information is essential to improve the shape estimation.
What Object Motion Reveals about Shape with Unknown BRDF and Lighting
Under orthographic projection, we prove that three differential motions suffice to yield an invariant that relates shape to image derivatives, regardless of BRDF and illumination.
