Search Results for author:
Found 78 papers, 35 papers with code
Unsupervised Video Object Segmentation with Joint Hotspot Tracking
We reformulate the problem of detecting and tracking of salient object spots as a new task called object hotspot tracking.
CLIFFNet for Monocular Depth Estimation with Hierarchical Embedding Loss
This paper proposes a hierarchical loss for monocular depth estimation, which measures the differences between the prediction and ground truth in hierarchical embedding spaces of depth maps.
SwapAnything: Enabling Arbitrary Object Swapping in Personalized Visual Editing
Compared with existing methods for personalized subject swapping, SwapAnything has three unique advantages: (1) precise control of arbitrary objects and parts rather than the main subject, (2) more faithful preservation of context pixels, (3) better adaptation of the personalized concept to the image.
IMPRINT: Generative Object Compositing by Learning Identity-Preserving Representation
Generative object compositing emerges as a promising new avenue for compositional image editing.
Holo-Relighting: Controllable Volumetric Portrait Relighting from a Single Image
At the core of portrait photography is the search for ideal lighting and viewpoint.
UniHuman: A Unified Model for Editing Human Images in the Wild
In this paper, we propose UniHuman, a unified model that addresses multiple facets of human image editing in real-world settings.
Relightful Harmonization: Lighting-aware Portrait Background Replacement
Portrait harmonization aims to composite a subject into a new background, adjusting its lighting and color to ensure harmony with the background scene.
SmartMask: Context Aware High-Fidelity Mask Generation for Fine-grained Object Insertion and Layout Control
To overcome these limitations, we introduce SmartMask, which allows any novice user to create detailed masks for precise object insertion.
Fast View Synthesis of Casual Videos
Specifically, we build a global static scene model using an extended plane-based scene representation to synthesize temporally coherent novel video.
Lasagna: Layered Score Distillation for Disentangled Object Relighting
Although generative editing methods now enable some forms of image editing, relighting is still beyond today's capabilities; existing methods struggle to keep other aspects of the image -- colors, shapes, and textures -- consistent after the edit.
Diffusion-Augmented Depth Prediction with Sparse Annotations
We leverage the structural characteristics of diffusion model to enforce depth structures of depth models in a plug-and-play manner.
Neural Video Depth Stabilizer
Video depth estimation aims to infer temporally consistent depth.
Ranked #16 on
Monocular Depth Estimation
on NYU-Depth V2
(using extra training data)
LightPainter: Interactive Portrait Relighting with Freehand Scribble
Recent portrait relighting methods have achieved realistic results of portrait lighting effects given a desired lighting representation such as an environment map.
3D Cinemagraphy from a Single Image
To animate the scene, we perform motion estimation and lift the 2D motion into the 3D scene flow.
PixHt-Lab: Pixel Height Based Light Effect Generation for Image Compositing
To compensate for the lack of geometry in 2D Image compositing, recent deep learning-based approaches introduced a pixel height representation to generate soft shadows and reflections.
Single View Scene Scale Estimation Using Scale Field
In this paper, we propose a single image scale estimation method based on a novel scale field representation.
Lens Parameter Estimation for Realistic Depth of Field Modeling
Instead, we go further and propose to use a lens-based representation that models the depth of field using two parameters: the blur factor and focus disparity.
GAIT: Generating Aesthetic Indoor Tours with Deep Reinforcement Learning
Placing and orienting a camera to compose aesthetically meaningful shots of a scene is not only a key objective in real-world photography and cinematography but also for virtual content creation.
ObjectStitch: Object Compositing With Diffusion Model
Object compositing based on 2D images is a challenging problem since it typically involves multiple processing stages such as color harmonization, geometry correction and shadow generation to generate realistic results.
Structure-Guided Image Completion with Image-level and Object-level Semantic Discriminators
Moreover, the object-level discriminators take aligned instances as inputs to enforce the realism of individual objects.
Perspective Fields for Single Image Camera Calibration
We propose perspective fields as a representation that models the local perspective properties of an image.
ObjectStitch: Generative Object Compositing
Object compositing based on 2D images is a challenging problem since it typically involves multiple processing stages such as color harmonization, geometry correction and shadow generation to generate realistic results.
SceneComposer: Any-Level Semantic Image Synthesis
We propose a new framework for conditional image synthesis from semantic layouts of any precision levels, ranging from pure text to a 2D semantic canvas with precise shapes.
Towards Accurate Reconstruction of 3D Scene Shape from A Single Monocular Image
To do so, we propose a two-stage framework that first predicts depth up to an unknown scale and shift from a single monocular image, and then exploits 3D point cloud data to predict the depth shift and the camera's focal length that allow us to recover 3D scene shapes.
Text-to-Image Generation via Implicit Visual Guidance and Hypernetwork
We develop an approach for text-to-image generation that embraces additional retrieval images, driven by a combination of implicit visual guidance loss and generative objectives.
Less is More: Consistent Video Depth Estimation with Masked Frames Modeling
Temporal consistency is the key challenge of video depth estimation.
Towards Domain-agnostic Depth Completion
Our method leverages a data-driven prior in the form of a single image depth prediction network trained on large-scale datasets, the output of which is used as an input to our model.
MPIB: An MPI-Based Bokeh Rendering Framework for Realistic Partial Occlusion Effects
Partial occlusion effects are a phenomenon that blurry objects near a camera are semi-transparent, resulting in partial appearance of occluded background.
Controllable Shadow Generation Using Pixel Height Maps
It can be used to calculate hard shadows in a 2D image based on the projective geometry, providing precise control of the shadows' direction and shape.
Dynamic Gradient Reactivation for Backward Compatible Person Re-identification
To address this issue, we propose the Ranking-based Backward Compatible Learning (RBCL), which directly optimizes the ranking metric between new features and old features.
BokehMe: When Neural Rendering Meets Classical Rendering
Based on this formulation, we implement the classical renderer by a scattering-based method and propose a two-stage neural renderer to fix the erroneous areas from the classical renderer.
Layered Depth Refinement with Mask Guidance
Depth maps are used in a wide range of applications from 3D rendering to 2D image effects such as Bokeh.
CM-GAN: Image Inpainting with Cascaded Modulation GAN and Object-Aware Training
We propose cascaded modulation GAN (CM-GAN), a new network design consisting of an encoder with Fourier convolution blocks that extract multi-scale feature representations from the input image with holes and a dual-stream decoder with a novel cascaded global-spatial modulation block at each scale level.
Ranked #1 on
Image Inpainting
on Places2
Interactive Portrait Harmonization
To enable flexible interaction between user and harmonization, we introduce interactive harmonization, a new setting where the harmonization is performed with respect to a selected \emph{region} in the reference image instead of the entire background.
Lite Vision Transformer with Enhanced Self-Attention
We propose Lite Vision Transformer (LVT), a novel light-weight transformer network with two enhanced self-attention mechanisms to improve the model performances for mobile deployment.
SSH: A Self-Supervised Framework for Image Harmonization
Image harmonization aims to improve the quality of image compositing by matching the "appearance" (\eg, color tone, brightness and contrast) between foreground and background images.
Black-Box Diagnosis and Calibration on GAN Intra-Mode Collapse: A Pilot Study
Existing diversity tests of samples from GANs are usually conducted qualitatively on a small scale, and/or depends on the access to original training data as well as the trained model parameters.
Single-image Full-body Human Relighting
We present a single-image data-driven method to automatically relight images with full-body humans in them.
Multimodal Contrastive Training for Visual Representation Learning
We first train our model on COCO and evaluate the learned visual representations on various downstream tasks including image classification, object detection, and instance segmentation.
Learning to Recover 3D Scene Shape from a Single Image
Despite significant progress in monocular depth estimation in the wild, recent state-of-the-art methods cannot be used to recover accurate 3D scene shape due to an unknown depth shift induced by shift-invariant reconstruction losses used in mixed-data depth prediction training, and possible unknown camera focal length.
Ranked #1 on
Indoor Monocular Depth Estimation
on DIODE
(using extra training data)
Semantic Layout Manipulation with High-Resolution Sparse Attention
A core problem of this task is how to transfer visual details from the input images to the new semantic layout while making the resulting image visually realistic.
Mask Guided Matting via Progressive Refinement Network
We propose Mask Guided (MG) Matting, a robust matting framework that takes a general coarse mask as guidance.
Deep Image Compositing
In this paper, we propose a new method which can automatically generate high-quality image compositing without any user input.
Attribute-conditioned Layout GAN for Automatic Graphic Design
In this paper, we introduce Attribute-conditioned Layout GAN to incorporate the attributes of design elements for graphic layout generation by forcing both the generator and the discriminator to meet attribute conditions.
Adversarial Knowledge Transfer from Unlabeled Data
While machine learning approaches to visual recognition offer great promise, most of the existing methods rely heavily on the availability of large quantities of labeled training data.
Open-Edit: Open-Domain Image Manipulation with Open-Vocabulary Instructions
We propose a novel algorithm, named Open-Edit, which is the first attempt on open-domain image manipulation with open-vocabulary instructions.
Shape Adaptor: A Learnable Resizing Module
We present a novel resizing module for neural networks: shape adaptor, a drop-in enhancement built on top of traditional resizing layers, such as pooling, bilinear sampling, and strided convolution.
SSN: Soft Shadow Network for Image Compositing
We demonstrate that our model produces realistic soft shadows in real-time.
High-Resolution Image Inpainting with Iterative Confidence Feedback and Guided Upsampling
To address this challenge, we propose an iterative inpainting method with a feedback mechanism.
Ranked #6 on
Image Inpainting
on Places2
Explainable Deep Classification Models for Domain Generalization
Explanations are defined as regions of visual evidence upon which a deep classification network makes a decision.
Is There Mode Collapse? A Case Study on Face Generation and Its Black-box Calibration
Generative adversarial networks (GANs) nowadays are capable of producing im-ages of incredible realism.
Scaling Object Detection by Transferring Classification Weights
Large scale object detection datasets are constantly increasing their size in terms of the number of classes and annotations count.
Multi-Channel Deep Networks for Block-Based Image Compressive Sensing
Specifically, with our multichannel structure, the image blocks with a variety of sampling rates can be reconstructed in a single model.
Towards High-Resolution Salient Object Detection
This paper pushes forward high-resolution saliency detection, and contributes a new dataset, named High-Resolution Salient Object Detection (HRSOD).
Ranked #11 on
RGB Salient Object Detection
on DAVIS-S
(using extra training data)
Multi-way Encoding for Robustness
However, it has been shown that deep models are vulnerable to adversarial examples.
LayoutGAN: Generating Graphic Layouts with Wireframe Discriminator
Layouts are important for graphic design and scene generation.
Multi-way Encoding for Robustness to Adversarial Attacks
Deep models are state-of-the-art for many computer vision tasks including image classification and object detection.
M2KD: Multi-model and Multi-level Knowledge Distillation for Incremental Learning
Instead of sequentially distilling knowledge only from the last model, we directly leverage all previous model snapshots.
LayoutGAN: Generating Graphic Layouts with Wireframe Discriminators
Layout is important for graphic design and scene generation.
Guided Zoom: Questioning Network Evidence for Fine-grained Classification
We propose Guided Zoom, an approach that utilizes spatial grounding of a model's decision to make more informed predictions.
Neural Rejuvenation: Improving Deep Network Training by Enhancing Computational Resource Utilization
By simply replacing standard optimizers with Neural Rejuvenation, we are able to improve the performances of neural networks by a very large margin while using similar training efforts and maintaining their original resource usages.
Sequence-to-Segment Networks for Segment Detection
Detecting segments of interest from an input sequence is a challenging problem which often requires not only good knowledge of individual target segments, but also contextual understanding of the entire input sequence and the relationships between the target segments.
DeepLens: Shallow Depth Of Field From A Single Image
To achieve this, we propose a novel neural network model comprised of a depth prediction module, a lens blur module, and a guided upsampling module.
GAPLE: Generalizable Approaching Policy LEarning for Robotic Object Searching in Indoor Environment
We study the problem of learning a generalizable action policy for an intelligent agent to actively approach an object of interest in an indoor environment solely from its visual inputs.
Learning to Blend Photos
Specifically, given a foreground image and a background image, our proposed method automatically generates a set of blending photos with scores that indicate the aesthetics quality with the proposed quality network and policy network.
Concept Mask: Large-Scale Segmentation from Semantic Concepts
Then, we refine and extend the embedding network to predict an attention map, using a curated dataset with bounding box annotations on 750 concepts.
Contemplating Visual Emotions: Understanding and Overcoming Dataset Bias
While machine learning approaches to visual emotion recognition offer great promise, current methods consider training and testing models on small scale datasets covering limited visual emotion concepts.
Good View Hunting: Learning Photo Composition From Dense View Pairs
Finding views with good photo composition is a challenging task for machine learning methods.
Excitation Dropout: Encouraging Plasticity in Deep Neural Networks
In this work, we utilize the evidence at each neuron to determine the probability of dropout, rather than dropping out neurons uniformly at random as in standard dropout.
Excitation Backprop for RNNs
Models are trained to caption or classify activity in videos, but little is known about the evidence used to make such decisions.
Predicting Foreground Object Ambiguity and Efficiently Crowdsourcing the Segmentation(s)
We propose the ambiguity problem for the foreground object segmentation task and motivate the importance of estimating and accounting for this ambiguity when designing vision systems.
Top-down Visual Saliency Guided by Captions
Neural image/video captioning models can generate accurate descriptions, but their internal process of mapping regions to words is a black box and therefore difficult to explain.
Top-down Neural Attention by Excitation Backprop
We aim to model the top-down attention of a Convolutional Neural Network (CNN) classifier for generating task-specific attention maps.
Salient Object Subitizing
We study the problem of Salient Object Subitizing, i. e. predicting the existence and the number of salient objects in an image using holistic cues.
Unconstrained Salient Object Detection via Proposal Subset Optimization
Our system leverages a Convolutional-Neural-Network model to generate location proposals of salient objects.
Do Less and Achieve More: Training CNNs for Action Recognition Utilizing Action Images from the Web
In contrast, collecting action images from the Web is much easier and training on images requires much less computation.
Ranked #14 on
Action Recognition
on ActivityNet
(using extra training data)
Minimum Barrier Salient Object Detection at 80 FPS
Powered by this fast MBD transform algorithm, the proposed salient object detection method runs at 80 FPS, and significantly outperforms previous methods with similar speed on four large benchmark datasets, and achieves comparable or better performance than state-of-the-art methods.
Ranked #6 on
Video Salient Object Detection
on VOS-T
(using extra training data)
