Code     Paper [CVPR 2020]     Paper [Arxiv]    

Abstract

We propose a novel method for combining syntheticand real images when training networks to determine geometric information from a single image. We suggest a method for mapping both image types into a single, shared domain. This is connected to a primary network for end-to-end training. Ideally, this results in images from two domains that present shared information to the primary network. Our experiments demonstrate significant improvements over the state-of-the-art in two important domains, surface normal estimation of human faces and monocular depth estimation for outdoor scenes, both in an unsupervised setting.

Idea


We propose to reduce the domain gap between synthetic and real by mapping the corresponding domain specific information related to the primary task (δs , δr) into shared information δsh, preserving everything else.

Method
Overview of the proposed architecture.

Results

Qualitative results for Monocular Depth Estimation on KITTI


We show qualitative comparison with GASDA on eigen test split of KITTI dataset. Notice that our method estimates better depth map and is relatively closer to the ground truth. The primary difference between our approach and GASDA is the presence of the SharinGAN module that presents a shared domain to the primary network.

Generalization to Make3D dataset (unseen during training)


We also show the generalization of our approach to a completely unseen dataset, Make3D and compare it with GASDA. We were able to capitalize on mapping to the shared domain that boosts the generalization performance of the primary network to unseen domains.

Qualitative results for Face Normal Estimation


We also show the efficacy of our approach for predicting face normals and we qualitatively compare it with SfsNet on the testset of the Photoface dataset.

Bibtex citation

@InProceedings{PNVR_2020_CVPR,
author = {PNVR, Koutilya and Zhou, Hao and Jacobs, David},
title = {SharinGAN: Combining Synthetic and Real Data for Unsupervised Geometry Estimation},
booktitle = {The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2020}
}

Contact

Koutilya PNVR