Depth Estimation from a Single RGB Image Scientific Director: Nassir Navab Contact Person(s): Iro Laina, Christian Rupprecht

Keywords: Computer Vision

Abstract

Depth estimation from multiple views (stereo vision) or other single-view assumptions (motion, shading, defocus) has been well studied in literature. However, estimating the depth map of a scene from a single RGB image remains an open problem due to the inherent ambiguity of mapping color/intensities to depth values, i.e. a 2D image could correspond to multiple 3D world scenarios. Within the spectrum of this project, we address this problem using a deep learning approach. We have investigated different Convolutional Neural Network (CNN) architectures and loss functions for optimization of the task at hand. Particular focus is given on in-network upsampling layers with learnable weights, aiming to optimally tackle the problem of high dimensional outputs without an excessive number of parameters. The best performing model encompasses residual learning and delivers state-of-the-art, real-time performance on depth prediction from images or videos of indoor and outdoor scenes. The methods developed within this project can be applicable to several other dense prediction problems as well.

Pictures

Figure 1: Results of depth estimation on benchmark datasets.

Figure 2: 3D reconstruction (point cloud) from a single image.

Team

Contact Person(s)

Dr. Christian Rupprecht

M.Sc. Iro Laina

Working Group

Dr. Christian Rupprecht

Federico Tombari

M.Sc. Iro Laina

M.Sc. Keisuke Tateno

Location

Technische Universität München Institut für Informatik / I16 Boltzmannstr. 3 85748 Garching bei München Tel.: +49 89 289-17058 Fax: +49 89 289-17059

Visit our lab at Garching.



internal project page

Please contact Iro Laina, Christian Rupprecht for available student projects within this research project.

Software

The trained models (indoors on NYU Depth v2; outdoors on Make3D) are publicly available and can be downloaded here:

• NYUDepth MatConvNet^?
• NYUDepth TensorFlow^?
• Make3D MatConvNet^?

Check out also our GitHub page, including the code for evaluation and inference.
The CNN-based depth estimation can be utilized by a monocular SLAM framework for successful scene reconstruction. For more information please visit this project .

Related Publications

2016
	I. Laina, C. Rupprecht, V. Belagiannis, F. Tombari, N. Navab Deeper Depth Prediction with Fully Convolutional Residual Networks International Conference on 3DVision (3DV), Stanford University, California, USA, October 2016. Oral presentation. The first two authors contribute equally to this paper. (bib)

2017
	K. Tateno, F. Tombari, I. Laina, N. Navab CNN-SLAM: Real-time dense monocular SLAM with learned depth prediction IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), Hawaii, USA, June, 2017. The first two authors contribute equally to this paper. (bib)