Improving high dimensional prediction tasks by leveraging sparse reliable data

Using CNN models for predicting high dimensional maps from RGB data has proved to be successful for several tasks, such as semantic segmentation [1] and depth prediction [2]. Dense depth prediction has been also recently employed to improve monocular SLAM [3]. In this project, we want to understand whether the accuracy of the predicted maps can be improved if given a sparse set of reliable measurements. In particular, in the case of depth prediction, we want to infer a dense depth frame from a single RGB + (very) sparse set of depth points.

The first goal is to generate an appropriate training set by taking an existing RGB-D dataset (e.g., Scannet) and creating training pairs including dense depth frames (ground truth) and sparse input by removing most of the depth values. The second goal is to design an appropriate deep learning architecture that can leverage the information from the sparse depth points to improve the accuracy of the predicted map.

We are looking for a motivated student with deep learning experience and preferably a good background in computer vision.

Related literature:

• [1] D. Eigen, C. Puhrsch, R. Fergus, "Prediction from a single image using a multi-scale deep network", NIPS 2014
• [2] I. Laina, C. Rupprecht, V. Belagiannis, F. Tombari, N. Navab, "Deeper Depth Prediction with Fully Convolutional Residual Networks", 3DV 2016
• [3] K. Tateno , F. Tombari, I. Laina, N. Navab, "CNN-SLAM: Real-time dense monocular SLAM with learned depth prediction", CVPR 2017