Deep Learning for Multi-View Stereo

Abstract

In the past few years Deep Learning has succesfully been employed to solve the Multi-View Stereo Problem. Deep Learning has been used to solve two important sub-tasks of the stereo problem.

The first step is finding correspondences between multiple frames, here classic heuristics based on patch comparisons using like sum of squared differences (SSD) or normalized cross-correlation (NCC) struggle under varying lighting conditions and depend heavily on the chosen patch size. Deep Learning has show to learn much more robust and adaptive comparison metrics.

Even when using learned correspondences, large patches of similar intensity as well as occlusions introduce erros into the estimated depth map.
Therefore traditionally a cost volume is build for all possible depth hypothesis for all pixels, now regularization techniques such as Semi-Global Matching (SGM) or Beleif-Propagation (BP) can be used to find better solutions.
Using Deep Learning instead has also show a great potential to extract a better depthmap from the cost volume.

One downside of the Deep Learning based appraoches is the lack of a confidence measure, as the existing models only provide a single depthmap as a result.

In this project we want to analyze and compare the state-of-the-art and develop methods to estimate confidence values of the depth predictions so they can be used in safetly critical applications such as autonomous driving.

Requirements

A general background in Computer Vision or Multi-View Stereo in particular will be useful.

Most of the existing works are written in Python, where some parts are accelerated by C++ and CUDA
Therefore good Python and basic C++/CUDA skills are required.

Literature

Deep Learned Multi-View-Stereo
Huang, Po-Han, et al. "Deepmvs: Learning multi-view stereopsis." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.
Wang, Kaixuan, and Shaojie Shen. "MVDepthNet: real-time multiview depth estimation neural network." 2018 International Conference on 3D Vision (3DV). IEEE, 2018.
Yao, Yao, et al. "Recurrent mvsnet for high-resolution multi-view stereo depth inference." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019.

If you are interested , please contact us via e-mail:

Nikolas Brasch

Federico Tombari