Towards Monocular Depth Benchmarking

Abstract

The field of monocular depth estimation has some inherent limitations due to the shortcomings of existing benchmarks and metrics used to evaluate network performance.

The standard metrics simply look at the depth residuals for each pixel allowing visually bad predictions to get very high metric scores, and vice versa. The new research gives us short incremental improvements which don’t often translate to improvement in the quality of predictions. The datasets used to train are collected by sensors and are hence noisy and full of gaps, they do not perform optimally under the existing metrics or the new ones we’ll propose.

The goal of this thesis is to establish a new benchmark with the 3RScan dataset and introduce novel metrics that are more expressive and take the geometry of the scene into account. The ground truth for 3RScan is recovered from textured 3D reconstruction of the environment. It has no sensor related issues and has less noisy, high resolution ground truth allowing us to propose new metrics that look at the 3D geometry of the scene for more informative and reliable quantitative results.

Literature

Wald, Johanna and Avetisyan, Armen and Navab, Nassir and Tombari, Federico* and Nießner, Matthias* "RIO: 3D Object Instance Re-Localization in Changing Indoor Environments" ICCV 2019.

Koch, Tobias and Liebel, Lukas and Fraundorfer, Friedrich and Körner, Marco. "Evaluation of CNN-Based Single-Image Depth Estimation Methods". ECCV-WS 2018.