MaSelfSupervisedIndoorDepthEstimation

Chair for Computer Aided Medical Procedures & Augmented Reality
Lehrstuhl für Informatikanwendungen in der Medizin & Augmented Reality

Self-supervised learning in indoor environment

Abstract

Self-supervised depth estimation shows promising results in outdoor environments. However, there are a few works targeting indoor and more arbitrary scenarios. After a series of experiments, we found that one reason may be the current architecture is not able to train the network with high variation ego-motion sequences. The existing methods usually rely on Kitti, Cityscape training dataset which mostly consists of onward motion. When testing existing methods on indoor datasets (such as TUM-RGBD, NYU ), on most of them, the training simply failed by outputting zero-depth image.

The goal of this project is to investigate this issue and try to find a solution for training a self-supervised indoor depth estimation network.
Possible directions:
1. Try to improve the pose estimation by using pre-trained pose networks, such as Sfmlearner, Flownet2.0, and fine-tuning.
2. Find a way to train the pose network end-to-end by improving the poseNet architecture, designing a good loss function and a good training method.

Requirement

Basic C++ and Python skill.
Moderate understanding of deep learning.
Moderate understanding of pose estimation and multiview geometry.

Literature

-Depth-
(vid2depth) Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints code
(monodepth2) Digging Into Self-Supervised Monocular Depth Estimation
(struct2depth) Depth Prediction without the Sensors: Leveraging Structure for Unsupervised Learning from Monocular Videos
(geoNet) Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose
-Pose-
(sfmleaner) Unsupervised Learning of Depth and Ego-Motion from Video flowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks


Students.ProjectForm
Title: Self-supervised learning in arbitrary image sequences
Abstract: Self-supervised depth estimation shows the promising result in the outdoor environment. However, there are few works target on the indoor or more arbitrary scenario. After a series of experiments, we found that one reason may be the current architecture is not able to train the network with high variation ego-motion sequences. The existing methods usually rely on Kitti, Cityscape training dataset which mostly consists of onward motion. When testing existing methods in the indoor dataset (such as TUM-RGBD, NYU ), on most of them, the training simply failed by outputting zero-depth image.

The goal of this project is to investigate this issue and try to find a solution for training a self-supervised indoor depth estimation.
Possible directions:
1. Try to improve the pose network by using pre-trained pose network, such as Sfmleaner, Flownet2.0, and fine-tuning.
2. Find a way to end-to-end train pose network correctly by improving the poseNet architecture, designing a good loss function or a structuring a good training method.

Student:  
Director: Federico Tombari
Supervisor: Shun-Cheng Wu
Type: DA/MA/BA
Area:  
Status: draft
Start:  
Finish:  
Thesis (optional):  
Picture:  


Edit | Attach | Refresh | Diffs | More | Revision r1.4 - 01 Oct 2019 - 08:22 - NikolasBrasch