Semantic Simultaneous Localization And Mapping

Abstract

This thesis aims to leverage 2D (or 3D) semantic information in a simultaneous localization and mapping (SLAM) system to handle typical difficulties, such as reconstructing/filtering moving objects.

The project can be split into two parts. The first one is mostly related to 3D reconstruction and the other one is related to tracking:

(1) 3D scene reconstruction often suffers from dynamic objects which lead to tracking failure and blurry reconstruction. Fusion++[1] proposed an object-level SLAM system which uses semantic information to reconstruct a map with the knowledge of objects. MaskFusion^?[2] on the other hand focuses on reconstructing moving 3D objects individually. The goal of this thesis is to combine the two methods above, to enable the reconstruction of a dynamic scene.

(2) The main goal of this part is to design a semantic tracking system based on 2D (or 3D) semantic information to improve the robustness of existing tracking methods. For a frame-to-model ICP tracking system [3], a huge amount of drifting occurs when the observed geometry is mostly uniform. With the help of semantic information, it is possible to maintain the performane of an ICP tracker through these difficult regions. For feature-based tracking systems, semantic information can reduce the number of incorrectly matched points in the matching process for points belonging to different classes.

Requirements

Very good C++
Python
Computer vision and multiview geometry

Literature

[1]Fusion++: Volumetric object-level slam

[2]Maskfusion: Real-time recognition, tracking and reconstruction of multiple moving objects

[3]KinectFusion: Real-time dense surface mapping and tracking