Planning on Dense Semantic Reconstructions

Abstract

In recent years different methods for real-time dense semantic reconstruction have been proposed. Also different planners based on those different representations have been proposed.
Most of the planners focus on the geometric information in the reconstruction to avoid collisions, for navigation in cluttered environments or in the case of autonomous driving the
geometry of the scene is not sufficient to plan an optimal safe trajectory (e.g. lane markings are usually not represented in the geometry of the environment).
Furthermore can the dense 3D reconstruction be leveraged to detect occlusions of areas, where other agents could be moving (e.g. a parked truck might occlude another vehicle or pedestrians).
Another important source of information for the planner is the detection, tracking and prediction of other dynamic objects to reason about preferred driving areas.

In this project we build up on an initial setup combining state-of-the-art reconstruction methods, levariging different sensor modalities (monocular, stereo and LIDAR), and a planner based on the geometry of the scene.
The goal is to extend the planner to also leverage semantic, occlusion information as well as the predicted trajectories of dynamic obejcts.

Due to the complexity of the problem, the project might be split into an semantic/occlusion and a dynamic objects part.

The following images show an example scene from the KITTI dataset for outdoor navigation represented as a surfel cloud.

Color	Semantics
Depth	Normals

Requirements

A general background on 3D reconstruction techniques and representations will be helpful for the first part.
A general background in Object detection, Tracking and Trajectory Prediction will be helpful for the second part.

The existing framework for 3D reconstruction and planning is written in C++ and uses ROS as an I/O interface.
The training of ML components, the evaluation and post processing are mostly done in Python.
Therefore basic C++ and Python skills are required.

Literature

3D Semantic Reconstruction
Chen, Xieyuanli, et al. "SuMa++: Efficient LiDAR^?-based semantic SLAM." 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2019.
Narita, Gaku, et al. "Panopticfusion: Online volumetric semantic mapping at the level of stuff and things." arXiv preprint arXiv:1903.01177 (2019).
Grinvald, Margarita, et al. "Volumetric instance-aware semantic mapping and 3D object discovery." IEEE Robotics and Automation Letters 4.3 (2019): 3037-3044.

Planning
Oleynikova, Helen, et al. "Voxblox: Incremental 3d euclidean signed distance fields for on-board mav planning." 2017 Ieee/rsj International Conference on Intelligent Robots and Systems (iros). IEEE, 2017.
Ruetz, Fabio, et al. "OVPC Mesh: 3D Free-space Representation for Local Ground Vehicle Navigation." 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019.

If you are interested , please contact us via e-mail:

Nikolas Brasch

Federico Tombari