Dynamic objects in dense reconstruction

Abstract

Dense reconstruction is an active field of research with the goal to provide a detailed 3D representation of an environment. It is essential for many robotic task like planning and navigation but also for occlusion aware and physics based augmented reality.

In recent years, several real-time dense reconstruction approaches for RGB-D sensors have been proposed and reached a level of performance, so they can be used in a diversity of applications. Due to the need of a special sensor for the reonstruction, recent work has focused on the reconstruction using only monocular RGB cameras. Combined with deep learning based methods for stereo matching and to learn geometric priors, like depth and nnormals, from the RGB image directly dense reconstruction of static environments possible.

Compared to active depth sensors dynamic objects pose a greater challenge to monocular approaches, due to the independent motions of camera and objects the detection of dynamic objects is not simple, especially when dealing with slow motions. Some works go even a step further and reconstruction the dynamic objects simultaneously with the static environment.

For outdoor applications like autonomous driving, msot of the objects can be mapped to a few classes, like vehicle, pedestrian, bicyclist etc., for which the shape variations are limited. In these cases recent work has shown that we can compute or learn compact shape representations and use them as regularizers to find a optimal representation for a dynamic object instance.

In this project we want to combine deep learned shape priors and monocular reconstruction methods to give us temporal consistent and fused reconstructions, measured using photometric and geometric confidences.

Requirements

A general background on 3D reconstruction (RGB-D, Multi-View-Stereo) techniques and representations (Surfel, Voxel, Mesh) should be gained before starting the thesis.

Most of the frameworks for 3D reconstruction are written in C++ and often use ROS as an I/O interface.
The training of ML components, the evaluation and post processing are mostly done in Python.
Therefore basic C++ and Python skills are required.

Literature

2011-Robust vSLAM for dynamic scenes
2015-Dynamic Body VSLAM with Semantic Constraints
2018-Stereo Vision-based Semantic 3D Object and Ego-motion Tracking for Autonomous Driving
2018-Robust Dense Mapping for Large-Scale Dynamic Environments
2018-3D Vehicle Trajectory Reconstruction in Monocular Video Data Using Environment Structure Constraints
2019-MID-Fusion: Octree-based Object-Level Multi-Instance Dynamic SLAM
2019-DirectShape: Photometric Alignment of Shape Priors for Visual Vehicle Pose and Shape Estimation
2019-ROI-10D Monocular Lifting of 2D Detection to 6D Pose and Metric Shape

If you are interested , please contact us via e-mail:

Nikolas Brasch

Federico Tombari