3D inpainting with Semantic Scene Completion

Introduction

Existing semantic scene completion methods target at complete missing geometry and predict semantic meaning in 3D reconstruction, but without predicting the texture of the completed regions. This project aims to fill this gap by introducing a method to also predict the texture of the predicted regions.

The achievement of recent literature in image in-painting has shown impressive results. Given an image of randomly masked out regions, this technique can in-paint these regions coherently and realistically.

On the other hand, the research area in semantic scene completion illustrates the possibility to complete partial geometry while understanding the scene (See Fig.1).

There is a certain similarity between these two research areas. First, they are both trying to complete missing information with known information. Second, in recent literature, both methods are achieved by not only using the information from the current input but also leverage the learned experience from latent space.

Inspired by these two research areas, this project aims to design a new method that is capable of predicting geometry, semantic meaning and texture information in 3D space.

Literature

Image in-painting
Image Inpainting for Irregular Holes Using Partial Convolutions
Free-Form Image Inpainting with Gated Convolution

Semantic scene completion
EdgeNet
ForkNet
SSCNet

Requirement

Knowledge in Python, Pytorch and deep learning.

Fig.1 Semantic scene completionimage. Predicted regions do not have texture information.