Medical Augmented Reality with SLAM-based perception

Problem statement:

Medical Augmented Reality (AR) systems display medical data such as volumetric images and segmented structures co-located with the patient’s anatomy. Such Medial AR applications range from surgery planning [1] to intra-operative guidance systems [2]. Pre-operative volumetric images such as CT or MRI scans are typically available. In order to correctly augment medical information onto the patient it is essential to accurately determine the transformation between the current pose of the patient, the viewing point of the observer, and the associated medical image data.

The required transformations can be estimated by registering the surface (skin) of the patient with a surface extracted from the associated volumetric scans. Depth sensors can be used to reconstruct the environment, for example a patient lying on an table in the operating room. Registration methods perform best if the represented surfaces correspond to each other. It is however challenging to extract only relevant parts from the reconstructed environment, for example the upper body of the patient that is contained in a CT scan.

Previous work on segmenting and reconstructing the environment using depth cameras [3] has shown that automatic segmentation of the reconstructed environment is possible without manual intervention. In particular, by exploiting SLAM-based algorithms, a segmentation of the entire scene can be obtained in a robust and computationally efficient way.

Once the relevant surface is segmented from the environment scan, the two surfaces can be registered with each other, which results in the required transformation between a well known world coordinate system and the medical data set. Once such a transform is known, augmentations based on the volumetric information can be rendered correctly.

Project Objectives:

• Use of extended joint reconstruction and segmentation to correctly segment body parts
• Use reconstructed/segmented pointcloud to register patient CT data
• Provide patient augmentations without requirement of manual registration

Assumptions:

• Patient is under anaesthesia and does typically not move

Literature:

• [1] P. Fallavollita, A. Winkler, S. Habert, P. Wucherer, P. Stefan, R. Mansour, R. Ghotbi, and N. Navab, “Desired-View--controlled positioning of angiographic C-arms.,” Med Image Comput Comput Assist Interv, vol. 17, no. 2, pp. 659–666, 2014.
• [2] C. Bichlmeier, F. Wimmer, S. M. Heining, and N. Navab, “Contextual Anatomic Mimesis Hybrid In-Situ Visualization Method for Improving Multi-Sensory Depth Perception in Medical Augmented Reality,” presented at the Proceedings of the 6th IEEE and ACM International Symposium on Mixed and Augmented Reality, Nara, Japan, 2007, pp. 129–138.
• [3] K. Tateno, F. Tombari, N. Navab, "Real-Time and Scalable Incremental Segmentation on Dense SLAM", IROS 2015

Supervision: Keisuke Tateno, Federico Tombari, Ulrich Eck

For further information please contact: Federico Tombari