BA/MA/IDP thesis: Intraoperative Ophthalmic Scene Reconstruction for surgical AR

Thesis by: Ekaterina Kanaeva
Advisor: Prof. Nassir Navab
Supervision by: Jakob Weiss, Federico Tombari

Abstract

Ophthalmic microsurgical interventions require a high level of handling precision for the surgeon as even little errors can damage intricate structures inside the human eye. Furthermore, the microscopic top-down view through the microscope is a limiting factor for the surgeon's depth perception, motivating the use of augmented reality to enhance the surgical view. With the use of volumetric intraoperative Optical Coherence Tomograph (OCT), 3D imaging can be performed during the surgery, yielding an additional source of information to guide the surgeon. However, to display this additional data to the surgeon, proper integration into the surgical view has to be considered. One very intriguing way to integrate the information is to use a focus and context visualization method already used by Bichlmeier et al. [1], see Figure 1. To be able to provide perceptually plausible in-situ rendering of the acquired volumes as a focus-and-context visualization, first the 3D surface of the retinal surface needs to be reconstructed as a 3D surface. This can be achieved by applying stereo block matching using a calibrated stereo image pair. To properly position the iOCT overlay, the surface must then be aligned to the coordinate system of the OCT volume. This, however, cannot be done preoperatively due to the complex optical setup involving the patient's own eye. Therefore, the second part of this project is to devise an online alignment that can compensate for the (potentially changing) optical pathway affecting all imaging modalities. This online alignment can involve a simple surface alignment step as well as more complex algorithms to account for deformations due to the different optical pathways between the two modalities.

Implementation of this project can be done either in Unreal Engine 4 [2] or in CAMPVis [3], using additional external libraries.

Figure 1: Examples of Anatomical Mimesis, from [1]

Figure 2: Available Images for scene reconstruction

Tasks

• Interface with the stereo cameras and evaluate existing algorithms for stereo depth reconstruction
• Devise a method to align the reconstructed surface with the OCT volume
• Implementation using either Unreal Engine or campvis, plus additional libraries library such as OpenCV^?
• Implementation of a proof-of-concept AR Application
• experimental evaluation

Depending on the desired thesis type and student's preferences, the tasks will be adapted to suit the scope of the thesis.

Prerequisites

• Knowledge of C++
• GPU programming experience will be helpful
• Basics in image registration will be helpful
• Background of CAMP I/II lectures will be helpful

Contact

Jakob Weiss

References

[1] Bichlmeier, C., Wimmer, F., Heining, S. M., & Navab, N. (2007). Contextual Anatomic Mimesis Hybrid In-Situ Visualization Method for Improving Multi-Sensory Depth Perception in Medical Augmented Reality. In 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality (pp. 1–10). IEEE. https://doi.org/10.1109/ISMAR.2007.4538837

[2] https://www.unrealengine.com/en-US/what-is-unreal-engine-4

[3] https://gitlab.lrz.de/CAMP/campvis-public