IDP / Bachelor's Thesis: Stereo AR Prototype for Ophthalmic Interventions using Unreal Engine

Thesis by: Aleksandra Dokic
Advisor: Prof. Nassir Navab
Supervision by: Jakob Weiss

Abstract

Opththalmic Interventions are amongst the most challenging surgical interventions as they require high handling precision of the surgeon by at the same time only providing a limited stereo view through a microscope. For vitreoretinal procedures, where the surgeon has to operate on the retina inside the eye, additional complexity is added by the even more limited field of view, high defocusing and the endoscopic access pathway. With the Zeiss Lumera 700 [1], Carl Zeiss Meditec has introduced the first ophthalmic microscope with integrated live OCT imaging. This allows the surgeon to assess cross-sectional slices in real time during the surgery, improving the decision making and guidance. In the current product, the OCT slices are augmented onto the surgical view using a semi-transparent display in one eye channel to provide the overlay (see Fig. 1).
The goal of this project is to recreate this system in a fully digital platform, using stereo cameras integrated into the microscope and the OCT data streamed from a web interface. The envisioned platform should recreate the views currently available inside the microscope using either a stereo screen or, alternatively, a head-mounted display such as the HTC Vive, to create a fully virtual surgical environment. However, it should display the OCT slices augmented on both eyes to provide a better depth placement. Based on this digital recreation, different experiments can be performed to optimize placement and visual parameters of the augmented views to minimize distraction from the direct view and occlusion problems.
The flexibility of Unreal Engine [2] shall be leveraged in this project to maintain extensibility of this project's outcome for future extensions.

This project can be done as a Bachelor's Thesis as well, the project scope will be adapted accordingly in this case.

Lumera_AR
Figure 1: Intraoperative augmentation inside the Lumera 700 Ocular. Image from [1]

Tasks

Interfacing with the Stereo camera interface and importing the streomed images into the Unreal Engine Framework
Integration of live OCT images from a C++ interface
Creation of a 3D environment integrating both modalities
Optional: Extending the 3D environment to a fully 3D interactive environment using the HTC Vive headset
Test of the prototype together with an Ophthalmic surgeon on an eye phantom or pig eyes

Prerequisites

Knowledge of C++
Experience with modern Game Engines (Unity/Unreal Engine/...) and/or high motivation to get familiar with UE4
Background of CAMP I/II lectures desirable

Contact

Jakob Weiss

References

[1] https://www.zeiss.de/meditec/produkte/ophthalmologie/katarakt/visualisierung/operationsmikroskope/opmi-lumera-700.html

[2] https://www.unrealengine.com/en-US/what-is-unreal-engine-4

Students.ProjectForm
Title:	Stereo AR Prototype for Ophthalmic Interventions using Unreal Engine
Abstract:	Opththalmic Interventions are amongst the most challenging surgical interventions as they require high handling precision of the surgeon by at the same time only providing a limited stereo view through a microscope. For vitreoretinal procedures, where the surgeon has to operate on the retina inside the eye, additional complexity is added by the even more limited field of view, high defocusing and the endoscopic access pathway. With the Zeiss Lumera 700 [1], Carl Zeiss Meditec has introduced the first ophthalmic microscope with integrated live OCT imaging. This allows the surgeon to assess cross-sectional slices in real time during the surgery, improving the decision making and guidance. In the current product, the OCT slices are augmented onto the surgical view using a semi-transparent display in one eye channel to provide the overlay (see Fig. 1). The goal of this project is to recreate this system in a fully digital platform, using stereo cameras integrated into the microscope and the OCT data streamed from a web interface. The envisioned platform should recreate the views currently available inside the microscope using either a stereo screen or, alternatively, a head-mounted display such as the HTC Vive, to create a fully virtual surgical environment. However, it should display the OCT slices augmented on both eyes to provide a better depth placement. Based on this digital recreation, different experiments can be performed to optimize placement and visual parameters of the augmented views to minimize distraction from the direct view and occlusion problems. The flexibility of Unreal Engine [2] shall be leveraged in this project to maintain extensibility of this project's outcome for future extensions.
Student:	Aleksandra Dokic
Director:	Prof. Nassir Navab
Supervisor:	Jakob Weiss
Type:	IDP
Area:	Computer-Aided Surgery, Medical Augmented Reality
Status:	finished
Start:
Finish:
Thesis (optional):
Picture: