Realtime Medical Volume Rendering on Microsoft Hololens

Advisor: Nassir Navab
Supervision by: Ulrich Eck

Abstract

The recent release of the Microsoft Hololens developer kit opens up exciting possibilities to further study the utility of head-mounted displays for augmented reality applications in medical procedures. An important feature is the visualization of volumetric images, such as MRI and CT. Due to the hardware limitations of the Hololens Graphics Processing Unit (GPU) and the missing OpenGL^? support, off-the-shelf solutions do not provide satisfactory results. This project aims to explore the feasibility of using the onboard GPU of the Hololens for volume raycasting.

Motivation

In all fields of surgery, maintaining asepsis within the sterile surgical field is one of the most important concerns of surgical staff as to minimize the patient’s risk of infection. In most operating rooms, the surgeon may only view patient data such as CT or MRI scans on wall-fixed screens, therefore requiring him to exit the surgical field, or view the information from a distance. This can be overcome by using head-mounted displays (HMDs) that remain on the surgeon’s head and do not contaminate the surgical site. These devices are capable of augmenting the surgeon’s view with images and information. This research project aims to study the feasibility of using a Microsoft Hololens HMD[1] to directly augment volumetric images onto patients for diagnosis and surgery. The processing capabilities of the Hololens are however not yet fully understood and the manufacturer does not release detailed technical specifications on the GPU, so that a bare-metal prototype implementation for rendering volumetric images using raycasting is needed to better understand its capabilities.

Task Description

Initially, the student will create a simple C++ application that loads a volumetric image and uses DirectX^? (HLSL) shaders to implement a simple raycasting algorithm[2]. In subsequent iterations, the student will benchmark the current implementation and optimize the algorithm until a sufficient frame rate was achieved. The benchmarks will investigate the relation between volume size and resulting frame rates. The optimizatios will include the integration of acceleration structures, such as empty-space skipping[3] and pre-integrated volume rendering[4]. Finally, the student will extend the capabilities of the rendering algorithm to improve visual quality, which will give us a more complete picture of the possibilities of using the Hololens GPU for volume rendering.

Required Skills

• Strong Computer Graphics Skills
• HLSL Shader Programming
• C++ Skills
• Visual Studio

Literature

[1] “Microsoft Hololens,” Microsoft Hololens. [Online]. Available: https://www.microsoft.com/microsoft-hololens/en-us. [Accessed: 14-Oct-2016].

[2] O. Kutter, A. Aichert, C. Bichlmeier, and J. Traub, “Real-time volume rendering for high quality visualization in augmented reality,” presented at the International Workshop on Augmented environments for Medical Imaging including Augmented Reality in Computer-aided Surgery, New York, USA, 2008.

[3] J. Kruger and R. Westermann, Acceleration Techniques for GPU-based Volume Rendering. IEEE Computer Society, 2003.

[4] K. Engel, M. Kraus, and T. Ertl, High-quality pre-integrated volume rendering using hardware-accelerated pixel shading. New York, New York, USA: ACM, 2001, pp. 9–16.