Doctoral Defense by Ma Meng

Speaker: Ma Meng Date: Monday, September 5th, 2016 Time: 11:00 AM Location: FMI-Building, Room 03.13.010

Personalized Perception and Interaction with Medical Information in Mixed Reality Environments

Abstract: Medical information is employed in different scenarios, such as education, training, diagnosis, and surgery. Perception and interaction of this information is fundamental, and augmenting the perception through personalized interaction with multi-model data is user specific. The good of this thesis if to investigate personalized perception through development of novel interaction method with the different medical information suitable for students, patients, and doctors.

At first, a Magic Mirror framework is presented to create a mixed reality environment with an in-situ augmented reality view. It tracks the user and achieves a registration between the medical information and the user. It inherently allows the user to perform interactive registration to enhance personalized perception. This opens interesting possibilities for personalized augmented reality in medical education and rehabilitation. User studies were designed to respectively assess the precision and acceptability of the Magic Mirror system for anatomy education. We also take advantage of the interactive mixed reality to generate personalized augmented reality applications for the learning of organs and muscle and for rehabilitation exercise. In addition, functions, e.g. “Organ explosion” and “Self-control virtual view,” are designed and implemented, and serious games are also developed in this framework.

We then introduce the Pointing At Several Targets (PAST) interaction method that offers an approach to calculate a virtual eye center as the origin for the eye-rooted pointing ray without tracking the eye gaze in an egocentric setting. Consequently, the user-specific pointing geometry is recovered for personalized interaction with ambient information without visual feedback. The proposed method is mathematically analyzed and shows that the calibration is feasible with a measured accuracy of the recovered pointing ray below 0.9°. Our user study shows that the calibration is independent of the environment and the pointing ray is recovered with an accuracy of 0.67° ± 0.71° without any visual feedback. Based on the recovered pointing gesture, a novel user interface is introduced, allowing the surgeon to perform touchless interaction with the various medical systems, and switch effortlessly among them, without accessing or modifying the software and hardware of targeted systems.

Finally, in conjunction with the proposed Magic Mirror framework and recovered pointing gesture, a framework for collaborative Mixed Reality (MR) is introduced. The human-human communication is enhanced via pointing gesture within an MR framework.