Interaction Concept for a Medical Augmented Reality System

Thesis by:
Advisor: Nassir Navab
Supervision by: Tobias Blum

Abstract

CAMPAR is a medical augmented reality framework we have developed, that has been used for many applications during the last years. The goal of this project is to design and implement a state-based interaction concept for this framework. A state machine will be used to model different states of a medical procedure and the AR system. The current state of the system can be changed by events like keybord input, user gestures, voice input or tracking data. Based on the state e.g. the AR visualization is adapted.

Another field of research where we are very active is workflow analysis. We automatically obtain data during a medical procedure, like the tool that is currently used, the position of the surgeon or biomedical data from the patient. Using statistical models like Hidden Markov Models (HMM) the workflow is modeled based on this data. Using probabilistic inference we can during a running procedure detect the current state. In this project the output of an HMM shall also be used as input for the state machine so that it is possible to build an AR system that is always aware of the current state of a procedure.

Action items

• Investigate interaction concepts of other medical imaging frameworks
  • MITK
  • IGSTK
• Investigate use of existing state machine implementations
  • e.g. Boost Statechart Library
• Implement or integrate a state machine into CAMPAR
• Implement model checking for the state machine
• Integrate with our probabilistic inference methods

Possible extensions

• Look a possible modifications towards using ontologies for describing the workflow
• Integrate an open source speech recognition engine