PhD Defense by Nicola Rieke

Speaker: Nicola Rieke Date: Monday, November 19, 2018 Time: 13:00 Location: FMI-Building, Room 01.09.014

Computer Vision-Assisted Surgery: Real-Time Instrument Tracking with Machine Learning

Abstract:

The accurate tracking of surgical instruments in medical video sequences is a key component of various computer-assisted interventions. Determining the instrument position and monitoring its movements enables for example surgical gesture recognition, visual servoing of medical robots and interventional workflow analysis. Moreover, real-time tracking paves the way for intelligent decision support during the intervention: crucial information such as the identification of a potentially dangerous situation could be displayed close to the surgeon’s center of attention, which is usually the tip of the surgical tool. A further value of tracking lies in the alignment of an additional intra-operative modality to the tool movement and hence the possibility to observe in-depth instrument-tissue interaction. Such deployment can significantly reduce the burden on the surgeon and have a positive impact on the surgical outcome.

It is therefore not surprising that instrument tracking has been an important topic of research in the last decade. However, despite great advances, accurate and robust tracking of instruments in the intra-operative setting has not yet been resolved to a satisfactory extent. One of the main difficulties arises from the fact that the image data in such a setting captures only a very restricted field of view of the highly dynamic environment. Especially the non-static directional light source complicates the task by creating shadows, uneven illumination and specular reflections in the images. At the same time, a tracking algorithm for such a setting must be robust, accurate and real-time-capable. This combination of requirements and constraints poses a particularly challenging computer vision problem.

Prior work in this field has mainly relied on explicit modelling and just started to explore the potential of data-driven approaches. This dissertation follows the idea of learning from data and introduces several approaches that leverage machine learning techniques to overcome the aforementioned challenges. In the feed-forward pipeline, a two-step approach with specialized Random Forests (RF) is introduced: An intensity-based RF template tracker first limits the image search space before a gradient-based RF determines the instruments’ 2D pose. Building on this dual RF, a robust pipeline is developed by adapting the offline model to online information and by “closing the loop” between the tracking and 2D pose estimation. Finally, a deep learning-based approach in the end-to-end pipeline is presented that simultaneously determines the segmentation and 2D pose of the instrument with a fully convolutional neural network. By reformulating the pose estimation task as a heatmap regression, the two objectives can leverage their spatial dependency and facilitate simultaneous learning. All presented methods achieve real-time performance and are evaluated in a cross-validation setup on in-vivo image sequences, demonstrating their applicability to various scenarios. The results demonstrate that machine learning-based instrument tracking has remarkable advantages with respect to the state of the art in terms of accuracy, robustness and generalization.