S.C. Lee, B. Fuerst, K. Tateno, A. Johnson, J. Fotouhi, G. Osgood, F. Tombari, N. Navab Multi-modal Imaging, Model-based Tracking and Mixed Reality Visualisation for Orthopaedic Surgery Healthcare Technology Letters Journal (IET HTL), September, 2017. This work won the outstanding paper award at AE-CAI 2017. The final publication is available at http://digital-library.theiet.org/content/journals/10.1049/htl.2017.0066 (bib)
	Orthopaedic surgeons are still following the decades old workflow of using dozens of 2D fluoroscopic images to drill through complex 3D structures, e.g. pelvis. This paper presents a mixed reality support system, which incorporates multi-modal data fusion and model-based surgical tool tracking for creating a mixed reality environment supporting screw placement in orthopaedic surgery. An RGBD camera is rigidly attached to a mobile C-arm and is calibrated to the Cone-Beam Computed Tomography (CBCT) imaging space via iterative closest point algorithm. This allows real-time automatic fusion of reconstructed surface and/or 3D point clouds and synthetic fluoroscopic images obtained through CBCT imaging. An adapted 3D model based tracking algorithm with automatic tool segmentation allows for tracking of the surgical tools occluded by hand. This proposed interactive 3D mixed reality environment provides an intuitive understanding of the surgical site and supports surgeons in quickly localizing the entry point and orienting the surgical tool during screw placement. We validate the augmentation by measuring target registration error (TRE) and also evaluate the tracking accuracy in presence of partial occlusion.
	This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each authors copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.