Research in Machine Learning for Medical Applications

Table of Content

Abstract

Machine learning offers methods for automatic learning of complex patterns from empirical data and making intelligent decisions based on learned behavior. This becomes often necessary in the area of medicine, where the large dimensionality of data and highly variable environments make analytical solutions impossible and human-generated, rule-based heuristics intractable. At CAMP, we develop and adapt state-of-the-art machine learning methods for medical applications. Applications include the learning of similarities across different image modalities for complex registration tasks, medical image understanding, organ localization, learning of anatomical shape, tissue classification and computer-aided diagnosis.

Research interests

• Similarity, metric, distance learning
• Computer aided diagnosis
• Pattern recognition and analysis
• Medical image understanding
• Tissue classication

Contact Person and Group Coordination

Diana Mateus	Nassir Navab

Research Projects in Machine Learning for Medical Applications

Shape Guided Segmentation of Cardiac Boundaries Prior shape information has been shown to be invaluable for segmenting cardiac boundaries. We develop new methods of exploiting such prior information to guide the segmentation by using techniques of machine learning or formulating the segmentation problem to fit our requirements in segmentation of 4D cardiac data.

patient-Specific Translational research on Atherosclerosis and Diagnosis (STAnD)

i2Sense: Development of an implantable health sensing system Atherosclerosis is deadliest disease in the world and occurs throughout vascular system like peripheral arteries, abdominal aorta, coronaries, etc. Balloon angioplasty and stenting are common treatment options, however, restenosis often happens with or without any sign of symptoms, which demands for repetitive expensive follow up routines. Within the i2Sense project we are working on an on-line/on-demand technology for monitoring of restenosis progression.

Big Data Analysis for Medical Applications Widespread use of electronic health records (EHR) led to vast amount of medical data being collected. The data is characterized by large sample size, heterogeneity of variables, unstructured information, missing information, and time-dependent variables, to name a few. These facts render commonly used statistical tools to perform analysis inadequate and require the development of sophisticated algorithms to overcome these challenges. For instance, the set of variables recorded for each patient can naturally be decomposed into groups, known as views. Most machine learning algorithms ignore this multi-view relationship. Instead, they can either be trained on each view separately or on a concatenation of all views to form a single view. Considering these relationships recently came to attention of the research community that proposed co-training, multiple kernel learning and subspace learning to address this problem. Although existing methods show promising results in their respective tasks, it is difficult to apply them to real-world clinical data, where highly heterogeneous features (continuous, categorial, ordinal) and missing values are common, because not all tests can be performed on all patients. As a result, multi-view learning algorithms have rarely been applied to medical problems.

Cross Modalities Image Synthesis One of the interesting talks in the Generative and Discriminative Learning for Medical Imaging Tutorial in the last MICCAI 2014 was presented by Jerry Prince regarding Image Synthesis and cross-modalities, motivating us to work on such a challenging topic. In our project for MLMI course, we would like to implement kind of Supervised Dictionary Learning incorporating the global consistency, discriminative labeling, and the Transformation-invariant.

Knowledge Propagation Models for Image Registration To register modalities with complex intensity relationships, we leverage machine learning algorithm to cast it into a mono modal registration problem. This is done by extracting tissue specific features for propagating anatomical/structural knowledge from one modalitiy to an other through an online learnt propagation model. The registration and propagation steps are iteratively performed and refined. For proof-of-concept, we employ it for registering (1) Immunofluorescence to Histology images and (2) Intravascular Ultrasound to Histology Images.

OCT Tissue Classification Optical coherence tomography (OCT), employing light rather than ultrasound, is a high-resolution imaging technology that permits a precise assessment of biological tissue. Used intravascular, OCT is increasingly used for assessing safety and efficacy of intracoronary devices, such as drug-eluting stents and bioabsorbable stents. Obtained images provide insights regarding stent malposition, overlap, and neointimal thickening, among others. Recent OCT histopathology correlation studies have shown that OCT can be used to identify plaque composition, and hence it is possible to distinguish “normal” from “abnormal” neointimal tissue based on its visual appearance. The aim of this project is to develop a novel method for the automatic analysis of tissue in IVOCT. Automatic tissue classification will allow for a quantitative and potentially more time-efficient and objective analysis of IVOCT data. For instance, classifying neointimal tissue as either “mature” or “immature” can be used to assess the disease state of patients; as a potential predictor of late stent-failure events such as stent thrombosis and restenosis.

Kooperationsprojekt SFB 824 (3. Förderperiode) & BFS The SFB824 (Sonderforschungsbereich 824: Central project for histopathology, immunohistochemistry and analytical microscopy) represents an interdisciplinary consortium which aims at the development of novel imaging technologies for the selection and monitoring of cancer therapy as an important support for personalized medicine. Z2, the central unit for comparative morphomolecular pathology and computational validation, provides integration, registration and quantification of data obtained from both macroscopic and (sub-)cellular in-vivo as well as ex-vivo imaging modalities with tissue-based morphomolecular readouts as the basis for the development and establishment of personalized medicine. In order to develop novel imaging technologies, co-annotation and validation of image data acquired by preclinical or diagnostic imaging platforms via tissue based quantitative morphomolecular methods is crucial. Light sheet microscopy will continue to close the gap between 3D data acquired by in-vivo imaging and 2D histological slices especially focusing on tumor vascularization. The Multimodal ImagiNg Data Flow StUdy Lab (MINDFUL) is a central system for data management in preclinical studies developed within SFB824. Continuing the close collaboration of pathology, computer sciences and basic as well as translational researchers from SFB824 will allow the Z2 to develop and subsequently provide a broad variety of registration and analysis tools for joint imaging and tissue based image standardization and quantification. The goal of the BFS Project: ImmunoProfiling using Neuronal Networks (IPN2) is to develop a method based on neuronal networks and recent advances in Deep Learning to allow characterization of a patient's tumor as ″hot″ or ″cold″ tumor depending on the identified ImmunoProfile. Recent research has shown that many tumors are infiltrated by immuno-competent cells, as well as that the amount, type and location of the infiltrated lymph nodes in primary tumors provide valuable prognostic information. In contrast to a ″cold tumor″, a ″hot tumor″ is characterized by an active immune system which the tumor has identified as threat. This identification provides the basis for selecting the therapy best suitable for the individual patient.

Non Invasive Histology of Atherosclerotic Plaque Stroke is the third leading cause of death in Germany. It is a neurology injury, whereby the oxygen supply to parts of the brain gets cut off. About 80% of these strokes are due to ischemia, i.e. an occlusion of a blood vessel leading to an interrupted blood flow. Stenosis inside the carotid artery imaged using four different MR weightings Special setting in this project is the arteria carotis. Plaque is most likely to develop at the branching of the arteria carotis communis into the arteria carotis interna (leading to the brain) and the arteria carotis externa. This can lead to an abnormal narrowing, called a stenosis. According to the American Heart Association these plaques can be divided into different types, based on their consistency and structure. Until now the decision about a surgery was only based on the degree of the stenosis and not on the type of plaque causing it. This is a faulty approach since there is a plaque type (Type IV) which constitutes a relevant clinical danger, although it does not necessary come along with a stenosis. Unlike most other image modalities MR images do not only give information about the degree of the stenosis, but also about the consistency of the plaque. Using different weighted MR images it is possible to correctly classify plaque into the types defined by the AHA. The main goal of this project is to create a classification tool based on T1, T2, Proton Density and 'Time of flight' weighted images. To achieve this goal the arteria carotis and the plaque have to be segmented from the images. Furthermore various features of the plaque have to be extracted in order to get information needed for the classification.

Developing Image Processing Techniques for Cryo Electron Tomography Cryo-ET has recently emerged as a leading technique to investigate the three-dimensional(3D) structure of biological specimens at close-to-native state. It also bridges the gap between low-resolution imaging techniques (e.g. light microscopy) and high-resolution techniques (e.g. single particle electron microscopy). The technique consists of acquiring many two-dimensional (2D) projections of the structure under scrutiny at various tilt angles under cryogenic conditions. The 3D structure is recovered through a number of steps including projection alignment, reconstruction and de-nosing. The resolution of the reconstructed tomogram, however, is aaffected by the low signal to noise ratio (SNR) of the projection images (typically 0.1 to 0.01) and the limited angular coverage (typically 60  to 70 ) resulting in wedge-shaped missing information in Fourier space, the so-called missing wedge; making the reconstruction process very challenging and demanding.

Decision Support System for Treating Patients with Coronary Artery Disease Efficacious treatment and management of patient with the Coronary Artery Disease (CAD) has drawn a lot of attention especially because of the high prevalence and high mortality rate. Two major treatment options are Coronary Artery Bypass Grafting (CABG) and Percutaneous Coronary Intervention (PCI). CABG is more expensive and invasive, however, the overall prognosis is better than PCI, which is widely considered as a less expensive and minimally-invasive option. It is documented that a large percentage of patients (up to one third) undergoing PCI procedure require repeat procedure after several years. It is beneficial to both patients and payers to identify this high-risk patient group as early as possible based on a series of in-vitro and imaging tests and possibly choose an alternate treatment option. Moreover, the identification of this high-risk patient group within the early months of PCI intervention could also trigger better management of these patients through administering possibly invasive tests (e.g., optical coherence tomography-OCT) to characterize and quantify the disease process and the associated risk. Our project partner at German Heart Center has collected large amount of data from patient cohorts that includes results of commercially available in-vitro tests, imaging tests, and for subset of patients the plasma samples and the follow-up data with clinical outcomes. The goal of this project is to identify features such as blood biomarkers that are predictive of PCI prognosis, to devise an algorithm decision support system to aggregate the biomarker test (including imaging) results and optimize the correlation to the clinical outcome, characterize and validate the system built based on retrospective data.

Crowdsourcing for Healthcare Today's clinical procedures often generate a large amount of digital images requiring close inspection. Manual examination by physicians is time-consuming and machine learning in computer vision and pattern recognition is playing an increasing role in medical applications. In contrast to pure machine learning methods, crowdsourcing can be used for processing big data sets, utilising the collective brainpower of huge crowds. Since individuals in the crowd are usually no medical experts, preparation of medical data as well as an appropriate visualization to the user becomes indispensable. The concept of gamification typically allows for embedding non-game elements in a serious game environment, providing an incentive for persistent engagement to the crowd. Medical image analysis empowered by the masses is still rare and only a few applications successfully use the crowd for solving medical problems. The goal of this project is to bring the gamification and crowdsourcing to the Medical Imaging community.

Similarity/Metric/Distance Learning for Medical Applications Many medical applications such as registration or tracking can be seen as the optimization of an objective function which involves a data term or similarity measure. Classical similarity measures rely for instance on image intensities, gradients or intensity statistics. In the case of noise or background clutter which is very frequent in the case of medical imaging, they might lead to registration/tracking errors. In this project, we investigate different approches and applications of learning a similarity measure directly from the data, leading to a more robust data term which is adapted to the image characteristics.

Organ Recognition Automatic localization of multiple anatomical structures in medical images provides important semantic information with potential benefits to diverse clinical applications. In this project, we investigate hierachical regression methods based on Random Forests and Random Ferns. Such hierarchical approaches permit to subdivide efficiently the feature space and to create a partition over it. In each cell of the resulting partition, data can be easily modeled using simple mathematical models such as constant or linear. The combination of these models over the whole partition results then in a complex non-linear model.

Ultrasound Based Tissue Characterization clinicians are challenged when colocated heterogeneous tissue backscatter mixed signals appearing as non-unique intensity patterns in B-mode ultrasound image. Tissue characterization algorithms have been developed to assist clinicians to identify such heterogeneous tissues and assess lesion stage. We propose a novel technique coined as Stochastic Driven Histology (SDH) that is able to provide information about co-located heterogeneous tissues. It employs learning of tissue specific ultrasonic backscattering statistical physics and signal confidence primal from labeled data for predicting heterogeneous tissue composition in plaques. We employ a random forest for the purpose of learning such a primal using sparsely labeled and noisy samples. In clinical deployment, the posterior prediction of different lesions constituting the plaque is estimated.

Hashing forests for morphological search and retrieval in neuroscientific image databases In this project, for the first time, we propose a data-driven search and retrieval (hashing) technique for large neuron image databases. The presented method is established upon hashing forests, where multiple unsupervised random trees are used to encode neurons by parsing the neuromorphological feature space into balanced subspaces. We introduce an inverse coding formulation for retrieval of relevant neurons to effectively mitigate the need for pairwise comparisons across the database. Experimental validations show the superiority of our proposed technique over the state-of-the art methods, in terms of recall for a particular code size. This demonstrates the potential of this approach for effective morphology preserving encoding and retrieval in large neuron databases.

Computer-Aided Ophthalmic Procedures In the current workflow of opthamologic surgeries, the surgeon observes the scene in an indirect way through a microscope while performing the surgery with the necessary high handling precision. During membrane peeling, for example, the surgeon has to grasp an anatomical layer of only 10 μm off the retina without damaging it. However, under this limited microscopic view, it becomes very challenging to infer the distance of the surgical instrument to the retina. Issues such as lens distortion, high level of blurriness and lack of haptic feedback complicate the task further. Recently, an intraoperative version of the Optical Coherence Tomography (OCT) was introduced, which provides the 3D information along a scanning line. On the one hand, this modality provides depth information during the surgery. On the other hand, the device has to be manually positioned to the region of interest which further complicates the current workflow of the surgeon (who already has to manipulate the surgical tool, the handheld light source and the microscope). The main goal of the project is support the surgeon in the current workflow and provide additional information during the surgery via advanced computer vision, visualization and augmented reality algorithms.

Related Publications

2016
	S. Albarqouni, U. Konrad, L. Wang, N. Navab, S. Demirci Single-View X-Ray Depth Recovery: Towards a Novel Concept for Image-Guided Interventions International Journal of Computer Assisted Radiology and Surgery (IJCARS), 2016, June 2016, Volume 11, Issue 6, pp 873-880. (bib)
	S. Albarqouni, C. Baur, F. Achilles, V. Belagiannis, S. Demirci, N. Navab AggNet: Deep Learning from Crowds for Mitosis Detection in Breast Cancer Histology Images IEEE Transactions on Medical Imaging (TMI), Special Issue on Deep Learning, vol. 35, no. 5, pp. 1313 - 1321, 2016. The first two authors contribute equally to this paper. (bib)
2013
	C. Rupprecht, O. Pauly, C. Theobalt, S. Ilic 3D Semantic Parameterization for Human Shape Modeling: Application to 3D Animation. (Oral Presentation) In Proc. International Conference on 3D Vision (3DV 2013) (bib)
	J. Lallemand, O. Pauly, L. Schwarz, D. J. Tan , S. Ilic Multi-task Forest for Human Pose Estimation in Depth Images. (Oral Presentation) In Proc. International Conference on 3D Vision (3DV 2013) (bib)
2012
	B. Glocker, O. Pauly, E. Konukoglu, A. Criminisi Joint Classification-Regression Forests for Spatially Structured Multi-Object Segmentation 12th European Conference on Computer Vision (ECCV), Firenze, Italy, October 7-13 2012. (bib)
	O. Pauly, A. Ahmadi, A. Plate, K. Bötzel, N. Navab Detection of Substantia Nigra Echogenicities in 3D Transcranial Ultrasound for Early Diagnosis of Parkinson Disease In Proc. International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), Nice (FR), October 01-05, 2012 (bib)
	Y. Chen, T. Hrabe, S. Pfeffer, O. Pauly, D. Mateus, N. Navab, F. Foerster Detection and Identification of Macromolecular Complexes in Cryo-Electron Tomograms Using Support Vector Machines IEEE International Symposium on Biomedical Imaging: From Nano to Macro (ISBI 2012), Barcelona, Spain, May 2 - 5, 2012 (bib)
	S. Atasoy, D. Mateus, A. Meining, G. Z. Yang, N. Navab Endoscopic Video Manifolds for Targeted Optical Biopsy IEEE Transactions on Medical Imaging. (bib)
2011
	O. Pauly, D. Mateus, N. Navab STARS: A New Ensemble Partitioning Approach ICCV Workshop on Information Theory in Computer Vision and Pattern Recognition (ITINCVPR 2011), Madrid, Spain, November 2011 (bib)
	A. Safi, M. Baust, O. Pauly, V. Castaneda, T. Lasser, D. Mateus, N. Navab, R. Hein, M. Ziai Computer-Aided Diagnosis of Pigmented Skin Dermoscopic Images MICCAI Workshop on Medical Content-based Retrieval for Clinical Decision Support, Toronto, Canada, September 2011 (bib)
	O. Pauly, D. Mateus, N. Navab Building Implicit Dictionaries based on Extreme Random Clustering for Modality Recognition MICCAI Workshop on Medical Content-based Retrieval for Clinical Decision Support, Toronto, Canada, September 2011 (bib)
	O. Pauly, B. Glocker, A. Criminisi, D. Mateus, A. Martinez-Möller, S. Nekolla, N. Navab Fast Multiple Organs Detection and Localization in Whole-Body MR Dixon Sequences To appear in Proc. Medical Image Computing and Computer-Assisted Intervention (MICCAI 2011), Toronto, Canada, September 2011 (bib)
2010
	O. Pauly, D. Mateus, N. Navab ImageCLEF 2010 Working Notes on the Modality Classification Subtask. Cross Language Image Retrieval Workshop (ImageCLEF? 2010), Medical Retrieval, Padua, Italy, September 2010 (bib)
	O. Pauly, H. Heibel, N. Navab A Machine Learning Approach for Deformable Guide-Wire Tracking in Fluoroscopic Sequences. Medical Image Computing and Computer-Assisted Intervention (MICCAI 2010), Beijing, China, September 2010 (bib)
2009
	O. Pauly, N. Padoy, H. Poppert, I. Esposito, H-H. Eckstein, N. Navab Towards Application-specific Multi-modal Similarity Measures: a Regression Approach. MICCAI Workshop on Probabilistic Models in Medical Image Analysis (PMMIA), London, UK, September 2009. (bib)
	A. Taki, S. Avansari, A. Roodaki, S.H. Rezatofighi, S.K. Setarehdan, N. Navab Developing new tool for automatic analysis of IVUS images: from border detection to plaque characterization 23nd International Congress and Exhibition June 23 - 27, 2009, Berlin, Germany (bib)
2007
	A.Soltanzadi, Z.Najafi, A. Roodaki, A. Taki, S.K. Setarehdan, R. Zoroofi, N. Navab Full automatic border extraction of coronary arteries in IVUS images using deformable models 14th Conference on Medical Engineering, Tehran, Iran, Dec.2007 (bib)

Working Group

Shadi Albarqouni

Tariq Mousa Bdair

Huseyin Coskun

Dhritiman Das

Beatrice Demiray

Azade Farshad

Seong Tae Kim

Magda Paschali

Daniel Roth

Matthias Seibold

Kuangyu Shi

Thomas Wendler

Yousef Yeganeh