ResearchIssueComputerVision

Chair for Computer Aided Medical Procedures & Augmented Reality
Lehrstuhl für Informatikanwendungen in der Medizin & Augmented Reality

Research in Computer Vision

About

The Computer Vision Team at the CAMP Chair mostly focuses its research activity on two main topics: real-time computer vision from 3D data and advanced machine learning for computer vision.

Regarding the first topic, the goal is to develop real-time computer vision algorithms based on 3D or RGB-D data obtained from consumer depth cameras. Specifically, our research is related to 3D reconstruction and SLAM, 3D object recognition, semantic segmentation, tracking and 3D human pose estimation.
The second cluster of our research concentrates on machine learning advances with a focus on computer vision tasks. Currently, the team is involved in developing deep learning paradigms and architectures well-suited for addressing vision problems, such as depth estimation, semantic segmentation, object detection and action recognition.

Our objective is to merge deep learning and 3D perception methodologies, providing real-time, robust and scalable learning technology that can be applied on RGB and 3D data.
Applications of our research activities are mainly found in the fields of robotic perception and scene understanding, augmented reality as well as medical image analysis.

The team is currently involved in several collaborations with prestigious companies and universities around the world, aiming to develop computer vision technology that can be deployed by research teams, companies and start-ups in order to tackle real problems and enable new applications.

Group Coordinators

Federico Tombari
Nassir Navab

Contact Person

Federico Tombari

  • Email: tombari[at]in.tum.de

  • Address:

Chair for Computer Aided Medical Procedures & Augmented Reality
Fakultät für Informatik
Technische Universität München
Boltzmannstr. 3
85748 Garching b. München

Open Positions

Oct 2018: We are looking for a candidate to fill a PhD? position on a fully funded project on the topics of deep learning and robotic perception, in particular regarding SLAM and 6D pose estimation. The candidate should have good skills in C++ and Python and a solid background in computer vision. Experience with deep learning and 3D computer vision is preferred. Please get in touch with Federico Tombari if interested, sending your CV and grade transcript.

Available Student Projects


Research Partners


Internal Members

Christian Rupprecht
Christian Rupprecht
Helisa Dhamo
Helisa Dhamo
David Tan
David Tan
Huseyin Coskun
Iro Laina
Keisuke Tateno
Keisuke Tateno
Fabian Manhardt
Johanna Wald
Shun-Cheng Wu
Yida Wang


External Members and Collaborators

Benjamin Busam
Benjamin Busam
Jakob Mayr
Jakob Mayr
Nicola Rieke
Nicola Rieke
Mira Slavcheva
Mira Slavcheva
Nikolas Brasch
Oliver Scheel
Markus Herb
Tolga Birdal
Tolga Birdal
Haowen Deng
Hemal Naik


Research Projects in Computer Vision

3D Object Reconstruction

3D Object Reconstruction

We propose a novel 3D object reconstruction framework that is able to fully capture the accurate coloured geometry of an object using an RGB-D sensor. Building on visual odometry for trajectory estimation, we perform pose graph optimisation on collected keyframes and reconstruct the scan variationally via coloured signed distance fields. To capture the full geometry we conduct multiple scans while changing the object’s pose. After collecting all coloured fields we perform an automated dense registration over all collected scans to create one coherent model. We show on eight reconstructed real-life objects that the proposed pipeline outperforms the state-of-the-art in visual quality as well as geometrical fidelity.
Real-time Incremental Segmentation on Dense SLAM

Real-time Incremental Segmentation on Dense SLAM

This work proposes a real-time segmentation method for 3D point clouds obtained via Simultaneous Localization And Mapping (SLAM). The proposed method incrementally merges segments obtained from each input depth image in a unified global model using a SLAM framework. Differently from all other approaches, our method is able to yield segmentation of scenes reconstructed from multiple views in real-time, with a complexity that does not depend on the size of the global model.
3D Temporal Tracker

3D Temporal Tracker

We propose a temporal tracking algorithm based on Random Forest that uses depth images to estimate and track the 3D pose of a rigid object in real-time. Compared to the state of the art aimed at the same goal, our algorithm holds important attributes such as high robustness against holes and occlusion, low computational cost of both learning and tracking stages, and low memory consumption. Due to these attributes, we report state-of-the-art tracking accuracy on benchmark datasets, and accomplish remarkable scalability with the number of targets, being able to simultaneously track the pose of over a hundred objects at 30 fps with an off-the-shelf CPU. In addition, the fast learning time enables us to extend our algorithm as a robust online tracker for model-free 3D objects under different viewpoints.
Spatio Temporal Modeling of Dynamic Scenes

Spatio Temporal Modeling of Dynamic Scenes

A great part of Computer Vision research has been dedicated to shape recovery, tracking and detection of 3D objects in images and videos. While excellent results have been achieved in these areas, the majority of the methods still assume static scenes and rigid objects, and rarely explore temporal information. However, the world surrounding us is highly dynamic, and in many situations objects deform over time. This temporal information provides a richer and denser source of information and have not yet been extensively exploited. Our objective within this project is to explore spatio-temporal information in order to recover 3D shapes and the motion of the deformable objects. Therefore, we refer to this area as spatio-temporal or four dimensional modeling (4D modeling). With the increased popularity of 3D content in film industry, TV, Internet and games, tools and methods that exploit spatio-temporal information and allow fast and automated 3D content production are going to be indispensable.
Learning Driving Strategies for Autonomous Cars

Learning Driving Strategies for Autonomous Cars

In this joint project of the Computer Vision Group at CAMP (TUM) and the BMW Group, we investigate the use of Machine Learning methods for Autonomous Driving. Of particular interest is the learning of driving strategies and the prediction of driving behavior, in order to be able to autonomously navigate through complex traffic scenarios. We take advantage of the multimodal sensor data available in the BMW environment model, consisting of tracked objects around the ego vehicle, detected lane markings, road boundaries and other bus data.
Depth Estimation from a Single RGB Image

Depth Estimation from a Single RGB Image

Depth estimation from multiple views (stereo vision) or other single-view assumptions (motion, shading, defocus) has been well studied in literature. However, estimating the depth map of a scene from a single RGB image remains an open problem due to the inherent ambiguity of mapping color/intensities to depth values, i.e. a 2D image could correspond to multiple 3D world scenarios. Within the spectrum of this project, we address this problem using a deep learning approach. We have investigated different Convolutional Neural Network (CNN) architectures and loss functions for optimization of the task at hand. Particular focus is given on in-network upsampling layers with learnable weights, aiming to optimally tackle the problem of high dimensional outputs without an excessive number of parameters. The best performing model encompasses residual learning and delivers state-of-the-art, real-time performance on depth prediction from images or videos of indoor and outdoor scenes. The methods developed within this project can be applicable to several other dense prediction problems as well.
Semantic segmentation based traffic light detection at day and at night

Semantic segmentation based traffic light detection at day and at night

Traffic light detection from a moving vehicle is an important technology both for new safety driver assistance functions as well as for autonomous driving in the city. In this paper we present a machine learning framework for detection of traffic lights that can handle in real-time both day and night situations in a unified manner. A semantic segmentation method is employed to generate traffic light candidates, which are then confirmed and classified by a geometric and color features based classifier. Temporal consistency is enforced by using a tracking by detection method. We evaluate our method on a publicly available dataset recorded at daytime in order to compare to existing methods and we show similar performance. We also present an evaluation on two additional datasets containing more than 50 intersections with multiple traffic lights recorded both at day and during nighttime and we show that our method performs consistently in those situations.
Framework for generation of synthetic ground truth data for driver assistance applications

Framework for generation of synthetic ground truth data for driver assistance applications

High precision ground truth data is a very important factor for the development and evaluation of computer vision algorithms and especially for advanced driver assistance systems. Unfortunately, some types of data, like accurate optical flow and depth as well as pixel-wise semantic annotations are very difficult to obtain.

In order to address this problem, in this paper we present a new framework for the generation of high quality synthetic camera images, depth and optical flow maps and pixel-wise semantic annotations. The framework is based on a realistic driving simulator called VDrift [1], which allows us to create traffic scenarios very similar to those in real life.

We show how we can use the proposed framework to generate an extensive dataset for the task of multi-class image segmentation. We use the dataset to train a pairwise CRF model and to analyze the effects of using various combinations of features in different image modalities.
Inside-Out Tracking

Inside-Out Tracking

Current tracking solutions routinely used in a clinical, potentially surgically sterile, environment are limited to mechanical, electromagnetic or classic optical tracking. Main limitations of these technologies are respectively the size of the arm, the influence of ferromagnetic parts on the magnetic field and the line of sight between the cameras and tracking targets. These drawbacks limit the use of tracking in a clinical environment. The aim of this project is the development of so-called inside-out tracking, where one or more small cameras are fixed on clinical tools or robotic arms to provide tracking, both relative to other tools and static targets.
These developments are funded from the 1st of January 2016 to 31st of December 2017 by the ZIM project Inside-Out Tracking for Medical Applications (IOTMA).
Rigid 3D Object Detection

Rigid 3D Object Detection

Object detection and localization is a crucial step for inspection and manipulation tasks in robotic and industrial applications. We present an object detection and localization scheme for 3D objects that combines intensity and depth data. A novel multimodal, scale- and rotation-invariant feature is used to simultaneously describe the object's silhouette and surface appearance. The object's position is determined by matching scene and model features via a Hough-like local voting scheme. The proposed method is quantitatively and qualitatively evaluated on a large number of real sequences, proving that it is generic and highly robust to occlusions and clutter. Comparisons with state of the art methods demonstrate comparable results and higher robustness with respect to occlusions.
Computer-Aided Ophthalmic Procedures

Computer-Aided Ophthalmic Procedures

In the current workflow of opthamologic surgeries, the surgeon observes the scene in an indirect way through a microscope while performing the surgery with the necessary high handling precision. During membrane peeling, for example, the surgeon has to grasp an anatomical layer of only 10 μm off the retina without damaging it. However, under this limited microscopic view, it becomes very challenging to infer the distance of the surgical instrument to the retina. Issues such as lens distortion, high level of blurriness and lack of haptic feedback complicate the task further. Recently, an intraoperative version of the Optical Coherence Tomography (OCT) was introduced, which provides the 3D information along a scanning line. On the one hand, this modality provides depth information during the surgery. On the other hand, the device has to be manually positioned to the region of interest which further complicates the current workflow of the surgeon (who already has to manipulate the surgical tool, the handheld light source and the microscope). The main goal of the project is support the surgeon in the current workflow and provide additional information during the surgery via advanced computer vision, visualization and augmented reality algorithms.

Recent Publications

2019
Y. Wang, D. J. Tan , N. Navab, F. Tombari
ForkNet: Multi-branch Volumetric Semantic Completion from a Single Depth Image
International Conference on Computer Vision (ICCV), Seoul, Korea, October 2019 (bib)
F. Manhardt, D. Arroyo, C. Rupprecht, B. Busam, T. Birdal, N. Navab, F. Tombari
Explaining the Ambiguity of Object Detection and 6D Pose From Visual Data
International Conference on Computer Vision (ICCV), Seoul, Korea, October 2019 (bib)
M. Herb, T. Weiherer, N. Navab, F. Tombari
Crowd-sourced Semantic Edge Mapping for Autonomous Vehicles
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Macau, China, November, 2019. (bib)
I. Laina, N. Navab, C. Rupprecht
Towards Unsupervised Image Captioning with Shared Multimodal Embeddings
International Conference on Computer Vision (ICCV), Seoul, Korea, October 2019. (bib)
H. Dhamo, N. Navab, F. Tombari
Object-driven Multi-Layer Scene Decomposition from a Single Image
International Conference on Computer Vision (ICCV), Seoul, Korea, October 2019 (bib)
J. Wald, A. Avetisyan, N. Navab, F. Tombari, M. Niessner
RIO: 3D Object Instance Re-Localization in Changing Indoor Environments
International Conference on Computer Vision (ICCV), Seoul, Korea, October 2019 [oral].
The last two authors share senior authorship.
(bib)
J. Postel, F. Ferroni, H. Coskun, N. Navab, F. Tombari
Sampling-free Epistemic Uncertainty Estimation Using Approximated Variance Propagation
International Conference on Computer Vision (ICCV), Seoul, Korea, October 2019 [oral]. (bib)
H. Dhamo, K. Tateno, I. Laina, N. Navab, F. Tombari
Peeking Behind Objects: Layered Depth Prediction from a Single Image
Pattern Recognition Letters, Vol. 125, 2019 (bib)
Y. Gao, Y. Wang, , N. Navab, F. Tombari
Variational Object-aware 3D Hand Pose from a Single RGB Image
IEEE Robotics and Automation Letters 2019 (bib)
T. Birdal, B. Busam, N. Navab, S. Ilic, P. Sturm
Generic Primitive Detection in Point Clouds Using Novel Minimal Quadric Fits
IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI), 2019 (bib)
2018
C. Rupprecht, I. Laina, N. Navab, G. D. Hager, F. Tombari
Guide Me: Interacting with Deep Networks
Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, Utah, USA, June 2018.
Spotlight. The first two authors contribute equally to this paper.
(bib)
J. Wald, K. Tateno, J. Sturm, N. Navab, F. Tombari
Real-Time Fully Incremental Scene Understanding on Mobile Platforms
IEEE Robotics and Automation Letters (presented at IROS), Madrid, Spain, October 2018 (bib)
K. Tateno, N. Navab, F. Tombari
Distortion-Aware Convolutional Filters for Dense Prediction in Panoramic Images
15th European Conference on Computer Vision (ECCV), Munich, Germany, September, 2018. (bib)
D. Rethage, J. Wald, J. Sturm, N. Navab, F. Tombari
Fully-Convolutional Point Networks for Large-Scale Point Clouds
European Conference On Computer Vision (ECCV), Munich, Germany, September 2018 (bib)
F. Manhardt, W. Kehl, N. Navab, F. Tombari
Deep Model-Based 6D Pose Refinement in RGB
European Conference On Computer Vision (ECCV), Munich, Germany, September 2018 [oral].
The first two authors contributed equally to this paper.
(bib)
T. Birdal, B. Busam, N. Navab, S. Ilic, P. Sturm
A Minimalist Approach to Type-Agnostic Detection of Quadrics in Point Clouds
IEEE Computer Vision and Pattern Recognition (CVPR), Salt Lake City, United States, June 2018 (bib)
G. Ghazaei, I. Laina, C. Rupprecht, F. Tombari, N. Navab, K. Nazarpour
Dealing with Ambiguity in Robotic Grasping via Multiple Predictions
Asian Conference on Computer Vision and Pattern Recognition (ACCV), Perth, Australia, December 2018. (bib)
2017
C.-H. Huang, B. Allain, E. Boyer, J.-S. Franco, F. Tombari, N. Navab, S. Ilic
Tracking-by-Detection of 3D Human Shapes: from Surfaces to Volumes
IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) (bib)
W. Kehl, F. Tombari, S. Ilic, N. Navab
Real-Time 3D Model Tracking in Color and Depth on a Single CPU Core
Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, USA, July 2017 (bib)
M. Slavcheva, W. Kehl, N. Navab, S. Ilic
SDF-2-SDF Registration for Real-time 3D Reconstruction from RGB-D Data
International Journal of Computer Vision (IJCV), Springer Verlag. The final publication is available at www.springerlink.com. (bib)
W. Kehl, F. Manhardt, F. Tombari, S. Ilic, N. Navab
SSD-6D: Making RGB-Based 3D Detection and 6D Pose Estimation Great Again
International Conference on Computer Vision (ICCV), Venice, Italy, October 2017 [oral].
The first two authors contributed equally to this paper.
(bib)
H. Coskun, F. Achilles, R. DiPietro, N. Navab, F. Tombari
Long Short-Term Memory Kalman Filters:Recurrent Neural Estimators for Pose Regularization
International Conference on Computer Vision (ICCV), Venice, Italy, October 2017 (bib)
D. J. Tan , N. Navab, F. Tombari
6D Object Pose Estimation with Depth Images: A Seamless Approach for Robotic Interaction and Augmented Reality
This was submitted as an extended abstract to the demo session and workshop.
A pre-print version is available online at arXiv.
(bib)
D. J. Tan , N. Navab, F. Tombari
Looking Beyond the Simple Scenarios: Combining Learners and Optimizers in 3D Temporal Tracking
IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Nantes, France, 2017 (bib)
K. Tateno, F. Tombari, I. Laina, N. Navab
CNN-SLAM: Real-time dense monocular SLAM with learned depth prediction
IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), Hawaii, USA, June, 2017.
The first two authors contribute equally to this paper.
(bib)
D. J. Tan , N. Navab, F. Tombari
Adaptive Learning-based Temporal Tracker for 3D Head Shape Models
British Machine Vision Conference, London, UK, September 4-7, 2017 (bib)
K. Tateno, F. Tombari, N. Navab
Large Scale and Long Standing Simultaneous Reconstruction and Segmentation
Computer Vision and Image Understanding (CVIU), Volume 157, Pages 138-150, April, 2017.
The final publication is available at http://www.sciencedirect.com/science/article/pii/S1077314216300662
(bib)
2016
W. Kehl, T. Holl, F. Tombari, S. Ilic, N. Navab
An Octree-Based Approach towards Efficient Variational Range Data Fusion
British Machine Vision Conference (BMVC), York, UK, September 2016 (bib)
W. Kehl, F. Milletari, F. Tombari, S. Ilic, N. Navab
Deep Learning of Local RGB-D Patches for 3D Object Detection and 6D Pose Estimation
European Conference On Computer Vision (ECCV), Amsterdam, The Netherlands, October 2016 (bib)
I. Laina, C. Rupprecht, V. Belagiannis, F. Tombari, N. Navab
Deeper Depth Prediction with Fully Convolutional Residual Networks
International Conference on 3DVision (3DV), Stanford University, California, USA, October 2016.
Oral presentation. The first two authors contribute equally to this paper.
(bib)
F. Achilles, A.E. Ichim, H. Coskun, F. Tombari, S. Noachtar, N. Navab
PatientMocap: Human Pose Estimation under Blanket Occlusion for Hospital Monitoring Applications
Proceedings of the 19th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), Athens, Greece, October 2016 (bib)
M. Slavcheva, W. Kehl, N. Navab, S. Ilic
SDF-2-SDF: Highly Accurate 3D Object Reconstruction
European Conference on Computer Vision (ECCV), Amsterdam, The Netherlands, October 2016 (supplementary) (bib)
C. Li , H. Xiao, K. Tateno, F. Tombari, N. Navab, G. D. Hager
Incremental Scene Understanding on Dense SLAM
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, Korea, October, 2016. (bib)
F. Achilles, F. Tombari, V. Belagiannis, A.M. Loesch, S. Noachtar, N. Navab
Convolutional neural networks for real-time epileptic seizure detection
Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization, Taylor & Francis, 2016 (bib)
C.-H. Huang, B. Allain, J.-S. Franco, N. Navab, S. Ilic, E. Boyer
Volumetric 3D Tracking by Detection
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, USA, June 2016 (spotlight presentation)
(supp., poster, spotlight oral video) The first two authors contribute equally to this paper.
(bib)
F. Grün, C. Rupprecht, N. Navab, F. Tombari
A Taxonomy and Library for Visualizing Learned Features in Convolutional Neural Networks
International Conference on Machine Learning (ICML) Workshop on Visualization for Deep Learning, New York, USA, June 23rd, 2016 (bib)
K. Tateno, F. Tombari, N. Navab
When 2.5D is not enough: Simultaneous Reconstruction, Segmentation and Recognition on dense SLAM
IEEE International Conference on Robotics and Automation (ICRA), Stockholm, May 2016 (bib)

D. J. Tan , T. Cashman, J. Taylor, A. Fitzgibbon, D. Tarlow, S. Khamis, S. Izadi, J. Shotton
Fits Like a Glove: Rapid and Reliable Hand Shape Personalization
IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, USA, June 2016 (bib)
2015
W. Kehl, F. Tombari, N. Navab, S. Ilic, V. Lepetit
Hashmod: A Hashing Method for Scalable 3D Object Detection
British Machine Vision Conference (BMVC), Swansea, UK, September 2015 (bib)
K. Tateno, F. Tombari, N. Navab
Real-Time and Scalable Incremental Segmentation on Dense SLAM
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Hamburg, Germany, September 2015 (bib)
C.-H. Huang, F. Tombari, N. Navab
Repeatable Local Coordinate Frames for 3D Human Motion Tracking: from Rigid to Non-Rigid
International Conference on 3D Vision (3DV), Lyon, France, Oct. 20, 2015 (bib)
T. Birdal, S. Ilic
Point Pair Features Based Object Detection and Pose Estimation Revisited
IEEE Computer Society Conference on 3D Vision (3DV), Lyon, France, October 2015 [oral] (poster) (bib)
F. Milletari, W. Kehl, F. Tombari, S. Ilic, A. Ahmadi, N. Navab
Universal Hough dictionaries for object tracking
British Machine Vision Conference (BMVC), Swansea, UK, September 2015 (bib)
C.-H. Huang, E. Boyer, B. do Canto Angonese, N. Navab, S. Ilic
Toward User-specific Tracking by Detection of Human Shapes in Multi-Cameras
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, USA, June 2015 (supp., poster) (bib)
U. Simsekli, T. Birdal
A Unified Probabilistic Framework for Robust Decoding of Linear Barcodes
IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brisbane, Australia, April 2015 (bib)
2014
W. Kehl, N. Navab, S. Ilic
Coloured signed distance fields for full 3D object reconstruction
British Machine Vision Conference (BMVC), Nottingham, UK, September 2014 (bib)
D. J. Tan , S. Ilic
Multi-Forest Tracker: A Chameleon in Tracking
IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, USA, June 24, 2014 (bib)
C.-H. Huang, E. Boyer, N. Navab, S. Ilic
Human Shape and Pose Tracking Using Keyframes
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, USA, June 24, 2014 (supp., poster) (bib)
D. J. Tan , S. Holzer, N. Navab, S. Ilic
Deformable Template Tracking in 1ms (Oral)
British Machine Vision Conference, Nottingham, UK, September 1, 2014 (bib)
2013
C. Rupprecht, O. Pauly, C. Theobalt, S. Ilic
3D Semantic Parameterization for Human Shape Modeling: Application to 3D Animation. (Oral Presentation)
In Proc. International Conference on 3D Vision (3DV 2013) (bib)
J. Lallemand, O. Pauly, L. Schwarz, D. J. Tan , S. Ilic
Multi-task Forest for Human Pose Estimation in Depth Images. (Oral Presentation)
In Proc. International Conference on 3D Vision (3DV 2013) (bib)
C.-H. Huang, E. Boyer, S. Ilic
Robust Human Body Shape and Pose Tracking
International Conference on 3D Vision (3DV), Seattle, USA, June 29, 2013 (oral, best paper runner up) (slides, poster) (bib)


Edit | Attach | Refresh | Diffs | More | Revision r1.79 - 11 Aug 2019 - 19:13 - MarkusHerb

Lehrstuhl für Computer Aided Medical Procedures & Augmented Reality    rss.gif