IDP / Dipl. / Master Thesis: Deformable Human Shape matching via Volumetric representation

Student: Bibiana do Canto Angonese
Advisor: Prof. Dr. Nassir Navab
Supervised by: Chun-Hao Paul Huang, Dr. Slobodan Ilic.

Overview

Recently there is a significant progress in human pose estimation with depth data. Thanks to decision forest, now people are able to infer correspondences (either sparse or dense) between a surface model and single depth image. Inspired by these works, in this project we would like to investigate the possibility to infer the correspondences between a surface model and a point cloud/visual hull obtained in multiple camera environments (right figure). One expected issue is, what could be the proper "feature" of input? This feature has to more or less describe the location of the input point along the point cloud/visual hull, such that it's sufficiently discriminative to infer correspondences. 3D shape descriptors that have long been studied in 3D shape matching is therefore worthwhile a look. This one-shot single frame pose estimation approach differs from standard ICP methods. It does not alternate between correspondence-estimation phase and energy-minimization phase. It is thus complement to normal human motion tracking approaches, which always rely on the result of previous frame to build correspondence pairs, and use it as the initialization of energy minimization. With the help of this one-shot approach, it is expected that one can prevent from error accumulation.

Requirements

• The student should have experience in C++ and/or Matlab. Knowledge in random forest is also helpful.

Literature & Resource

Random forest for correspondence estimation:

• J. Taylor, J. Shotton, T. Sharp, and A. Fitzgibbon, The Vitruvian Manifold: Inferring Dense Correspondences for One-Shot Human Pose Estimation (CVPR 2012)
  
• R. Girshick, J. Shotton, P. Kohli, A. Criminisi, and A. Fitzgibbon, Efficient Regression of General-Activity Human Poses from Depth Images (ICCV 2011)
  
• J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, and A. Blake, Real-Time Human Pose Recognition in Parts from a Single Depth Image (CVPR 2011)
  

Keypoints and local descriptors of 3D shapes:

• A. Zaharescu, E. Boyer, R. Horaud, Keypoints and Local Descriptors of Scalar Functions on 2D Manifolds (IJCV 2012), code
• F. Tombari, S. Salti, L. Stefano, Performance Evaluation of 3D Keypoint Detectors (IJCV 2013)
  

Deformable surface tracking:

• C.-H. Huang, E. Boyer, and S. Ilic, Robust Human Body Shape and Pose Tracking (3DV 2013)
  
• C. Cagniart, E. Boyer, and S. Ilic, Probabilistic Deformable Surface Tracking From Multiple Videos (ECCV 2010)
  
• C. Cagniart, E. Boyer, and S. Ilic, Free-From Mesh Tracking : a Patch-Based Approach (CVPR 2010)
  
• Related project page: Spatio temporal modeling.
  
  

Publications

2017
	C.-H. Huang, B. Allain, E. Boyer, J.-S. Franco, F. Tombari, N. Navab, S. Ilic Tracking-by-Detection of 3D Human Shapes: from Surfaces to Volumes IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) (bib)
2015
	C.-H. Huang, F. Tombari, N. Navab Repeatable Local Coordinate Frames for 3D Human Motion Tracking: from Rigid to Non-Rigid International Conference on 3D Vision (3DV), Lyon, France, Oct. 20, 2015 (bib)
	C.-H. Huang, E. Boyer, B. do Canto Angonese, N. Navab, S. Ilic Toward User-specific Tracking by Detection of Human Shapes in Multi-Cameras IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, USA, June 2015 (supp., poster) (bib)

Contact

• Mail to Chun-Hao Paul Huang, or drop by room 03.13.039.