Dipl. / Master Thesis: 3D Human Pose/Shape Tracking: when bottom-up meets top-down

Student: -
Advisor: Prof. Dr. Nassir Navab
Supervised by: Chun-Hao Paul Huang, Dr. Slobodan Ilic
Due date:

Overview

Human pose estimation can in general be categorized in two classes: top-down vs. bottom-up. The former estimates poses by deforming a reference 3D human surface frame by frame, and avoids unreasonable poses by enforcing constraints on surface deformations. The latter starts with body part detections in 2D images, back-projects the detected body parts from each view, and attempts to assemble the skeletons in 3D space. The following two videos provide one typical example for each category:

         top-down approach by surface tracking (Huang et al. 3DV 2013)          bottom-up approach by pictorial structure (Belagiannis CVPR 2014)

Although top-down methods provide visually and temporally smoother results, bottom-up approaches hold the advantage of not assuming background subtraction, which is suitable to more generalized applications. In this project, we investigate the possibility to find the middle ground, in order to retain the merits from both. Specifically, one first recovers rough 3D skeletal poses with state-of-the-art bottom-up approach as initializations, and deform pre-defined human surfaces to fit to those poses. The generic human surface has to be adapted according to the size/scale of the skeleton. Skeleton-based animation such as linear-blend skinning, dual-quaternion-blend skinning is also worthwhile a look.

Requirements

• The student can use whatever language he/she finds comfortable for fast prototyping, but the final solution is expected to be delivered in C++.
• Knowledge in machine learning, and mesh processing is also helpful.

Literature & Resource

Bottom-up human pose estimation:

• V. Belagiannis, S. Amin, M. Andriluka, B. Schiele, N. Navab, S. Ilic, 3D Pictorial Structures for Multiple Human Pose Estimation (CVPR 2014)

Human shape adaptation/estimation:

• C. Rupprecht, O. Pauly, C. Theobalt, S. Ilic 3D Semantic Parameterization for Human Shape Modeling: Application to 3D Animation (3DV 2013)
• M. Ye, and R. Yang. Real-time Simultaneous Pose and Shape Estimation for Articulated Objects with a Single Depth Camera (CVPR 2014)

Skinning method:

• L. Kavan, S. Collins, J. Zara, C. O'Sullivan, Geometric Skinning with Approximate Dual Quaternion Blending (ACM Transaction on Graphics 2008)

Top-down human pose/shape tracking:

• C.-H. Huang, E. Boyer, and S. Ilic, Robust Human Body Shape and Pose Tracking (3DV 2013)
• C. Cagniart, E. Boyer, and S. Ilic, Probabilistic Deformable Surface Tracking From Multiple Videos (ECCV 2010)
• C. Cagniart, E. Boyer, and S. Ilic, Free-From Mesh Tracking : a Patch-Based Approach (CVPR 2010)
• Related project page: Spatio temporal modeling.

Contact

• Mail to Chun-Hao Paul Huang, or drop by room 03.13.042.