Natural Feature Based Head- and Eye-Tracking

Subject

With recent advances of Head-up Display (HUD) technology in cars, Augmented Reality becomes interesting in supporting the driving task. At the chair for ergonimocs of the faculty for mechanical engineering, we have a driving simulator and are currently integrating a HUD.

Sketch of the existing HUD (Courtesy of BMW)

In order to render location fixed virtual objects, we have to know about the position of the driver's head. For example, when the head moved to the left, virtual objects must be rendered on a leftmore position on the HUD.

This SEPs focus resides in integration of a normal USB camera and a infrared USB camera into an Augmented Reality application for use in a Head-Up Display. The infrared camera is intended to give two unique spots from the drivers pupils, because these are extremely reflective. On the other hand, the normal light camera allows natural feature tracking of the nose's root and the eyeball. Computer Vision algorithms shall be used to track the points and areas of relevance. In addition we got some code for skin independent tracking of the root of a nose.

As a interesting addition to the SEP, we want to track the drivers gaze. We need this information to monitor the drivers distraction caused by virtual objects in the HUD.

Content

This section defines taks that have to be absolved within this project.

Calibration

Whenever the setup of the cameras changes (does not happen inside the car or a new users mounts te car, the system needs to be recalibrated. So, a novel, easy and fast calibration mechanism must be considered.

Tracking

Head-Tracking is the main goal of this project. So we focus on this and give Eye-Tracking a low priority, even if we need the position of the eyes for comptation of the distance of the head.

What we do is to compare some existing head-tracking libraries, respectively computer-vision libraries. Afterwards, we implement a tracking system by use of the most interesing systems.

• Initial Face Detection
• Face Tracking

3D Pose Reconstruct

Use 2 cams, find an easy calibration mechanism and calibrate both cams to some origin. Reconstruct 3D Pose of nose root (head pose, Nasenwurzel) and both eyes.

Usability, Accuracy/Performance measurement

During time the system is set up, a study has to be developed, how the usabiltiy of the system can be evaluated. In addition, it is important to know about the accuracy and the performence, the system provides.

Sourcecode

Find Matlab stuff, Eigenfaces and other stuff: on janus under /home/shared/public/Students/arnaudov-yordanov

Find source code in SVN repository: svn+ssh://@svnnavab.informatik.tu-muenchen.de/svn/studwork/trunk/seps/arnaudov-yordanov

Find documentation (SEP-Report) in CVS repository on janus: /cvs/tumdwarf, module doc/seps/arnaudov-yordanov

Installation

To install the head tracker on a new machine first make sure that OpenCV(Version 1.0) and FlyCapture API(1.6 Beta 06) have already been installed. The whole source tree is then copied/checked out to a desired location. The application could then be compiled and started form the command line or from within Visual Stusio.

Compiling

Open the HeadTrackingSEP VC++ project file and adjust include in lib paths to point to the right locations of the OpenCV and FlyCapture APIs. Compile.

Running Application

• Go the the folder where the binary has been placed
• Run the program: >HeadTrackingSEP.exe <janus-base-path>/Students/arnaudov-yordanov/newMask2 [image-file-path]

The first command line parameter is compulsory and is the absolute path to the folder containing all the training data. A second command-line parameter, if existent, is used as a path to a still image. In this case tracking is performed on this single image only. See SEP-report for details.

Key-bindings

During initial mask setup use:

• a - Translate Left
• d - Translate Right
• w - Translate Up
• s - Translate Down
• e - Scale Up
• q - Scale Down
• f - Start Tracking

During tracking use:

• SPACE - Make One Step(available when tracking in a single image)
• r - Reset Tracking

-- StanimirArnaudov - 22 Mar 2006
-- MichailYordanov - 22 Mar 2006
-- MarcusToennis - 07 Jul 2005