Chair for Computer Aided Medical Procedures & Augmented Reality
Lehrstuhl für Informatikanwendungen in der Medizin & Augmented Reality

Visual Speech Processing - Winter Term 2017/2018

Administrative Info

Lecture by Guest Lecturer Helen Bear

Type: Lecture IN3200
Programs: Informatics (Bachelor, Master)
Biomedical Computing (Master)
Computational Science and Engineering (Master)
Wirtschaftsinformatik (Bachelor, Master)
SWS: 4
ECTS: 5 Credits
Course Language: English
Contact: h.bear@surrey.ac.uk
(on appointment, send an email beforehand)

Time, Location & Requirements

Course information and certificate requirements:

  • The classes as well as the exercises will be held in English.
  • Software requirements: Unity, FFmpeg, HTK, Matlab

Site Content


  • 15/08: Registration on TUMOnline is possible
  • 01/09: Lectures will be held Monday's and Thursday's from 1pm to 2:30pm


It is a common misconception that speech is only about the comprehension of sounds. In reality, we also use visual cues to understand what someone is saying. This means articulated speech is bimodal, meaning there are two channels of information; acoustic and visual. Research in recognising acoustic speech is maturing, think of Siri and Cortana. However, research into visual speech understanding is in its infancy. Sometimes called machine lip reading, work in understanding the visual speech channel, (which you can also think of as a sequence of lip gestures) is showing just how difficult this is, and how different visual speech is compared to acoustic speech.

In this module we will cover the latest work in this field. Understanding the challenges it provides, reviewing the latest results for recognition and synthesis (for animations) and how we are striving to exploit that which we know for application in new domains such as speech therapy and historical research.

This module requires proactive participation and independent study in addition to the contact time. Some prior knowledge of signal processing will help but is not essential, plus good mathematical skills. This module does not relate to processing specific medical images such as MRI’s, or x-rays. Rather we use image processing of videos to understand visual speech.

Some lip reading data sets are legally restricted to protect the identity and confidentiality of the data subjects, therefore there is strict policy of no photographs or recordings permitted during lectures. Sharable lecture slides and other teaching materials will be provided.

The module is structured to be 4-6 hours of contact time per week as a mixture of lectures, seminars, and practical tutorials. Additionally, the Module Organiser will commit to two hours every week where students can drop in without appointment. Appointments can be arranged for other times by email in advance.

Learning Outcomes

By the end of this module, students will gain;

  • knowledge of the conventional lip-reading system and its parameters,
  • understanding of conventional and cutting edge lip features - representation of small non-skeletal shapes, and
  • Knowledge of the complex function between audio to visual speech.
  • Students will understand how linguistics & speaker individuality affect the visual channel,
  • gain practical skills in recognising and synthesizing realistic speech,
  • methodological skills in unit and label selection for modelling visual speech gestures &,
  • practical skills in audio speech recognition.
  • Students will also obtain knowledge of expressive speech component variations and,
  • understand how to recognise languages from visual speech channel alone.

Preliminary Lecture & Exercise Schedule

Date Time Place Material/Literature Topic Conducted by Material
19.10.17 13:00 - 15:00 01.13.010 Lecture Introduction to Lip Reading Helen Bear Exercise Sheet
Code to test
23.10.17 13:00 - 15:00 01.13.010 Lecture Data and challenges Helen Bear  
26.10.17 13:00 - 15:00 01.13.010 Lecture Units and labels Helen Bear  
30.10.17 13:00 - 15:00 01.13.010 Lecture/seminar Phonemes-to-visemes Helen Bear  
02.11.17 13:00 - 15:00 01.13.010 Lecture Speakers Helen Bear  
06.11.17 13:00 - 15:00 01.13.010 Lecture/seminar Multispeakers and adaptation Helen Bear  
09.11.17 13:00 - 15:00 01.13.010 Lecture Visual speech features 1 Helen Bear  
13.11.17 13:00 - 15:00 01.13.010 Lecture/Seminar Visual speech features 2 Helen Bear  
16.11.17 13:00 - 15:00 01.13.010 Lecture Classification 1 Helen Bear  
20.11.17 13:00 - 15:00 01.13.010 Lecture Classification 2 Helen Bear  
23.11.17 13:00 - 15:00 01.13.010 Lecture A Visual Speech Deep Learning Review Helen Bear  
27.11.17 13:00 - 15:00 01.13.010 Lecture/Seminar Learning sentences Helen Bear  
30.11.17 13:00 - 15:00 01.13.010 Lecture Introduction to boosting Helen Bear  
04.11.17 13:00 - 15:00 01.13.010 Lecture/seminar Hierarchical training for the audio-visual function Helen Bear  
07.12.17 13:00 - 15:00 01.13.010 Lecture Audio-visual 1 Helen Bear  
11.12.17 13:00 - 15:00 01.13.010 Lecture Audio-visual 2 Helen Bear  
14.12.17 13:00 - 15:00 01.13.010 Lecture/Demo Building a voice recogniser - Acoustic word recognition Helen Bear  
18.12.17 13:00 - 15:00 01.13.010 Lecture/Demo Building a visual voice recogniser - A conventional visual speech recogniser Helen Bear  
08.01.18 13:00 - 15:00 01.13.010 Lecture Expressive and emotional visual speech Helen Bear  
11.01.18 13:00 - 15:00 01.13.010 Lecture/Seminar Visual speech synthesis with AAMs Helen Bear  
15.01.18 13:00 - 15:00 01.13.010 Lecture Dynamic visemes Helen Bear  
18.01.18 13:00 - 15:00 01.13.010 Lecture/Seminar Visual speech synthesis with DNNs Helen Bear  
22.01.18 13:00 - 15:00 01.13.010 Tutorial Animation with Unity Helen Bear  
25.01.18 13:00 - 15:00 01.13.010 Tutorial Animation with Unity Helen Bear  
29.01.18 13:00 - 15:00 01.13.010 REVISION Come with your questions Helen Bear  
05.02.18 13:00 - 15:00 01.13.010 EXAM EXAM - in class. Helen Bear  

Title: Visual Speech Processing
Professor: Guest Lecturer Helen Bear
Type: Lecture
Information: 4 SWS, 5 ECTS Credits IN3200
Term: 2017WiSe

Edit | Attach | Refresh | Diffs | More | Revision r1.9 - 22 Jan 2018 - 14:29 - HelenBear

Lehrstuhl für Computer Aided Medical Procedures & Augmented Reality    rss.gif