DA/MA/BA Thesis: Webly supervised human activity recognition

In recent years, significant progress in image classification and object recognition has been achieved by learning effective visual representations via Convolutional Neural Networks (CNN) and R-­CNN. Although large datasets such as ImageNet^? have played a key role in this process, as we move forward a key question arises: can we continue to manually annotate images with the hope of scaling up to billion images? This question is even more important in the video domain, where the cost of labeling is more prominent, especially if we want to address also the temporal localization problem.

This project aims at investigating this problem by analyzing the following aspects: 1. “Webly supervised learning” has recently emerged as an alternative to the classical approach based on large sets of manually annotated images. In their recent paper, Chen and Gupta (ICCV 2015) show that a CNN trained only with web images performs very competitively to the ImageNet^? pretrained network architecture for object detection without even using a single ImageNet^? training label. 2. From images to videos: in this project we explore this approach in the context of a large-­scale video collection of human activities. The key idea is to incrementally learn CNNs starting from the images retrieved from a search engine. So the initial seed of images is enriched with the "easy” frames extracted from the classified videos, then the initial classifier is incrementally adapted to the video domain. 3. Modeling the temporal structure: we will analyse different architectures for representing the video, starting from a simple early/late fusion scheme and then proceeding towards more complex architectures which explicitly model temporal sequences (e.g. LSTM).

As for the experimental analysis, the developed approach will be tested on ActivityNet^?, a large-­scale video benchmark for activity recognition and temporal localization; we will provide an in-­depth analysis of the performances that can be obtained without using a single training labelled video.

Supervision

The work will be carried out together with Dr. Lamberto Ballan, member of the Stanford Vision Lab at Stanford University.

Requirements

The student should have experience in C++ and/or Matlab. Knowledge in machine learning, in particular Deep Learning, is also helpful.

References

• Ikizler-­Cinbis, “Learning actions from the Web”, ICCV 2009
• Chen and Gupta, “Webly Supervised Learning of Convolutional Networks”, ICCV 2015
• Heilbron et al, "ActivityNet: A Large-Scale Video Benchmark for Human Activity Understanding”, CVPR 2015

Contact

Mail to Federico Tombari, or drop by room 03.13.042