Sensornets and Sensor Fusion
Hauptseminar Sommersemester 2006
Prof. Gudrun Klinker, Ph.D.
, Martin Bauer
, and Daniel Pustka
Anmeldung per email an
This course's topic
(AR) applications are highly dependent upon accurate and precise tracking data. Since current tracking technologies do not always provide such information everywhere in real-time application developers must combine several trackers to minimize negative properties of one tracker by another. The
result are sensor networks. They can be used to inform applications about the current position and orientation of objects they are concerned about. Furthermore, such pose data can be evaluated with respect to several criteria of quality.
Currently most AR applications use their own customized solution to this problem. Typically, these solutions are hardly reusable in other systems. This inhibits the development of large-scale sensor networks because there are no standard interfaces between these technologies. In this course, we will look at the necessities to form ubiquitous tracking environments consisting of several sensor networks.
The seminar will take place on two days near the end of the summer term (Blockseminar)
July 10th: 10h-12h, 13h-15h
July 17th: 10h-12h, 13h-15h
Room: to be announced
How to get the Schein
- The outline of the talk and the report must be discussed with the supervisor at least one month before the presentation
- The presentation slides and handout should be finalized and discussed with the supervisor one week before the presentation (here is a template)
- A preliminary version of the report is given to the supervisor one week before the talk
- Give a well prepared talk of about an hour and handle follow up discussion
- Write a report of approximately 15 pages (in LaTex and english, here is a template)
- Be present at all talks
Theses papers should be read by all participants not yet fimilar with the concepts presented therein, to get a common basis of understanding to build on.
What is Augmented Reality and Tracking?
This presentation should give an overview of state-of-the-art Augmented Reality
systems and technologies. It will explain the general processing pipeline of AR systems and highlight the key subcomponents of every AR system: tracking, 3D scene generation and multimodal user interaction. This talk will present a selection of AR systems, ranging from the Boeing Wire Assembly demo to current AR software frameworks. It will try to identify reoccurring problems and the evolution of ideas.
- R. Azuma, A Survey of Augmented Reality
- R. Azuma, Y. Baillot, R. Behringer, S. Feiner, S. Julier, B. MacIntyre, Recent advances in augmented reality
- Paul Milgram, Herman Colquhoun Jr., A Taxonomy of Real and Virtual World Display Integration, In: Y. Ohta and H. Tamura, eds., "Mixed Reality -- Merging Real and Virtual Worlds", March 1999.
- G. Klinker, K. Ahlers, D. Breen, P.-Y. Chevalier, C. Crampton, D. Greer, A. Kramer, E. Rose, M. Tuceryan, R. Whitaker Confluence of Computer Vision and Interactive Graphics for Augmented Reality PRESENCE - Teleoperators and Virtual Environments, Special Issue on Augmented Reality 6(4), 1997.
- C. Sandor, A. MacWilliams, M. Wagner, M. Bauer, G. Klinker, SHEEP: The Shared Environment Entertainment Pasture
- M. Bauer, B. Bruegge, G. Klinker, A. MacWilliams, T. Reicher, S. Riß, C. Sandor, M. Wagner, Design of a Component-Based Augmented Reality Framework
- S. Feiner, B. MacIntyre, T. Höllerer, and T. Webster. A Touring Machine: Prototyping 3D Mobile Augmented Reality Systems for Exploring the Urban Environment. In Proceedings of ISWC 1997, Boston (MA), 1997.
- D. Schmalstieg, A. Fuhrmann, G. Hesina, Z. Szalavari, L. Miguel Encarnacao, M. Gervautz, and W. Purgathofer. The Studierstube Augmented Reality Project. Technical report, TU Wien, 2000.
Ubiquitous and Context Aware Computing: Overview and Systems
Ubiquitous Computing (UbiComp) is a new paradigm that aims at making computer systems invisibly interwoven with the user's environment. This talk will give an overview of the first ideas of UbiComp and applications that have been implemented to date. Context-aware systems, especially those that are mobile, are a fairly new field, but several successful systems have already been built. All of these are heavily under development and are used as research platforms. This presentation should give an overview of context awareness as a term, existing systems and present a few in more detail, showing goals, approach and current status.
- Mark Weiser. The Computer for the 21st Century. Scientific American, 1991.
- Lars Erik Holmquist, Hans-Werner Gellersen, Gerd Kortuem, Albrecht Schmidt, Martin Strohbach, Stavros Antifakos, Florian Michahelles, Bernt Schiele , Michael Beigl and Ramia Maze, Building intelligent environments with Smart-Its. In Computer Graphics and Applications, IEEE, Jan/Feb 2004, page 56 - 64, Volume: 24 , Issue: 1, ISSN: 0272-1716.
- Florian Michahelles, Stavros Antifakos, Jani Boutellier, Albrecht Schmidt and Bernt Schiele, Instructions immersed into the real world. How your Furniture can teach you. Poster Submission, The Fifth International Conference on Ubiquitous Computing (Ubicomp), Seattle, USA, October 2003.
- Anind K. Dey. Providing Architectural Support for Building Context-Aware Applications. PhD thesis, Georgia Institute of Technology, November 2000. 1st chapter.
- D.Garlan, D.Siewiorek, A.Smailagic, P.Steenkiste: Project Aura: Toward Distraction-Free Pervasive Computing. IEEE Pervasive Computing, Vol.1, No.2, 2002, pp. 22-32
- Tony Jebara, Bernt Schiele, Nuria Oliver, and Alex Pentland. DyPERS: Dynamic Personal Enhanced Reality System. Technical Report 463, MIT Media Laboratory, Cambridge, MA, 1997
Overview and Mathematics of Tracking Technologies
Tracking is the process of repeatedly determining the position and orientation of objects or people. It is one of the core technologies of Augmented Reality and is at the center of this course's attention. In this talk, an overview of existing tracking technologies should be given, along with an intense discussion of advantages and drawbacks of different technologies. The representation of position and orientation in three-dimensional space and reoccuring problems therein will be discussed.
- Miguel Ribo, State of the Art Report on Optical Tracking, 2001
- K. Meyer, H. L. Applewhite and F. A. Biocca. A Survey of Position Trackers. Presence: Teleoperators and Virtual Environments, Vol. 1, No. 2, pp. 173-200, 1992.
- Rolland, J.P., L. Davis, and Y. Baillot, A Survey of Tracking Technology for Virtual Environments, in Augmented Reality and Wearable Computers, Ch. 3, Ed. Barfield and Caudell, Mahwah, NJ., 2001.
- Gary Bishop, Greg Welch, and B. Danette Allen, Tracking: Beyond 15 Minutes of Thought, SIGGRAPH 2001 Course Notes
- J. Kuipers, Quaternions and Rotation Sequences: A Primer with Applications to Orbits, Aerospace, and Virtual Reality
Introduction & Concepts of Ubiquitous Tracking
: Alexander Perzylo
In providing the possibility for wide area Augmented Reality
applications, combining sensor information into complicated tracking setups becomes more and
more an issue. With Ubiquitous Tracking a formal framework is introduced which
addressees the problems with describing these setups.
The framework defines three different
kinds of spatial relationship graphs to create an abstract view of the
relations between objects and sensors. This gives such a general and flexible way
to describe spatial relations that it kan be used for all kinds of
setups also from existing applications.
- J. Newman, M. Wagner, T. Pintaric, A. MacWilliams, M. Bauer, G. Klinker, D. Schmalstieg, Fundamentals of Ubiquitous Tracking for Augmented Reality
- M. Wagner, G. Klinker. An Architecture for Distributed Spatial Configuration of Context Aware Applications
- G. Klinker, T. Reicher, B. Bruegge, Distributed User Tracking Concepts for Augmented Reality Applications
Existing Architectures & Systems
In this work some existing systems for Ubiquitous Tracking are presented: OpenTracker
, which implements a static dataflow model for streams of sensors readings;
DWARF - a framework for component-based peer-to-peer systems; VRPN - a static network-transparent abstraction between applications and pre-defined
trackers and Trackd - a commercial system from VRCO Inc., which abstracts the tracking from the other applications and offers a common API.
- OpenTracker, TU Wien
- G. Reitmayr, D. Schmalstieg, An Open Software Architecture for Virtual Reality Interaction, In Proc. VRST'01, Banff, Canada, Nov. 15 - 17, 2001.
- G. Reitmayr, D. Schmalstieg. OpenTracker -- an open software architecture for reconfigurable tracking based on XML. In Proc. IEEE Virtual Reality 2001, pages 285--286, Yokohama, Japan, March 13--17 2001.
- Russell M. Taylor II, Thomas C. Hudson, Adam Seeger, Hans Weber, Jeffrey Juliano, Aron T. Helser,
VRPN: A Device-Independent, Network-Transparent VR Peripheral System
Proceedings of ACM Symposium on Virtual Reality Software & Technology (VRST 2001)
- Commercial software: trackd, by VRCO, Inc.
Tracker Alignement: Algorithms and Procedures
: Rainer Kofler
Given two sensors rigidly connected to each other, tracker alignment is used to determine the transformation between those two. Once correctly calibrated, the measurements made by the sensors can be related each other. A short scenario: A camera can be tracked by attaching a fiducal to it that is tracked by another device; knowledge of the pose of the camera can be used together with the images obtained by the camera to generate three dimensional models of the scene perceived. This paper presents techniques for obtaining the transformation between two coordinate systems obtained from trackers, starting from the classical way from Tsai, a nonlinear two stepped algorithm, and ending at a linear one stepped algorithm from Daniilidis which makes use of dual quaternions.
The Mathematics of (Auto-)Calibrating AR Systems
In AR setups, some relationships between objects and properties of cameras, displays etc. have to be determined before the system can be used. The mathematics to do so will be discussed in this talk. Additionally, this presentation gives an overview of research results aimed at automatic calibration of AR setups. Calibrating large setups without the user's intervention will be a key requirement to make the vision of Ubiquitous Tracking work.
- R. Y. Tsai. A Versatile Camera Calibration Technique for High Accuracy 3D Machine Vision. IBM Technical Report 1985.
- M. Tuceryan, D. Greer, R. Whitaker, D. Breen, C. Crampton, E. Rose, and K. Ahlers, Calibration requirements and procedures for a monitor-based augmented reality system. IEEE Transactions on Visualization and Computer Graphics, vol. 1, no. 3, pp. 255–273, 1995.
- A. State, M. Livingston, W. Garrett, G. Hirota, M. Whitton, E. Pisano, and H. Fuchs, Superior augmented reality registration by integrating landmark tracking and magnetic tracking. in Computer Graphics Proceedings, Annual Conference Series: SIGGRAPH ’96 (New Orleans, LA), pp. 429–438, ACM SIGGRAPH, New York, August 1996.
- D. Koller, G. Klinker, E. Rose, D. Breen, R. Whitaker, M. Tuceryan, Automated camera calibration and 3D egomotion estimation for augmented reality applications. Proc. 7th Int. Conf. on Computer Analysis of Images and Patterns (CAIP'97), 1997.
The Kalman Filter and its Extensions
Sensor Fusion aims to coveniently integrate data from different sensors in order to compute a dynamic systems's state.
Here problems arise out of uncertainty. In general the knowledge about the system is incomplete and the
given data is corrupted with noise. Hence the state of the system can only be estimated.
The Kalman Filter is a very robust and popular approach for stochastic estimation given noisy measurements. This paper
therefore concentrates on the ideas of Kalman Filtering for linear and non-linear process models and the application of
the filter for the task of combining different data sets. Furthermore some extensions of the Kalman Filter are presented
which are useful for sensor fusion: the SCAAT method and the Federated Kalman Filter.
- R.E. Kalman, A New Approach to linear Filtering and Prediction Problems, 1960
- A. Gelb (editor), Applied Optimal Estimation
- G. Welch and G. Bishop. An Introduction to the Kalman Filter. Technical Report TR 95-041, University of North Carolina, Department of Computer Science, 1995.
- G. Welch: The Kalman Filter Page
- Neal A Carlson, Federated square root filter for decentralized parallel processors, IEEE Transactions on Aerospace and Electronic Systems, Vol. 26 (3) May 1990, pp. 517-525
- G. Welch: SCAAT: Incremental Tracking with Incomplete Information
- David A. Forsyth, Jean Ponce, Tracking with Nonlinear Models, Chapter 17 from "Computer Vision: A Modern Approach"
Sensor Fusion: Particle Filters
: Daniel Lang
In recent years, particle filters have found widespread application in domains with noisy sensors, such as computer vision and robotics, as well as in many other technology fields. Particle filters are powerful tools for Bayesian state estimation in non-linear systems. The key idea of particle filters is to approximate a posterior distribution over unknown state variables by a set of particles, drawn from this distribution. This paper addresses a primary definition and methods of particle filters: Particle filters are insensitive to costs that might arise from the approximate nature of the particle representation. Their only criterion for generating a particle is the posterior likelihood of a state. This paper gives also short introduction to Bayesian filters, continuing with the most known methods, as well as the advantages and disadvantages of the particle filters as one of the most use tracking methods. There is also one short overview of particle filter aplication in the area of tracking people.
- David A. Forsyth, Jean Ponce, Tracking with Nonlinera Models, chapter that did not make it into the book "Computer Vision: A Modern Approach"
- M. N. Rosenbluth and A.W. Rosenbluth, Monte Carlo calculation of the average extension of molecular chains, Journal of Chemical Physics 23 (2) 1956.
- N. J. Gordon, D. J. Salmond, and A. F. M. Smith, Novel approach to nonlinear/non-Gaussian Bayesian state estimation, IEE Proceedings on Radar and Signal Processing, 140, 1993
- M. S. Arulampalam, S. Maskell, N. Gordon, and T. Clapp, A tutorial on particle filters for online nonlinear/non-gaussian Bayesian tracking, IEEE Transactions on Signal Processing, 50 (2) 2002.
- Matt Rosencrantz, Geoffrey Gordon, Sebastian Thrun, Decentralized Sensor Fusion with Distributed Particle Filters
- Carine Hue, Jean-Pierre Le Cadre, Patrick Pérez, A Particle Filter to Track Multiple Objects (PDF)
- D. Fox, J. Hightower, L. Liao, D. Schulz, and G. Borriello. Bayesian filtering for location estimation IEEE Pervasive Computing, 2(3):24–33, July-September 2003.
- A. Doucet, N. de Freitas, and N. Gordon, Eds., Sequential Monte Carlo Methods in Practice, Springer, 2001.
- B. Ristic, S. Arulampalam, N. Gordon, Beyond the Kalman Filter: Particle Filters for Tracking Applications, Artech House Publishers, 2004.