Hand-Held Augmented Reality

Panel Discussion at the fourth International Symposium on Mixed and Augmented Reaality.
Friday, Oct. 7, 2005

Panel

Tolga Capin, Nokia Research Center, USA, Panelist
Clemens Kujawski, Siemens Communications, Germany, Panelist
Fumihisa Shibata, Ritsumeikan University, Japan, Panelist
Andreas Binder, Vodafone Pilotentwicklung GmbH, Germany, Panelist
Leif Oppermann, The University of Nottingham, UK, Panelist
Dieter Schmalstieg, Graz University of Technology, Austria, Panelist
Oliver Bimber, Bauhaus-University Weimar, Germany, Panel Organizer

Abstract

More than 500 million mobile phones have been sold worldwide last year. It has been estimated that by the end of the year 2005 over fifty percent of all cell phones will be equipped with digital cameras. Today, a large variation of communication protocols allows the transfer of data between individual units, or accessing larger networks – such as the Internet. Leading graphics board vendors are about to release new chips that will enable hardware-accelerated 3D graphics on mobile phones – including geometry processing and per-pixel rendering pipelines. Some exotic devices even support auto-stereoscopic viewing, GPS navigation, or scanning of RFID tags. Due to the rapid technological advances of cell phones, the distinction between PDAs and mobile phones might be history soon. Obviously, compact hand-held devices, such as PDAs –but especially mobile phones– are becoming platforms that have the potential to bring augmented reality to a mass market. This will influence application areas, such as entertainment, edutainment, service, and many others. This 90-minutes panel aims at bringing together experienced developers and users of hand-held augmented reality systems and enabling technology, who can make clear position statements on one or more of the following questions (or on related points): What will be the role of hand-held devices for augmented reality in the year 2010? What are the technological and conceptual challenges? What are the future directions of hardware manufacturers and service providers? What will be potential applications, and who will benefit from such approaches?

Position Statements

Tolga Capin

Given the power of the CPUs and graphics engines being designed for the next-generation mobile devices, it will be possible to exploit augmented reality technologies in a few years. The most widespread processor and camera platform will be the mobile device. The mobile use cases of augmented reality will not be the same as on desktop. Users carry their mobile devices wherever they go, and the pattern of use of mobile services is different than a PC. Therefore, future successful AR services will likely exploit users' location and context. Furthermore, AR will improve user experience while interacting with mobile devices, e.g. by mapping users' physical movements to device UI. For mass market use, these solutions should be as robust as possible.
From a platform perspective, the key problems to solve for daily use of augmented reality are hardware/software architectures, battery power, and performance. An easy-to-use software infrastructure is likely the key component in platforms: high-level tools are necessary to allow non-expert developers to build AR applications. There are many technical problems to be addressed for mobile AR. These include new software/hardware architectures, robust algorithms for tracking, use of multiple sensors for input and output, use of larger or wearable displays, user interaction techniques, and visualization techniques on small displays.

Tolga Capin is a Research Manager at the Computer Graphics and Vision Group at Nokia Research Center. In the last five years, he has been overseeing Nokia's various graphics/vision research activities. He received a Ph.D. in Computer Science from Swiss Federal Institute of Technology at Lausanne (EPFL) in 1998. His research interests include mobile graphics architectures, augmented reality, virtual environments, and animation. He has authored or co-authored more than 35 scientific publications and a book, and various patents and patent applications. Tolga Capin, Research Manager, Multimedia Technologies Laboratory Nokia Research Center, Dallas, USA, Panelists

Clemens Kujawski

Within the last two decades computer science went through several hypes such as Fuzzy Logic, Neural Networks and Expert Systems. Today, we often can find the relics of these former "universal remedies" only in some niche markets. Today's hype is called Augmented Reality and it runs the risk of suffering the same fate. While today a lot of people are fond of wearing see-through displays and watching square-edged artificial objects bumping through realities plastered with markers, things might change very soon. A good approach to make Augmented Reality successful in lots of future products is to get many people accustomed to it today. As a modern mobile phone is equipped with a color screen, a strong processor and a camera which can point in the opposite direction of the screen and as many people already possess such a phone it seems to be the ideal device for spreading the idea of Augmented Reality. Of course, the processing power of even a smart-phone is low compared to a personal computer. But we can start with games like "Mozzies". Augmented Reality games condone a lot of errors in registration and presentation without driving the user crazy. So Augmented Reality games on mobile phones can smooth the way for a lot of future "serious" AR applications on mobile phones and computers in general.

Clemens Kujawski studied computer science and physics at the University of Hamburg and began his career at Siemens in 1986. At the Corporate Technology division he worked in the area of Artificial Intelligence and was involved in the development of expert system tools. After participating in the development of an autonomously driving vehicle in the context of PROMETHEUS (Programme for a European Traffic with Highest Efficiency and Unprecedented Safety) at the University of the Federal Armed Forces, Munich, he joined Siemens again in 1997 and contributed to the development of the world's first working prototype of a mobile video phone. Since 2000 Clemens Kujawski is responsible for innovative Game Technologies for mobile phones. Under his responsibility Siemens developed the "Global Pl@yer", a prototype of a cordless HMD with 3D capabilities and "Mozzies", the first series-production ready Augmented Reality game for mobile phones.

Fumihisa Shibata

Cellular phones with digital cameras are rapidly spread, especially in Japan. They will have the same performance as PDAs due to the recent technological advances. No other device seems to be an ideal candidate for handy AR/MR platform. Although current cellular phones do not have sufficient CPU power to display videos and to calculate their camera parameters, AR/MR on cellular phones will be popularly used in 2010. It is simply because it is natural to have appetites to merge electronic data into the real world anytime and anywhere. A lot more R&D should be done for such a future use. In those days, state-of-the-art systems for indoor stationary use and wearable systems with high-performance laptop computers will certainly advance in AR/MR field. If so, we should establish a general framework that is independent to mobile platforms and applications, so that applications for high-performance systems might be easily infiltrated into low performance systems. Actually we have been developing such a framework. From this experience, we demand the following two issues. First, to the mobile platform manufacturers; Assuming that the AR/MR functions become popular, the API and SDK should be designed and provided. For example, fast matrix computing libraries and/or compact database system are expected for AR/MR use. Second, to the researchers of the tracking mechanism; A pressing need is to develop a tracking mechanism for outdoor use. This can be attained independent to our general framework. Therefore, if a compact and stable method for cellular phones and PDAs is invented, it is inevitable that it spreads rapidly. I personally prospect that a hybrid method of inertia sensor and vision-based algorithm is promising.

Fumihisa Shibata is an associate professor in the College of Information Science and Engineering at the Ritsumeikan University, Japan. His research interests include mobile computing, mixed reality, computer vision and computer-human interaction. My current research topic is design of general framework for mixed reality systems applicable to various kinds of mobile devices and wearable computers. He received Ph.D. at the Osaka University in 1999. Shibata worked during four years at the Osaka University as a research associate in the Institute of Scientific and Industrial Research.

Leif Oppermann

Pervasive games extend gaming experiences out into the real world – be it on the city streets or in remote wilderness. Players with mobile displays move through the world. Sensors capture information about their current context, including their location, and this is used to deliver them a gaming experience that changes according to where they are, what they are doing, and even how they are feeling. The player becomes unchained from their console and experiences a game that is interwoven with the everyday world and that is potentially available at any place and anytime. Since 2000 we have developed, deployed and studied a variety of pervasive games as part of the UK’s Equator project (www.equator.ac.uk) and now the European iPerG project (www.pervasive-gaming.com), including Can You See Me Now?, Uncle Roy All Around You, I Like Frank in Adelaide, and Savannah. We will draw on these experiences to highlight key design challenges for pervasive games including approaches to dealing with uncertainty and new tools for designing, configuring and managing location based content. We will show how these challenges are being addressed in our latest designs of pervasive games for mobile phones.

Leif Oppermann is a research associate in the Mixed Reality Lab at the University of Nottingham. He studied Media Informatics at the Hochschule Harz (FH) in Wernigerode. His diploma thesis, about Interaction Surfaces in Augmented Reality, was awarded with the best of faculty prize in 2003. His current research interests lie in technologies for Mixed Reality and Pervasive Gaming and he is working towards a PhD in this area.

Dieter Schmalstieg

Handheld AR has the potential to finally make AR a mainstream technology. Smartphones and PDAs provide a platform which is cheap, ergonomically reasonable and socially acceptable. Hardware development and distribution is driven by consumer desire for multimedia applications; and given suitable hardware, AR application developers can finally focus on software and services. However, this requires open standards and interoperable hardware, which are still largely missing in the area of handheld devices. Despite these shortcomings of today, it is reasonable to assume that handheld devices will be sufficiently mature by 2010. Therefore, research of today can focus on the unique challenges posed by handheld AR, such as massive multiuser AR (now that many people own an AR device, how does collaboration work?), or meaningful location based services (now that you can really take AR anywhere, how do you turn the world into an interactive application?). Our Handheld AR research project is an important vehicle to systematically answer these questions.

Dieter Schmalstieg is full professor of Virtual Reality and Computer Graphics at Graz University of Technology, Austria, where he directs the "Studierstube" research project on augmented reality. His current research interests are augmented reality, virtual reality, 3D user interfaces, and ubiquitous computing. He received Dipl.-Ing. (1993), Dr. techn. (1997) and Habilitation (2001) degrees from Vienna University of Technology. He is author and co-author of over 90 reviewed scientific publications, member of the steering committee of the IEEE International Symposium on Mixed and Augmented Reality, and chair of the EUROGRAPHICS working group on Virtual Environments. In 2002, he received the START career award presented by the Austrian Science Fund to excellent young researchers.