Chair for Computer Aided Medical Procedures & Augmented Reality
Lehrstuhl für Informatikanwendungen in der Medizin & Augmented Reality

Bachelor's Thesis: Augmented Virtual Space: Interactions in a Multi-Device Virtual Environment


Within the last few years video games have become increasingly more involved in our everyday life. With the goal in mind to create even more immersive virtual realities, new hardware has become available in order to achieve higher levels of quality in the field of computer graphics and visualization, as well as in the field of user interaction and input devices. Head-mounted displays like the Oculus Rift have become available that allow its users to experience a higher level of immersion by offering a much greater field of view and lens distortion. Additionally depth sensoring and body tracking have led to a completely new interaction experience in video games that feels more natural. This thesis shows the implementation of a card game as an example of how this hardware can be used for natural interaction in an augmented virtual space using Oculus Rift, Leap Motion and a tablet PC. It describes how limitations of a single device can be overcome by their combination in a multi-device setup and how they interact with each other.


With a lot of new hardware for Human Computer Interaction having been developed over the last 40 years, new ways of interactions have become possible in the area of augmented and virtual reality. Especially in the area of body tracking many things have changed, since new and smaller devices like the Kinect or the Leap greatly facilitate body tracking. They are relatively cheap compared to other devices and do not require extra markers or a complex camera setup for tracking. These devices allow the setup of new virtual realities that are suitable for home use, using interaction techniques performed in mid-air, by tracking the hands or other parts of the human body. Since this kind of Human Computer Interaction is something new to most people and not as common as interaction in WIMP (Windows, Icons, Menus, Pointer) based 2D user interfaces, new interaction techniques have to be found for interactions in a 3D environment. Therefore, a multi device system will be presented that enables its user to play cards in an augmented virtual space using natural interaction techniques.

Use of a Multi-Device Setup

Since there are different kinds of interactions available in a virtual card game and not all input devices provide the feeling of a virtual reality for every interaction, it makes sense to use different input devices for these different kinds of interactions. So a multi-device setup is used to ensure that all interactions work in their desired way and provide a natural feeling. This setup includes the Oculus Rift, the Leap Motion controller and a tablet PC.


Oculus Rift:

For the visualization of the game world, the Oculus Rift (DK2) is used. It is a head-mounted display (HMD) developed by Oculus VR and is used to provide a strong feeling of immersion within virtual environments. Therefore the Oculus Rift uses a 75 Hz display with a resolution of 960x1080 per eye and two lenses, that distort the image, in order to create a realistic 3D effect and a huge field of view. It is also fitted with a gyroscope, an accelerometer and a magnometer for internal headtracking. This enables the user to look around in the virtual reality the same way as in the real world. To detect even fast head movements it updates these sensors with a rate of 1000 Hz. For additional position tracking, the Oculus Rift is delivered together with an external camera that has to be placed in front of the HMD, using an infrared sensor to measure the position of the HMD.

Leap Motion Controller:

For all interactions with virtual cards (hand) in mid-air, the Leap Motion controller is used. It is a small controller designed to create "intuitive" 3D interfaces, in which users can interact using the motion of their own hands. This is achieved by the controller being able to track the hands and fingers of the user in a hemispherical area in front of the controller. Therefore it uses two CCD cameras combined with three infrared emitters. Using this technology for card interactions within the virtual space makes it possible for the user to control his hand much in the same way as in real card games. To achieve a stronger feeling of immersion, the Leap Motion controller is attached to Oculus Rift using the official VR Developer mount. This has the advantage that whenever the hands of the user would become visible in his field of view, they are also within the Leap’s tracking area and can be replaced by animated virtual hand models in the virtual reality that behave in the same way his real hands behave. This makes interactions with the virtual cards displayed in his virtual hands a lot more intuitive.


Since real card games also require some placing or picking up gestures to be performed on a table, a tablet with a touch screen is used to handle all table-based interactions. This tablet is represented by a highlighted interactive 2D area on the surface of a virtual table, visualized in the virtual environment. To guarantee an interaction area that is large enough to play, at least an 18" tablet is required. Otherwise it will hardly be possible to interact with the cards lying on the table, due to space limitations and the size of the cards. Another issue might be that performing gestures on smaller displays will become more difficult, since a higher level of precision will be required to touch cards properly. Moreover, it is used to provide touch input and passive haptic feedback for better user experience and a greater feeling of immersion. Although the tablet provides no active haptic feedback, it can detect the user’s touch positions that are used to perform the detection of gestures required for interactions with cards lying on the virtual table surface. It additionally provides kinesthetic feedback, since direct manipulation can be used to change the position of virtual cards. When interacting with these cards, this highly improves the user’s experience because he is able to sense the position of the tablet and the cards by proprioception. Nevertheless, it also compensates some shortcomings of the Leap Motion controller. The first one is probably caused by its hardware architecture. The Leap is not able to detect and track the hand or fingers when they are close to other objects or the table surface. To still be able to interact with cards then would require a setup where no haptic feedback is available, but this would result in a lack of user experience. The other shortcoming that is compensated by the tablet is the low accuracy, caused by occlusion of some fingers when the user has to perform complex gestures like picking up cards from the table surface. There, the Leap would have to predict the position and orientation of the occluded fingers by the use of its internal tracking model. Since performing these gestures requires a high level of precision, the touchscreen of the tablet is the better choice for the task, as it provides precise finger positions.

Card Game Interactions using Multiple Devices

To be able to play the virtual card game and interact with the virtual cards in the same way as in a real card game, many of the typical card game actions have to be implemented. This includes the actions that are being executed on a table, but also the mid-air interactions affecting the player’s hand. Therefore a system was developed to be able to deal with both, the detection of touch gestures the player needs to perform on the tablet, as well as collision-based interactions using the virtual hands that are updated by the Leap.

Interactions using the Tablet:

In nearly every card game, there are some basic game play mechanics that involve interactions performed on a table. One very simple example for such an interaction is playing a card. But often it is also necessary to move or pick up cards. All these interactions have in common that they are mainly touch-based. So the player is either directly touching the table, a card or a card touches the table. This can be utilized by using the tablet and its touchscreen to detect these kinds of interactions by their gestures. To provide an experience that feels as natural as playing with real cards, a framework for the detection of these natural gestures was used. This framework was developed by Andreas Dippon to reveal a more intuitive alternative to conventional interaction techniques. This enables the system to detect gestures that are really close to their natural equivalent in a real card game and helps the player to perceive the virtual environment in a much more realistic way. The framework provides the implementation for the detection of the main gestures used in card games. These are: to "place", "pick up"," lift" or "move" a card. So a virtual card can be placed on the in-game table by simply touching the screen at the desired target location. By dragging a card, it can additionally be moved around. To pick up a card, the user can perform either the "pick up" or the "lift" gesture. While the "pick up" gesture can be performed by touching the card with two fingers and then moving both fingertips together, the "lift" gesture requires only one finger to touch the card before the fingertips have to be moved together.


But this raises the question of how the player is able to interact or even find the tablet in the real world with only seeing the virtual world around him. For this reason, the whole system has to be well calibrated to exactly match the proportions of the real world in the virtual environment. So the virtual table and a highlighted interaction area illustrating the tablet should be visible at exactly the same location where they are positioned in reality with respect to the user’s body proportions. Since the player’s hands are tracked by the Leap, the positions of the virtual hands provide additional visual feedback, which allows the user to estimate the distance of the tablet using proprioception. To match the player’s real right hand outside the virtual reality, the position of the right virtual hand is changed to the touch position of the tablet in an interacting posture slightly above the virtual table. By doing this, the player is always able to see where he actually touches the tablet, even if the calibration is very inaccurate. This could be confirmed by a small group of people, that were testing the system with a wrong calibration. Another interesting result was that the people, regardless of the wrong calibration, even managed to precisely interact with the tablet, since they were able to remember its position relative to their body.


Interactions using the Leap:

When interacting with cards being held in the hand, there are three typical interactions that exist in almost every card game. The player can choose to play a card, add a card to his hand or sort the cards in his hand according to their properties. While the first one only requires the player to be able to grab a certain card with his other hand, the other two also demand the possibility to insert a grabbed card at a specific position. Since the Leap can track the user’s hands in mid-air, we tried to implement these gestures in a way that is as close to reality as possible to offer the player the same experience as playing with real cards.


Collisions are used to detect if the player is adding a card to his hand. By finding the index of the nearest card and determining whether the card being added is left or right of it, it is even possible to insert the card at a specific location which is required when sorting the hand. When a new card is inserted, the position and orientation of all the cards in the left hand is instantly being updated according to their new index.



By combining different input devices in a multi device setup many of the limitations each device brings along can be overcome. This is shown by the development of an augmented virtuality application, which combines the Oculus Rift, the Leap and a tablet to afford an immersive experience playing a virtual card game. Therefore, each of the devices is used in a different context and is completely controllable by using only natural interaction techniques as user input. The Oclulus Rift is mainly used for the visualization of the game world but, due to the built-in headtracking, also allows the user to look around like in reality. To actually be able to play, virtual hand models holding the player’s cards are shown within the user’s field of view. Due to the Leap's hand tracking capability, these models behave like the user’s real hands and can be used to interact with the virtual cards in mid-air. For all interactions affecting the cards lying on the virtual table, the touchscreen of the tablet provides a convenient way to detect the natural touch gestures performed by the player.

When implementing this system, many issues and bugs appeared, which greatly affected the overall game play experience of the user. While some of them were the result of general hardware limitations and were caused by insufficient or inaccurate tracking data, others were induced by the device communication and a result of slow transmission of the input data. Dealing with these issues during the implementation of the system not only led to new approaches that show how most of these issues could be overcome, but also provided valuable information about the Leap’s current state of development. So it was basically possible to establish an augmented virtuality which is controllable by using only natural gestures for interactions that are really close to the gestures performed in real card games. However, there were still many adjustments and restrictions necessary in order to be able to guarantee an experience that is enjoyable for its users.

Edit | Attach | Refresh | Diffs | More | Revision r1.3 - 28 Apr 2015 - 13:57 - MaxWandinger