Jurassic pARk - Paleontological Augmented Reality Museum

Abstract

In my bachelor thesis, I created an app in cooperation with the paleontological museum in Munich to make a tour through the museum more interactive.
Markers were used throughout the museum.
If you find one you can see a 3D model with your android phone and move it around.
Your phone will remember the pieces you found and add them to your augmented reality museum, so you can show your family and friends what you saw on your visit to the museum.
The app will have a playful character with mini-games like puzzles and a scavenger hunt to unlock all the different puzzle pieces.

Creating the app

The first question you should ask yourself if you want to create an AR application is what platform you want to work with.
There are examples of other museums around the world which are using specially manufactured hardware like a hand-held, lens-shaped mobile computer.
The Louvre in Paris is using an even more complex system which consists of a handheld guidance computer and a stationary computer.
Both have to be connected via a cable and thus are not mobile at all.
In this approach, every single exhibit you want to include in your guide has to have his own computer which is highly impractical and unaffordable for smaller museums like the Paleontological Museum Munich I am working with.
Therefore, I decided to use simple smartphones with Android OS. The advantage is, that a lot of people have a smartphone and most of those are using Android OS as their operating systems.
On the other side, if you do not own such a smartphone, you are not able to use the app.
Nonetheless, this approach is far more inexpensive and opens another feature field which was not possible with a device you have to rent and bring back to the museum.
The second question you should ask yourself is what engine you want to use to run your app on the selected platform.

Unity Engine

Unity is one of the best choices if you want an inexpensive solution.
The alternative would be to use the Android SDK and program the app from scratch, but if you want to show some 3D models in your app, I would recommend using Unity since it gives you all the features you need to do so much faster and easier.
Moreover, Unity is taught a lot at TUM and I used it a lot in university and in my spare time.

Marker tracking

For marker tracking, I am using Vuforia because it has Unity support and is as free for non-commercial use as Unity is.
They provide a very good library you can easily include in your Unity project.
To get your own markers you have to upload some images to their servers and they will give you back a small package with all the marker information to include in your Unity project.

Model Creation

Since I decided to include 3D models of exhibition pieces in my application I need a fast and simple way to model complex meshes of dinosaur skeletons.
For that, I used the free Autodesk 123D service where you can basically upload a bunch of photos of the object you want and get a 3D mesh of it.
So the first thing you need is a pretty good camera. I am not a photographer and thus do not own a very good camera, so I had to borrow a camera from a friend.
Second thing is that you should be able to move freely around the object you want to photograph because you will need up to 70 photos from different angles.
Around 30 photos from different angles on the same height as the object and 10-15 photos slightly above worked best for me.
If the exhibit is very large or very complex you might want to split it up into several parts and stitch it together later.
Furthermore, if the lighting is very bad or you do not choose a small enough angle to take your photos in, you will have to stitch photos together manually, which means you have to look for at least 3 similarities on every photo and despite the effort get a bad result.
If you download your model from the Autodesk website you will notice that not only your exhibit but the whole room it stood in, is now part of your 3D mesh.
So you will have to edit your model in some kind of 3D model software like Blender.
By using the provided cutting features of Blender, it is rather easy to remove additional vertices. The trickier part is to remove the ground, the exhibit is standing on.
Since it is not guaranteed that the ground is a flat and simple plane, you have to carefully cut away every single piece of ground around the area the exhibit is standing on.
In the case of the Paleontological Museum Munich, almost every bigger exhibit is standing on some kind of wavy gravel ground, which made it pretty time consuming to remove.
Other small tricks like mirroring 3D models are very helpful for completing otherwise incomplete models.
The triceratops head is photographed from around 180 degrees on one side, which does not result in a complete model. By a clean cut in the middle of the skull and the mirroring function of Blender, it is possible to complete the head without getting all the photos, which was not possible at that time.
To increase the application's performance, it is advisable to drastically reduce the amount of vertices.
To do that, we can reduce the amount by using the decimate function.
Blender will generate a similar looking model from the old 3D data with fewer vertices.
Every model in the application only has about 5\%-10\% of the original vertices, which greatly reduces the amount of storage space and rendering power the application needs on a smartphone.
Blender is a very powerful tool to edit meshes and it is free to use, but it is very complex and almost always an overkill of functionality.
As a highlight, I used Blender to animate one of the more important models.

Museum Mode

For the Museummode, there will be distributed 4 different markers around the corresponding models they reflect.
The camera will be active at all times using this mode and fill out the background.
If you point your camera at the marker you will get a 3D model version of the exhibit you can rotate and re-size through touch commands. The phone will remember every model you found and store it in its own database.
On top of that, you can blend in some extra information about the exhibit by pressing the information button.
At first, the information was displayed in the 3D space beneath the actual model. We decided that this method of information sharing was not very practical since the user had to keep pointing the camera at the marker to keep reading.
The new approach takes care of that problem, by moving the information text into the 2D GUI space, to ease the reading process.
Furthermore, the camera does not have to eye the marker to show the information. That way, the user can first look at the model and on the way to the next station, he can get additional information about the model.
Moreover, there is some kind of scavenger hunt mini-game you can play. On the top of the screen will be a clue which indicates the information board the user has to look for.
The boards are distributed in a way that the user will have seen the whole museum if they are finished with the scavenger hunt.
Since the museum has three different floors, there are two boards on each floor you have to hunt down.
If the user is able to find the correct board he can point at it with his camera and the application will recognize it almost instantly.
After that, the user will have to answer a question that is related to the information he can obtain by reading the board carefully.
A keyboard can be opened by pressing the small input field on the bottom. The user has to type in the correct answer to get to the next step in the scavenger hunt.
For every board the user discovers, he unlocks one of six puzzle pieces for the puzzle mode.
If the scavenger hunt is completed, the phone will activate all the content of the application even if you restart.

Home Mode

For the Homemode, the visitors will be given a printed version of the home marker. Every exhibit marker the visitor found on his tour through the museum will be saved and stored in the database.
Now the visitor can scroll through his collection of fossils he obtained at the museum and show it to friends and family. That might gain the museum more visitors.
The functionality is the same as in the museum mode. The user will be able to rotate and re-size the models on his phone via touchscreen, as well as getting the extra information about the selected exhibit.
By that, we are providing the museum visitor with something new and innovative. In a regular museum, a visitor is naturally not allowed to take home anything from the museum at all.
To compensate for that, they are taking photos or they are visiting a museum intern shop to buy some souvenirs.
Some visitors might go further and buy a book related to the topic about the museum, but not every museum has a shop.
Furthermore, certain museums forbid to take pictures, because the flash of modern cameras can damage certain exhibits like very old paintings.
Even if a museum lays out some flyers or handouts the visitor is unlikely to learn and think about the visit again.
By providing 3D models in an AR environment, we enable the visitor to almost start some kind of own collection of exhibits. The possibilities of that thought will be discussed more in the chapter 'Future work'.

Puzzle Mode

The Puzzlemode uses the home marker as well to guarantee that the user can look at and do the puzzle again at home.
If the scavenger hunt was completed there will be puzzle pieces all over the marker and you will have to puzzle together the last, biggest and most important exhibit of the museum via touchscreen.
To simplify the puzzling-process, the movement of the puzzle pieces is restricted to a 2D plane in front of the marker.
Puzzling in a 3D space without that restriction is very frustrating because the user has to regard the rotation of the pieces.
That would not be a problem at all if the user could touch the pieces with their hands, but since he has the touchscreen barrier, navigating the pieces through 3D space by rotating around the marker or rotate the marker itself is very difficult.
If you complete the puzzle, the animation starts running and you can make the exhibit walk to the right by touching the screen.
It will disappear at a certain point and re-spawn at the left of the marker. So it is possible to let the exhibit walk around. That supports the playful character of the application.
Moreover, the manual cutting of the exhibit was rather problematic, because meshes are hollow objects and mostly can not be seen from the inside, which restricted the cutting options a lot.
The torso of the model is the anchor point to make some kind of puzzle snapping mechanism possible. Every other piece can be freely moved.
If a puzzle piece gets very close to his correct position it snaps into place to make the puzzle more enjoyable and to give some feedback about the progress.

Future Work

Considering the general idea of the application, there are a lot of things that could not be done because of the limited time of a bachelor's thesis and a lot of things that could not be done because there was only one person to create the application.
Since 3D printers are getting more and more affordable, the application could provide the user with the 3D exhibits. It would be an easy task to upload them to a website, to enable the user to 3D print the models they saw at the museum.
Furthermore, there could be a feature to take photos within the application to encourage the user to use the application.
With a bigger team, the number of exhibits in the application could be drastically increased. Almost every exhibit in the Paleontological Museum Munich is convertible with the cheap methods that were used to get all the other models.
In addition to that, the completion of damages exhibits is very easy in an AR environment.
If an exhibit is missing some crucial parts to stand in the museum, it is very easy to digitally add the missing parts in a 3D space to complete it.
Through mirroring and artistic completion, almost every exhibit could be completed.
Original fur and skin could be added to the models to enhance the museum experience and increase the degree of realism of animated exhibits.
In closer collaboration, the quality of information provided by the AR tour could be increased. Additionally, a better tour route could be worked out to optimize the information content.
For models that are more accurate and professional, other techniques like laser scanning could be used to provide the user with better 3D printing material.
Moreover, the application could be used to build some kind of network between different museums in bigger cities or even countries so the user can collect exhibits from every museum he likes to expand his own collection in his own virtual museum.
Even a connection to big social networks like Facebook or Google+ could be implemented to show off his collection.
On top of that, we could provide museums with some kind of sandbox editor for AR applications with guides on how to scan exhibits and creating own AR tours, which would help to grow the number of compatible museums by a lot.

Conclusion

In conclusion, it is safe to say that an AR guide is no science fiction anymore.
As described in the sections above, it is fairly easy and inexpensive to create an AR guide due to free software like the Unity engine and Autodesk's free photogrammetry service.
v The only two things you need to create fairly good 3D models of your exhibits are an above average camera and a computer for editing.
Even very small teams or single persons can create an application that makes the visitor feel positively influenced and sparks interest for the new technology.
It can potentially add content to the museum by reconstructing and enhancing already existing exhibits, and it can be done by almost every type of museum.
Furthermore, the museum does not have to take huge expenses due to new powerful smartphones that are capable of rendering complex exhibits onto the camera screen.
Perhaps augmented reality is the solution to the slightly shrinking interest in museums and is able to make a visit to a museum more appealing for a bigger audience again.