School of Computer and Information Science

Size: px

Start display at page:

Download "School of Computer and Information Science"

Trevor Wade
6 years ago
Views:

School of Computer and Information Science CIS Research Placement Report Augmented Reality on the Android Mobile Platform Jan-Felix Schmakeit Date: 08/11/2009 Supervisor: Professor Bruce Thomas

1 School of Computer and Information Science CIS Research Placement Report Augmented Reality on the Android Mobile Platform Jan-Felix Schmakeit Date: 08/11/2009 Supervisor: Professor Bruce Thomas Abstract This project aims to evaluate the use of the Android platform for Augmented Reality applications. The main outcome will be an AR application for the T-Mobile G1 phone that demonstrates the use of the Android platform for AR. One problem faced by AR applications running on mobile, distributed clients is the backend in which data is stored and processed. The Hyper Place system developed at the University of Nottingham deals with hypermedia in a geospatial context and could be extended for the use of AR applications.

2 Table of Contents 1 Introduction Background Augmented Reality on the Android Platform Development Platform Sensors, Camera and GPS OpenGL Sound Vibrator Back-end Developed Application: Fairy Hunt Limitations of the G1 Phone Sensors GPS Accuracy Battery Life CPU and Memory Conclusion Bibliography...13

3 1 Introduction The aim of this project is to develop an Augmented Reality (AR) application on the Android T-Mobile G1 Phone (HTC Dream) 1 and to show its limitations and possibilities as an AR platform. Several AR applications already exist for the Android or iphone mobile platforms, such as Wikitude 2, an AR Browser for Wikipedia, or Layar 3, the Reality Browser that displays real-time digital information in an AR-environment. However, so far no standard for the communication between client devices and back-end servers, especially in a geospatial context, exists. The Hyper Place project of the University of Nottingham is currently under development and will provide such a framework for location-based hypermedia, which could be utilised for this purpose and, furthermore, could provide a unified approach to the geospatial use of media in general. The ultimate goal of this project is the integration of a mobile AR client into the Hyper Place back-end and the extension of the Hyper Place system to include AR-specific information. The following section provides some background information on the use of mobile, handheld AR systems. Section 3 shows the development process for an AR application for the Android platform; section 4 shows the application that was developed as part of this project; and finally section 5 evaluates the use of the T-Mobile G1 phone and the Android platform as a base for mobile, handheld AR systems

4 2 Background Augmented Reality systems aim to supplement the real world with virtual objects (Azuma et al. 2001) and, as outlined by Azuma et al., have overcome many technical obstacles in recent years that have enabled AR to be used for a multitude of different applications. The availability of portable computers and sensors have enabled the development of mobile applications, such as Hollerer T. et al. (1999) or even a mobile AR implementation of the game Quake, ARQuake (Thomas et al. 2000). However, all of these AR systems require specialised hardware, including special headworn glasses or displays to provide a fully immersive environment. However, more recently, mobile phones and handheld devices have been used to provide a magic lense style view(schmalstieg & Wagner 2008). Initially work was focussed on PDAs running the Windows CE platform, such as the Studierstube ES framework by Schmalstieg and Wagner (2007) and Symbian phones, such as the work by Mohring, Lessig & Bimber (2004). Due to the unavailable of sophisticated sensor and GPS technology in these handheld devices, these implementations relied on visual tracking and other implied locations, such as bluetooth, WLAN cells or RFID tags (Rashid et al. 2006). Amongst others, the Mixed Reality Laboratory (MRL) at the University of Nottingham has illustrate the use of portable, consumer-grade GPS devices and handheld computers for mixed reality applications through games such as Can you see me now? (Benford et al. 2006) or Bystander, as evaluated by Flintham et al. (2003). With the availability of cheap consumer devices that include accelerometers, a camera and GPS, such as the Apple iphone 4 or Android 5 phones, the potential for interactive, location-based AR applications that are accessible by anyone becomes imminent

5 3 Augmented Reality on the Android Platform This section describes the development process of an augmented reality application on the Android platform. 3.1 Development Platform The software development kit (SDK) for the Android platform includes debugging and maintenance tools and a plug-in for the Eclipse IDE that was used for this project. The SDK also contains a simulator which can be used to test applications before deploying them onto a real physical device. However, it was found that the simulator was comparatively slow and unreliable, especially when using advanced features, such as OpenGL rendering, access to the built-in camera and GPS. Therefore, a T-Mobile G1 phone was used throughout the development for testing and debugging, due to the nature of the application. 3.2 Sensors, Camera and GPS The Android platform caters for a number of external sensors, such as pressure, light, proximity and temperature sensors, however, the Android-based phones that are currently available only have a magnetic orientation and accelerometer sensors. These sensors are periodically read through a SensorManager that provides access to the device's sensors. During development it was found that the delay between readings has an impact on the performance of the application and the overall battery life of the device. (refer to section 5.) It is not possible to gain direct access to the camera on the Android platform. Therefore the camera was independently embedded into the application as a preview View in the

6 background, behind the OpenGL rendering. This was achieved by utilising a FrameLayout that allowed the layering of different views. (Refer to section XX) The built-in GPS device is accessed through a LocationManager, which can be configured to pause between GPS updates to conserve battery power. However, this functionality was disabled to provide the most recent, up-to-date GPS location readings to the application, which is critical for an AR application. 3.3 OpenGL The Android platform provides full-support for OpenGL ES 1.0, and a limited and incomplete support for OpenGL ES 1.1. The object loader available from 6, as published under the LGPL, was modified and updated to work on the Android SDK 1.6 and was incorporated into the custom GLSurfaceView.Renderer that was written for this application. 3.4 Sound Sound can be provided through two means, a SoundPool can be used to handle short sound effects, for example for games, whereas a MediaPlayer offers playback support for longer audio files. A vast number of audio formats are supported on the Android platform, however, for ease of use, the sound used in this application was encoded as a 128kbit MP3-file. Due to the length of the file, a MediaPlayer instance was created to handle audio playback. 3.5 Vibrator The SDK provides direct access to the built-in vibrator, which can be activated for a specified time or a repeating pattern. 6

7 3.6 Back-end Due to time constraints it was not possible to implement the Hyper Place system as a backend for this application. However, the application was structured in a way that allows for hyper place support to be added in the future. 4 Developed Application: Fairy Hunt The FairyHunt application was developed as part of this project. 3D Models are placed at specified locations on the UniSA Mawson Lakes campus and the user has to find these objects through the use of AR on the T-Mobile G1 phone. Once the user is within a certain range (60 m), music starts and increases in volume the closer the user is to an object. Once the user is within 7.5m of the object, it is marked as found and the user can proceed to find the next object. The overall goal is to find all objects on the campus. It is to note that this application serves as a proof-of-concept to show the use of AR on the Android platform, rather than a fully usable game. Standard user interface elements, such as an option menu, pop-up notifications and alert boxes that are natively available on this platform were used It proved to be difficult to locate freely available 3D models of fairies in the correct format. The model of a warrior 7 was included instead to show the possibility of loading a 3D-object from a file into the OpenGL ES environment on the G1 phone. The following screenshots show various features of the application: Visual Artifacts are only visible in these screenshots and were caused by the screenshot technique used. 7 The model was made by Antonio Della Rocca and is freely available from

Illustration 1: Start Screen showing instructions (sample text)

8 Illustration 1: Start Screen showing instructions (sample text) Illustration 2: The default User Interface, showing the closest fairy and the total number of fairies collected. Yellow indicates that the user is within the range of this object and that the music has started to play. Illustration 4: Selection menu showing all objects and the distances from them Illustration 3: Detailed Object View, showing information about sensors and location details.

9 Illustration 6: Model outside the D-Building Illustration 5: Notification message when an object has been found. Illustration 7: User Interface: Menu Illustration 8: Model outside the library Illustration 9: The GPS location can be manually overridden for debugging purposes. Illustration 10: The "Petcube" that follows the user and is always located directly underneath the device. It was used for debugging OpenGL.

10 5 Limitations of the G1 Phone During the development, and subsequent evaluation and testing of the FairyHunt AR application, several issues with the G1 Phone and the Android platform became apparent. Most problems were caused by hardware issues, the Android platform itself proved to be very accessible and stable. 5.1 Sensors Dedicated AR systems rely on fine and very accurate accelerometers and sensors, for example, the ARQuake application, for example, relied on a dedicated, high precision compass 8 (Thomas et al. 2000). A dedicated compass is more robust and provides more accurate readings than the built-in compass of the G1 mobile phone. During testing it was found that it is very sensitive to electromagnetic interference and close metallic objects. The accelerometers that were accessed to determine the pitch and roll of the device were simiarly noisy and caused inaccurate readings that resulted in uncontrollable, shaky object movements on the screen, even when the phone was not physically moved in any direction. Sophisticated algorithms exist to mitigate these inaccurate readings, such as the stabilisation and motion tracking techniques proposed by Azuma et al. (1999) or the visual detection of objects and the horizon. However, only a very simplistic sensor smoothing technique was added to the system to illustrate the improvements possible with a more sophisticated smoothing technique. The last 5 readings of the pitch sensor are average and passed to the system, where as the last four compass readings are averaged, with minimal updates (less than 3 degrees in either 8 Precision Navigation TCM2-80

11 direction) are ignored. This showed a significant improvement over the initial, raw sensor readings when deployed into the system, but did not solve this issue completely. 5.2 GPS Accuracy Similarly to the sensors, the GPS chip used in the G1 phone was significantly less accurate than military-grade GPS devices that provide coordinates with high accuracy that can be even further improved through the use of specialised DGPS receivers, as used by Thomas et al. (2000). During tests, GPS accuracy as reported by the device varied between 2 6m, with considerably lower (16m) accurate readings during cloudy weather. High accuracy is required for AR systems, especially when objects are to be placed in absolute positions in the environment. Additional techniques, such as cell phone tower triangulation, could have been used to possibly improve the accuracy of the location readings. Sensors could also be used to calculate the current movement, and predict the position of the device. 5.3 Battery Life It became evident that the constant access to the GPS receiver and the sensors, combined with OpenGL rendering on the screen resulted in significant power usage. Power-saving techniques, such as delaying sensor readings, or accessing sensors less frequently, could be introduced to improve the battery life. Further testing needs to be done in this area to determine the best trade-off between accurate readings and power savings. However, it is assumed that future devices will contain a larger battery and more power-efficient processors that will also improve on this.

12 5.4 CPU and Memory Mobile platforms only have a limited amount of memory and processing resources, which means that intensive applications need to be specifically optimised for the platform. It was found during the development that the OpenGL model loader was quite inefficient and often caused the application to crash due to insufficient memory. It would be advised to re-write the model loader and optimise it for the latest available Android SDK. Although special programming techniques were used for this platform, such as the minimisation of large data structures and the creation of objects, slow performance was observed when interacting with the application through OpenGL, which was primarily caused by the model loader. 6 Conclusion Augmented Reality applications have seen a vast improvement in recent years. This project showed the ease with which AR applications for inexpensive consumer-grade devices, such as Android phones, can be written. Tests showed that the orientation and location sensors are reasonably accurate, but need further refinement to improve to acceptable levels. The application that was produced as part of this project is fully functional and shows the possibilities which AR applications have on these handheld devices. Today, AR-based tube maps 9, real estate advertisements 10 or even an ARenhanced traffic camera viewer 11 exist that only scratch the surface of what can be achieved when AR technologies are combined with the ubiquitous connectivity of widely available smartphones

13 7 Bibliography Azuma, R, Baillot, Y, Behringer, R, Feiner, S et al. 2001, Recent Advances in Augmented Reality, IEEE Computer Graphics and Applications, vol. 21, no. 6, pp Azuma, R, Hoff, B, Iii, HN & Sarfaty, R 1999, A Motion-Stabilized Outdoor Augmented Reality System, in Virtual Reality Conference, IEEE, vol. 0,, IEEE Computer Society, Los Alamitos, CA, USA, p Benford, S, Crabtree, A, Flintham, M, Drozd, A et al. 2006, Can you see me now?, ACM Trans. Comput.-Hum. Interact., vol. 13, no. 1, pp Flintham, M, Benford, S, Anastasi, R, Hemmings, T et al. 2003, Where on-line meets on the streets: experiences with mobile mixed reality games, in Proceedings of the SIGCHI conference on Human factors in computing systems, ACM, Ft. Lauderdale, Florida, USA, pp Hollerer T., Feiner S., Terauchi T., Rashid G. & Hallaway D. 1999, Exploring MARS: developing indoor and outdoor user interfaces to a mobile augmented reality system, Computers and Graphics, vol. 23, pp Mohring, M, Lessig, C & Bimber, O 2004, Video See-Through AR on Consumer Cell- Phones, in Proceedings of the 3rd IEEE/ACM International Symposium on Mixed and Augmented Reality, IEEE Computer Society, pp Rashid, O, Mullins, I, Coulton, P & Edwards, R 2006, Extending cyberspace: location based games using cellular phones, Comput. Entertain., vol. 4, no. 1, p. 4. Schmalstieg, D & Wagner, D 2007, Experiences with Handheld Augmented Reality, in Proceedings of the th IEEE and ACM International Symposium on Mixed and Augmented Reality, IEEE Computer Society, pp Schmalstieg, D & Wagner, D 2008, Mobile Phones as a Platform for Augmented Reality, in Proceedings of the IEEE VR 2008 Workshop on Software Engineering and Architectures for Realtime Interactive Systems, Shaker Publishing, Reno, NV, USA, pp Thomas, B, Close, B, Donoghue, J, Squires, J et al. 2000, ARQuake: An Outdoor/Indoor Augmented Reality First Person Application, in Proceedings of the 4th IEEE International Symposium on Wearable Computers, IEEE Computer Society, p. 139.

Roadblocks for building mobile AR apps

Roadblocks for building mobile AR apps Jens de Smit, Layar (jens@layar.com) Ronald van der Lingen, Layar (ronald@layar.com) Abstract At Layar we have been developing our reality browser since 2009. Our