Chapter 1 Augmented Reality: An Overview

Transcription

1 Chapter 1 Augmented Reality: An Overview Julie Carmigniani and Borko Furht 1 Introduction We define Augmented Reality (AR) as a real-time direct or indirect view of a physical real-world environment that has been enhanced/augmented by adding virtual computer-generated information to it [1]. AR is both interactive and registered in 3D as well as combines real and virtual objects. Milgram s Reality-Virtuality Continuum is defined by Paul Milgram and Fumio Kishino as a continuum that spans between the real environment and the virtual environment comprise Augmented Reality and Augmented Virtuality (AV) in between, where AR is closer to the real world and AV is closer to a pure virtual environment, as seen in Fig. 1.1 [2]. Augmented Reality aims at simplifying the user s life by bringing virtual information not only to his immediate surroundings, but also to any indirect view of the real-world environment, such as live-video stream. AR enhances the user s perception of and interaction with the real world. While Virtual Reality (VR) technology or Virtual Environment as called by Milgram, completely immerses users in a synthetic world without seeing the real world, AR technology augments the sense of reality by superimposing virtual objects and cues upon the real world in real time. Note that, as Azuma et al. [3], we do not consider AR to be restricted to a particular type of display technologies such as head-mounted display (HMD), nor do we consider it to be limited to the sense of sight. AR can potentially apply to all senses, augmenting smell, touch and hearing as well. AR can also be used to augment or substitute users missing senses by sensory substitution, such as augmenting the sight of blind users or users with poor vision by the use of audio cues, or augmenting hearing for deaf users by the use of visual cues. J. Carmigniani ( ) Department of Computer and Electrical Engineering and Computer Sciences, Florida Atlantic University, Boca Raton, Florida, USA jcarmign@fau.edu B. Furht (ed.), Handbook of Augmented Reality, DOI / , Springer Science+Business Media, LLC

2 4 J. Carmigniani and B. Furht Fig. 1.1 Milgram s reality-virtuality continuum [1] Azuma et al. [3] also considered AR applications that require removing real objects from the environment, which are more commonly called mediated or diminished reality, in addition to adding virtual objects. Indeed, removing objects from the real world corresponds to covering the object with virtual information that matches the background in order to give the user the impression that the object is not there. Virtual objects added to the real environment show information to the user that the user cannot directly detect with his senses. The information passed on by the virtual object can help the user in performing daily-tasks work, such as guiding workers through electrical wires in an aircraft by displaying digital information through a headset. The information can also simply have an entertainment purpose, such as Wikitude or other mobile augmented reality. There are many other classes of AR applications, such as medical visualization, entertainment, advertising, maintenance and repair, annotation, robot path planning, etc. 2 History The first appearance of Augmented Reality (AR) dates back to the 1950s when Morton Heilig, a cinematographer, thought of cinema is an activity that would have the ability to draw the viewer into the onscreen activity by taking in all the senses in an effective manner. In 1962, Heilig built a prototype of his vision, which he described in 1955 in The Cinema of the Future, named Sensorama, which predated digital computing [4]. Next, Ivan Sutherland invented the head mounted display in 1966 (Fig. 1.2). In 1968, Sutherland was the first one to create an augmented reality system using an optical see-through head-mounted display [5]. In 1975, Myron Krueger creates the Videoplace, a room that allows users to interact with virtual objects for the first time. Later, Tom Caudell and David Mizell from Boeing coin the phrase Augmented Reality while helping workers assemble wires and cable for an aircraft [1]. They also started discussing the advantages of Augmented Reality versus Virtual Reality (VR), such as requiring less power since fewer pixels are needed [5]. In the same year, L.B Rosenberg developed one of the first functioning AR systems, called Virtual Fixtures and demonstrated its benefit on human performance while Steven Feiner, Blair MacIntyre and Doree Seligmann presented the first major paper on an AR system prototype named KARMA [1]. The reality virtuality continuum seen in Fig. 1.1 is not defined until 1994 by Paul Milgram and Fumio Kishino as a continuum that spans from the real environment to the virtual environment. AR and AV are located somewhere

3 1 Augmented Reality: An Overview 5 Fig. 1.2 Ivan Sutherland s HMD [5] in between with AR being closer to the real world environment and AV being closer to the virtual environment. In 1997, Ronald Azuma writes the first survey in AR providing a widely acknowledged definition of AR by identifying it as combining real and virtual environment while being both registered in 3D and interactive in real time [5]. The first outdoor mobile AR game, ARQuake, is developed by Bruce Thomas in 2000 and demonstrated during the International Symposium on Wearable Computers. In 2005, the Horizon Report [6] predicts that AR technologies will emerge more fully within the next 4 5 years; and, as to confirm that prediction, camera systems that can analyze physical environments in real time and relate positions between objects and environment are developed the same year. This type of camera system has become the basis to integrate virtual objects with reality in AR systems. In the following years, more and more AR applications are developed especially with mobile applications, such as Wikitude AR Travel Guide launched in 2008, but also with the development of medical applications in Nowadays, with the new advances in technology, an increasing amount of AR systems and applications are produced, notably with MIT 6th sense prototype and the release of the ipad 2 and its successors and competitors, notably the Eee Pad, and the iphone 4, which promises to revolutionize mobile AR.

4 6 J. Carmigniani and B. Furht 3 Augmented Reality Technologies 3.1 Computer Vision Methods in AR Computer vision renders 3D virtual objects from the same viewpoint from which the images of the real scene are being taken by tracking cameras. Augmented reality image registration uses different method of computer vision mostly related to video tracking. These methods usually consist of two stages: tracking and reconstructing/recognizing. First, fiducial markers, optical images, or interest points are detected in the camera images. Tracking can make use of feature detection, edge detection, or other image processing methods to interpret the camera images. In computer vision, most of the available tracking techniques can be separated in two classes: feature-based and model-based [7]. Feature-based methods consist of discovering the connection between 2D image features and their 3D world frame coordinates [8]. Model-based methods make use of model of the tracked objects features such as CAD models or 2D templates of the item based on distinguishable features [7]. Once a connection is made between the 2D image and 3D world frame, it is possible to find the camera pose by projecting the 3D coordinates of the feature into the observed 2D image coordinates and by minimizing the distance to their corresponding 2D features. The constraints for camera pose estimation are most often determined using point features. The reconstructing/recognizing stage uses the data obtained from the first stage to reconstruct a real world coordinate system. Assuming a calibrated camera and a perspective projection model, if a point has coordinates (x,y,z) T in the coordinate frame of the camera, its projection onto the image plane is (x/z,y/z,1) T. In point constraints, we have two principal coordinate systems, as illustrated in Fig. 1.3, the world coordinate system W and the 2D image coordinate system. Let p i (x i,y i,z i ) T,wherei = 1,...,n, with n 3, be a set of 3D non-collinear reference points in the world frame coordinate and q i (x i,y i,z i )T be the corresponding cameraspace coordinates, p i and q i are related by the following transformation: where R = q i = Rp i + T (1.1) r T 1 r T 2 r T 3 t x and T = t y (1.2) t z are a rotation matrix and a translation vector, respectively. Let the image point h i (u i,v i,1) T be the projection of p i on the normalized image plane. The collinearity equation establishing the relationship between h i and p i using the camera pinhole is given by: h i = 1 r T 3 p i + t z (Rp i + T) (1.3)

5 1 Augmented Reality: An Overview 7 Fig. 1.3 Point constraints for the camera pose problem adapted from [9] The image space error gives a relationship between 3D reference points, their corresponding 2D extracted image points, and the camera pose parameters, and corresponds to the point constraints [9]. The image space error is given as follow: ( u E p i = rt 1 p ) 2 ( i + t x i r3 T p + v i rt 2 p ) 2 i + t y i + t z r3 T p (1.4) i + t z where m i ( ui, v i,1) T are the observed image points. Some methods assume the presence of fiducial markers in the environment or object with known 3D geometry, and make use of those data. Others have the scene 3D structure pre-calculated beforehand, such as Huang et al. s device AR-View [10]; however, the device will have to be stationary and its position known. If the entire scene is not known beforehand, Simultaneous Localization And Mapping (SLAM) technique is used for mapping fiducial markers or 3D models relative positions. In the case when no assumptions about the 3D geometry of the scene can be made, Structure from Motion (SfM) method is used. SfM method can be divided into two parts: feature point tracking and camera parameter estimation. Tracking methods in AR depend mostly on the type of environment the AR device will be introduced to as well as the type of AR system. The environment might be indoor, outdoor or a combination of both. In the same way, the system might be mobile or static (have a fixed-position). For example, if the AR device is a

6 8 J. Carmigniani and B. Furht fixed-position device for an outdoor real environment, such as Huang et al. s device AR-View [10], the developers can use mechanical tracking since the movements to be tracked will all be mechanical, as the position of the device is known. This type of environment and system makes tracking of the environment for augmenting the surroundings easier. On the other hand, if the AR device is mobile and designed for an outdoor environment, tracking becomes much harder and different techniques offer some advantages and disadvantages. For example, Nilsson et al. [11] built a pedestrian detection system for automotive collision avoidance using AR. Their system is mobile and outdoor. For a camera moving in an unknown environment, the problem for computer vision is to reconstruct both the motion of the camera and the structure of the scene using the image and additional sensor data sequences. In this case, since no assumption about the 3D geometry of the scene can be made, SfM method is used for reconstructing the scene. Developers also have the choice to make use of existing AR libraries, such as the ARToolKit. ARToolKit, which was developed in 1999 by Hirokazu Kato from the Nara Institute of Science and Technology and was released by the University of Washington HIT Lab, is a computer vision tracking library that allows the user to create augmented reality applications [12]. It uses video tracking capabilities to calculate in real time the real camera position and orientation relative to physical markers. Once the real camera position is known, a virtual camera can be placed at the same exact position and 3D computer graphics model can be drawn to overlay the markers. The extended version of ARToolKit is ARToolKitPlus, which added many features over the ARToolKit, notably class-based APIs; however, it is no longer being developed and already has a successor: Studierstube Tracker. Studierstube Tracker s concepts are very similar to ARToolKitPlus; however, its code base is completely different and it is not an open source, thus not available for download. It supports mobile phone, with Studierstube ES, as well as PCs, making its memory requirements very low (100KB or 5 10% of ARToolKitPlus) and processing very fast (about twice as fast as ARToolKitPlus on mobile phones and about 1 ms per frame on a PC) [13]. Studierstube Tracker is highly modular; developers can extend it in anyway by creating new features for it. When first presenting Studierstube in [13], the designers had in mind a user interface that uses collaborative augmented reality to bridge multiple user interface dimensions: Multiple users, contexts, and locales as well as applications, 3D-windows, hosts, display platforms, and operating systems. More information about Studierstube can be found at [13 15]. Although visual tracking now has the ability to recognize and track a lot of things, it mostly relies on other techniques such as GPS and accelerometers. For example, for a computer to detect and recognize a car it is very hard. The surface of most cars is both shiny and smooth and most of the feature points come from reflections and thus are not relevant for pose estimation and even sometimes recognition [16]. The few stable features that one can hope to recognize, such as the windows corners or wheels, are extremely difficult to match due to reflection and transparent parts. While this example is a bit extreme, it shows the difficulties and challenges faced by computer vision with most objects that have irregular shape, such as food, flowers, and most objects of art.

7 1 Augmented Reality: An Overview 9 Fig. 1.4 HMD from [17] A recent new approach for advances in visual tracking has been to study how the human brain recognizes objects, also called the Human Vision System (HVS), as it is possible for humans to recognize an infinite number of objects and persons in fractions of seconds. If the way of recognizing things by the human brain can be modeled, computer vision will be able to handle the challenges it is currently facing and keep moving forward. 3.2 AR Devices The main devices for augmented reality are displays, input devices, tracking, and computers Displays There are three major types of displays used in Augmented Reality: head mounted displays (HMD), handheld displays and spatial displays. HMD is a display device worn on the head or as part of a helmet and that places both images of the real and virtual environment over the user s view of the world (Fig. 1.4). HMD can either be video-see-through or optical see-through and can have a monocular or binocular display optic. Video-see-through systems are more demanding than optical-see-through systems as they require the user to wear two cameras on his head and require the processing of both cameras to provide

8 10 J. Carmigniani and B. Furht Fig. 1.5 Handheld displays from [18] both the real part of the augmented scene and the virtual objects with unmatched resolution, while the optical-see-through employs a half-silver mirror technology to allow views of physical world to pass through the lens and graphically overlay information to be reflected in the user s eyes. The scene as well as the real world is perceived more naturally than at the resolution of the display. On the other hand, in video-see-through systems, augmented view is already composed by the computer and allows much more control over the result. Thus, control over the timing of the real scene can be achieved by synchronizing the virtual image with the scene before displaying it while in an optical-see-through application, the view of the real world cannot be delayed, so the time lag introduced in the system by the graphics and image processing is perceived by the user. This results in image that may not appear attached with the real objects they are supposed to correspond to, they appear to be unstable, jittering, or swimming around. Handheld displays employ small computing devices with a display that the user can hold in their hands (Fig. 1.5). They use video-see-through techniques to overlay graphics onto the real environment and employ sensors, such as digital compasses and GPS units for their six degree of freedom tracking sensors, fiducial marker systems, such as ARToolKit, and/or computer vision methods, such as SLAM. There are currently three distinct classes of commercially available handheld displays that are being used for augmented reality system: smart-phones, PDAs and Tablet PCs [18]. Smart-phones are extremely portable and widespread, and with the recent advances present a combination of powerful CPU, camera, accelerometer, GPS, and solid state compass, making them a very promising platform for AR. However, their small display size is less than ideal for 3D user interfaces. PDAs present much of the same advantages and disadvantages of the smart-phones, but they are becoming a lot less widespread than smart-phones since the most recent advances, with Android-based phones and iphones. Tablet PCs are a lot more powerful than

9 1 Augmented Reality: An Overview 11 Fig. 1.6 SAR from [21] smart-phones, but they are considerably more expensive and too heavy for single handed, and even prolonged two-handed, use. However, with the recent release of ipad, we believe that Tablet PCs could become a promising platform for handheld AR displays. Spatial Augmented Reality (SAR) make use of video-projectors, optical elements, holograms, radio frequency tags, and other tracking technologies to display graphical information directly onto physical objects without requiring the user to wear or carry the display (Fig. 1.6) [19]. Spatial displays separate most of the technology from the user and integrate it into the environment. This permits SAR to naturally scale up to groups of users, thus allowing collaboration between users, increasing the interest for such augmented reality systems in universities, labs, museums, and in the art community. There exist three different approaches to SAR which mainly differ in the way they augment the environment: video-see-through, optical-see-through and direct augmentation. In SAR, video-see-through displays are screen based; they are a common technique used if the system does not have to be mobile as they are cost efficient since only off-the-shelf hardware components and standard PC equipment is required. Spatial optical-see-through displays generate images that are aligned within the physical environment. Spatial optical combiners, such as planar or curved mirror beam splitters, transparent screens, or optical holograms are essential components of such displays [19]. However, much

10 12 J. Carmigniani and B. Furht like screen-based video see-through, spatial optical-see-through does not support mobile applications due to spatially aligned optics and display technology. Finally, projector-based spatial displays apply front-projection to seamlessly project images directly onto physical objects surfaces, such as in [20]. More details about SAR can be read in [19]. Table1.1 shows a comparison of different types of displays techniques for augmented reality Input Devices There are many types of input devices for AR systems. Some systems, such as Reitmayr et al. s mobile augmented system [17] utilizes gloves. Others, such as ReachMedia [22] use a wireless wristband. In the case of smart-phones, the phone itself can be used as a pointing device; for example, Google Sky Map on Android phone requires the user to point his/her phone in the direction of the stars or planets s/he wishes to know the name of. The input devices chosen depend greatly upon the type of application the system is being developed for and/or the display chosen. For instance, if an application requires the user to be hands free, the input device chosen will be one that enables the user to use his/her hands for the application without requiring extra unnatural gestures or to be held by the user, examples of such input devices include gaze interaction in [23] or the wireless wristband used in [22]. Similarly, if a system makes use of a handheld display, the developers can utilize a touch screen input device Tracking Tracking devices consists of digital cameras and/or other optical sensors, GPS, accelerometers, solid state compasses, wireless sensors, etc. Each of these technologies has different level of accuracy and depends greatly on the type of system being developed. In [24], the authors identified the general tracking technology for augmented reality to be: mechanical, magnetic sensing, GPS, ultrasonic, inertia, and optics. In [25], the authors use a comparison from DiVerdi [26] based on range, setup, resolution, time, and environment. We further adopted their comparison method to this survey in Table Computers AR systems require powerful CPU and considerable amount of RAM to process camera images. So far, mobile computing systems employ a laptop in a backpack configuration, but with the rise of smart-phones technology and ipad, we can hope to see this backpack configuration replaced by a lighter and more sophisticated looking system. Stationary systems can use a traditional workstation with a powerful graphics card.

11 1 Augmented Reality: An Overview 13 Table 1.1 Comparison of different techniques for different types of display Types of displays HMD Handheld Spatial Video-see- Optical-see- Video-see-through Video-see- Optical-see- Direct Techniques through through Types of displays HMD Handheld through through augmentation Advantages complete visualization control, possible synchronization of the virtual and real environment Disadvantages requires user to wear cameras on his/her head, require processing of cameras video stream, unnatural perception of the real environment employs a half-silver mirror technology, more natural perception of the real environment time lag, jittering of the virtual image portable, widespread, powerful CPU, camera, accelerometer, GPS, and solid state compass portable, powerful CPU, camera, accelerometer, GPS, and solid state compass small display becoming less widespread, small display more powerful cost efficient, can be adopted using offthe-shelf hardware components and standard PC equipment more expensive and heavy does not support mobile system more natural perception of the real environment does not support mobile system displays directly onto physical objects surfaces not user dependent: everybody sees the same thing (in some cases this disadvantage can also be considered to be an advantage)

12 14 J. Carmigniani and B. Furht Table 1.2 Comparison of common tracking technologies (adapted from Papagiannakis et al. [25] and DiVerdi et al. [26]). Range: size of the region that can be tracked within. Setup: amount of time for instrumentation and calibration. Precision: granularity of a single output position. Time: duration for which useful tracking data is returned (before it drifts too much). Environment: where the tracker can be used, indoors or outdoors Range Setup Precision Technology (m) time (hr) (mm) Time (s) Environment Optical: in/out marker-based Optical: in/out markerless Optical: in outside-in Optical: in/out inside-out GPS 0 5,000 out WiFi ,000 in/out Accelerometer 1, in/out Magnetic in/out Ultrasound in Inertial in/out Hybrid in/out UWB in RFID: active when needed 500 in/out RFID: passive when needed 500 in/out 3.3 AR Interfaces One of the most important aspects of augmented reality is to create appropriate techniques for intuitive interaction between the user and the virtual content of AR applications. There are four main ways of interaction in AR applications: tangible AR interfaces, collaborative AR interfaces, hybrid AR interfaces, and the emerging multimodal interfaces Tangible AR Interfaces Tangible interfaces support direct interaction with the real world by exploiting the use of real, physical objects and tools. A classical example of the power of tangible user interfaces is the VOMAR application developed by Kato et al. [27], which enables a person to select and rearrange the furniture in an AR living room design application by using a real, physical paddle. Paddle motions are mapped to intuitive gesture based commands, such as scooping up an object to select it for movement or hitting an item to make it disappear in order to provide the user with an intuitive experience.

13 1 Augmented Reality: An Overview 15 A more recent example of a tangible AR user interface is TaPuMa [28]. TaPuMa is a table-top tangible interface that uses physical objects to interact with digital projected maps using real-life objects the user carries with him as queries to find locations or information on the map. The advantage of such an application is that using objects as keywords eliminates the language barrier of conventional graphical interfaces (although most of them do have multiple languages, they are often mistranslated). On the other hand, keywords using objects can also be ambiguous, as there can be more than one mapping to actions or information possible, and different people from different places, age-group, and culture have different meanings for different objects. So although this system might seem rather simple to use, it opens the door to a main problem in user interfaces: showing the user how to utilize the real objects for interacting with the system. White et al. s [29] solution was to provide virtual visual hints on the real object showing how it should be moved. Another example of tangible AR interactions includes the use of gloves or wristband such as in [22]and[30] Collaborative AR Interfaces Collaborative AR interfaces include the use of multiple displays to support remote and co-located activities. Co-located sharing uses 3D interfaces to improve physical collaborative workspace. In remote sharing, AR is able to effortlessly integrate multiple devices with multiple locations to enhance teleconferences. An example of co-located collaboration can be seen with Studierstube [13 15]. When first presenting Studierstube in [13], the designers had in mind a user interface that uses collaborative augmented reality to bridge multiple user interface dimensions: Multiple users, contexts, and locales as well as applications, 3Dwindows, hosts, display platforms, and operating systems. Remote sharing can be used for enhancing teleconferences such as in [31]. Such interfaces can be integrated with medical applications for performing diagnostics, surgery, or even maintenance routine Hybrid AR Interfaces Hybrid interfaces combine an assortment of different, but complementary interfaces as well as the possibility to interact through a wide range of interaction devices [7]. They provide a flexible platform for unplanned, everyday interaction where it is not known in advance which type of interaction display or devices will be used. In [32], Sandor et al. developed a hybrid user interface using head-tracked, seethrough, head-worn display to overlay augmented reality and provide both visual and auditory feedbacks. Their AR system is then implemented to support end users in assigning physical interaction devices to operations as well as virtual objects on which to perform those procedures, and in reconfiguring the mappings between devices, objects and operations as the user interacts with the system.

14 16 J. Carmigniani and B. Furht Multimodal AR Interfaces Multimodal interfaces combine real objects input with naturally occurring forms of language and behaviors such as speech, touch, natural hand gestures, or gaze. These types of interfaces are more recently emerging. Examples include MIT s sixth sense [20] wearable gestural interface, called WUW. WUW brings the user with information projected onto surfaces, walls, and physical objects through natural hand gestures, arms movement, and/or interaction with the object itself. Another example of multimodal interaction is the work from Lee et al. [23], which makes use of gaze and blink to interact with objects. This type of interaction is now being largely developed and is sure to be one of the preferred type of interaction for future augmented reality application as they offer a relatively robust, efficient, expressive, and highly mobile form of human-computer interaction that represent the users preferred interaction style. They have the capability to support users ability to flexibly combine modalities or to switch from one input mode to another depending on the task or setting. In addition, multimodal interfaces offer the freedom to choose which mode of interaction the user prefers to use depending on the context; i.e. public place, museum, library, etc. This freedom to choose the mode of interaction is crucial to wider acceptance of pervasive systems in public places [75]. 3.4 AR Systems Augmented reality systems can be divided into five categories: fixed indoor systems, fixed outdoor systems, mobile indoor systems, mobile outdoor systems, and mobile indoor and outdoor systems. We define a mobile system as a system that allows the user for movement that are not constrained to one room and thus allow the user to move through the use of a wireless system. Fixed system cannot be moved around and the user must use these systems wherever they are set up without having the flexibility to move unless they are relocating the whole system setup. The choice of the type of system to be built is the first choice the developers must make as it will help the developers in deciding which type of tracking system, display choice and possibly interface they should use. For instance, fixed systems will not make use of GPS tracking, while outdoor mobile system will. In [25], the authors conducted a study of different AR systems. We conducted a similar study using 25 papers that were classified according to their type of system, and determined what the tracking techniques, display type and interfaces were for each. Tables 1.3 and 1.4 show the results of the study and Table 1.5 the meaning of the abbreviation used in Tables 1.3 and 1.4. The papers used for the study were all published between 2002 and 2010 with a majority of papers (17 papers out of 25) published between 2005 and Note that in the mobile indoor and outdoor systems, one of the system studied (Costanza s eye-q [49]) does not use any tracking techniques, while others use multiple type of tracking techniques. This is due to the fact that this system was

15 1 Augmented Reality: An Overview 17 Table 1.3 AR system comparison Application domain Tracking and registration Ent. Nav. System and and type Method Adv. ed. info. Med. Gen. Optical GPS Mechanical Sensors Wireless Display Interface Fixed indoor Fixed outdoor Mobile indoor Bichlmeier et al. [34] Luo et al. [35] Mistry et al. [28] X IO, OI, MB HMD tangible X IO HMD tangible X OI RFID spatial tangible Magic Mirror X OI, spatial tangible MB Botella et al. X IO, HMD tangible [36] MB Cooper et al. X OI, spatial shared [30] MB Takahashi X MB spatial tangible et al. [37] Akinbiyi X force HMD tangible et al. [38] Sandor et al. X IO HMD hybrid [32] Huang et al. X X spatial tangible [10] handheld tangible Miyashita et al. [39] Bruns et al. [40] X ML rotation, RFID X ML Bluetooth handheld tangible (continued)

16 18 J. Carmigniani and B. Furht Table 1.3 (continued) Application domain Tracking and registration Ent. Nav. System and and type Method Adv. ed. info. Med. Gen. Optical GPS Mechanical Sensors Wireless Display Interface Caruso et al. X MB IR LEDs HMD tangible [41] Mobile outdoor Mobile indoor and outdoor Lee et al. [23] X IO HMD multimodal Arvanitis X IO HMD tangible et al. [42] Reitmayr X MB inertial HMD tangible et al. [17] Stutzman X X inertial handheld shared et al. [43] Brown et al. X X X inertial HMD shared [44] Dahne et al. X ML X EC HMD tangible [45] Chen et al. X ML X inertial spatial tangible [46] Schneider X X X rotation HMD tangible et al. [47] Wang et al. X X X inertial handheld tangible [48] Mistry et al. X X spatial multimodal [20] X handheld multimodal Feldman et al. [22] Costanza et al. [49] X RFID, inertial X HMD tangible

17 1 Augmented Reality: An Overview 19 Table 1.4 Systems preferred type of tracking, display and interfaces Application domain Mobile indoor and Fixed indoor systems Fixed outdoor systems Mobile indoor systems Mobile outdoor systems outdoor systems Number Percentage Number Percentage Number Percentage Number Percentage Number Percentage Adv. 1 11% 0 0% 1 14% 0 0% 0 0% Ent. and ed. 1 11% 1 100% 3 43% 2 40% 0 0% Nav. and 2 22% 0 0% 1 14% 3 60% 0 0% info. Med. 4 44% 0 0% 0 0% 0 0% 0 0% Gen. 1 11% 0 0% 2 29% 0 0% 3 100% Tracking Optical 7 78% 0 0% 2 29% 0 0% 1 33% GPS 0 0% 0 0% 0 0% 0 0% 0 0% Sensor 1 11% 0 0% 0 0% 0 0% 0 0% based Wireless 0 0% 0 0% 0 0% 0 0% 0 0% Mechanical 0 0% 1 100% 0 0% 0 0% 0 0% Hybrid 1 11% 0 0% 5 71% 5 100% 1 33% Display HMD 5 56% 0 0% 4 57% 3 60% 1 33% type Handheld 0 0% 0 0% 3 43% 1 20% 1 33% Spatial 4 44% 1 100% 0 0% 1 20% 1 33% display Interface Tangible 7 78% 1 100% 5 71% 5 100% 1 33% Shared 1 11% 0 0% 1 14% 0 0% 0 0% Hybrid 1 11% 0 0% 0 0% 0 0% 0 0% Multimodal 0 0% 0 0% 1 14% 0 0% 2 67% Total number of papers: 9 36% 1 4% 7 28% 5 20% 3 12%

18 20 J. Carmigniani and B. Furht Table 1.5 Abbreviations Abbreviation Meaning Abbreviation Meaning Adv Advertising IO inside out Ent. and ed. Entertainment and education OI outside in Med. Medical MB marker-based Nav. and info. Navigational and informational ML markerless Gen. General EC electronic compass developed as a personal, subtle notification display. Also notice that there was only one paper studied that discussed fixed outdoor systems because this type of system is not popular due to its inflexibility. The results from this study thus can also be used to show the most popular type of system developed so far. However, we need to take into account the fact that mobile system are now on the rise and while mobile indoor and outdoor systems represented only 12% of the papers studied, developers are looking more and more into this type of system as they have the most chance for making it into the market. Although these results cannot be used as a general rule when building an AR system, they can serve as a pointer of what type of tracking techniques, display, or interface is more popular for each system type. Developers should also keep in mind that these choices depend also on the type of applications, although as can be seen, the application does not necessarily guide the type of system. From this study, we see that optical tracking is mostly preferred in fixed systems, while a hybrid approach is most often preferred for mobile systems. HMDs are often the preferred type of display choice; however, we predict that they will need to become more fashionably acceptable for systems using them to reach the market. When it comes to interfaces, the most popular choice is tangible interfaces, but we predict that multimodal interfaces will become more famous with developers within the next years as we believe that they also have a better chance to reach the public industry. 4 Augmented Reality Mobile Systems Augmented reality mobile systems include mobile phones applications as well as wireless systems such as MIT s sixth sense [20] andeye-q[49]. AR mobile systems involve the use of wearable mobile interfaces for the user to interact with digital information overlapped on physical objects or surfaces in a natural and socially acceptable way. Mobile phones for augmented reality present both advantages and drawbacks. Indeed, most mobile devices nowadays are equipped with cameras making mobile phones one of the most convenient platforms on which to implement augmented reality. In addition, most cell phones provide accelerometers, magnetometers and GPS from which AR can benefit. However, in spite of rapid advances in mobile phones, their computing platform for real-time

19 1 Augmented Reality: An Overview 21 imaging applications is still rather limited if done using the cell phone s platform. As a result, many applications send data to a remote computer that does the computation and sends the result back to the mobile device, but this approach is not well adapted to AR due to limited bandwidth. Nevertheless, considering the rapid development in mobile devices computing power, it can be considered feasible to develop real-time AR application locally processed in the near future. We define a successful AR mobile system as an application that enables the user to focus on the application or system rather than on the computing devices, interacts with the device in a natural and socially acceptable way, and provides the user with private information that can be shared if necessary. This suggests the need for lightweight, wearable or mobile devices that are fashionably acceptable, private and possess robust tracking technology. 4.1 Socially Acceptable Technology Many research groups have raised the problem of socially acceptable technology. Mobile systems in particular are constantly faced with social acceptance issues to go from the laboratories to the industry. For systems to be successful in the market, developers need to take into account that the device needs to be socially acceptable, natural to interact with, and fashionably acceptable Social Acceptance Mobile phones and PDAs reminders, messages, calls, etc. have been judged to be distractions and are mostly considered not to be socially acceptable as they not only disrupt the person whose phone or PDA is receiving a message or reminding its owner of more or less important information, but also the other persons present in the same room, whether they are having a conversation with the disruptor (as this is how the person whose phone is disrupting the room will be seen) or the disruptor is in a public place, such as a bus. As a result, research groups such as [22, 49], [others], have decided that interaction with Augmented Reality systems implemented in mobile applications need to be subtle, discrete and unobtrusive, so to not disrupt the user if s/he is under a high load of work and the disruption is not of priority level. A system that is subtle, discrete and unobtrusive becomes socially acceptable. Indeed, the main problem with social acceptance comes from the level of disruption portable devices create in public places and during conversations. In [49], the authors study the peripheral vision and adapt their mobile device, eyeq, so that it does not occlude the foveal field of view, which is the main focus in the human field of view. The cues become less and less visible depending on the level of concentration and work-load of the user, making it naturally adaptive to

20 22 J. Carmigniani and B. Furht users cognitive workload and stress. And since the cues are only visible to the user, they can be considered socially acceptable as they will only disrupt the user depending on his level of concentration and he can choose not to answer to the cues. In addition, Sorce et al. [33] have come up to the conclusion that multimodal interfaces are crucial to wider acceptance of pervasive systems in public places as they offer the user the freedom to choose from a range of interaction modes. As a result, users are presented with the freedom to choose the most appropriate and socially acceptable mean of communication with their devices Natural Interaction Another important factor to socially acceptable devices is that the user has to be able to interact with them in a natural way. If the interaction between the user and the device is unnatural, it will appear awkward of use in public places. In [22], the authors have created an augmented reality system that uses a wireless wristband that includes an RIFD reader, 3-axis accelerometer and RF communication facilities, a cell phone and a wireless earpiece to allow the user to interact with services related to objects using RFID tags through implicit touch-based gestures. Once an object is detected in the user s hand, the user can interact with information about this object using natural slight wrist gestures while previous commercial interfaces that supported hands-free and eyes-free operations required speech recognition, which not only suffers from poor performance and noisy conditions, but is also not socially acceptable Fashion Acceptance Mobile AR systems that wish to step from the laboratories to the industry will also be facing fashion issues as the users will not want to wear a HMD or other visible devices. As a result, developers of mobile systems should take into account fashion trends as this might be a big obstacle to overcome. Groups such as MIT Media Lab, constantly try to reduce the amount of undesirable visible devices or arrange them in different design choice. WUW s first development stage integrated the camera and projector into a hat and the second development integrated it into a pendant. The group is also researching a way to replace the need to wear colored markers on fingers tips [20]. Although these might not yet look very fashionable, they are a good step in the direction of fashionably acceptable AR mobile systems. Another example is ReachMedia [22] which integrates a wristband with a smartphone in order to interact with everyday objects using natural hand and wrist slight gestures. Here the authors avoid the use of unstylish head mounted display for using acceptable wristband and phones. However, the audio interaction with the phone might require the use of headphones in order to remain discrete and subtle.

21 1 Augmented Reality: An Overview Personal and Private Systems Augmented reality mobile systems need to be personal, meaning that the displayed information should only be viewed by others if the user allows it. MIT s SixthSense technology [20] although very advanced, does not offer such privacy to its user due to the use of direct augmentation technique without using any viewing device for protecting the information. Anyone can see the same thing as the user at any time. This poses a dilemma as not needing to wear or carry any extra viewing device for the WUW is an advantage for fashionable acceptable devices; however, it is a problem when it comes to privacy. Systems such as Costanza et al. eye-q [49] or Babak Parviz s contact lens [50] offer such privacy to the user with information that can only be viewed by the user. These systems can also be considered socially acceptable as they are discrete and subtle as well as fashionably correct. However, these systems do not offer the ability of sharing information if the user desires to. A successful AR mobile system should provide the user with private information that can be shared when the user wishes to. In addition, AR mobile systems need to be careful not to violate other users and non-users privacy in new ways. Indeed, information that are available and not considered private on social networks, for instance, can be considered private in everyday life. As a result, technologies such as WUW [20] that make use of online available information about other persons to display them for the user might face privacy issues due to the way the information are being disclosed. 4.3 Tracking Technology for Mobile AR Systems It is well known that for AR to be able to trick the human senses into believing that computer-generated information coexist with the real environment, very accurate position and orientation tracking is required. As was seen in the AR systems section, the most common type of tracking systems for mobile systems is by combining a few complimentary tracking techniques to comprise the advantages of both and support the disadvantages of the other, which creates hybrid tracking. Outdoors systems make mostly use of GPS and inertial tracking technique with the use of accelerometers, gyroscopes, electronic compasses and/or other rotation sensors, along with some computer vision tracking techniques. GPS system, although lacking in precision, provide an easy tracking system for outdoor systems that allows for better estimating the position of the user and its orientation once coupled with some inertial sensors. In this way, the user s interest point is narrowed down and allows for easier visual tracking with fewer options. Indoors systems were GPS cannot be used unite visual tracking with inertial techniques only. Visual tracking achieve the best results with low frequency motion, but are highly likely to fail with rapid camera movement, such as the ones that will occur with HMDs. On the other hand, inertial tracking sensors perform best with

22 24 J. Carmigniani and B. Furht high frequency motion while slow movements do not provide good results due to noise and bias drift. The complementary nature of these systems leads to combining them together in most hybrid systems. Other systems, such as [23] and[42], rely on computer vision for tracking, but most are indoor systems with which the environment can be somewhat controlled. When it comes to visual outdoor tracking, a couple of factors, such as lightning, make tracking extremely difficult. Moreover, some objects present tracking difficulties. One of the most advanced visual tracking mobile systems is Google Goggles [51]; however, this application can only track objects of regular form such as barcodes and books, or places thanks to its GPS and accelerometer that help the application recognize where the user is standing and the user s orientation to narrow down the options. Google Goggles cannot recognize things of irregular shapes such as leaves, flowers or food. 5 Applications While there are many possibilities for using augmented reality in an innovative way, we have cornered four types of applications that are most often being used for AR research: advertising and commercial, entertainment and education, medical, and mobile application for iphones. Below, we study why AR could bring a better solution to some areas, a cheaper solution to others, or simply create a new service. We also discuss the challenges augmented reality is facing to go from the laboratories to the industry. Note that it was decided here to replace the navigational and informational domain application that was encountered in the AR systems section by a study of the augmented reality mobile applications as these applications most often have navigational and informational use. 5.1 Advertising and Commercial Augmented reality is mostly used by marketers to promote new products online. Most techniques use markers that the users present in front of their webcam either on special software or simply on the advertising company s website. For example, in December 2008, MINI [52], the famous car company, ran an augmented reality advertisement in several German automotive magazines [53]. The reader simply had to go to the MINI s website [52], show the ad in front of their webcam, and a 3-D MINI appeared on their screen, as seen in Fig Beyond Reality [54] released a marker less advertisement magazine of 12 pages that could be recognized and animated by a software the user could download on the publisher s website as a starting point to their Augmented Reality Games. They see that with such a system, they could add a paid option on the software that would allow the user to access

23 1 Augmented Reality: An Overview 25 Fig. 1.7 MINI advertisement [53] additional content, such as seeing a trailer and then being able to click on a link to view the full movie, turning the magazine into a movie ticket [54]. AR also offers a solution to the expensive problem of building prototypes. Indeed, industrial companies are faced with the costly need to manufacture a product before commercialization to figure out if any changes should be made and see if the product meets the expectations. If it is decided that changes should be made, and this is more often than not the case, a new prototype has to be manufactured and additional time and money are wasted. A group of the Institute of Industrial Technologies and Automation (ITIA) of the National Council of Research (CNR) of Italy [21] in Milan works on AR and VR systems as a tool for supporting virtual prototyping. The ITIA-CNR is involved in the research for industrial contexts and application using VR, AR, realtime 3D, etc. as a support for product testing, development and evaluation. Some examples of applied research projects where the above technologies have been applied include motorcycle prototyping (Fig. 1.8), virtual layout of a factory and an office (Figs. 1.9 and 1.10), virtual light simulation (Fig. 1.11), and virtual trial of shoes with the Magic Mirror interface, which will be discussed next. Shoes are the accessories that follow fashion trends the most and are renewed annually, especially for those who live in fashion capitals, such as Milan, New York and Paris. For these people, it is more important to wear trendy shoes at the sacrifice of comfort. With the Magic Mirror, the ITIA of CNR in Milan [21] has created a system which, combined with high-tech footwear technology for measurement, enables the user to virtually try on shoes prior to buying/ordering them. The user is able to see his/her reflection in the Magic Mirror with a virtual model of the pair of shoes s/he would like to try on. The advantage of such a system over going to the

24 26 J. Carmigniani and B. Furht Fig. 1.8 Right picture: Picture of a virtual motorcycle prototype (on the right) next to a physical motorcycle prototype (on the left) in the real environment; left picture: virtual prototype in a virtual environment [21] Fig. 1.9 Virtual factory prototype [21] store is that once the user selects shoes for trial, s/he has the ability to change a few details, such as the color, the heel, and/or even the stitching. To do so, the user puts on some special socks with spherical, infrared reflective painted-on markers that serve as a tracking system for the Magic Mirror, which is in fact an LCD screen that actually processes the information from the electronic catalog and inputs data from the client to see if the model chosen is approved, to detect and mirror the movements of the customer. To build such a system, the ITIA-CNR in Milan created their library, called GIOVE library. The GIOVE library has been developed (and is continuously under development) not only to approach this specific project, but for the ITIA-CNR to use as a software library when needed for any application since any type of functionality can be added to the GIOVE library as it has been made from scratch by the ITIA- CNR. The system first has to be calibrated using a bigger version of ARToolKit markers onto which one of the small tracking sphere (similar to the ones place on

25 1 Augmented Reality: An Overview 27 Fig Virtual office prototype [21] Fig Virtual testing of a lighting system [21] the socks ) is placed in the middle. The marker is then laid on the orange pad that is used to help the system recognize the position where virtual elements have to be inserted. A couple of infrared lights camera track the IR lights reflected from the markers and since the system knows in advance the geometric placement of the

26 28 J. Carmigniani and B. Furht Fig User trying on virtual shoes in front of the Magic Mirror [21] markers, the virtual position can be reconstructed accurately. The socks are of the some orange color, which is the color specified to the system for tracking. This color was chosen by the group because it is an unusual color for pants; however, the system could track any color it is indicated. Another challenge the ITIA group had to face with such a system was due to the camera lens distorting the real image while the virtual image to be added to the environment would remain perfect. This detail is not always perceived in every system, but with the Magic Mirror application, it was too noticeable. The solution to such a problem is to either compensate the real image so that the distortion is not so perceptible anymore or to distort the virtual image so that it is not so perfect anymore. The ITIA group chose to compensate their real image using some of the MatLab s software formulas to figure out the degrees of distortion to compensate are (Fig. 1.12). Similar examples to Magic Mirror use of AR for advertising and commercial applications lies in fully replacing the need to try on anything in stores, thus saving considerable amount of time for clients, which would most likely be used for trying on more clothing (shirts, dresses, watches, pants, etc.) and thus increasing the stores chances for selling (Fig. 1.13). Augmented reality has not fully reached the industrial market in advertisement application mostly because a few improvements need to be made to systems similar to the Magic Mirror or Cisco s retail fitting room. Indeed, for the product to be viable in the market it needs to provide the user with a flawless representation of the

27 1 Augmented Reality: An Overview 29 Fig Cisco s AR commercial where a client is trying on clothes in front of a magic screen prototype; the user should have the impression that they are looking at a physical prototype. In the case of the Magic Mirror system, this would mean flawless tracking so that when the user looks at the magic mirror s/he feels like s/he is actually wearing the shoes and can really see what the shoes would look like. 5.2 Entertainment and Education Entertainment and education applications include cultural apps with sightseeing and museum guidance, gaming apps with traditional games using AR interfaces, and some smart-phone apps that make use of AR for an entertainment and/or educational purpose. In cultural application, there exists a few systems that uses AR for virtually reconstructing ancient ruins, such as in [10] (Fig. 1.14), or for virtually instructing user about site s history such as in [47]. There are also a few systems that exploit AR for museum guidance such as [40] and[39]. In [40] and[39], both systems are mobile, but [40] alsousesa mobile phone as an interface while [39] simply uses a magic lens configuration. In [40], the authors identified the benefits of using augmented reality as an interface for their cultural applications as: efficient communication with the user through multimedia presentations, natural and intuitive technique and low maintenance and acquisition costs for the museum operators presentation technology in the case of smart-phone being used as an interface. And indeed, using a smart-phone or even

28 30 J. Carmigniani and B. Furht Fig Augmented view of Dashuifa from [10] another hand-held display is a more intuitive and natural technique than looking up a number randomly assigned to the object in a small written guide, especially when the user can simply make use of his/her own phone in a world where everybody already possesses one. Similarly, users can relate easier to multimedia presentations brought to them and will more willingly listen, watch and/or read about information that they can acquire by simply pointing at an object using their phone rather than have to look it up in a guide (Figs and 1.16). AR can also be used for a learning purpose in the educational field. In fact, AR recently emerged in the field of education to support many educational applications in various domains, such as history, mathematics, etc. For instance, Mark Billinghurst et al. [55] developed the Magic Book, a book whose pages incorporated simple AR technology to make the reading more captivating. Malaka et al. [56,57] built a mobile outdoor augmented reality system application using their previously developed GEIST project for assisting users in learning history through a story-telling game where the user can release ghost from the past. AR gaming applications present many advantages other physical board with, for example, the ability to introduce animations and other multimedia presentations. The ability to introduce animations can not only add excitement to a game, but it can also serve a learning purpose with, for example, indication to help players learn the game or know when they are making an invalid move. In [30], the authors create an augmented reality Chinese checkers game, called ARCC that uses three fiducial markers and cameras fixed on the roof to track the markers. Two of their fiducial markers are used for positioning the checkerboard and the third is used for manipulating the game pieces. Having only one tool for manipulating the pieces

29 1 Augmented Reality: An Overview 31 Fig Mobile phone-enabled guidance in a museum from [40] Fig Visitor with guidance system from [39]

30 32 J. Carmigniani and B. Furht Fig ARCC from [30] allows the authors the adapt their setup to different types of games as all they have to change is the GUI (Graphic User Interface) that is the board game and the game s logic (Fig. 1.17). Beyond Reality [54], which was the first to introduce a marker less magazine, presents two board games, PIT Strategy and Augmented Reality Memory. In PIT Strategy, the player is the pit boss in a NASCAR race and must act according to given weather condition, forecast and road condition. In Augmented Reality Memory, the player turns a card and sees a 3D object, turns a second card and sees another 3D object. If they match, a celebration animation will appear; otherwise, the player can keep looking for matches. These two games are still under development and further information can be found on [54]. Here again, augmented reality has not fully reached its potential to enter the industrial market. Once again, this is mostly due to technological advances such as tracking system. For example, we saw that the few museum guidance systems developed were only applicable to the museum or exhibition they were developed for and could not be utilized for other museums. This is due to the fact that both these systems relied on the organization of the museum or the exhibition to recognize the artifacts as opposed to detecting the artifacts solely using computer vision. So why hasn t computer vision been used to recognize the objects instead of relying on knowing the user s position in the museum? As was seen in the Computer Vision Methods in AR section, some objects have irregular forms and although it might seem easy for us to recognize them, it is very hard for a computer to detect what these objects are, and this is the case of most artifacts. Paintings do not present such

31 1 Augmented Reality: An Overview 33 Fig Bichlmeier et al. system for viewing through the skin [34] a big trouble for system such as Google Goggles [51] due to their regular shape, but objects, such as objects of modern art have very irregular shape that makes it hard to track a defined feature. 5.3 Medical Applications Most of the medical applications deal with image guided and robot-assisted surgery. As a result, significant research has been made to incorporate AR with medical imaging and instruments incorporating the physician s intuitive abilities. Significant breakthrough has been provided by the use of diverse types of medical imaging and instruments, such as video images recorded by an endoscopic camera device presented on a monitor viewing the operating site inside the patient. However, these breakthroughs also limit the surgeon s natural, intuitive and direct 3D view of the human body as the surgeons now have to deal with visual cues from an additional environment provided on the monitor [34]. AR can be applied so that the surgical team can see the imaging data in real time while the procedure is progressing. Bichlmeier et al. [34] introduced an AR system for viewing through the real skin onto virtual anatomy using polygonal surface models to allow for real time visualization. The authors also integrated the use of navigated surgical tools to augment the physician s view inside the human body during surgery (Fig. 1.18). Teleoperated robot-assisted surgery provide the surgeons with additional advantages over minimally invasive surgery with improved precision, dexterity,

32 34 J. Carmigniani and B. Furht Fig Sequential shots of knot tying task from [38] and visualization [58, 59]; however, implementing direct haptic feedback has been limited by sensing and control technology and thus is restricting the surgeon s natural skills. The lack of haptic feedback has been proved to affect the performance of several surgical operations [60]. In [38], the authors propose a method of sensory substitution that provides an intuitive form of haptic feedback to the user. The force applied by the surgeon is graphically represented and overlaid on a streaming video using a system of circles that discretely change colors across three pre-determined ranges (Low Force Zone (green), Ideal Force Zone (yellow) and Excessive Force Zone (red)) according to the amount of bending forces detected by strain gages (Fig. 1.19).

33 1 Augmented Reality: An Overview 35 The need to reduce surgical operations is not the only one to depend upon seeing medical imaging data on the patient in real time; the necessity to improve medical diagnosis also relies on it. In this research field, the ICAR-CNR group of Naples [62,77] is working on an AR interactive system for checking patient s hand and wrist for arthritis by overlaying in real time 3D MR imaging data directly on top of the patient s hand. Since arthritis disabilities are strongly associated with pain intensity and so require a direct manipulation of the hand and wrist region to be diagnosed, the system may support physicians by allowing them to perform morphological and functional analyses at the same time [62].ARcouldalsobeusedtomanageclients medical history. Imagine if all a doctor had to do to check a patient s medical history was to put on a head mounted display and look over the patient to see virtual labels showing the patient s past injuries and illnesses. The use of AR in the medical field to provide better solutions to current problems than already existing solutions is infinite. In [35], the authors use AR to provide a low cost and smaller in size solution to the post-stroke hand rehabilitation problem, which has the potential to being use in clinics and even at home. In [36], the authors use AR to help patients fight against the phobia of cockroaches and thus show that AR can be used to treat psychological disorders as well. Additionally, AR can be used to assist the impaired, such as for instance supporting the visually impaired through augmented navigation. In [63], Hara et al. developed a multimodal feedback strategies for augmented navigation of the visually impaired. The feedback device consisted of a Wiimote which provided audio and haptic feedback to operate as a guiding tool and warn the user when they were getting close to walls and other obstacles. Unfortunately, on top of facing a few technological advances issues such as displays and tracking issues, medical applications also face privacy concerns. Displays challenges mostly arise from the fact that the preferred type of display to use for medical applications is a HMD as it allows the physician not only to use both hands, but it is also easier to track where the doctor is looking to augment the right surfaces; however, it is challenging to implement HMD to medical applications. There are challenges that arise because of the HMD itself, such as accurately placing and applying depth perception to 3D models, and challenges that are due to the medical field itself, such as for a surgeon to still be able to see his tools through the projected images. Another possible type of display that could be used would be spatial display to allow the whole surgical team to see the same thing at the same time; however, it is then very hard to track where the surgeon is looking and what the desired place for augmenting is. Privacy concerns always arise in the medical field, especially when discussing the treatment of the very confidential medical history of patients. Another type of issues that medical application in augmented reality will most likely have to face is the problems that arise with retraining the medical staff for using new tools. Most AR applications aim at simplifying the use of AR tools such that they correspond to what the physician is used to; for instance in [38], the feedback system developed by the authors did not require the surgeons to truly learn how to use it as the application was easily integrated onto the da Vinci Surgical

34 36 J. Carmigniani and B. Furht System that most surgeons know how to use. Even with this system, the surgeons still have to get use to this type of haptic feedback system, although the training is rather short and inexpensive. However, there are some systems that will require a complete retraining of the staff to interact with the application. For example, applications that will require the user to interact with a 3D input device as opposed to 2D input devices such as a mouse, will present some training problems as they might be too costly for the medical field to judge them viable. 5.4 Mobile (iphone) Applications Many AR mobile applications for iphone already exist; however, no development has been done for the ipad due to the lack of camera in the first generation of ipad. Moreover, most iphone applications either have an entertainment and/or educational purpose or a navigational and/or informative purpose, such as orienting the user. Examples of such applications include WikitudeDrive [64], which is a GPS-like application that allows the user to keep his/her eyes on the road while glancing at the GPS; Firefighter 360, which has an entertainment purpose that permits the user to fight a virtual fire like a real firefighter; and Le Bar Guide that has a navigational function to guide the user to the nearest bar that serves Stella Artois beer. Websites such as Mashable, the Social Media Guide [65] and iphoneness [66] haveall come up with the best augmented reality applications for iphone and we encourage interested readers to have a look at them. Due to the relatively new side of adding AR to mobile applications, there currently are not many libraries, kits or codes available for iphone program developers to add some augmented reality to their application. Studierstube Tracker and Studierstube ES support iphone platforms; however, they are not open sources [14]. We found two sources to help iphone developers in using AR for the mobile application. SlideShare, Present Yourself [61] is a slideshow presentation that shows the viewer how to develop augmented reality applications on iphone with codes for retrieving GPS position, using the compass and accelerometer, and getting the image from the camera. iphone ARKit [67] is a small set of class that can offer developers augmented reality on any iphone application by overlaying information, usually geographic, over the camera view. The iphone ARKit s APIs were modeled after Map Kit s [68], which is a framework for providing an interface to embed maps directly into the windows and views and to support annotating the map, adding overlays, and performing reverse-geocoding lookups to determine place mark information for a given map coordinate (Figs ). Augmented reality mobile applications are one of the only augmented reality applications that we can find in the general public. However, even these applications are facing a few challenges issues. There are, for instance, problems due to GPS systems not being accurate enough for other applications that require very precise placement of virtual tags. There are issues with working with limited hardware capability when we require image processing power. In the iphone category, there

35 1 Augmented Reality: An Overview 37 Fig WikitudeDrive [64] Fig Firefighter 360 [65] were challenges due to accessing the video APIs as Apple would not open its APIs enough for the developers to be able to get in there and work with the video. However, with the release of the iphone 4/iOS4, augmented reality sees more doors opening for AR applications on mobile phones: the developers now have the ability to access the camera images APIs, enhanced image tracking, gyroscopic motion sensing, and faster processor and high resolution display [69]. Similar to mobile systems, mobile applications also face social acceptance issues, as the applications also need to be subtle and discrete. Mobile applications should not be able to make random noises at inopportune times for the user. No matter

36 38 J. Carmigniani and B. Furht Fig Le Bar Guide [65] how accustomed to mobile phones our society has become, it is still considered to be rude and obnoxious when someone is on the phone in a public place; when a person s phone is ringing in public, the first reflex for this person is to search for their phone to turn off the sound and then check who is calling or what the reminder is. Of course, social acceptance has the advantage of changing through the generations much like fashion. 6 Future of Augmented Reality Applications AR is still in infancy stage, and as such, future possible applications are infinite. Advanced research in AR includes use of head-mounted displays and virtual retinal displays for visualization purposes, and construction of controlled environments containing any number of sensors and actuators [1]. MIT Media Lab project Sixth Sense (Fig. 1.23) [20] is the best example of AR research. It suggests a world where people can interact with information directly without requiring the use of any intermediate device. Other current research also include Babak Parviz AR contact lens (Fig. 1.24)[50] as well as DARPA s contact lens project (Fig. 1.23)[70], MIT Media Lab multiple research applications such as My-Shopping Guide [71] and TaPuMa [28]. Parviz s contact lens opens the door to an environment where

37 1 Augmented Reality: An Overview 39 Fig From top to bottom and left to right: Examples of futuristic augmented reality [74, 76] and Babak Parviz s contact lens [50] information can only be viewed by the user. Of course, this can also be done by using glasses as opposed to contact lens, but the advantage in both cases over using a cell phone, for instance, is that no one else but the user can see the information projected, making it very personal. Cisco has imagined a world where AR could be used for replacing the traditional fitting rooms by trying on virtual clothes, thus saving time and providing the ability to try on more clothes, increasing the chance for stores to sell. Augmented reality also brings the possibility of enhancing missing senses for some users. For example, AR could be used as a sensory substitution device. Hearing-impaired users could receive visual cues informing them of missed audio signals and sightless users could receive audio cues notifying them of unknown visual events. We believe that new mobile devices, such as iphone, Android-based devices, and ipad are not well used in AR. Indeed, most of the current applications include gaming, entertainment and education, and while most already believe that these are amazing apps [65].

38 40 J. Carmigniani and B. Furht Fig From top to bottom and left to right: DARPA s contact lens project [70], MIT s Sixth Sense [20], Contactum s AR Solutions [72] Even the future is not far from challenges for augmented reality. We see social acceptance issues, privacy concerns, and ethical concern arising with the future of augmented reality applications in the industry. Social acceptance mostly arise from mobile devices with the need for the devices to be subtle, discrete and unobtrusive as well as fashionably acceptable as was discussed in the Augmented Reality Mobile Systems section, but also with systems that will require retraining of the personnel and staff in order to be utilized. We have seen that this might be the case with some medical applications and that the health system might decide against the use of augmented reality if they decide that the retraining is too costly. A system for easy integration of such system will have to be developed to avoid such issues. Privacy concerns arise not only with medical applications, but also with technologies that have the ability to detect and recognize people. For instance, MIT s WUW technology video presentation [73] has an application that is capable of recognizing