ABSTRACT. variety of operating systems, communication protocols or equipment. Those differences

Transcription

1 iii ABSTRACT As technology improves, the number of devices, sensors around us are increasing. Different devices and sensors designed by different companies can use a variety of operating systems, communication protocols or equipment. Those differences make communications between different types of devices challenging. Specific industrial standardization is required for this problem. Internet of Things standardization is the expected solution for increasing number of digital objects. Although standardized communication of these digital entities creates new challenges, managing these objects and interacting with them is challenging with traditional user interface solutions. Augmented Reality allows enhancing and supporting the real world environment with computer generated user interfaces. Augmented reality provides real-time effective UI to manage and interact with several objects in real time. The project aims the ability to achieve communication and collaboration of different users on Internet of Things network with augmented reality. Users need to be able to manage and manipulate the data which is generated from different objects or users. Changed information needs to be reported to other users. Users need to be able to see differences from their unique perspective.

2 iv TABLE OF CONTENTS CHAPTER Page ABSTRACT iii TABLE OF CONTENTS iv LIST OF TABLES vi LIST OF FIGURES vi 1 BACKGROUND AND RATIONALE Introduction to Augmented Reality Augmentation of Senses Hardware and Display Systems Registration and Tracking Interaction Distributed Augmented Reality Collaborative Augmented Reality Internet of Things NARRATIVE Problem Statement Motivation Project Objective Project Scope SYSTEM DESIGN AND INTEGRATION Actors Presentation Actor Interaction (I/O) Actor Tracking Actor Things Actor Manager Collaboration

3 v CHAPTER Page Collaboration of Managers Collaboration of Actors Software Requirements Hardware Requirements Implementation Windows Application Android Application Google Glass Application TESTING AND EVALUATION Unit Testing Integration Testing Tracking Testing Usability Testing FUTURE WORK CONCLUSION REFERENCES

4 vi LIST OF TABLES TABLE Page 1 Usability test questions and scores

5 vii LIST OF FIGURES FIGURE Page 1 Milgram s Virtuality Continuum [45] Generic augmented reality reference architecture Collaboration of managers with peer to peer Collaboration of managers with collaboration server Collaboration of managers with actor server architecture Actor peer to peer architecture Shared actors Connecting the Alljoyn server CommClass Smartphone screenshot Smartphone side menu Google Glass screenshot Manager server s data traffic logs Vuforia feature detection ARToolkit detected feature points

6 1 CHAPTER 1 BACKGROUND AND RATIONALE Augmented reality is the manipulation of real world information with virtual data. In augmented reality, computer generated graphics are overlaid into the user s field-of-view to provide extra information or guidance about their tasks and surroundings. We are living in the era that increasing number of devices are connected the devices. In the future, this growth is expected to be exponential. All of the internet connected devices are going to create massively distributed networks called the Internet of Things (IoT). IoT is not just about a network of devices. IoT can contain several sensors, infrastructures, and related architecture. Internet of Things is a complex distributed architecture that is hard to control and maintain. In this problem, classic separate User Interface (UI) solutions are not able to respond to the complexity. A new UI solution needs to support 4 awareness issues; 1. Environment awareness, 2. Situation awareness, 3. User awareness, and 4. Context awareness. The UI solution needs to be environmentally aware. The IoT network provides several sensors and devices (things). Information generated from things needs to be filtered or customized for usage. Things or application generated events needs to be handled by UI. Situations can change UI simultaneously. The UI needs to

7 2 be informative, understandable and users should be able to understand UI without problems and can interact with UI or data smoothly. UI needs to simplify complex data flow for the user. Applications can change UI needs massively. Depending on application, devices, sensors and user s need change. UI needs to solve all of this pervasive computing awareness problems. Augmented reality can answer all of these problems altogether. However, the complexity of the network and custom complex architecture needs for different problems are decreasing development and usability. Also, augmented reality can be created for all senses. On the other hand, current augmentation architectures, are designed for specific senses. This project creates an architecture for augmented UI solutions for different senses for the complex internet of things environment. Augmented reality is the manipulation of real world information with virtual data. In augmented reality, computer generated graphics are overlaid into the user s field-of-view to provide extra information or guidance about their tasks and surroundings. Augmented reality can provide simple or more complicated user interface tools. 1.1 Introduction to Augmented Reality Augmented Reality (AR) is a human-computer interaction technique that allows combining virtual digital elements with the real world environment. AR allows computer generated data to be aligned or registered with the real world environment in real time. Augmentation might give a feeling of removal of real objects from senses by applying virtual data [8]. With AR, the user can interact with the real world environment in a natural way, while using and interacting with the computer generated data to utilize related

8 3 Figure 1.: Milgram s Virtuality Continuum [45] information [65]. As seen in Figure1, Milgram s Virtuality Continuum Diagram [45] explains the position of AR in the reality domain. Azuma s definition is commonly accepted as an explanation of augmented reality. Azuma defines AR as a combination of real and virtual imagery with the registration of the virtual imagery on the real world, accompanied by real-time interaction [9]. Augmented reality is an interdisciplinary field that overlaps a wide array of areas including human-computer interaction, computer vision, wearable and distributed computing. AR applications can be applicable for several domains such as engineering, aviation, telecommunication, entertainment, healthcare, marketing, law enforcement, construction, manufacturing, planning, training, maintenance/repair, design/art and education. 1.2 Augmentation of Senses AR can be applied to all kinds of senses. Vision, hearing, touch, smell and taste can be augmented for different purposes. Different augmentation types have their challenges. As a UI solution, audio has been used for a long time. The sound makes us aware

9 4 of the environment. With sound, we assume the existence of the source object and with echo, we can understand the distance of the objects. However, the standalone effect of sound augmentation is not widely explored. Olfactory augmentation is one of the undeveloped augmentation areas. There are no discovered primary odors. Lack of primary odors makes it difficult to generate scents. Humans can detect approximately 400k chemical compounds as odor. Based on the research of Buck and Axel [16] and recent genome analysis, humans have approximately 350 scent receptors. Some researchers accept 350 as the prime odor number. Nambu et al. [48] and Hyung-Gi et al. [17] try to decrease the number of required scents based on vision because some scents have similarities, and human s vision-smelling relation may be tricked. Gustatory augmentation is challenging due to the complicated nature of taste sensation. Narumi et al. [49] created gustatory augmentation with a combination of visual and olfactory augmentation. Nakamura et al. [47] try to differentiate taste with electricity. If they become successful, sensing of the tongue can be increased, and we can understand part of the complexity of tasting. Touch is the primary interaction method of humans. Unlike gustatory and scent augmentation, haptic augmentation can be simulated. Tactile displays can be categorized as, Braille display for the visually impaired and the haptic device to make the virtual world get tactile textures that seem more real [35]. 1.3 Hardware and Display Systems Combining the virtual and real world is one of the most challenging design factors for successful AR systems. This can be achieved with several hardware systems. Researchers are developing hardware for different kinds of augmentation. However,

10 5 AR research is mostly focused on the sense of sight. Mainly computers, electronic glasses, headgears, mobile devices and projectors are used with visual augmented reality applications. Different hardware platforms have their own advantages and disadvantages. There are studies [64] that show the effect of the screen size or type on task performance, but they produce mixed results. Selection of hardware depends on application development and purpose. In recent years different tools are released for augmented reality purposes. Google Glass [30], Epson Smart Glass [25] are examples of daily user focused electronic glasses on the market. Also, specific application related tools are released. Daqri industrial smart helmet [23] is an example of it. Wearable headsets are usually using LCDs or laser image displays (Retinal Image Display-RID). LCD displays can be designed for different sizes. Furthermore, other display improvements are enhancing AR. Magic Leap [41] created AR device based on glasses-free 3D tensor displays [63] for more natural, user-friendly environment. Microsoft has designed their holographic display solution [44]. The holographic display can be considered for mixed reality solution. HoloLens contains CPU, GPU, and HPU (Holographic Display Unit). Stereoscopic displays usually are considered for virtual reality solutions. However, there are several solutions for using stereoscopic displays for augmented reality. Different headset designs have a large variety of field of views. Smartphones and tablets are multi-purpose mobile devices. They have increasing popularity and usage. Having parts like camera, GPS, gyroscope, accelerometer and mobility advantage, makes these devices effective for registration and tracking for augmented reality. Projection approach projects the desired virtual information directly on physical objects to be augmented. Cassinelli at al. [18] created laser based projector that is

11 6 capable of augmenting all kind of surfaces while simultaneously scanning projection surface information like shape, texture, reflectivity, etc. with another laser beam. That allows feature based registration abilities. When manipulating visual data captured by the eyes, changing information at eye level with contact lenses is considered. Focusing close range is a problem for augmented reality solutions. Usually, the optical object needs to be used between eye and display. For audio augmentation, we use auditory displays to create sound. Curran surveyed audio-haptic, tangible audio displays in his book [22]. In his book, he points out tangible user interfaces and auditory displays needs to improve. SmartTool [51] are used for haptic augmented reality. Tools like a pen or surgeon equipment are integrated with sensors and feedback devices. Recently haptic augmented reality devices started to be on the market. Touch based augmented reality device named Gloveone [29] has been announced. It gives the user the feeling effect with vibrations. Olfactory augmentation can be created with specific scent generator. Scent distributing systems or scent generators require specific multimedia devices. Scent delivery can be created in two different ways. It can be provided with direct inhaling with scent mask or tubes. Scent also can be distributed to the user s nose by diffusion. 1.4 Registration and Tracking Registration is one of the main components of AR. Integrating virtual information in real data requires proper registration. Registration is the process of real detection with distinct features. The features can be provided from a different type of sensors like camera or GPS. In most AR applications, user s movement, position,

12 7 and environmental changes need detection to be tracked with previously configured reference data. Proper augmentation requires user situation (user location, distance to the physical object, etc.) and user actions (user movement, head movement, eye movement, etc.). There are several methods for registering augmented data on the real world. Hardware based or software based tracking techniques can be used. GPS, gyroscope information can be utilized for outdoor location based solution. Ultrasonic, magnetic, optical, radio frequency and inertial technologies can be used to achieve very accurate positional data. LIDAR solutions can gather position and orientation information as well as physical object features. Using ultrasonic waves for distance calculation is a popular, simple method. RF technology has increasing popularity for tracking and localization. RFID tags are used in assets, inventories, industries, groceries, logistic facilities, hospitals etc. RF technology can be categorized mainly in two methods for tracking. In the first method, environmental RF module acts as an anchor hold its position information. Environmental RF module contains position coordinates. When the user contacts with the environmental module, user s location is calculated. Another method is known as a fingerprint based method. Unique identifiers and their signal strength are known as fingerprints. User s position is calculated by mapping the fingerprints locations. High accuracy registration is desired for all kinds of augmentation. Although the accuracy needs of AR applications changes depending on the usage, applications of advertising, entertainment, etc. might not need high precision. However, manufacturing, health-care need high accuracy registration. Outdoor AR systems use inertial systems and GPS for tracking purposes. These sensors have high error rate therefore precision and accuracy are bad. For indoor applications, GPS is not

13 8 applicable. Usually, a sensor supported computer vision solutions are used for indoor registration purposes. However, when high frequency motion or rapid camera movement occurs, image processing techniques are failing. Other than hardware based tracking and techniques, a significant amount of software based tracking solutions is designed. Software based solutions have increasing popularity, because of decreasing requirement to bulky hardware. Software based solutions mostly use computer vision techniques. Image processing solutions for tracking can be categorized as marker based, feature based and mapping-model based. Markers are pre-designed artificial landmarks planted on the environment for getting data with specific techniques. Mostly marker based tracking techniques use a barcode, QR code, or tag image as a marker. Also, specific texture, image or image pattern can be utilized as a marker. Augmented reality software can determine the marker and give results based on that. Easily detectable feature points and higher response time makes marker based solutions effective. ARToolkit [39] is the most popular marker based library. Markers with asymmetric patterns are designed to be an easily detectable by augmented reality trackers seamlessly. There are various projects ongoing porting ARToolkit to different platforms. osgart is a C++ based cross platform library which is supporting marker tracing and AR rendering with the integration of OpenSceneGraph graphic library and ARToolkit library or ARToolkit fork ARToolkitPlus. AndAR [4] is a Java library for developing Android based AR applications. ARToolkit has been forked for a different project. NyARToolkit [52] supports multiple programming languages (C++, C#, Java, ActionScript, Unity) and it can work on different operating systems. NyARToolkit has several branch projects. FLARToolkit [27], is based on NyARToolkit, has been aimed to flash based

14 9 AR application development with ActionScript. FLARManager [26], which is also support FLARToolkit, also is lightweight AR library for flash based applications. SLARToolkit [60] is also NyARToolkit based library for Silverlight and Windows Phone development. ARTag [6] is another open source tool for fiducial marker based augmented reality. ARTag was released several years later than ARToolkit. ARTag has better image processing and digital symbol processing capabilities. Also, ARTag has better performance under illumination changes. ARMES [5] is a commercialized SDK that uses unique circular markers. Layar [40] is a commercial SDK that uses geotagging and predesigned markers. Geotagging allows GPS based location registration for outdoor purposes. On the other side, in most cases, markerless solutions are more desirable. A markerless solution does not need to change the environment for tracking purposes. Feature based solutions allow natural integration of augmented data or virtual objects on real environments. Most of the current natural feature tracking methods use the robust points as a feature for matching. Various algorithms are used for defining, detecting and matching. Binary Robust Independent Elementary Features (BRIEF), ferns feature, Scale Invariant Feature Transform (SIFT) features and other SIFT variations like Speeded Up Robust Features (SURF), Affine Scale Invariant Feature Transform(ASIFT) etc. are most popular feature detection algorithms. For AR purposes Qualcomm released commercial feature based AR library called Vuforia [59]. Vuforia has both markers based and feature based AR solutions. Also, FastCV [58], an image processing library that optimized for mobile application released for faster image processing AR solutions. ARToolkitNFT [50] is feature tracking library which is expanding ARToolkit s capabilities with natural feature tracking. BazAR [57] is a computer vision library that detects and registers known planar objects in

15 10 images with feature points detection and matching with a key point based method. Mapping-model based solutions are recently becoming popular among researchers and SDK developers. Several researchers improved 3D environmental mapping and simultaneous localization and mapping (SLAM) techniques for registration purposes. In previous techniques, camera pose has a relation with feature generated target. Also, the target is usually planar. However, SLAM provides pose estimation in an unknown environment and 3D reconstruction of the problem while moving in 6 degrees of freedom. Klein has proposed a parallel tracking and mapping (PTAM) solution [37]. PTAM is monocular SLAM system that uses dense map of lower quality features which is useful for tracking in static and limited small scenes. Parallel Tracking and Multiple Mapping (PTAMM) extents PTAM to allow usage of multiple independent cameras to create multiple maps. PTAMM allows linking different workspace maps which in various AR applications to each other. Open Tracking Library (OpenTL) [55] use pre-produced models, such as texture, appearance, movement, etc. to match environment features. It is not designed for augmented reality purposes, but user friendly API and multithreading support makes OpenTL considerable. RGB-D sensor-based techniques have lowered the error rate of tracking and registration of image processing. Registration, mapping techniques are continuously improving related to hardware improvement. 1.5 Interaction Augmented reality is usually used as an output method. virtual reality are naturally applicable to augmented reality. Input methods for Augmented reality can also use equipment that is used for registration and tracking purposes like a gyroscope, GPS for an interaction inputs. Also, separate input equipment can be

16 11 employed. Input devices can be different for distinct setups. Haptic feedback provides interaction tools and creates immersive and interactive application environment for the user. Body tracking devices like ControlVR [21], Manus [43] are effective tools for augmented reality. Also, a combination of different AR senses can give input information. 1.6 Distributed Augmented Reality Augmented reality is a complex field that combines an extensive range of algorithms (computer vision, registration, localization etc.) and several devices and sensors (cameras, displays, tracking equipment etc.). Because of the complexity, developing and managing the functionality of AR frameworks is becoming harder. AR contains different components. Separating AR components and managing them separately increases re-usability of AR libraries and allows extensibility. In parallel, to the growth of AR research, several distributed solutions emerge. Dwarf (Distributed Wearable Augmented Reality Framework) [15] is a decentralized component model based framework. It is using common object request broker architecture (CORBA) to provide communication with different languages. Dwarf uses the concept of independent services. Services work on their needs and abilities between each other. Every network node contains a service manager that controls its local services. Service managers can cooperate each other to connect remote services. DWARF aims rapid prototyping of AR applications. Independent services allow decentralized development with Dwarf. Studierstube [61] is mainly developed for multi-user, groupware AR and VR applications. It is built on device management and tracker framework OpenTracker. Studierstube has named each of its components as application objects. Each applica-

17 12 tion object inherits from an application type. Application objects can be instantiable from different application types in same time. These applications can communicate with each other. In Studierstube architecture, service manager manages overall system. Every framework has their distinct advantages. Dwarf and Studierstube are both component based and expendable. Bauer et al. [11] wanted to use both systems advantages by integrating this two system together. Extending their component for communicating both of the systems allow developers to use both of the systems advantages. MORGAN [54] is proprietary component based framework. It is CORBA based like Dwarf. However, it is also using a different proxy pattern for other protocols. MORGAN is designed for making AR application development with multiple users easier. In this architecture, components subscribe to input devices that they are interested. The system creates publisher subscriber architecture between components. MORGAN framework has its rendering engine for support multi user solution in distributed system. Component management is provided with a specific component called a broker. Tinmith evo-5 [56] is a modular architecture that aims to solve complex development problems like data distribution, rendering, interaction, tracker abstractions, modular extensions. It is written in C++. Modules can be application specific or generic. The communication between modules is provided with client server style architecture. When data is changed on the server, the new values are sent to all clients that subscribed their interest. The system is asynchronous and uses data flow methodology. If there are no data changes, there won t be any action. VARU [33] is designed for VR, AR or ubiquitous computing development. VARU

18 13 uses XML based configuration descriptors. In VARU, components can use these reusable XML configurations for different development projects. Users can interact each other like co-located in same space without depending on their reality (AR, VR, and ubiquitous). VARU framework contains two main part: VARU server, and VARU client. VARU server synchronizes objects in space. Objects are described in three level: Class, Individual, and Extension. A Class is an abstract group of objects with an assigned purpose. The Individual is an instance of Class. An Extension is a representation of an Individual in the interaction space. Each application store objects in the database with an individual s and extensions tables. VARU uses different libraries for different requirements. OpenSceneGraph is main rendering library for VARU. For managing an AR application, osgart, ARToolkit or OpenSceneGraph can be utilized. For smart device connection, it uses CAIM middleware, UPnP protocol and VRPN for the interaction peripherals. ARCS (Augmented Reality Component System) [24] is designed for rapid prototyping of distributed AR applications. It is written in C++ with multi platform support (Unix, Linux, Windows). It can provide centralized or decentralized distributed architecture. ARCS is using component orient programming. ARCS components can be configured and composed of other components. ARCS works with finite state machines. Applications are described as threads and threads are controlled with finite state machines that indicate specific configuration. The configuration is called sheet. Sheet contains pre-connection, connection, post-connection and cleanup states. Preconnection states configure components. Connection states configure links between components and manage data transfer. Post connection invokes application configuration. Cleanup state restores component states. When states change, global configuration of the components also change, and global configuration reconfigures

19 14 the connections. There is also ARCS.js JavaScript library under development with same principles. It is intended to run both browser and the node.js environment. AMIRE [31] uses a component oriented architecture and consists of the minimal set of components required for a demonstrator, a reusable augmented/ mixed reality software collection and a visual authoring tool for building AR applications. 1.7 Collaborative Augmented Reality Desktop computer supported collaborative work (CSCW) applications might separate users from each other and from their tools. This is an important disadvantage for a collaborative environment. Also, classic CSCW applications provide hard to use solutions to three dimensional problems. AR enables us to enhance the physical environment with a computer generated virtual environment. That allows us a more natural communication and removing desktop type CSCW limitations. Collaborative AR allows physical interaction and virtual interaction simultaneously that allows the developer to use virtuality and reality as different parameters. This provides application development capabilities without depending time or space constraints. The research of StudierStube [13] defines collaborative AR environments features are: 1. Virtuality: Objects that don t exist in the real world can be viewed and examined. 2. Augmentation: Virtual annotations can augment real objects. 3. Cooperation: Multiple users can see each other and cooperate in a natural way. 4. Independence: Each user controls his independent viewpoint.

20 15 5. Individuality: Displayed data can be different for each viewer. Collaborative augmented reality (CAR) is powerful user interface tool that can be applied in all kind of position on time/space matrix. In same time/ same space domain, CAR applications can create identical or user customized augmentation for a different user to assist interaction. Tabletop solutions which have surface or equipment tools that provide shared augmentation are the main application area for same time/ same space domain. Table top approach is a traditional collaboration solution. Several applications designed for productivity, education, design, training, maintenance/repair etc. use this approach. Researchers designed several co-located systems that use the same database platform. Also, manipulating 3D models that augment in the user view are tested in several experiments. In the same time/ different space domain, Kato et al. [36] and Barakonyi et al. [10] experimented face to face video conferencing tool with CAR. Using avatar is one of the main approaches for distributed collaboration. Avatars provide character recognition of different users in different places. Users can communicate and interact with avatars that are representing other users. Archaeological digging is an example one time gatherable sensible data type of work. Benko et al. [12] created mixed space CAR tool for solving the problem. Augmented sensation allows several users to inspect the historical area. Also, remote users can get information about the work environment. Freer [28] is provided a collaborative messaging system on augmented reality. Different time aspect of the collaborative work for AR is not widely evaluated. In same space situation, location registration is enough for AR collaboration. In various environments, overall UI needs to adapt to new data. Challenges in synchronous AR situation are mostly answering asynchronous AR problems. Thomas et al. [62] provide examples of the different time problems with the logistic area.

21 16 Mueller-Tomfelde created a hybrid sound reproduction in audio augmented reality [46]. This solution can create direct undisturbed inter-individual communication within the team and at the same time a personalized augmented sound environment in collaborative augmented reality. Knoerlein et al. designed visual and haptic AR solution for entertainment purposes [38]. Their ping-pong game is an example collaborative visuo-haptic AR solution. 1.8 Internet of Things Internet of Things (IoT) is an application technology which is promoted originally with the electronic product code (EPC) technology developed by Auto-ID Center of MIT [14] and the research reports written by International Telecommunication Union [34]. The paradigm of this concept is the pervasive presence around the user of a variety of things or objects (sensors, ID-tags, connectible devices etc.) which, though unique addressing schemes, are able to interact and communicate each other to reach common goals [7]. The most important issue about Internet of Things is the standardization problem. Devices need to be able to communicate each other with wired or wireless networks. These networks are including, but not limited to, RFID, DSL, Wi-Fi, LAN and mobile networks (GPRS, 3G, 4G, and 5G). For standardization, several consortiums have been formed such as Industrial Internet Consortium (IIC) [20], OpenInterconnect Consortium (IOC) [19], Allseen Alliance [2], IPSO Alliance [3] and Internet of Things Consortium (IoTC) [53]. The IoTC aims to educate technology firms, retailers, insurance companies, marketers, media companies and the wider business community about the IoT. The IPSO Alliance was formed to promote the Internet Protocol as the network technology of choice for connecting Smart Objects

22 17 around the world. The mission of the IIC is the establishment of use cases and standards requirements for IoT. The stated goal of the OIC and Allseen Alliance is to define IoT interoperability specifications and enable branded implementations of them. Allseen Alliance and OIC have published their proposed architectures and related frameworks; AllJoyn [1] and Iotivity [32]. Both Alljoyn and Iotivity have similar architectural design approaches. However, when this paper was written Iotivity was not complete. Researched or developed augmented reality solutions usually focus on building one augmentation device, and all sensor and peripherals connect the same device. However, Internet of Things environment provides devices for work standalone for their purpose. Getting information and interacting with them separately is becoming a complex problem. Augmented reality solutions need to manage and act respectively with this device as system actors. The purpose of the augmented reality is to provide better human-computer interaction environment for the user. Because of this, the proposed architecture designed for the user-centric approach. Naturally, Internet of Things is distributed environment. Mature Internet of Things environment is going to handle communication between devices. Allowing the user to manage the complex system with combined UI increase user experience. Augmented reality can provide the user unified UI experience. In this architecture, AR system also prefers separation of modules on different devices. Main reason is separation can give higher modularity advantage. Also, separation of the system allows developing sub-architectures for a specific module.

23 18 CHAPTER 2 NARRATIVE 2.1 Problem Statement Every day, hardware becomes cheaper and powerful. Improvements on hardware allow development of different solutions around the users. User s interaction with the devices is increasing. However, managing different interactions are becoming a challenging problem. To solve this problem, we need to manage to unify user interfaces of different devices. Single Interface could provide required information. Because of the manipulative power of the real environment, augmented reality is an effective candidate to solve the problem. Unification of multiple user interfaces on one interface, allows the user to manage multiple device-generated contents. Communication between different hardware is a challenging problem because of its nature. Devices can use varying technologies to talk each other like Bluetooth, Wi-Fi standards ( a/b/c/n/ac), NFC etc. Also, data sending protocols and data types need to be considered. Internet of things (IoT) is a unification of communication standards. Complicated hardware interactions which are created by different vendors and different technologies needs to use IoT standards to manage communication. Nature of this project requires usage of different hardware solutions with varying operating systems. Using IoT libraries can simplify communication procedures and increase the extensibility of the project.

24 Motivation The increasing amount of hardware creates a managing problem for the user. Interacting devices user interface separately decreases user experience.the user needs to control and interact with devices independently or device s interfaces and communication methods should be unified. Augmented reality can provide respective UI for devices without creating a complex user experience. AR can be generated with a tracking solution to generate respective UI for different interfaces. Also, same as device collaboration, user collaboration for different applications are highly desired. This project aims to provide collaboration capabilities for users while using different hardware or operating systems. Meanwhile, the solution should be expandable for future hardware implementations and communication solutions. 2.3 Project Objective This project also aims to create collaboration among peers with improved user experience. Collaborative augmented reality can provide improved joint effort when securing individual user experience. Users can provide and interact with data and share with the other peers. The design of the system within the current technologies and building the application with a minimum amount of complexity are desired. Several different hardware and operating systems cause a heterogeneous environment for a hard to develop collaborative environment. The system aims to provide a solution over the network and operating system complexity with ongoing standardizations.

25 Project Scope The primary task of this project is showing collaboration and manipulation capabilities of the augmented reality system. The project runs on three different hardware which are Windows PC, Android smartphone, and Google Glass. This system can be expandable for various hardware and operating systems with the same architecture. Devices use their interaction methods to provide collaboration. This project is not going to provide multiple types of augmentations. The current system only provides visual augmentation for displaying manipulation of the data. However, the system can be expandable for future developments.

26 21 CHAPTER 3 SYSTEM DESIGN AND INTEGRATION An Augmented Reality (AR) aims to offer the ability to integrate virtual entities into the real environment in real time. Developing an AR requires the use of different interaction and registration - tracking algorithms, hardware (sensors, cameras, tracking devices, feedback equipment) based on the application. Because of these multiple heterogeneities, several AR s dedicated frameworks and solutions are developed. However, the AR is growing and has increasing complexity based on application. For solving these problems, component based distributed solutions are designed. The AR has specific needs for developing this modular system. Based on these requirements MacWilliams et al. [42] analyzed and created a generic architecture for augmented reality applications. As seen in Figure 2, it aims to break down any AR system into six core components (application, interaction, presentation, tracking, context, and world model). Separating components allows re-usability of the system parts, and abstract patterns. Functional independence allows developing and improving varying architectures and numerous algorithms for different components. This system requires several applications depending on the use cases. In the development process, Windows OS, Android OS, and Android OS for Google Glass are used as a client platform. In the Windows side development, C++ is used as a language preference. For Android and Google Glass side of development, Java is used as a language preference. Communication between the applications is provided with Alljoyn IoT library.

27 22 Figure 2.: Generic augmented reality reference architecture

28 Actors Presentation Actor The purpose of the presentation actor is to provide data to users. The user can see required information from presentation actor. In this project, a visual display is used for a presentation. Computer generated graphics visualize to the user. In the project, generic graphical elements and teapot are utilized for an example. In the development procedure, OpenGL is used for Windows based actor solutions. In mobile versions (Android and Google Glass) OpenGL ES is used for graphical content generation Interaction (I/O) Actor Interaction actor allows the user to interact with data. Interaction can be provided with different input devices. Touch screen, keyboard, voice recognition can provide user input to manipulate or create content. In this project, Android application with touch screen based gesture recognition is used to rotate the 3D generated content. User s keyboard inputs also provide the same manipulation on the desktop version Tracking Actor Tracking actor provides tracking and registration of the augmented data. Tracking can be provided with image registration or sensors like GPS. Unified sensors in the system can be used with several AR clients. Tracking actor provides the ability to integrate standalone tracking and registration system for single or multiple clients. In this project, image recognition based tracking solution is used. For this

29 24 purpose, mobile versions are used Vuforia AR tracking library and Windows version is used ARToolkit AR tracking library to generate tracking. Image-based tracking requires an image with a high number of feature points to be effective. The sample image is created to provide effective tracking Things Actor Things actor is the possible data source for the system. Things actor covers the need of non-ar based module needs. These actors are an additional component of the system. Based on the application needs things data provide non-ar related data flow to the system. Things are data sender and receivers with no other features. 3.2 Manager The purpose of the manager is allowing communication between actors and be able to manage the augmented reality system. Every peer has their manager. The overall architectural scheme provides user based application design. The manager controls all actors and main communication. Control of the manager allows the user to control its communication and application flow. Other AR modules communicate with the manager. The Manager also handles communication between peers. 3.3 Collaboration In this system, collaboration is considered in two different aspects. When users need to collaborate each other, their whole AR system needs to communicate each other. In this architecture, managers are handling system control. Because of this, user-based collaboration is called collaboration of managers.

30 25 Figure 3.: Collaboration of managers with peer to peer Also, in some cases, different AR systems can use same actors or actor s architecture to provide their roles. The situation is called collaboration of actors Collaboration of Managers As previously explained, managers define different peers. For collaboration among peers, communication between managers is required. Communication can be established both peer to peer and server architectures. As seen in Figure 3, managers can communicate each other with direct communication. In this architecture, managers listen to possible communication hosting, inside the network. If there is no host broadcast inside the network, the manager declare itself as a host and broadcast communication between peers. Name prefixes of AllJoyn network to provide different communication channels for the manager to manager and manager to actor communication. Also, as seen in Figure 4, managers can communicate each other with predefined collaboration server. In this architecture, similar to peer to peer architecture managers listens to possible communication hosting inside the network. Therefore, there

31 26 Figure 4.: Collaboration of managers with collaboration server is predefined server for communication, managers don t need to host communication themselves. They find hosting server and join the communication Collaboration of Actors Actors may need to communicate each other for different purposes. For example, actors may provide a different type of augmentation to the user. Managing different augmentation actor type on the same manager allows user based architectural design. Managers can have a similar type of actors for various purposes. Sending same type of data to related actors are inconvenient and causes unnecessary traffic in the network. To solve the network traffic issue, actors can be managed by the additional server to control traffic over actors. Additional layer allows actor based communication and customization possible for actors data. As previously explained, especially image processing based tracking and registration requires high computation power. In addition, things actors may need client server architecture in the system. Figure 5 symbolizes possible overall communication between managers and actors. Peer to peer communication of actors may also require for different application

32 27 Figure 5.: Collaboration of managers with actor server architecture Figure 6.: Actor peer to peer architecture

33 28 Figure 7.: Shared actors designs. For example, standalone tracking sensors may communicate each other to better tracking and registration. In this process, the actor can be used by several managers. Allowing actors sharing decreases unnecessary data and communication needs. As previously explained, actor sharing is depended on the purpose of an application and may or may not be needed. 3.4 Software Requirements Windows Language: C++ Operating System: Windows IDE: Visual Studio 2015

34 29 Libraries: Alljoyn 15.04, OpenGL, Artoolkit 5 Android Language: Java Operating System: Android 5 IDE: Android Studio Libraries: Alljoyn 15.04, OpenGL ES, Vuforia, Android Support Glass Language: Java Operating System: Android 4.4 IDE: Android Studio Libraries: Alljoyn 15.04, OpenGL ES, Vuforia, Glass Development Kit (GDK) 3.5 Hardware Requirements Processor: X86 (Windows), ARM-Qualcomm (Phone-Glass) Wi-Fi: IEEE a/b/c/n/ac Camera: Standard camera HDD: 100MB

35 Implementation Windows Application Window application uses ARtoolkitNFT library for registration and tracking purposes. Communication uses Alljoyn standard core bus technology. For manager communication, org.alljoy.bus.comm is used for bus name prefix. All communications run on port 27. The difference between actor and manager communication is the different name prefixes.if the separate server handles the communication, server extends the name prefix with server name tag for declaring itself. In the other case (managers declare themselves as a server), server manager extends the name prefix with CollArHost name tag to differentiate between independent and self-initiate server. While controlling outside actors, manager server uses org.alljoy.bus.actor to communicate with actors. When a manager wants to communicate with the others, firstly it looks for possible running servers. If it is able to find, it connects to the server. If the manager cannot find the server, it self-initiates and creates a server and connects its self-defined server. Figure 8 shows this algorithm. In this project, all data transmission is provided as a string. When data transmit, the first word provides the type of the data; the second word declares the data.

36 31 Figure 8.: Connecting the Alljoyn server Android Application ActivitySplashScreen activity is a dummy activity that provides splash screen before application run. The manager runs CollAr class which is the main activity class for the application that runs all submodules inside of it. CollAr calls all the core classes (Registration/tracking, communication, rendering) and handles interaction inside itself. Also, contains left slide menu which is created with AppMenu class. AppMenu class uses AppMenuGroup class to add views as a component. Animations of the AppMenu managed by AppMenuAnimator class. Rendering of the 3D graphics is handled by OpenGL ES on CollArRenderer class.colllarrenderer implements GLSurfaceView.Renderer to provide OpenGL ES library. CollArRenderer uses shader based system to generate 3D graphics with OpenGL ES.

37 32 CollAr class implements a vuforiacontrol interface to provide Vuforia library s registration and tracking capabilities. Vuforia provides pre-trained natural feature based image registration and tracking capabilities. Android application is using the touch screen for interaction purposes. When interaction happens, data sent to the server. Android application uses same communication settings as a Windows application. To communicate with other application broadcast need to enable on left swipe menu. After that Figure 8 applies as similar as a Windows application Google Glass Application Google Glass application uses immersion mode to provide ongoing activity on the screen. The application can be started with Google Glass menu or OK Glass > Show me a demo with > Collar LiveCard voice command chain. When the application runs, CollArService starts. CollARService runs CollarGlass activity that contains all of the apps features like CollAr activity on Android smartphone version. Same as a smartphone app, CollArRenderer is used OpenGL ES for rendering. Unlike smartphone application, Google Glass application initiate communication as soon as it starts.

38 33 Figure 9.: CommClass

39 34 CHAPTER 4 TESTING AND EVALUATION 4.1 Unit Testing All of the methods of each class in client and server have been tested individually to avoid exceptions while run time and have been handled properly in the try, catch blocks of the Java. Also, logging enabled with different tags for different classes inside the code for easier debugging and problem solving. Logging provided collecting specific unit exception on the system. Exceptions are levied on development stage. Applications on the different devices should work and register AR without a problem.figure 10 shows the registering and rendering 3d model on display. Figure 11 shows side menu of the application. Side menu allows the enabling connection, managing outside actors. Figure 12 shows registering and rendering 3d model on display. Unlike smartphone version, glass menu doesn t have managing tools. Whenever the application runs, it creates communication procedures. 4.2 Integration Testing In this system, different application types communicate each other on the network. Communication between applications is separated with different communication name prefixes. Ability to communicate on specific name prefixes allow communication integrity. Integration Testing was successfully implemented in this client server architecture to check the function of the name prefixes on a specific port. Integration and possible server connection can be tested with test server logs

40 35 Figure 10.: Smartphone screenshot

41 36 Figure 11.: Smartphone side menu

42 37 Figure 12.: Google Glass screenshot and runtime app logs. Android and Google Glass applications are generating log records on android activity monitor. Also, every data transmission on the server is written on standard io. Figure 13 shows an example data stream on the server. 4.3 Tracking Testing Augmented reality is highly dependable on registration and tracking techniques. In this project, two different visual augmentation libraries used. Visual augmentation is usually implemented with image processing techniques. In this project, Vuforia and ARToolkitNFT augmented reality libraries are used. Both Vuforia and AR- ToolkitNFT support natural feature point detection from the video feed. For proper augmentation, selected registration image should be trained carefully.the high number of feature point extraction is desirable. In this section, generated feature point is compared.

43 38 Figure 13.: Manager server s data traffic logs Registration of the augmented reality requires easy to use feature detectable images. Image selection for registration is an essential to the process of augmented reality application development. Feature detection of the image is also tested. Figure 14 shows Vuforia s generated detectable features of selected registration image of the system. ARToolkit also supports natural feature point extraction with ARToolkitNFT extension. Figure 15 shows trained feature points for ARToolkitNFT. Both ARToolkitNFT and Vuforia can generate enough feature point to provide effective tracking. However, Vuforia has a better success rate in real time experiment.

44 39 Figure 14.: Vuforia feature detection

45 40 Figure 15.: ARToolkit detected feature points