Evaluation of Advanced Mobile Information Systems Falk, Sigurd Hagen - sigurdhf@stud.ntnu.no Department of Computer and Information Science Norwegian University of Science and Technology December 1, 2014 1 Introduction The rapid evolution in mobile information systems changes not only the way new systems are developed, but also users expectations to new systems. Ubiquitous computing (ubicomp) and augmented reality (AR) are examples of such systems that, if widely accepted by users, will change the way we think of and interact with mobile devices and computers. These two categories of systems have yet to be a part of users everyday use of mobile information systems. A measure for speeding up development and adoption of such systems is for researchers to evaluate their work on their respective technology based on standard evaluation frameworks. User satisfaction need to be an essential part of such frameworks. Standard frameworks for evaluation represents a way to easily reuse research and ideas. This paper compares three papers on ubicomp and AR and frameworks for evaluation. First, the relations between the papers are discussed. Second, how they relate to mobile information systems are presented and finally, a conclusion summarizes the findings. The papers in question are presented below: The Computer for the 21st Century The author of this paper, Weiser [1999], are referred to as the father of ubicomp. He presents his visions for this technology, presents his work on a early implementation of a ubicomp workspace and points out both hardware and social barriers that need to be overcome for this technology to reach its potential. Mobile Augmented Reality for Cultural Heritage: A Technology Acceptance Study 1
Haugstvedt and Krogstie [2012] identifies the lack of effort to sufficiently evaluate the user acceptance of AR applications with cultural heritage resources such as historical pictures and information. They developed an AR application using historical local photographs of Trondheim. Using device GPS an camera the application give users the ability to see how structures look like in a historical perspective. Using this as an artifact they present a model for evaluating such applications. To confirm their hypothesis both a Internet survey and street interviews were performed. Toward a Framework for Evaluating Ubiquitous Computing Applications Scholtz and Consolvo [2004] introduces his paper by pointing out the lack of a widely accepted framework for user evaluations of ubiquitous applications hampers their efforts. He proposes a user evaluation framework developed specifically for ubiquitous computing to help researchers compare results, create ubiquitous computing design guidelines, develop effective discount evaluation techniques, understand the appropriateness of different evaluation techniques, and develop a more complete structure so they can avoid overlooking key areas of evaluation. Finally the proposed framework is tested on a number of existing applications. 2 Discussion The discussion focuses on similarities between the papers presented in the introduction. Relevant technologies will be compared and how the emergence of these technologies reveals a need for methods for evaluation in order stimulate further enhancement are presented. Finally these papers relevance to mobile information systems are discussed. 2.1 Relation between the papers 2.1.1 Ubiquitous computing vs. augmented reality In the papers presented, two different technologies are present - ubicomp and AR. In this section both technologies will first be defined and then compared. Ubicomp was first articulated by Weiser in 1988. He is described as the father of ubicomp. In his vision, ubicomp is roughly the opposite of virtual reality. Virtual reality forces people to live inside the computers world, ubicomp forces the computer to live in the human world. To accomplish this the computer has to be invinsible. It has to exist in a natural way so that people can use it without thinking about using it. On the other hand AR aims to enhance our view of the world by superimposing virtual objects on the real world in a way that persuades the viewer that the virtual object is part of the real environment [Butchart, 2011]. The 2
application developed by Haugstvedt and Krogstie [2012] is a good example of this type of technology. They project information in form of images and text alongside real world objects of Trondheim s streets captured from the device camera. Another exiting example is Google Glass developed by Googles. AR changes the way information on mobile devices is presented. To precept virtual object, the user has to see through a invisible screen these objects can be projected on. As explained above, camera feed from a mobile device or glasses such as Google Glass are commonly used to achieve this. To avoid projection of to much information, the user also has to interact with the virtual objects, expanding or minimizing them to extract the appropriate amount of information. Ubicomp build on the idea of the computer to be invisible. This requires other forms of input such as the user is walking, the user is driving a car and the traffic is heavy. By incorporating such input, computers will dynamically customize to meet the needs at hand in a given situation. In the future these technologies might work together. AR provides a intuitive and informative interface for decisions made and information accumulated by a ubicomp system. However, by doing so ubicomp will surely lose some of the early intentions visioned by Weiser. AR s need for a screen to project virtual objects denies it to be a invisible force implemented in peoples everyday life. Both systems provide a useful way of enhancing interactions with computers by presenting information in forms more intuitive to people. To find out if combining them further improves their abilities tests of user acceptance and evaluations is needed. 2.1.2 Accepting mobile information systems When Weiser [1999] wrote his paper on ubicomp, hardware needed to fulfill his visions was not yet invented or existed on a not satisfying level. Nevertheless he states requirements for such systems and tries to predict the development of relevant technologies. An example of this is quoted below: Flat-panel displays containing 640x480 black-and-white pixels are now common (...) As long as laptop, palmtop and notebook computers continue to grow in popularity, display prices will fall, and resolution and quality will rise. By the end of the decade, a 1000x800- pixel high-contrast display will be a fraction of a centimeter thick and weigh perhaps 100 grams. A small battery will provide several days of continuous use. Indeed his prediction was correct. Weiser lay down standards for developing ubicomp systems. These standard are still accepted and are a foundation for developers, but the rapid evolution in hardware and the development of usable ubicomp systems presents the need to further refine these standards and also stimulate to produce new rules regarding such systems. To properly implement Weisers vision of ubicomp there is a need to possess large amounts of personal information about the users. Today, the distribution of shared sensitive information to third parties, for example to targeted marketing companies, has made 3
users more concerned about their privacy and more retracted when sharing information [Chin et al., 2012]. There are also a need to improve pure technical aspects such as how systems interact with users and how systems interact with systems [Bellotti et al., 2002]. Models for evaluation and measurement of user satisfaction are important to improve existing systems as well as when developing new systems. Both Haugstvedt and Krogstie [2012] and Scholtz and Consolvo [2004] recognizes the lack of effort to define good models for evaluation. As described in the introduction, Haugstvedt and Krogstie focuses on acceptance or willingness to use AR applications with cultural heritage resources. Their research model is shown in figure 1. Figure 1: Haugstvedt and Krogstie [2012] research model. The boxes in the figure represents the major parts in the questionnaire and arrows represents positive relations between boxes. These relationships were used as hypothesis for evaluation. Results indicates that people would like to use mobile AR applications with cultural heritage resources both for enjoyment and to achieve some learning goal. They also found that fun and usefulness are important incentives for people to use such applications. Scholtz and Consolvo hope to improve comparability across research efforts with their proposed framework of ubicomp evaluation areas. They highlight the importance of a common framework for researchers in order to enhance the development of ubicomp. Their framework incorporates areas from previous research on different fields of ubicomp. It is built on conceptual measures (an observable value) and metrics associated with that value for human judgement. When evaluating there must be two groups of users effected by the system, stakeholders and indirect stakeholders, to make sure everyone affected by the evaluating application is counted for. Interaction, impact and side effects and appeal are areas incorporated in the framework. Some metrics among these areas are user satisfaction, utility and fun. Hence, there are some similarities to 4
what Haugstvedt and Krogstie [2012] found is important incentives to use AR applications. They also propose trust as an area of evaluation, taking Weiser [1999] initial concerns about privacy regarding ubicomp into account. Their results when applying the framework to existing applications show promise and the need for further enhancement. The papers involved in this discussion all includes exiting technologies for mobile information systems, but in essense they try to highlight the importance of pointing development of these technologies in the right direction. Weiser and researchers at PARC worked with an early ubicomp system. They identified the need for new and better hardware for them to accomplish their visions and goals. Several years later, this hardware has emerged and ubicomp applications are allready a part of the daily life for most people. This presents new challenges. Researchers are trying to implement systems in the way Weiser envisioned. These systems should be used as benchmarks for the development for further research and evaluation. As presented earlier there is a need for better frameworks for evaluation of such systems. User satisfaction need to be weighted high in such evaluations. Researchers need to establish a common ground on how to do this. They need to agree on terms and methods which will lead to the ability to reuse and enhance ideas in the community surrounding advanced mobile information systems. Haugstvedt and Krogstie [2012] and Scholtz and Consolvo [2004] have made their models based on previous research and made them fit their systems. Their approach fronts evolution above revolution, hence reuse and improvement of other researchers ideas. 2.2 Relation to Mobile Information Systems The curricula of this course focuses primarily on work on technologies in the field of mobile information systems. Many of these technologies are being used but has yet to reach their full potential. Krogstie et al. [2004] states that due to the arise of devices constantly connected to the Internet it is predicted that the way people use information resources will be radically transformed. Ubicomp, AR and smart environments are examples of technology aiming to provide better use of such devices. However, the adoption of many advanced mobile services has been slower than expected [Carlsson et al., 2006]. This might be an consequence of developers not being able to adapt to users expectations. Hence frameworks for evaluation of user satisfaction and information describing pattern of use are needed. As described in the previous chapter, these frameworks need to be standardized and specialized to fit different types of applications. Proper implementation of good privacy mechanisms is crucial when developing systems that collects personal information. Modern mobile systems has to be able to gain users trust. This is a problem addressed in almost every paper in the curriculum. It is easy to imagine how a network of devices with a variety of sensors could be misused. Large scale surveillance is an obvious example of this. When collecting and storing information it is important to notify the users how this information is used, hence minimizing the asymmetry in information flow [Xiaodong Jiang, 2002]. To successfully evaluate systems, user satisfaction 5
regarding privacy should be taken into account. This is an area that probably will experience a lot of changes if systems like ubicomp are successfully adopted. Hence the need to constantly evaluate systems with respect to this subject is important to keep up with changes. 3 Conclusion Efforts to improve evaluation of new technologies in mobile information systems are needed to speed up development as well as achieve a higher degree of user acceptance. Both AR and ubicomp tries to bring the computer closer to peoples everyday life. This will definitely redefine the way we think of computing today. Mark Weiser argue the most profound technologies are those that disappear. Development without continuing user feedback probably wont be able to achieve this. This involves interviews of people initially unaware of the technology and lab studies. How such methods are performed should be guided by widely accepted frameworks. Doing so will make research comparable and form a foundation for improving system performance and interaction with users. It will also most likely speed up the development of advanced mobile information systems and help to overcome difficulties and barrierss. References Victoria Bellotti, Maribeth Back, W. Keith Edwards, Rebecca E. Grinter, Austin Henderson, and Cristina Lopes. Making sense of sensing systems: Five questions for designers and researchers. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2002. B. Butchart. Augmented reality for smartphones. Technical report, JISC Observatory, 2011. C Carlsson, J Carlsson, K Hyvonen, J Puhakainen, and P Walden. Adoption of mobile devices/services searching for answers with the utaut. IEEE Computer Society, 2006. Erika Chin, Adrienne Porter Felt, Vyas Sekar, and David Wagner. Measuring user confidence in smartphone security and privacy. Proceedings of the Eighth Symposium on Usable Privacy and Security, 2012. Googles. Google glass. URL https://www.google.com/glass. Visited November 25, 2014. Anne-Cecilie Haugstvedt and John Krogstie. Mobile augmented reality for cultural heritage: A technology acceptance study. Proceedings of the 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2012. 6
John Krogstie, Kalle Lyytinen, Andreas Lothe Opdahl, Barbara Pernici, Keng Siau, and Kari Smolander. Research areas and challenges for mobile information systems. Int. J. Mob. Commun., 2004. J. Scholtz and S. Consolvo. Toward a framework for evaluating ubiquitous computing applications. IEEE Pervasive Computing, 2004. Mark Weiser. Ubiquitous computing. URL http://www.ubiq.com/weiser/ubihome.html. Visited November 25, 2014. Mark Weiser. The computer for the 21st century. SIGMOBILE Mob. Comput. Commun. Rev., 1999. James A. Landay Xiaodong Jiang, Jason I. Hong. Approximate information flows: Socially-based modeling of privacy in ubiquitous computing. 2002. 7