Methodologies for researching the usability of applications within mobile communities Eric Duran Helsinki University of Technology eric.duran@hut.fi Abstract Mobile communities are different from traditional social networks. They have unique characteristics such as being accessible from anywhere at any time, and the advantage of not requiring face-to-face meetings in order to develop their inner links. Consequently, the research of mobile users behavior poses new challenges due to the mobile nature and unique features of these users. Researchers have to decide whether to take a passive role while observing the subjects of their studies or participate actively in their activities in order to have a better understanding of their behavior. Since the use of traditional methods for studying mobile users behavior might not necessarily yield the best results, researchers may use alternative methods for applying the advantages provided by modern technologies. Finally, the location of the research is no longer limited to the laboratory since researchers may conduct their tests and studies in a non-intruding fashion, within the user s environment, as the result of applying technology innovations. This study presents and compares different alternatives to the previously mentioned research issues. KEYWORDS: Mobile users, research methods, usability. 1 Introduction During the last decade, we have seen an incredible growth in the use of mobile devices. The customer base of these devices has shifted from typically high-profile niches such as top business executives to all kinds of sectors such as college students, team members distributed across different geographical locations and even parents that give pre-paid cell phones to their children as a medium for reaching them in emergencies [1]. Mobile devices have become very popular since their prices began to decrease, making them more accessible to the mainstream market. Consequently, the extensive use of these devices has created a new social group that merges some of the characteristics of virtual communities and mobile communications. Mobile communities are the result of this merge. There are unique features that make mobile communities different from other kind of communities. First, mobile communities use telecommunication networks that are accessible from anywhere, at any time. These mobile networks allow the members of the community to be instantly available to each other without requiring them to be present in the same geographical location. Secondly, members of a mobile community share common interests, bringing them together to coordinate their actions, as a group, in order to explore their affinities [2]. Because of the previously mentioned characteristics, traditional methods may not be the best choice for researching the behavior of members of mobile communities. Therefore, researchers should use alternative, unconventional methods that are more appropriate for mobile research situations [3]. In a similar manner, researchers have been conducting their experiments in laboratory settings, but the dynamic context and physical distribution of mobile users make fieldbased evaluations more appealing, and in some scenarios, even necessary [5]. 2 Research methods for mobile communities Research methods for studying communities have been in use for a long time within sciences that deal with humanities studies [1]. Ethnography is a complete, formal, research method that aims to understand the cultural practices and social phenomena that take place within a social system. Furthermore, the results of sociologic and cultural studies are the input of ethnography studies that have been in use for business purposes showing different characteristic attitudes, behavior patterns and values for the categories participating in common social systems. Other research methods share the same objective and through different techniques, they are able to product a variety of results that together, provide a holistic understanding of a community. The following subsections present some traditional and alternative methods for researching the behavior of mobile users, and compare their advantages and drawbacks. In addition, they describe supporting technologies and group scalability of each method. 2.1 Traditional methods 2.1.1 Ethnography One of the possible uses of ethnography, as previously said, is to conduct long-term studies where researchers immerse themselves in the lives of the people they study, keeping records of the subject s activities while usually having minimal impact on the situation under research.
However, this approach is very time consuming because it demands complete availability from the researcher in order to spend, at least, the same amount of time for tracking the subject s activities as the subject of the study uses in performing them. In addition, when studying a mobile community, making record notes might be difficult because of the constant and rapid movement between different locations. Finally, there is a risk of ending up with large quantities of unstructured descriptive data that researchers might not be able to use for testing future scenarios [5]. On the other hand, ethnography has the benefit of providing very rich accounts of behavior and an improved understanding of the way in which work is socially organized, which makes easier the task of identifying undiscovered trends. Nevertheless, researchers have not used this method remotely, that is, researchers have to be on-site, very close to the participants of the study. Furthermore, the use of ethnography as a research method might not be appropriate for a mobile environment but it is suitable for studying a group of subjects at the same time [5]. 2.1.2 Laboratory experiments Another traditional research method is laboratory experiments. These experiments aim to test an interface in an artificial, controlled environment. Because of its isolated nature, experimenting in a lab allows for precise metrics; therefore, comparisons and evaluations are more accurate. Moreover, external factors are less likely to affect the results. However, the artificial environment may make people uncomfortable or act differently, and some activities are not feasible to do in the lab. Besides, researchers cannot conduct laboratory experiments remotely nor while moving, although this method applies to group researching. 2.2 Alternative methods The following subsections will explore research methods that utilize unconventional approaches for analyzing and testing the behavior of mobile users. 2.2.1 Researching backpackers While researching community interaction amongst backpackers, researchers Jeff Axup and Stephen Viller [6] realized that existing methods have difficulties when applied to mobile groups. Therefore, they decided to adapt common methods and try less used ones as part of their research process. Backpackers are travelers that move frequently across scattered locations and for long periods, but they wish to organize group activities when they make a stop to enjoy a particular place. However, they encounter challenges such as the lack of collaboration methods available to them, and since many of them never meet each other face to face, it is very difficult to exchange information although they share common needs when visiting the same places at different times. Based on contextual interviews and site surveys, the researchers discovered that social ties depended on location and circumstances; therefore, they stopped conducting evaluations at only one place and time and decided to try alterna- Figure 1: Backpackers discussing how they build social relatioships between them. Photography taken from [6]. tive research methods. Figure 1 depicts the use of an interactive social network board in order to have a better understanding of the relationships between backpackers. One of these methods was a participatory social pairing exercise. First, the researchers created small groups of backpackers that shared an affinity, such as intending to visit the same place. Those backpackers that had already visited a place of common interest to the group talked about their experiences in that place. Then, the researchers discussed about trust of travel information and possible uses for an information sharing system between backpackers. However, the pairing process was more demanding than expected, because it required plenty of time and knowledge of local geography. In the end, this exercise provided a good example of a social pairing system because it demonstrates the benefits of social networking, in spite of the enormous efforts it requires to build a well-interconnected network. 2.2.2 Digital ethnography Another alternative method is the one created by adapting a traditional one: digital ethnography. Researchers Davis Masten and Tim Plowman [1] used remote sensing to conduct their research because sensing technology has become cheaper and it allows getting richer, more detailed data. With the Internet revolution, researchers can connect to multiple physical points at the same time. The main difference between traditional ethnography and digital ethnography is that digital ethnography can gather data from participants without location or time restrictions.
Therefore, digital ethnography makes possible for researches to conduct simultaneous and remote researches. Consequently, their Valentine s Day case study used multiple techniques including email, cell phones, digital cameras, chat rooms, online questionnaires, and digitized audio diaries to gather data about user s behavior. In this study, the participants were more involved in the analysis of their own behavior, which allows them to realize the richness of their own observations. Sharing the results of the study encourages them to become active contributors to the research project. 2.2.3 Fashion as a research tool Finally, in March 2003, a team of researchers of the Interaction Design Institute Ivrea [7] worked on an unconventional research project. They base their work on the fact that mobile devices have become very popular, but nobody has researched the consequences of their extensive use on the surroundings of the user s environment. Therefore, they worked on a project that creates awareness of the impact of mobile communications on the user s environment. In order to achieve the project s goal, the researchers decided to use clothing as a form of illustrating how mobile phone calls affect our private space. They chose clothing because people use clothes to make social and fashion statements, aside from the traditional use of covering the body. Figure 2: Fashion as a research tool. Picture taken from [7]. Next, the researchers designed a bag that reacts to mobile communications within the user s environment. The bag has electronic components that sense the surrounding radiations originated from mobile calls and it displays a bigger or smaller stain based on the amount of radiation sensed. Figure 2 shows how the stain mechanism functions. The researchers tested the bag in an exhibition, as well as in many public spaces in the city of Torino. The results of the experiment showed that the bag fulfilled its goal by revealing how a mobile phone call is an intruder of private spaces in a public context. However, the experiment failed to create a reaction on the person making the phone call because the bag was not able to draw her attention. 3 Usability evaluation of mobile applications Since laboratory facilities have been the traditional location of user interface experiments, it is necessary to investigate if a laboratory environment can help to identify problems related to the mobility characteristics of mobile applications (described already in section 1). The following section compares the results of evaluating a mobile application s usability in the laboratory versus in the field, and explores new techniques for usability evaluation of mobile systems. 3.1 Comparison between laboratory and field testing Usability researchers gather information about the natural way of interacting with an application by asking users to describe how they accomplish given tasks. Usability tests are important because they help to identify severe problems that prevent users to perform a task. Usability test laboratories are isolated, quiet areas where the user can focus on performing the requested tasks while being under observation by researchers. Consequently, these facilities lack the interruptions, movement and noise that mobile users face in the real world. Therefore, usability studies should consider these requirements, that is, taking the effect of distractions and interruptions into account when designing usability tests in order to replicate the characteristics of the environment that potential users of the application will face. Meanwhile, testing in the field has become easier and cheaper, although researchers still conduct the majority of usability tests in the laboratory because it is less time and effort consuming. Thus, the objective of the study developed by a group of Finnish researchers [4] is to discover if field tests are critical when testing a mobile application s usability or if laboratories can simulate mobility characteristics sufficiently well. This study compares a low-budget laboratory test with a field test, using otherwise similar conditions and with similar tasks and methodology, such as think-aloud method. In order to make a clear distinction between the laboratory and the field, the former did not have any modifications for simulating field characteristics and the application under evaluation was a consumer application, where users can explore and be more creative while performing the tasks. The evaluations consisted of giving predefined tasks to 20 users in a laboratory and asking them to think aloud what they did, their expectations and unexpected experiences they encountered. There were four moderators giving written instructions to the users. The profile of the users was young people (22 to 35 years old) with experience using mobile devices and familiar with Nokia s Series 40 phones. The laboratory test did not have any unexpected situations. Three video cameras recorded the user s movements, the phone s keyboard and display, and an overall picture of the user. The moderator gave instructions to the user in oral form; then, the user described the actions performed in order to accomplish the task.
On the other hand, the field tests involved traveling from Ruoholahti to Itäkeskus (districts in the city of Helsinki), cross a busy street during rush hours, walking in a large shopping center, while executing the tasks given by the moderator. The moderator used a video camera to record the movements of the test user when it would have been impossible to see them. In a similar fashion as in the laboratory environment, three cameras recoded the user s expressions, phone and surroundings. The tested application was still under development, so the users did not have previous knowledge of it. The application, called Mobile Wire, allows the user to transfer files between computers and the mobile device. The requested tasks involved a series of steps required in order to download the application, view images shared by friends, take pictures for later use at home, give right to friends to view a saved picture and delete the application. The results of this comparison are very interesting. The researchers found identical problems in the laboratory and in the field. Nine of these problems were critical, making impossible for the user to finish the task, while ten detected problems were severe enough to cause problems in the application s use. Next, the researchers noted a statistically significant difference in the number of occurrences of problems #5, #12 and #19. These problems were more common in the field. However, the severity of the problems found in the field was, on average, the same as the severity of the problems found in the laboratory. Similarly, the time required for executing the tasks was the same in both environments. Accordingly, the results prove that the main difference between the two environments is on the qualitative properties found during the execution of the tasks. For instance, when a task required more attention from the users in the field, they stayed still or stepped aside in order to focus on the task. This clearly signals that level of difficulty of the task was higher. This insight is not possible to obtain in the laboratory. Another example is the different behavior exposed by the users in the field while waiting for the completion of an application s tasks. The users tend to get distracted by the environment, so the application s design has to be careful enough to avoid short-timed alerts that the user may miss. Finally, the study concludes that there are no significant performance differences between the laboratory and the field, but total time required for conducting the testing in the laboratory is less than the required in the field. The main reason of this difference was that the moderator needed more time to travel from each location to the next one. In summary, testing on both locations yields the same results. Evidently, field-testing is the recommended method when testing not only the usability of a system, also user behavior and environment. However, field-testing requires more time for preparing the experiment and the duration of the experiments is longer. 3.2 New techniques for usability evaluation Figure 3: Problems found in usability tests of a mobile application. Based on results from [4]. A study of mobile human-computer interaction shows that 71% of the mobile system s evaluations take place in a laboratory, and very few of these laboratory evaluations engage in customized techniques developed to meet the unique characteristics of evaluating a mobile system [8]. Capturing key situations in the user s context is not easy with field-based evaluations. First, it is difficult to apply well-known, conventional evaluation techniques such as observation and think-aloud in a field setting. Second, many unknown variables could potentially affect the setup of field evaluations, making the process of collecting data more difficult. The traditional concepts that describe mobility (such as application, type, and context) are difficult to use for recreating mobile scenarios in the laboratory. Instead of focusing on these traditional concepts, Kjeldskov and Stage have created
two different frameworks for defining and testing mobility. In the first framework, they define mobility with a matrix having in one dimension the attention needed to navigate, with values such as none or conscious. In the second dimension, the matrix has values for the type of body motion, such as none, constant and varying. For each cell in the matrix, they have a support method for simulating the conditions described in the cell, such as walking on a treadmill, or stepping on a stepping machine, or walking on a track with obstacles. As another alternative, the notion of divided attention is the basis of the second framework. The main idea behind it is that users divide their attention between physical motion and the use of a mobile system. Therefore, the framework aims to recreate a setting where the user has to perform a cognitive task that serves as a distraction from the use of a mobile system. For the first framework, the researchers conducted six usability tests; one for each combination of attention needed and body motion and the last one as a typical real world situation that served as a reference for the other techniques. The test asks the user to solve a number of predefined tasks using a mobile system in 10 minutes. Each evaluation collected data about usability problems, performance (time spent on each task) and workload (user s subjective experience of the task s workload and the factors that contribute to it). The purpose of the experiment is to prove that the ideal laboratory techniques would not differ from the "real life" technique 6 in terms of the previously described measures. Regarding the first measure, which is usability problems, the results of the tests showed that users with no body motion and no attention needed to navigate discovered significantly more usability problems (specially, cosmetic problems) than the users tested with any of the other techniques. However, every technique supported the identification of usability problems for each category (critical, serious and cosmetic). For the second criterion, the researchers measured the time spent on solving each of the five tasks. The results showed no significant difference between the techniques. However, the technique with the fastest task completion time changed from task to task. Finally, the workload differences showed between the techniques were very large. The less physical demanding technique (sitting at a table) required less than half of the effort required by the most demanding technique (walking in a pedestrian street). However further comparisons between the techniques regarding physical demands, effort, and overall workload do not reveal any significant results, aside from the fact that varying speed and conscious attention effectively put a greater workload on the test subject. For the second framework, the researchers designed an experiment in order to compare two mobile phones. The experiment should prove if testing in the laboratory supports discovering more usability problems than testing in the field. Accordingly, the researchers simulated mobility in the laboratory by requiring the user to move on a mat connected to a computer game. The results of the experiment of the second framework show that testing in the field helped to discover critical problems that were not identified in the laboratory, although the total amount of errors found with both techniques is roughly the same. Moreover, the performance of the tasks executed in the laboratory proves that the user had to spend more time on them because the action (namely, dancing on the mat) was more distracting than the environment in the field-testing. In conclusion, the previously described techniques aim to explore different alternatives for recreating challenges of physical movement while using a mobile system. The results of the experiments showed that all the techniques are relatively similar in terms of workload, required time to perform given tasks and amount of usability problems detected. 4 Conclusion We have seen that traditional research methods, such as the ones described in section 2.1 may yield too much information that reveal very little key findings about the interaction of users with mobile applications. Furthermore, these methods may not even be appropriate for researching mobile usability because they do not support remote use, mobile environments or group settings. On the other hand, alternative research methods leverage the benefits of technology in order to address the challenges of distributed, mobile communities with a creative approach. As described in section 2.2, researchers are continuously seeking for novel, and even unconventional methods that could improve the evaluation of the usability of an application. Therefore, finding innovative methods or adapting common ones should be a standard part of the research process of mobile communities. Whether researchers should take a passive role while observing the subjects of their studies or participate actively in their activities is a matter of the chosen research methodology. For techniques that take place in the field, such as the one described in section 2.2.1, it is clear that researchers should take an active role in the experiment and immerse themselves into the user s environment in order to be fully aware of the context s impact. Based on the comparison presented in section 3.1, we can conclude that evaluating the usability of mobile applications in the field or in the laboratory, lead to similar results. In both locations, researchers are able to discover the same amount of usability problems with similar severity. However, the main difference is the amount of time necessary to conduct the experiments in the field, since it requires a considerable larger amount of time than testing in the laboratory. Clearly, field-testing does not provide enough benefits to justify the required extra effort and time. Thus, laboratory testing remains as the preferred location for conducting usability research. Finally, researchers should feel encouraged to explore new techniques that could facilitate evaluating mobile systems in a controlled setting while being as similar as possible to a natural mobile environment.
References [1] Davis Masten and Tim Plowman. Digital Ethnography: The Next Wave in Understanding the Consumer Experience. DMI Journal, Spring 2003. [2] Howard Rheingold The virtual community: homesteading on the electronic frontier. MIT Press, 2000 [3] John Lofland and Lyn H. Lofland Analyzing social settings: a guide to qualitative observation and analysis. Wadsworth, Belmont, Calif., 1995. [4] Anne Kaikkonen, Aki Kekäläinen, Mihael Cankar, Titti Kallio, and Anu Kankainen. Usability Testing of Mobile Applications: A Comparison between Laboratory and Field Testing. Journal of Usability Studies, Issue 1, Volume 1, November 2005. [5] Jesper Kjeldskov and Jan Stage. New techniques for usability evaluation of mobile systems. International Journal of Human-Computer Studies (IJHCS), I60(2004):599-620, October 2003. [6] Jeff Axup and Stephen Viller. Formative Research Methods For The Extremely Mobile: Supporting Community Interaction Amongst Backpackers. In workshop: Appropriate Methods for Design in Complex and Sensitive Settings at OZCHI, Canberra, Australia 2005. [7] Agnelli Davide, Buzzini Dario, Drori Tal. Fashion Victims: an unconventional researchapproach in the field of mobile communication. Proceedings of the 2004 conference on Designing interactive systems: processes, practices, methods, and techniques, ACM Press. August 2004. [8] Jesper Kjeldskov and Connor Graham A Review of Mobile HCI Research Methods. Proceedings of the 5th International Mobile HCI, Udine, Italy 2003.