Faculty of Cognitive Science and Human Development

Faculty of Cognitive Science and Human Development COLLABORATIVE VIRTUAL ENVIRONMENT : COMPARING THREE DIFFERENT USABILITY EVALUATION TECHNIQUES FOR CVE SYSTEMS Norrasyidah Binti Jerani QA 76.9 U83 N822 Bachelor of Science with Honours (Cognitive Science) 2006 2006

P. KHIDMAT MAKLUMAT AKADEMS( UNIMAS i'usat Khidrnat Maktumat AKadenmº UNIVERSIT3` MA4ý4 SARAWAIC 44'i(Kl Kota Samardhaa ýinniiýiiinvflinia 1000143754 Collaborative Virtual Environment: Comparing Three Different Usability Evaluation Techniques for CVE Systems NORRASYIDAH BINTI JERANI This project is submitted in partial fulfillment of the requirements for Bachelor of Science with Honours (Cognitive Science) Faculty Of Cognitive Science and Human Development UNIVERSITI MALAYSIA SARAWAK 2006 t

BORANG PENGESAHAN STATUS TESIS JUDUL: COLLAgpRPTIUE yirtualg-nvironment ; COMPARINGt T40. EE t)lrpe0. tn7 UypºBIL-iTY t-yalnation, tcctwl6lul5 for WE gystemc SESI PENGAJIAN : 03 06 Saya NDRR-ASYIbAN ginti ý3erani (HURUF BESAR) mengaku membenarkan tesis * ini disimpan di Pusat Khidmat Maklumat Akademik, Universiti Malaysia Sarawak dengan syarat-syarat kegunaan seperti berikut: I. Tesis adalah hak milik Universiti Malaysia Sarawak 2. Pusat Khidmat Maklumat Akademik, Universiti Malaysia Sarawak dibenarkan membuat salinan untuk tujuan pengajian sahaja 3. Pusat Khidmat Maklumat Akademik, Universiti Malaysia Sarawak dibenarkan membuat pendigitan untuk membangunkan Pangkalan Data Kandungan Tempatan 4. Pusat Khidmat Maklumat Akademik, Universiti Malaysia Sarawak dibenarkan membuat salinan tesis ini sebagai bahan pertukaran antara institusi pengajian tinggi $. ** sila tandakan (4) SULIT (Mengandungi maklumat yang berdarjah keselamatan atau kepentingan seperti termaktub di dalam AKTA RAHSIA RASMI 1972) TERHAD (Mengandungi maklumat Terhad yang telah ditentukan oleh organ isas i/badan di mana penyelidikan dijalankan) V-11, TIDAK TERHAD yý/ rl (TANDATANGAN PENULIS) (TANDATANGAN PENYELIA) Alamat Tetap: N0-14.1t_N DRAHNaAIV l'ma, 6100, sa2l1le1. SA RAWA 'L ' Tarikh : Tarikh: O q- o ud6 Catatan: * Tesis dimaksudkan sebagai tesis bagi Ijazah Doktor Falsafah, Sarjana dan Sarjana Muda * Jika tesis mi SU LIT atau TERHAD, sila lampirkan surat daripada pihak berkuasa/organisasi berkenaan dengan menyatakan sekali schab dan tempoh tesis ini perlu dikelaskan sebagai TERHAD. I

The project entitled `Collaborative Virtual Environment: Comparing Three Different Usability Evaluation Techniques for CVE Systems' was prepared by Norrasyidah Jerani and submitted to the Faculty of Cognitive Science and Human Development in partial fulfilment of the requirements for a Bachelor of Science with Honours (Cognitive Science) Receivedifor examination by: (Mohd. Kamal bin Othman) Date: OA - 05.2001 11

PENGHARGAAN / ACKNOWLEDGEMENT I would like to thank Allah S. W. T. for giving me good health and chances to complete this project. First of all, I would like to express my gratitude and thanks to my supervisor Mr. Mohd. Kamal Bin Othman for the supervision from the start of this project until the end. I also would like to thank him for the support, motivation and understanding that he gave me throughout this project and also for spending his time to provide and guide me until this project is completed. Next, I would like to thanks my evaluators Pn. Wan Norizan, Dr. Tan Kok Wah, Mr. Ng Giap Weng and Dr. Chen for their comments and guidelines in order to make this project a better one. Then, I would like to thank my respondents for their participation and the time that they spent in helping me to get the data that I needed in order to complete my project. I would also like to thanks my family and friends for being there for me when I needed them the most and for the moral support, motivation and understanding through all the hard and good times that I've gone through when completing this project. Lastly, I would like to thank all the people whom their names I did not include here who have contribute and help me to complete this project whether direct and indirectly. III

Yusat Khidmat Maktumat Alcadenft IJPiIYERSTii MALAYSIA SARAWAY V43(1(1 Kots Smnmakm TABLE OF CONTENTS Penghargaan/Acknowledgement Jadual Kandungan/Table of Contents Senarai Jadual/List of Figures Senarai Gambarajah/List of Tables Abstrak/Abstract Abstrak/Abstract Ill iv vii ix a ai 1. CHAPTER 1 INTRODUCTION 1.0. Background 12 1.1. Statement of Problem 15 1.2. Research Objectives 16 1.3. Research Significance/Contribution 17 1.4. Research Limitation 18 1.5. Definition of Terms 18 2. CHAPTER 2 LITERATURE REVIEW 2.0. Introduction 23 2.1. Collaborative Virtual Environment (CVE) 24 2.2. Usability Issues in CVE/VE 25 2.3. Usability Framework 29 2.4. User-Centred Design 31 2.5. User Interface 31 2.6. Interaction and Communication 32 2.7. Avatar 33 2.8. Navigation 34 2.9. Evaluation Techniques 35 2.10. Questionnaires 37 2.11. Cognitive Walkthrough 38 2.12. Focus Group 40 2.13. Principle of Design 40 2.14. Current States of Art in CVE and How It Can Be Improved 42 3. CHAPTER 3 METHODOLOGY 3.0. Introduction 44 3.1. Research Design 44 3.2. The Population 45 3.3. The Sample 45 iv

3.4. The Research Instrument 3.4.1. Task Set 46 3.4.2. Questionnaires 46 3.5. Technique Used 3.5.1. Cognitive walkthrough 46 3.5.2. Focus Groups 47 3.5.3. Questionnaires 47 3.6. Data Collection Procedures 48 4. CHAPTER 4 RESULTS AND DISCUSSIONS 4.0. Introduction 50 4.1. Questionnaires 51 4.1.1. Structure of the software 51 4.1.2. Navigation 51 4.1.3. Avatar / Human Embodiment 52 4.1.4. Interaction 53 4.1.5. User Interface 53 4.1.6. Communication 54 4.1.7. Environment 54 4.2. Cognitive Walkthrough 55 4.2.1. Post Test Close-Ended Questionnaires for Cognitive Walkthrough 55 4.2.2. Post Test Open-Ended Questionnaires for Cognitive Walkthrough 61 4.2.2.1.3D Environment 61 4.2.2.2. Navigation 61 4.2.2.3. User Interface 61 4.2.2.4. Human Embodiment/Avatar 62 4.2.2.5. Interaction/Communication 62 4.3. Focus Group 62 4.3.1. Navigation 62 4.3.2. Avatar 63 4.3.3. Communication 63 4.3.4. Environment 63 4.3.5. Interaction 64 4.3.6. User Interface 64 4.4. Usability Framework 65 4.5. Discussion 66 4.5.1. The Problem With the Avatar 66 4.5.2. The Problem With the Environment 67 4.5.3. The Problem With the Communication 68 V

4.5.4. Comparison of the test (Questionnaires, Cognitive Walkthrough, Focus Group) 69 4.5.5. Needs Analysis 70 5. CHAPTER 5 CONCLUSIONS AND RECOMMENDATION 5.0. Introduction 74 5.1. Conclusions 74 6. REFERENCES 7. APPENDIX vi

LIST OF FIGURES Figure 1 Group Communication and user interfaces (Heldal, 2001) 24 Figure 2 Building process of CVE (COVEN, 1997) 25 Figure 3 Usability criteria assessed by MAUVE (Stanney et al., 2003) 27 Figure 4 Usability problem categories and trade-off design areas (Tromp, 1999) 28 Figure 5 Framework within which the development of COVEN CVE evaluation and design strategy took places (Tromp, Steed & Wison, 2003) 30 Figure 6 Methodology for the user-centred design and evaluation of VE user interaction (Gabbard, Hix, & Swan 11,1999) 37 Figure 7 Pie Chart of Question I of Post Test Questionnaires in Percentage 56 Figure 8 Pie Chart of Question 2 of Post Test Questionnaires in Percentage 57 Figure 9 Pie Chart of Question 3 of Post Test Questionnaires in Percentage 57 Figure 10 Pie Chart of Question 4 of Post Test Questionnaires in Percentage 58 Figure 11 Pie Chart of Question 5 of Post Test Questionnaires in Percentage 58 Figure 12 Pie Chart of Question 6 of Post Test Questionnaires in Percentage 59 vii

Figure 13 Pie Chart of Question 7 of Post Test Questionnaires in Percentage 59 Figure 14 Pie Chart of Question 8 of Post Test Questionnaires in Percentage 60 Figure 15 Pie Chart of Question 9 of Post Test Questionnaires in Percentage 60 Figure 16 Usability Framework for CVE 65 VIII

LIST OF TABLES Table 1 Results of the structure of the software by percentage. 51 Table 2 Results of the navigation of the software by percentage. 52 Table 3 Results of the avatar of the software by percentage. 52 Table 4 Results of the interaction of the software by percentage. 53 Table 5 Results of the user interface of the software by percentage. 54 Table 6 Results of the communication of the software by percentage. 54 Table 7 Results of the environment of the software by percentage. 55 Table 8 Results of the post test close-ended by percentage. 56 Table 9 Comparison of Cognitive Walkthrough, Questionnaires and Focus Group. 69 ix

ABSTRAK COLLABORATIVE VIRTUAL ENVIRONMENT: COMPARIING THREE DIFFERENT USABILITY TECHNIQUES FOR CVE SYSTEMS NORRASYIDAH BINTI JERANI Kajian ini bertujuan untuk membuat perbandingan diantara ujian kebolehgunaan bagi CVE/VE sistem. Adalah penting untuk melihat dan mengethaui teknik yang sesuai untuk bahgian-bahagian yang dikaji terutama CVE/VE sistem. Objektif utama projek ini adalah untuk membandingkan ujian kebolehgunaan untuk CVE/VE sistem. Objektif spesifik projek ini pula adalah untuk membandingkan diantara 3 teknik ujian kebolehgunaan iaitu dengan menggunakan borang soal selidik, Cognitive Walkthrough, dan Focus Group, memberi cadangan untuk framework kebolehgunaan bagi CVE/VE sistem dan memberi analisis kehendak untuk sistem yang dikaji. Seramai 90 orang responden telah dipilih secara rawak daripada para pelajar Universiti Malaysia Sarawak untuk menjadi sampel dalam kajian ini. Perisian Microsoft Excel telah digunakan untuk menganalisa data-data yang diperolehi daripada sampel. Analisis kajian mempamerkan bahagian- bahagian yang bermasalah untuk sistem yang dikaji, cadangan untuk memperbetulkan sistem yang dikaji dan kelemahan dan kelebihan ketiga-tiga ujian kebolehgunaan yang digunakan. X

ABSTRACT COLLABORATIVE VIRTUAL ENVIRONMENT: COMPARING THREE DIFFERENT USABILITY EVALUATION TECHNIQUES FOR CVE SYSTEMS NORRASYIDAH BINTI JERANI This research aims to compare the different usability testing to evaluate CVE/VE systems. It's crucial to know which evaluation techniques is good for -each part of the research especially in CVE/VE. The general objective - is to compare the usability testing for the CVE/VE system. Specific objectives for this project are to compare the three usability testing used for CVE/VE systems, which are the questionnaire, cognitive walkthrough, and focus group; suggest and develop usability framework for the CVE/VE system; and to provide the need analysis for the CVE/VE system that being tested. 90 respondents have been randomly chosen from the Universiti Malaysia Sarawak students. Microsoft Excel has been used to analyze the data from the research. The analysis of the research point up the problematic part of the system that being tested and make recommendation on how to improve the system and also discuss the advantages and the weaknesses of the three tests. XI

CHAPTER 1 INTRODUCTION 1.0. Background As the technologies grow, Virtual Reality/Virtual Environment has become an apparent trend nowadays. It is a technology that widely used especially in the entertainment such as 3D games, learning, training, architecture, and etc. One of the technologies that emerged from this technology is the Collaborative Virtual Environment (CVE) or also known as Networked Virtual Environment (NVE). CVE is a system that allows users to communicate with each other when they are in physically different locations and thus not be able to communicate in the usual face-to-face manner (Preece et al., 2002). CVE 12

supports human-human communication in addition to human-machine communication and uses a Virtual Environment (VE) as the user interface. CVE is a computer-based, distributed, virtual space or set of places where people can meet and interact with others, agents or with virtual objects (Cavallo, 2004). CVE incorporate 3D graphical spaces, 2.5D and 2D environments, to text-based environments. Access to CVE is not limited to desktop devices, but might as well include mobile or wearable devices, public kiosks, and etc. (Cavallo, 2004). The emergence technologies also give rise to the significant of Human Computer Interaction (HCI) in VEs. It aims to incorporate human factors to VE (in this case CVE). Designing usable and effective interactive VE systems are a new challenge for system developers and human factors specialists because traditional usability principles do not consider characteristics which are unique to VE systems, such as the design of way finding and navigational techniques, object selection and manipulation, as well as integration of visual, auditory and haptic system outputs (Stanney et al., 2003). Traditional usability inspections still can be applied to VE systems but it was not comprehensive enough to characterize usability attributes specific to three-dimensional (3D) spatially immersive and interactive environments (Stanney et al., 2003) and there are certain limitation in traditional usability 13

methods for assessing VE systems, for example as stated by Stanney et at. (2003), traditional single user-task-based assessment methods do not consider VE system characteristics in which two or more users interact in the same environment. All of this resulting in only few of the traditional usability evaluation techniques applied for VE systems (Gabbard, Hix, & Swan II, 1999). This research compared the usability techniques that can be used to evaluate the CVE. The techniques involved in this study are Questionnaires, Cognitive Walkthrough, and Focus Group. Findings from this research can determined/suggests the technique that can be used to evaluate CVE/VE software/systems/prototype. Final phase of this research will focus on building the usability framework for the CVE. This research also highlights the need analysis that has been made to the systems. This research focuses and aims on the user interfaces of the system. User Interfaces in this research refer to the Graphical User Interface (GUI) and the user interface (UI) itself (i. e. the icon, menus, dialog boxes, and etc. ). This research also looks at the interaction in the CVE/VE system (not limited to user-user interaction but user-object as well). It also looks at the navigation and the avatar. 14

russt Khidmat Makiumat Akadet UNIVERSM MALAYSIA SARAWAK Q43M Kota Samarshom Outcome of this research are as follows: i. Comparison of the approach for evaluating CVE/VE usability ii. Develop usability framework for the CVENE systems iii. The need analysis for the CVE/VE The structures of this thesis are as follows. The first chapter focuses on the introduction to the research that been carried out including the problem statement, objectives, the research significance and its contribution. The second chapter of this thesis will deal with different aspects of relevant background (often called the literature review study). Chapter 3 will focus on the methodology of the research. This includes the research design, population of samples, instrumentation, data collection procedures and data analysis procedures. Chapter 4 will discuss what has been gathered from the research and the analysis of the result from this research. Finally, Chapter 5 will conclude this research work by reviewing what has been achieved and what has not been achieved. This report also suggests what may need to be improved in the future. 1.1. Statement of Problem Nowadays, VE and CVE are the technologies that are widely used around the world. Developers are trying to design a CVE system that met ideal criteria 15

of CVE system. The ideal CVE system is a system that can overcome the human constraint in the sense that the environment has to be effective and intuitive and a system that can overcome the machine constraint in the sense that the VE has to utilize minimum computational load and network traffic (Tromp, 1999). The current system now is lacking in this criteria. So, it is important to evaluate the system in order to incorporate the current system with these criteria. Problem statement: What are the differences between the usability technique approaches to evaluate CVE/VE software/prototype/system. 1.2. Research Objectives General objectives of this research are to study and evaluate existing CVENE software using various usability techniques available such as Summative, Focus Group and Cognitive Walkthrough. Specific objectives for this research are: i. To evaluate the existing CVENE software by various evaluation techniques and compare the findings. ii. To compare the approach to evaluate CVE/VE software. iii. To suggest what the CVE system that is being evaluated needs (need analysis). 16

iv. To suggest and develop the usability framework that suitable for VE/CVE. 1.3. Research Significance/Contribution The importance of conducting this research because there are many usability techniques available and not all of these techniques can be used to evaluate CVE/VE software. Without this research, other researchers will have to differentiate which technique are appropriate for their system and a guidelines about techniques that are good for their software/systems and researchers will facing some problems if using techniques that not appropriate. The research conducted can provide guidelines for other researchers that want to evaluate CVE/VE software/systems as it suggests the usability techniques to evaluate CVE/VE software/systems. The contributions of this research are: i. Provide the usability framework for CVE. ii. Reducing the "not user friendly" CVE system. iii. Help the researchers to build the systems based on appropriate guidelines. 17

1.4. Research Limitation There are many research limitations when conducting this research, some of this are: i. There are many usability techniques available and this research will limit the techniques accordingly. ii. The capability of the available systems and the inadequate component of the systems. iii. The different level of experiences and exposure-ness of the sample will affect the result of the research. 1.5. Definition of Terms 1.5.1. Collaborative virtual environment " Literal definition A system that allows multiple users who are distributed across a computer network to enter a shared Virtual Environment constructed from 3D computer graphics where each participant has a visual embodiment called an avatar (Yang, 2002). " Operational Definitions A system that allows people to communicate with each other when they are in different locations, in single Virtual 18

Environment as if they were physically present in front of each other. 1.5.2. Virtual Environment " Literal definition An environment that provides the user with access to information that would not otherwise be available at that place or time capitalizes upon natural aspects of human perception by extending visual information in three spatial dimensions and may supplement this information with other sensory stimuli and temporal changes (Wann & Mon-Williams, 1996). " Operational Definitions 3-Dimensional environment. 1.5.3. Evaluation " Literal definition An attempt to asses the worth or value of some value innovation or intervention, some services or approach (Robson, 1993). " Operational Definitions An attempt to assess the usability aspect of CVE/VE software. 19

1.5.4. Usability " Literal definition Ensuring that interactive products are easy to learn, effective to use, and enjoyable from the user's perspective (Preece, Rogers & Sharp, 2002). " Operational Definitions The extent to which the users can interact efficiently with the system without unnecessary mental effort. 1.5.5. Usability testing " Literal definition Involves measuring typical users' performance on carefully prepared tasks that are typical of those for which the system was design (Preece, Rogers & Sharp, 2002). " Operational Definitions Measuring user performances by using certain type of evaluation techniques. 20

1.5.6. User Interfaces " Literal definition Visual interface component that allows him/her to view and manipulate the virtual world (Rickel & Johnson, 2000) " Operational Definition Part of the system that the users interact with in order to interact with the system. 1.5.7. Graphical User Interfaces (GUI) " Literal definition User-friendly interfaces that allow non-experts to quickly learn to manipulate a given environment (Gronegress et at., 2001). " Operational Definition The only part of the system with which the user is in direct contact and it incorporates graphical presentation instead of the conventional user interfaces. 1.5.8. Avatar " Literal definition Humanoids embodiments generally referred to as avatars that represent inhabitants in CVE (Fabri et al., 1999). 21

" Operational Definition Visual representation of the users in CVE. 1.5.9. Sense of presence " Literal definition The sense of immediacy and control created by presence: the feeling of `being there' (Gerhard et al., 2001) " Operational Definition The feeling of `being there' in the CVENE. 22