Evaluating Relative Contributions of Various HCI Activities to Usability

Evaluating Relative Contributions of Various HCI Activities to Usability Anirudha Joshi and NL Sarda IIT Bombay, Mumbai 400076 India {anirudha,nls}@iitb.ac.in Abstract. Several activities related to human-computer interaction (HCI) design are described in literature. However, it is not clear whether each HCI activity is equally important. We propose a multi-disciplinary framework to organise HCI work in phases, activities, methods, roles, and deliverables. Using regression analyses on data from 50 industry projects, we derive weights for the HCI activities in proportion to the impact they make on usability, and compare these with the recommended and assigned weights. The scores of 4 HCI activities (user studies, user interface design, usability evaluation of the user interface, and development support) have the most impact on the Usability Goals Achievement Metric (UGAM) and account for 58% of variation in it. Keywords: HCI activities, design process, weights. 1 Introduction A human-computer interaction (HCI) design process is made up one or more phases, each of which may consist of one or more HCI activities. Each activity may be associated with one or more methods. A method may require specific skills. An activity may result in a specific deliverable that may be an end in itself, or may be an input for another activity in the HCI design process or the software development process. For example, usability evaluation is an HCI activity that is a part of almost every HCI design process. Usability evaluation could be performed by several methods such as a think-aloud test, a performance test, a heuristic evaluation, a cognitive walkthrough, or an expert review. Performing each method requires a specific set of skills e.g. the think-aloud test requires skills in prototyping, qualitative test design, user recruitment, interviewing users, and analysing data. The activity results in deliverables such as usability problems with the design, potential ideas to improve the design, and possibly a decision about the future course of development. Several HCI activities are described in literature. One or more methods and deliverables are prescribed for each activity. Authors of HCI design processes often express their preference for one method over the other. However, it is not clear whether each HCI activity is equally important. In a specific project, some activities may happen with high level of fidelity, other activities may be cut short, and some activities may not happen at all. Given the context, skipping an HCI activity may have a significant impact on the usability of the product, while skipping or cutting short another activity may only have a marginal impact. R. Bernhaupt et al. (Eds.): HCSE 2010, LNCS 6409, pp. 166 181, 2010. IFIP International Federation for Information Processing 2010

Evaluating Relative Contributions of Various HCI Activities to Usability 167 In section 2, we review traditional design literature and HCI literature to articulate the characteristics relevant to design of interactive artefacts. In section 3, we identify 8 HCI activities that we believe are important in any HCI design process and organise them in a multi-disciplinary framework along with their associated methods, roles, and deliverables. In section 4, we propose a method to express the relative importance of these activities. In section 5, we describe a study that we conducted with 50 industrial projects in India to arrive at the relative importance of these activities empirically. In section 6, we present our conclusions. 2 Design Activities 2.1 Activities in Traditional Design Process Archer defines design as a goal-driven problem-solving activity [1]. According to Jones [2], the effect of designing is to initiate a change in man-made things that in turn affect the manufacturers of those products, the distributors, the purchasers, the users, and ultimately the society. An important job of the designer is to predict each of those behaviours and responses at each stage in the life of the product. One of the ways to understand design is to chart the design process [3]. Several authors agree that at its bare bones, a systematic design process comprises of three fundamental activities [1], [2 ], [3], [4 ]: Analyse the user needs, the problems and the opportunities to identify the goals and the constraints Synthesise alternative solutions Evaluate them against goals and redesign the product where necessary. Authors also agree that the design processes are iterative. Problems found with the proposed design at the time of evaluation are fixed in a new design solution and this is done until the most appropriate solution is found. As iterations progress, the design also moves from generic to detailed. Designers have evolved many methods to carry out these activities. The main effect of the design methods is to externalise what good designers do intuitively to allow design of complex and innovative systems that might be beyond the experience of any one designer. The need for expanding upon the design brief itself before converging to a solution has been expressed by several authors, including Jones [2] and Laseau [5]. Jones broadly divides design methods into three categories that correspond to the three stages of design divergence, transformation, and convergence [2]. Divergence refers to the act of extending the boundary of the design situation to have a large search space in which to seek a solution. The aim of divergent search is to restate the original brief while identifying the features of the design situation that will permit a valuable and feasible degree of change. Key characteristics of the divergence stage are its tentativeness and instability. The objectives, the problem boundary, and the sponsor s brief are unstable, and evolve during this stage and evaluation is deliberately deferred so that nothing relevant is disregarded. Design methods related to this stage often require both rational and intuitive actions, and many of them require legwork rather than armchair speculation [2].

168 A. Joshi and NL. Sarda Transformation is the creative and the most interesting step of design when the objectives, the brief, and the problem boundaries are fixed, the critical variables are identified, the constraints are recognised, and the opportunities are taken. Jones warns that this could also be the stage where big blunders are made, and where experience and sound judgement are necessary. Design methods for searching for new ideas (such as brainstorming and synectics), and design methods to explore the problem structure (such as mind mapping, interaction matrix, and affinity) enable this transformation. Jones calls many of these methods as black-box methods as these depend on the chief designer s creativity and intuition 2]. In convergence, the designer s aim is to reduce the secondary uncertainties rapidly so that an optimum solution can be arrived at with minimal effort. During this stage, the designer is working with the most details in design and if he does not converge fast, the number of alternatives available can explode. Design methods related to convergence stage are related to evaluation, measurement, and analysis. Jones calls these as glass-box methods as these are very rational and analytical [2]. 2.2 Activities in HCI Design Process Many authors have prescribed process models specifically for the design of interactive products. Several of these (particularly the early authors) came from backgrounds in psychology, and their process models reflect a stronger emphasis on analysis, usability evaluation, and convergent thinking. Nevertheless, there are many overlaps with the traditional design processes, particularly in the later literature. The basic ideas for design of interactive systems were already articulated by the 1980s. Gould and Lewis recommended three principles of design, which easily translate into the steps of a process: early focus on users and tasks, empirical measurement of user performance on prototypes, and iterative design to fix problems found during usability tests as they will be [6]. They also acknowledged the importance of the process to ensure meeting usability goals. More detailed process models have been proposed by other authors. Nielsen suggests a 11-stage usability engineering lifecycle model [7]. Kreitzberg identifies a 6- stage design methodology [8]. Dix et al. relate the HCI design process to software development lifecycle [9]. Preece et al. emphasise the need to look beyond HCI into interaction design [10], [11]. Contextual Design process developed by Beyer and Holtzblatt explicitly brings in divergence and transformation in addition to convergence [12]. Divergence is enabled mainly by the technique of contextual inquiry, an interview technique that draws upon ethnography and allows designers to gain deep understanding of users tasks, roles, artefacts, environment, and culture. Transformation is brought in by consolidating findings across users through techniques such as affinity and redesign of users work with a vision of the design that drives changes to the organisational work practice. Cooper and Riemann s goal-directed design process is driven by roles such as managers, designers, programmers, software testers and usability testers [13]. Their design process consists of steps that reflect divergence, transformation, and convergence: Research users and the domain; model users and use contexts (personas and their goals); define requirements of users, business, and technology; create a framework to define the design structure and flow through scenarios; refine the framework; design the interface details; and validate them.

Evaluating Relative Contributions of Various HCI Activities to Usability 169 Mayhew brings the perspective of an external usability consultant to the product development process 14]. Mayhew suggests a variety of techniques to carry out each task, but her approach is open and flexible one may substitute a quicker / cheaper technique to do a task, but each task must be done. (Our approach to activity and method is similar to Mayhew s approach to task and technique). Garrett divides users experience of a website in five layers surface, skeleton, structure, scope, and strategy [15]. Garrett s model of user experience is not a process model in itself, but it has important implications for the process. Decisions at lower layers affect the choices available at the higher layers. Therefore, a strategic decision will ripple through the scope, the structure, the skeleton, and the surface. Similarly, changes in the scope will affect the structure, the skeleton, and the surface. Therefore, one needs to consider as many alternatives as possible before freezing upon the strategy and the scope. Gulliksen et al. review HCI literature and list key principles of user centred design 16]. One of the principles they emphasise is that the design should be holistic, considering all aspects including impact of design on the users work, on the organisation, roles, etc. All parts of the product (task organisation, user interface, online help, user training, health and safety aspects etc.) should be influenced by common design thinking. 3 Framework for HCI Design By combining the essential characteristics of the various processes discussed above, we propose a process framework for HCI design. In our framework, we prescribe 8 Fig. 1. Proposed framework for HCI design process

170 A. Joshi and NL. Sarda HCI activities, and associate them with typical methods, and deliverables. We organise the activities in phases, which we describe in terms of four questions derived from [12]: What matters? How should we respond? How should the design be detailed? How are we doing? Figure 1 captures the phases, activities, methods, skills, and deliverables in a visual form. Table 1 summarises the same in a tabular view. Below, we describe this framework in detail. The framework is a flexible way of understanding and communicating the work of HCI designers in different contexts. Our objective is not to prescribe a one-size-fits-all HCI design process, but rather to articulate the typical HCI activities within which several methods and deliverables can be assimilated. Not all activities or methods may be essential in each instance of the process. 3.1 What Matters? It is not only about what is required by someone. What matters? is a broader question and asks the design team to look at the problem at hand as holistically as possible. This question is answered through divergent thinking, looking beyond what had been specified in the design brief. The HCI activity associated with this phase is user studies, user modelling, and market analysis (activity 1 in Table 1). To understand the key concerns of the users, the stakeholders, the domain, and the context deeply, the team uses methods such as stakeholder interviews, contextual user interviews, focus groups, field observations, log analyses etc. The team may also study related issues such as the environmental, social, or cultural aspects. Most teams would do a benchmark analysis of competitive products. This is a very multi-disciplinary phase where ethnographers, business analysts, domain experts, client / business stakeholders, designers, and potential users are involved. At the end of this phase, the team gets a good understanding of users needs, problems, goals, and constraints. They also have a good understanding of the design opportunities. The phase ends with identifying product goals, including usability goals. 3.2 How Should We Respond? This is a holistic question as well. Now the design team is not just describing the situation but also transforming the problem space so that one or a few solutions become evident. The team begins with ideation (activity 2 in Table 1). With their creativity, but also using a range of ideation techniques such as brainstorming, synectics, participatory design, quality function deployment, and TRIZ (theory of inventor's problem solving) the team comes up with a range of design ideas that solve the problems and realise the opportunities. The ideas could be wild and divergent to begin with, but eventually the team reaches a coherent understanding and articulation of the context and creates a meaningful, holistic response a high-level product definition (activity 3 in Table 1). In interaction design, the product definition is often expressed through personas and scenarios. Sometimes, low-fidelity prototypes are created to support the scenarios. If design involves a new hardware, the form factor is modelled. If it involves software, wireframes of the screens are created. Buxton calls these techniques as sketching user experiences [17].

Evaluating Relative Contributions of Various HCI Activities to Usability 171 In this phase, a multi-disciplinary team is involved, including designers, business analysts, engineers, and client / business stakeholders. If the method of participatory design is used, users are also involved. The product goals guide the team in this phase, but the product goals are also reviewed. The first loop in the framework is the feasibility loop that occurs just after the product definition. Here business and technical feasibility of the proposed product definition are assessed. If competing product definitions are still in contention, a choice is made. If none of the proposed product definitions is found to be feasible, the team goes back and think of more alternatives. At this stage, the design team may do a formative evaluation of the product definition (activity 4 in Table 1), possibly by lightweight methods such as a heuristic evaluation or a cognitive walkthrough. The product definition would be refined to fix any problems found. At the end of this activity, product goals are finalised and technically and financially feasible product definition is agreed. 3.3 How Should the Design Be Detailed? Once a feasible product definition is agreed upon, the detailed user interface is designed (activity 5 in Table 1). This activity completes the transformation and initiates the convergence. Designers explore the details of the user interfaces such as labels, icons, and behaviour of widgets. The text is written. Information is visualised. Visual elements such as typography, colours, fonts, and layouts are designed. Product form is finalised. Design decisions that seem particularly risky are prototyped first so that feedback on these can be sought early. This activity is primarily the designers responsibility, though truly innovative designs may require collaboration between design, technology, and business. The output of this phase is one or more prototypes capturing and representing the design decisions. 3.4 How Are We Doing? In this phase, the team seeks to converge to a usable solution quickly. The purpose of creating a prototype is to evaluate it. As it may happen, the initial design decisions do not fit all the users needs. When a prototype is ready, a formative usability evaluation is done against the usability goals to identify potential problems with the design (activity 6 in Table 1). Card sorts and think-aloud tests are the most preferred method of evaluation at this stage, but other methods may also be used. Usability evaluators do the evaluations, though designers may also participate to get a first-hand feedback. The evaluation generates a list of problems and design ideas to fix them. The second loop in the framework is that of redesign. As Gould and Lewis, interactive systems are particularly prone to having problems in the early designs [6]. After a round of evaluation, problems are fed back and products are redesigned until an acceptable solution is found. The fidelity of the prototypes keeps increasing through the iterations as more details are added. This cycle of design, prototype, evaluate, redesign needs to be tight, quick and consume as few resources as possible. Changes to the design inevitably happen during software development. Some changes are inadvertent slip-ups that need to be corrected (e.g. an accidental change of typeface, colour, or layout). Other changes have to be made because the original design was not feasible. Yet other set of changes happen because there is a change in

172 A. Joshi and NL. Sarda Table 1. A multi-disciplinary framework for the HCI design process. Asterisk (*) denotes necessary deliverables. Phases Disciplines HCI Activities Methods Deliverables What matters? Ethnographers, business analysts, domain experts, client / business stakeholders, designers, users 1. User studies, user modelling, market analysis Stakeholder interviews Contextual inquiry Focus groups Competitive product analysis Analysis of individual interviews User models such as affinity, work models, mind-maps, personas User needs, problems, goals and constraints* Opportunities for design interventions Product goals (including usability goals)* How should we respond? Designers, business analysts, engineers, client / business stakeholders, ethnographers, users 2. Ideation Brainstorming Participatory design TRIZ QFD Design ideas 3. Product definition Interaction design Information architecture High-level use scenarios, storyboards Low fidelity prototypes, wireframes of software, foam models of hardware Business model Strategy, scope and structure of Garrett s model Feasibility Engineers, client / business stakeholders, usability experts 4. Formative usability evaluation 1 and refinement Heuristic evaluation Refined and approved product definition and product goals* Technology feasibility approval* Business feasibility approval* How should the design be detailed? Designers, engineers 5. Design detailing Interface design Information design Navigation design Visual design Product form design Medium to high fidelity UI prototypes through iterations Structure, skeleton and surface of Garrett s model How are we doing? Usability experts, designers, users 6. Formative usability evaluation 2 and... Heuristic evaluation Cognitive walkthrough Think aloud test Card sorting Usability problems Metrics... refinement Same as in design detailing Refined, detailed UI prototypes* UI specification* Dev. support Designers, usability experts 7. Development support Reviews during development Minor tweaks Usability experts, users 8. Summative usability evaluation 3 Usability performance test Field trials Usability approval* Metrics

Evaluating Relative Contributions of Various HCI Activities to Usability 173 requirements or change in technology platform. In all cases, ongoing collaboration between the design and engineering teams is important during software development we call this development support (activity 7 in Table 1). When an early version of the production code becomes available, it is a good idea to do a summative usability evaluation against the usability goals (activity 8 in Table 1). Often summative evaluation is done in a lab-based quantitative performance test. In some cases, it may be done by deploying the product in the field. Preferably, a summative evaluation is done by an external evaluator. The main outcome of a summative evaluation is (hopefully) a usability approval. A set of metrics could also emerge. Though summative evaluation is not supposed to affect the design, if serious usability problems are found, these ought to be fixed before release. 4 Recommended Weights for HCI Activities The HCI activities in our framework must be integrated with the software engineering process model in use, so that they are applied in the practice of software development. Further, each activity may not be equally important in all situations. The importance of an activity would depend on the nature of the product, the context, and the experience of the team. In this section, we recommend the importance to be assigned to each HCI activity in our framework appropriate for typical contexts. However, note that the importance may vary in specific cases (some examples of which we point out). We express the importance of an activity by assigning it a weight on the scale of 0-5, where 0 indicates that the activity is not relevant, 1 indicates the activity is somewhat relevant, 2 indicates the activity is of typical importance, 3 indicates the activity is more important than usual, 4 indicates that the activity is very important, and 5 indicates that the activity is extremely important. Expressing the importance of these activities in this manner helps in direct evaluation of process metrics, as we describe in [18] and [19]. We will demonstrate the use of this framework with the waterfall model of software engineering. Despite criticisms, the waterfall model is still popular in the industry. In a survey of 200 practitioners, Neill and Leplante reported that the waterfall model was the most dominant and 35% of the practitioners claim using it [20]. In our experience, the waterfall model is used even more extensively in the Indian software industry. To integrate our framework with the phases of the waterfall model, we suggest that the Communication phase of the waterfall model should include activities 1-4 of our framework, the Modelling phase should include activities 5-6 and the Construction phase should include activities 7-8 [21]. Table 2 lists our weight recommendations for each HCI activity when integrated with the waterfall model in this manner. Below, we describe our rationale for these weights. In the beginning of a project, it is very important to understand the context of the user and the market scenario. Hence, the activities related to user studies and competitive product analysis is recommended a weight of 3 to 4. The weight can increase if the team is especially unfamiliar with the domain and the context, and can decrease if the team is very familiar with the domain and the context.

174 A. Joshi and NL. Sarda Table 2. Initial weights recommendations for HCI activities in the waterfall model HCI Activity Recommended weights 1. User studies, user modelling, competitive product analysis 3 4 2. Ideation with a multidisciplinary team 2 3. Product definition 1 3 4. Usability evaluation 1 of the product definition and refinement 1 3 5. User interface prototyping 4 5 6. Usability evaluation 2 of the user interface and refinement 4 5 7. Development support: ongoing reviews by usability team during development 3 8. Usability evaluation 3 of an early version 1 3 Ideation is an important activity. However, doing ideation formally as an independent activity may not be as important as generating ideas. Since user studies may also generate many ideas, the importance of explicit ideation is somewhat less. We therefore give it a weight of 2. However, extensive user studies may not be done if the product is not based on contextual data but on ideas, (for example, a toy, or an interactive installation). In such cases, the weight of ideation will go up. Product definition is given a weight of 1 to 3 because we feel this activity can vary in importance. In situations where the product is very innovative or particularly unpredictable, the weight of this activity can go up. On the other hand, if the product is very predictable and what needs to be done is clearly understood by all, the weight can go down. Detailed UI prototyping is the crux of the HCI design process as the main deliverables of HCI professionals come forth from it. This activity is therefore recommended a weight of 4 to 5. In our framework, we identify three occasions where usability evaluation can be done just after the product definition, after detailing out the user interface, and after an initial version of the working product becomes available. Of these, the first two are formative (aimed at improving the design), while the last one is summative (aimed at ensuring that all goals have been met). The formative evaluations are important as they directly affect the design. Between the two formative evaluations, we expect usability evaluation 2 of the user interface to be more important in many contexts as it will evaluate the design with many of its details in place. This evaluation is therefore recommended a weight of 4 to 5. We assume that if this formative evaluation was done well, the importance of doing the other two usability evaluations will be less. Usability evaluation 1 of the product definition will usually have to be done on a very low fidelity prototype under very tight deadlines. Hence, we recommend a low weight for this step. In practice, the situations may vary somewhat. There may be opportunities (e.g. high-fidelity prototype was available early) and reasons (e.g. to demonstrate ideas to investors) to give more importance to the first formative evaluation. In this case, the weight of this usability evaluation can be increased and correspondingly, the weight for the next usability evaluation can be decreased. Usability evaluation 3 of an early release is a summative evaluation and is expected to have little impact on design. Hence, it was also assigned a weight of 1.

Evaluating Relative Contributions of Various HCI Activities to Usability 175 However, in projects where user is expected to do critical tasks, this step will gain weight of up to 3. Finally, we reckon that a lot depends on the continued contact between the HCI professionals and the development teams after the activity of detailed UI prototyping has been completed. Unanticipated UI changes may arise late in the project. In many companies, the HCI professionals are a shared resource and they keep moving from one project to the next before the earlier project is over. To emphasise the importance of development support and reviews of design changes during software development, we assign this step a weight of 3. 5 Validating Recommended Weights 5.1 Method We derived the relative contributions of HCI activities in our framework to usability (and validated the weights proposed in section 4) with the help of simple linear regressions of each activity and a stepwise multiple linear regression of all activities on the usability of products in real-life industrial projects. As a measure of the usability, we selected Usability Goals Achievement Metric (UGAM), a product metric that measures the extent to which the design achieves the usability goals. To calculate UGAM, high-level user experience goals are broken down into detailed, measurable goal parameters. For example, parameters for the high-level goal of learnability could be: options / data / information should be easy to find, user should take little time to learn, user should be able to learn on his own, the product should be consistent with its earlier version, etc. Each goal parameter is assigned a weight between 0-5. During a usability evaluation, each goal parameter is assigned a score between 0-100. UGAM is the sum of the weighted average of the scores, where W p is the weight of the goal parameter p and S p is its score. UGAM is described in more detail in [18] and [19]. Goals and goal parameters are described in more detail in [22]. HCI professionals working in the Indian IT industry were invited to participate in the study. Participants were taught the method of calculating UGAM. They were also walked through the HCI activities in our process framework. First, participants were asked to calculate UGAM scores of the products delivered by their projects. Participants were then asked to assign a weight to each HCI activity based on their judgement of the importance of that activity in the context of their project. While they were shown the recommended weights described above, they were also given the freedom to assign a different score if they wished. Finally, participants were asked to assign a score to each HCI activity from 0 to 100, where 100 represents the best case situation i.e. the activity was done in the best possible manner, with the highest fidelity, in the most appropriate phase of software

176 A. Joshi and NL. Sarda development and with the best possible deliverables; 75 represents that the activity was somewhat toned down, but was still well-timed and well-executed; 50 represents an undecided state where the activity was done with some shortcuts or perhaps was not timed well; 25 represents that the activity was done with many shortcomings; and 0 represents the worst case situation where the activity was not done at all. To help participants assign a score to each activity, we came up with detailed guidelines for evaluating each activity. For example, following are the guidelines for the activity 1 user studies, user modelling, and competitive product analysis: 1. Both organizational data gathering and user studies are done before requirements are finalized. 2. User studies are done in the context of the users by the method of contextual inquiry. 3. User studies are done with at least 20 users in each profile. 4. User studies are done by people with experience in user studies in a similar domain of at least 2 projects. 5. The findings including user problems, goals, opportunities, and constraints are analyzed, documented, and presented in an established user modelling methodology such as personas, work models, affinity diagram, etc. 6. Competitive / similar products and earlier versions of the products are evaluated for potential usability problems, at least by using discount usability evaluation methods such as heuristic evaluation, and are benchmarked. 7. User experience goals are explicitly agreed upon before finalizing requirements. 100 = All the above are true, the activity was performed exceptionally well, 75 = At least five of the above are true, including point 7, or all the above are true, but point 3 had fewer than 20 users per profile, the activity was performed reasonably well, 50 = At least three of the above are true, including point 7, the activity was done with some shortcuts and / or perhaps was not timed well, 25 = Only two of the above are true, the activity was done poorly with many shortcomings, 0 = None of the above are true, the activity was not done. Detailed guidelines for all activities are available online [23]. 5.2 Weights Assigned by Participants A total of 36 participants submitted 50 projects (some participants submitted more than one project). The HCI related experience of participants was between 1-7 years. The participants came from a wide variety of companies including large contracted software development companies, smaller contracted software development companies, multi-national companies with large product development centres in India, one large, internationally popular internet company, and five smaller product development companies. Only the projects following the waterfall model were used for the analyses presented in this paper. Table 3 lists the averages of weights actually assigned by participants for HCI activities and their standard deviations. Participants do not seem to have deviated substantially from our recommendations.

Evaluating Relative Contributions of Various HCI Activities to Usability 177 Table 3. Initial recommendations for weights of the HCI activities and the average and the standard deviation of weights actually assigned by participants to those HCI activities (N = 50). HCI Activity Recommended weights Assigned weights average Assigned weights SD 1. User studies, user modelling... 3 4 3.7 0.8 2. Ideation with a multidisciplinary team 2 2.5 0.7 3. Product definition 1 3 3.1 0.7 4. Usability evaluation 1 of product definition... 1 3 2.0 1.1 5. User interface prototyping 4 5 4.5 0.6 6. Usability evaluation 2 of the user interface... 4 5 3.8 0.8 7. Development support... 3 3.2 0.8 8. Usability evaluation 3 of an early version... 1 3 1.9 1.0 5.3 Weights Derived from Regression Analysis The score of each HCI activity is a measure of the fidelity of that HCI activity. UGAM is a measure of usability goal achievement in the project. The UGAM score is arrived at independently of the scores of HCI activities. If we can find the relative effect of the scores of HCI activities on the UGAM scores, this could be a way of evaluating the impact of HCI activities on the usability. Separate simple linear regressions were performed assuming the scores of each of the eight HCI activities to be the predictor variables and UGAM to be the criterion variable (Table 4). In case of each HCI activity, a significant model emerged and the activity score had a positive significant Pearson s correlation with UGAM (0.56 > R > 0.33, 0.32 > R 2 > 0.11, 0.30 > adjusted R 2 > 0.09, 22.399 > F > 5.796, p <= 0.02, twotailed). All coefficients were positive. All lower bounds of the 95% confidence intervals of the coefficients were also positive. We can conclude that all HCI activities recommended in Table 2 affect UGAM positively. The scores of the HCI activities seem to be affecting the UGAM scores to varying degrees some HCI activities have a larger effect on UGAM than others. The strongest correlations, largest adjusted R 2 values, and largest coefficients were observed for the HCI activities of user interface prototyping, usability evaluation of the user interface and refinement, development support, and user studies, user modelling, competitive product analysis. This justifies our 3+ weight recommendations for these activities (Table 2) and also the 3+ average weight assigned by participants (Table 3). The adjusted R 2 value in a simple linear regression represents the extent to which a predictor variable affects the criterion variable. We could possibly assign weights to the HCI activities derived in proportion to the adjusted R 2 values we show below in column 4 of Table 6. Using the stepwise method, a multiple regression was performed assuming the scores of the eight recommended HCI activities as predictor variables and UGAM as the criterion variable. The most significant model returned these values: R = 0.784, R 2 = 0.614, adjusted R 2 = 0.580, F = 8.533, p < 0.005. The four HCI activities identified above also emerged as significant predictors in this model (Table 5). The scores on these four HCI activities predicted 58% of variation in UGAM (adjusted R 2 = 0.580).

178 A. Joshi and NL. Sarda Table 4. Summary of simple linear regressions on UGAM as criterion variable and the scores of individual HCI activities as predictor variables on merged project scores (N = 50). The top four correlating activities have been highlighted. Model R R 2 Adj. F Sig. B t Sig. Lower R 2 Bound 95% conf. interval for B Upper Bound User studies 0.445 0.207 0.190 12.517 0.001 0.221 3.538 0.001 0.095 0.346 Ideation 0.384 0.148 0.130 8.326 0.006 0.190 2.886 0.006 0.057 0.322 Prod Def 0.406 0.165 0.148 9.481 0.003 0.227 3.079 0.003 0.079 0.375 UE 1 0.351 0.123 0.105 6.748 0.012 0.162 2.598 0.012 0.037 0.287 UI Proto 0.564 0.318 0.304 22.399 0.000 0.299 4.733 0.000 0.172 0.426 UE 2 0.534 0.285 0.270 19.126 0.000 0.249 4.373 0.000 0.134 0.363 Dev Support 0.532 0.283 0.268 18.967 0.000 0.216 4.355 0.000 0.116 0.315 UE 3 0.328 0.108 0.089 5.796 0.020 0.134 2.407 0.020 0.022 0.246 These four HCI activities had a positive, significant coefficient (p <= 0.023) and the lower bound of the 95% confidence interval for all coefficients was positive. The variance inflation factors (VIFs) of all predictor variables are well below 4, indicating that there is no multi-collinearity among the predictor variables. This implies that the assumption that the HCI activity scores are independent variables was acceptable for the purpose of the stepwise multiple regression. Table 5. The most significant model in the SPSS output of stepwise multiple linear regression on UGAM as criterion variable and the ratings of HCI activities as predictor variables (n = 50) R R 2 Adj. R 2 Std. Error of the Estimate R Square Change F Change Change Statistics df1 df2 Sig. F Change 0.784 0.614 0.580 7.702 0.073 8.533 1 45 0.005 Unstandardised Coefficients B Std. Error Standardised coefficients t Sig. 95% Confidence Interval for B L. Bound U. Bound (Constant) 33.794 4.024 8.398 0.000 25.690 41.889 Collinearity Statistics Usability Eval 2 0.154 0.048 0.332 3.208 0.002 0.057 0.250 1.247 Dev Support 0.123 0.040 0.306 3.064 0.004 0.042 0.204 1.165 User studies 0.138 0.047 0.286 2.921 0.006 0.043 0.233 1.116 UI Prototyping 0.133 0.057 0.253 2.346 0.023 0.019 0.247 1.354 VIF Brace et al. suggest that the standardised coefficients of the predictor variables in a multiple regression can be used to compare the relative contribution of each predictor variable to the criterion variable and assess the strength of the relationship [24]. We could possibly assign weights to the HCI activities derived in proportion to these standardised coefficients as shown in column 5 of Table 6.

Evaluating Relative Contributions of Various HCI Activities to Usability 179 Table 6. A comparison of our recommended weights, average weights assigned by participants, weights derived by scaling up adjusted R 2 values from simple linear regressions (SLRs) and from scaling up the standardised coefficients of the stepwise multiple regression (MR) HCI Activity Recommended weights Assigned weights Derived weights scaled from 1. User studies, user modelling... 3 4 3.7 3.1 4.3 2. Ideation with a multidisciplinary team 2 2.5 2.1-3. Product definition 1 3 3.1 2.4-4. Usability evaluation 1... 1 3 2.0 1.7-5. User interface prototyping 4 5 4.5 5.0 3.8 6. Usability evaluation 2 of the UI... 4 5 3.8 4.4 5.0 7. Development support... 3 3.2 4.4 4.6 8. Usability evaluation 3... 1 3 1.9 1.5 - SLRs MR 6 Conclusions Drawing from literature, we proposed a framework comprising of 8 HCI activities. By using simple linear regressions, we could demonstrate that each of these activities had a significant positive correlation with the usability metric UGAM. In a stepwise multiple regression, four of these HCI activities accounted for 58% of the variation UGAM. We can conclude that while all activities in the framework affect usability, the identified four HCI activities are relatively more important. The statistical analyses were in consonance with our original recommendations and with the weights assigned by practitioners, as summarised in Table 6 above. Perhaps the most underestimated HCI activity during recommendation and assignment was the support that HCI teams need to give during the software development, though it was not a complete surprise. A possible critique of our method could be that we showed the recommended weights to the participants before they assigned theirs. While this could have been an approach, it must be noted that that neither the recommended weights, nor the weights assigned by participants play a role in the regression analyses, which are based on the UGAM scores and activity scores alone. The weights derived from the regression analyses validate both the recommended and the assigned weights. Another possible critique could be about our assumption that the scores of HCI activities are independent variables. Although the activity scores are naturally related (teams likely to score well on some HCI activities are likely to score well on others), it was essential to use them as predictor variables as it is the only way to establish their effect on usability. We minimised the bias by prescribing guidelines for evaluating each activity. The statistics did not show any multi-collinearity among the HCI activity scores. Knowing which HCI activities are important would be useful in many contexts, particularly when resources are scarce and tradeoffs need to be made. Designers can use the rigorous, higher fidelity methods on activities that are more important, and make do with discount methods on less important activities. This knowledge would be

180 A. Joshi and NL. Sarda useful in integrating HCI activities in software engineering processes HCI professionals can insist on including the important activities, while conceding the relatively less important ones. The weighted average of the scores of activities could be used as a process metric as we describe in [18] and [19]. We used our framework of HCI activities, the waterfall model, UGAM as the product metric, and projects from the Indian IT industry to find the relative contribution of various HCI activities. Our results may be generalised within these choices. Other researchers could use other frameworks, other process models, other product metrics, and / or other contexts in a similar way to identify the activities that matter in those contexts. Acknowledgements We thank Sanjay Tripathi, Pramod Khambete, Ved Prakash Nirbhay, Deepak Korpal, Atul Manohar, Aniruddha Puranik, Prof. UA Athavankar, Prof. Umesh Bellur, Prof. S Sudarshan, and all participants in the studies for their suggestions, assistance, and contributions. References 1. Archer, B.: Systematic Method for Designers. Council of Industrial Design (1965) 2. Jones, C.: Design Methods, Seeds of Human Futures. Wiley- Interscience, Chichester (1970) 3. Lawson, B.: The Design Process Demystified. Butterworth Architecture, Butterworths (1980) 4. Cross, N.: Engineering Design Methods, 3rd edn. John Wiley & Sons, Chichester (2000) 5. Laseau, P.: Graphic Thinking for Architects and Designers. Van Nostrand Reinhold Company, New York (1980) 6. Gould, J., Lewis, C.: Designing for Usability: Key Principles and What Designers Think. Communications of the ACM 28(3) (1985) 7. Nielsen, J.: Usability Engineering. Morgan Kaufmann, San Francisco (1993) 8. Kreitzberg, C.: Managing for Usability. In : Multimedia: A Management Perspective. Wadsworth, Belmont (1996) 9. Dix, A., Finlay, J., Abowd, G., Beale, R.: Human-Computer Interaction, 2nd edn. Prentice Hall, Englewood Cliffs (1998) 10. Preece, J., Rogers, Y., Sharp, H.: Interaction Design, Beyond Human-Computer Interaction, 1st edn. John Wiley & Sons, Chichester (2002) 11. Sharp, H., Rogers, Y., Preece, J.: Interaction Design, Beyond Human-Computer Interaction, 2nd edn. Wiley India, Chichester (2007) 12. Beyer, H., Holtzblatt, K.: Contextual Design. Morgan Kaufmann, San Francisco (1998) 13. Cooper, A., Reimann, R.: About Face 2.0. Wiley, Chichester (2003) 14. Mayhew, D.: Usability Engineering Lifecycle. Morgan Kaufmann, San Francisco (1999) 15. Garrett, J.: The Elements of User Experience. New Riders, Indianapolis (2003) 16. Gulliksen, J., Göransson, B., Bovie, I., Blomkvist, S., Persson, J., Persson, J., Cajander, A.: Key Principles for User-centred System Design. Behaviour and Information Technology 22(6) (2003) 17. Buxton, B.: Sketching User Experiences. Morgan Kaufmann, San Francisco (2007)

Evaluating Relative Contributions of Various HCI Activities to Usability 181 18. Joshi, A., Tripathi, S.: User Experience Metric and Index of Integration: Measuring Impact of HCI Activities on User Experience. In: International Workshop on the Interplay between Usability Evaluation and Software Development, Pisa (2008) 19. Joshi, A., Sarda, N., Tripathi, S.: Measuring Effectiveness of HCI Integration in Software Development Processes. Journal of Software System (in press, 2010) (Corrected Proof), doi: 10.1016/j.jss.2010.03.078 20. Neill, C., Laplante, P.: Requirements Engineering: The State of the Practice. IEEE Software 20(6) (2003) 21. Joshi, A., Sarda, N.: HCI and SE: Towards a Truly Unified Waterfall Process. In: Aykin, N. (ed.) HCII 2007. LNCS, vol. 4559, pp. 108 112. Springer, Heidelberg (2007) 22. Joshi, A.: Usability Goals Setting Tool. In: 4th Workshop on Software and Usability Engineering Cross-Pollination: Usability Evaluation of Advanced Interfaces, Uppsala (2009) 23. Joshi, A.: Index of Integration, http://www.idc.iitb.ac.in/~anirudha/ioi.htm (accessed 2009) 24. Brace, N., Kemp, R., Snelgar, R.: SPSS for Psychologists, 2nd edn. Palgrave Macmillan, China (2003)