Perspectives to system quality. Measuring perceived system quality

Transcription

1 1 Perspectives to system quality 2 Measuring perceived system quality UX SIG on UX measurability System (product, service) Heli Väätäjä heli.vaataja@tut.fi TUT - Human-Centered Technology (IHTE) Manufacturer, Vendor Customer (Organization) i Example of a measurement model for user experience Mahlke, S. (2008) User Experience of Interaction with Technical Systems Downloadable at: 3 User s perception of system quality (according to Mahlke s model) 4 System properties p User characteristics Context parameters Human-technology interaction Instrumental quality - Accomplishment of tasks and goals Non-instrumental quality Needs beyond instrumental UX components Perception of instrumental qualities (utility, usability) Emotional user reactions (subjective feelings, psysiological reactions etc.) Consequences of the UX (overall judgments, choice bw alternatives, usage behavior) Perception of noninstrumental qualities (Aesthetic, symbolic and motivational aspects) Utility Does what is needed? Usability How well can be used? Aesthetic Symbolic Communicative Associative Motivational Ability to motivate the user Mahlke, S. (2008)

2 Example of a research design for a quasi experiment in field conditions case reader reporters, applied UX model by Mahlke 5 Another perspective to system quality DeLone and McLean model of IS success (lists of system qualities to measure in the literature, see also TAM and user satisfaction) 6 Väätäjä, H., Vainio, T., Sirkkunen, E. andsalo, K. (2011) Crowdsourced News Reporting Supporting News Content Creation with Mobile Phones. Proc. MobileHCI2011, ACM Press. DeLone, W.H., McLean, E.R. (2003) The DeLone and McLean Model of Information Systems Success: A Ten-Year Update. Journal of Management Information Systems, 19 (4), pp Measuring quality perceptions p Quantitative & qualitative measurement 1. Quantitative & qualitative measurement Quantitative methods 2. Using a baseline for comparison - Expectations 3. Timing of measurement For example SERVQUAL (Parasuraman et al. 1988, 1994) AttrakDiff, AttrakDiff2 (Hassenzahl, 2000 ) Usability questionnaires Qualitative methods Qualitative descriptions using e.g. Critical incident method, DRM (Day Reconstruction Method), narratives, Interviews for descriptions of quality etc.

3 Measuring user s perception of system quality (examples) 9 AttrakDiff2 - example 10 Instrumental quality (SUS, SUMI, ASQ, etc.) Utility Usability Efficiency Effectiveness Satisfaction Learnability Etc. Non-instrumental quality e.g. Aesthetic Lavie & Tractinsky, 2004 Classical visual aesthetics, expressive Symbolic Hassenzahl - AttrakDiff2: identification Etc. Inventive Professional Scale above: Semantic differential Other scales for assessment, e.g.: Likert scale - agreement/disagreement Scale between 0-10 for quality assessment Conventional Unprofessional Example of applying SERVQUAL in test procedure for speech recognition based interaction 11 Expectations vs. Perceptions 12 Mikko Hartikainen, i Esa-Pekka Pkk Sl Salonen and Markku Turunen. Subjective Evaluation of Spoken Dialogue Systems Using SERVQUAL Metho In Proceedings of ICSLP 2004: From Hartikainen, Salonen, Turunen, 2004

4 Disconfirmation measures (SERVQUAL) 13 Example of an alternative approach 14 MSS Measure of Service Quality = 1. Assessment of system quality Perceived level l Desired levell MSA Measure of Service Adequacy = 2. Assessment of whether the quality is acceptable (in a situation/context) Perceived level Accepted level See e.g.: Parasuraman, et al. 1988, 1994 Brown et al. (1993) Improving pov the measurement esue e of service quality YES/NO Sketched experience of a new axe retrospectively When to measure? Buying moment + see Karapanos et al. 2010, Kujala et al (IwC, DPPI) for sketching Joyful waiting The almost - accident It is more difficult to use and learn than expected Teaching starts sketching Rationality takes over it is useful etc., but lack of personal relationship a hint of disappointment Timing for measuring quality perceptions Timing affects the measured perceptions (momentary, episodic, cumulative, anticipated)! Information from other methods may provide explanations for the perceptions of system quality. On the other hand: - What are the meaningful and important qualities in different phases of use or related to specific incidents? id 16

5 Take Aways 17 Quantitative instruments for measuring perceived system quality exist Find and use suitable ones for the study and system One may need to make changes according to the context Or create own instruments/measures use theory, prior user studies for identifying qualities and/or attributes, or use for example goals set for system quality in design and development Measuring and using expectations as a baseline has challenges other ways of measuring acceptable quality exist Timingi of measurement effects results of assessed quality perceptions