USERS IMPRESSIONISM AND SOFTWARE QUALITY

Transcription

1 USERS IMPRESSIONISM AND SOFTWARE QUALITY Michalis Xenos * Hellenic Open University, School of Sciences & Technology, Computer Science Dept. 23 Saxtouri Str., Patras, Greece, GR ABSTRACT Being software engineers, we have spent years of studies and practice in software engineering, which has enabled us to deploy an analytical way of thinking. Our engineering approach makes us decompose abstract terms such as software quality until we reach quantifiable, measurable characteristics. It is this approach with regard to software quality that had lead to the development of commonly applied quality standards. But what about users of our software, particularly those who rarely take an engineering approach similar to ours simply because the deploy a different way of thinking? How do they perceive software quality? If we are to improve the quality of software that we produce, it is important to understand and address their way of thinking too. We argue that the way in which these software users perceive software quality compares to the way impressionist painters perceived visual reality in terms of transient effects of light and color. In order to prove this point we present data from software projects during the last ten years. Analysis of these data has guided us to conclude that the initial user impression of software quality is hard to change. In particular, the positive impressions may be revised in time, but not significantly, whilst the negative impressions are very hard to change over time. KEYWORDS Software Quality, User Perceived Quality, Usability Laboratory. 1. INTRODUCTION Impressionism (source: Wikipedia, the free encyclopedia, was a 19th century art movement that began as a loose association of Paris-based artists who began publicly exhibiting their art in the 1860s. The name of the movement is derived from Claude Monet's Impression, Sunrise (Impression, soleil levant). The influence of Impressionist thought spread beyond the art world, leading to Impressionist music and Impressionist literature. Characteristics of impressionist painting include visible brushstrokes, light colors, open composition, emphasis on light in its changing qualities (often accentuating the effects of the passage of time), ordinary subject matter, and unusual visual angles. Impressionism also describes art done in this style, but outside of the late 19th century time period. The techniques of impressionism gradually grew more specific to the movement, and encompassed what its adherents argued was a different way of seeing. They painted en plein air (outdoors) rather than in a studio as was the custom, capturing the momentary and transient aspects of sunlight. * Tel , Fax , xenos@eap.gr 11

2 What we argue in this paper is that sometimes software users perceive software quality in a way that compares to the way impressionist painters perceived visual reality. For many users the first impression has a strong impact on their final opinion. If we are to improve the quality of software that we produce, it is important to understand and address this way of thinking. In order to prove our point, in this paper we present and discuss statistical data from various software projects, in which we have collected user-perceived quality measurements, during the last ten years. This paper is organized as follows: section 2 is a brief literature review of the perceived software quality assessment methods and tools. Section 3 presents the methods and tools which we have used to collect our data, emphasizing in our software quality laboratory. Section 4 discusses results from users first impressions and how these impressions affect their entire attitude towards software quality, while section 5 summarizes the conclusions of this paper. 2. ASSESSING PERCEIVED SOFTWARE QUALITY Many definitions of software quality have been published, which in general agree on what quality means and their agreement can be enshrined by the phrase satisfaction of user requirements. In simple language, software must do what the user expects it to do. The user plays an important role in software quality and his/her perceived quality opinion is important for software developers. The methods used for assessing perceived software quality can be distinguished into two basic categories: Inquiry methods and Experimental (or laboratory) methods. Inquiry methods focus on collecting opinions directly from the users, while laboratory methods allow the specialists to conclude about the users perceived software quality by observing the users under controlled conditions inside a quality laboratory. 2.1 Inquiry methods The most commonly used inquiry methods are the Focus groups, the User Interviews and the Questionnaires. In these methods the quality assessment team members make direct questions to the users about the software being evaluated. The answers given by the users can help the quality assessment team to draw conclusions about the perceived software quality. Using focus groups so as to identify perceived software quality, the quality assessment team must know that focus group interviewing is about listening (Krueger & Casey, 2000). Focus group interviewing is about being open to hear what people have to say and about being nonjudgmental. It is also about being careful and systematic with the things people tell you. Focus groups could be used together with individual interviews. For instance, preliminary focus groups can provide useful starting point for individual interviews that involve unfamiliar topics (Morgan, 1997). The basic idea is to use an explanatory focus group to reveal the range of quality issues and experiences prior to the individual interviews. Alternatively, preliminary individual interviews can help generate focus group discussion guides by giving a feel as to how people think and talk about software perceived quality before the final focus group discussion. Questionnaires are a valuable tool for the quality assurance team. Modern software companies have recognized the importance of questionnaires and they are using them to measure user perceived quality. As argued (Kaplan, 1995), questionnaires allow focusing just on the issues of interest, since they offer complete control on the questions being asked. Furthermore, questionnaires are quantifiable and therefore are not only indicators in themselves, but also allow the application of more sophisticated analysis techniques appropriate to organizations with higher levels of quality maturity. The simple truth is that subjectivity of measurements will remain a problem in all inquiry methods, regardless of the measurements methodology. However, the adoption and application of simple rules when planning the survey or designing the questionnaire will improve the quality of the measurements. The quality assessment team must follow guidelines in order to minimize subjectivity due to various interpretations of questions or choice levels (Xenos, 1997). 12

3 2.2 Experimental methods The Experimental methods involve the observation of individual users performing specific tasks with the software under evaluation. In these methods the subjects performing the tasks are appropriate end-users that perform representative tasks, under the discrete attendance of the quality assessment team. The observation takes place in a properly designed and organized laboratory (Rubin, 1994). The use of this laboratory (often called quality laboratory or in some cases more specifically usability laboratory) is essential for the experimental methods. The most commonly used laboratory methods are the Performance Measurement, the Thinking Aloud Protocol, the Co-Discovery and the User logging. In the Performance measurement the system performance is evaluated against pre-defined criteria, like time to complete a task or numbers of errors made. Thinking Aloud Protocol requires the users to express loud their thoughts, feelings and opinions while interacting with the system. Co-Discovery requires two users to discuss the software while using it. Finally, User logging involves the record of the user s activities with the use of special equipment like microphones, cameras and specialized capturing software. In most cases, the use of laboratory methods may lead to finding software quality problems emphasizing in usability problems (Nielsen & Mack, 1994) as well as allowing the quality assessment team to make recommendations for fixing the problems thus improving the quality of the software. The main idea behind the laboratory methods is that while users are using the software the quality assessment team record a variety of data (audio, video, screen captures, benchmarks). The analysis of these data will aid the quality assessment team to conclude regarding users perceived quality. Figure 1. The Questionnaire Tool 3. CASE DESCRIPTION The results presented in this paper have been collected by the Software Quality Research Group of the Hellenic Open University ( As regards the inquiry methods used to assess user perceived software quality focus groups and interviews were used in a typical manner, while questionnaires 13

4 used by the aid of an online tool which allows the quality assessment team to apply various techniques to ensure the validity of the results as well as applying belief revision techniques (Stavrinoudis et al., 2005). Screen shots of the Online Questionnaires tool are shown in figure 1. QWCO S = n i= 1 O E S i i S n i= 1 E i S S i T i T (1) Questionnaires are usually based on the QWCO S (Qualifications Weighed Customer Opinion with Safeguards) technique (Xenos, 1997), shown in equation (1). This technique weighs users opinions according to their qualifications. In order to achieve this Oi, measures the normalized score of user i opinion, Ei measures the qualifications of user i, and n is the number of user interviewed or participating in the questionnaire-based survey. Therefore, each customer contributes to the average according to his/her qualifications. Furthermore the QWCO S technique uses safeguards to handle possible errors. Safeguard is defined as a question placed inside the questionnaire so as to measure the correctness of responses. Therefore, safeguards are not questions aiming to measure customer perceived quality, but control questions aiming to detect errors. In equation (1) Si is the number of safeguards that the user i has replied correctly to, and ST is the total number of safeguards. Since the use of the QWCU S technique implies the use of at least one safeguard in the questionnaire, division by ST is always valid. As mentioned earlier experimental methods take place in laboratories. Such a laboratory (Software Quality Assessment laboratory) has been developed and used by the SQRG team. This laboratory is suitably equipped for the performance of quality assessment of various systems. A schematic of the laboratory is shown in figure 2. Figure 2. Software Quality Assessment Laboratory The Software Quality Assessment Laboratory of SQRG consists of one room designated as the testing room and a second room designated as an observation and control room. The observation room is separated by the testing room by one-way mirror so that the members of the quality assessment team can watch the users whilst the users cannot see them. The maximum number of individuals inside the testing room is two users and in some cases a member of the assessment team, depending on the case of study. All other members of the quality assessment team are seated inside the observation room viewing the proceedings. The equipment installed in this laboratory consists of one roof-mounted video camera that records the participant s behavior, such as hand movements, facial expression and general body language throughout the experiment. A microphone is also placed near where the participants are positioned to record their utterances. Another microphone is installed in the observation room to record evaluators comments on the behavior of 14

5 the participant. Video from the camera as well as the real time image from the participant s monitor are both directed to video monitors into the observation room where they are recorded for further evaluation. A strong point of the particular Software Quality Assessment Laboratory is its setting. First of all, the members of the quality assessment team are not in the same room with the users. This eliminates almost entirely any possible biasing effects due to inadvertent non-verbal communications or mannerisms. The test room is structured in such a way that gives the user the feel of a normal office and not a laboratory. Finally the camera (roof-mounted) and the microphone are placed in such a way that are not easily observable, despite the fact that the participant is informed of their existence before the beginning of the experiment. 4. USERS FIRST IMPRESSION During the past ten years we have completed 26 software projects of various sizes developing customized software that has been delivered to organizations and has been used by a significant number of users. In all of these projects we have conducted several measurements of perceived user quality during all stages of software development, using various techniques. Our discussion hereinafter focuses on projects that have already released the software to the users, which means that what users have in their hands has already undergone several quality improvements. In such cases, a synthetic metric Subj M that combines users opinions is expected to be normally distributed around a mean value. This value may vary significantly depending on the particular case, while the same applies to skewness and kurtosis. Subj M is used to represent combined measurements that include subjective users opinions regarding software quality and it is derived by combining various metrics that may slightly differ depending on the case. By definition its values range from 0 (negative opinion) to 1 (positive opinion). Figure 3 shows the histogram for a typical project using measurements from 317 users. Regardless of what Subj M combines, similar histograms (with various means, standard deviation, skewness and kurtosis) have been derived for all measured projects. Assuming that Subj M combines metrics that cover all factors of perceived software quality, it is expected that this will be somehow the case for all projects that have released software to their users. Figure 3. The measurements for perceived software quality. In order to isolate the effect of the initial user impression on the perceived software quality, we have excluded all values that lie within 1 standard deviation of the mean. By definition, in a normal distribution this excludes approximately two thirds of the values, thus leaving this one third of the values representing the extreme opinions. The left part of figure 4 illustrates these values (98 values) that lie outside of the area {mean std. dev., mean + std. dev}. 15

6 Figure 4. The one third extreme values (left) and the revised measurements (right). Focusing on these users and analyzing their opinions we have found that approximately 20% of them have formed their opinion based on a false assumption that has resulted from their first impression of the software. An example of such case is a user claiming that a particular functionality is missing from the software and basing his negative opinion mainly on this omission, despite the fact that the particular functionality is actually available. An important observation based on hundreds of similar experiments is that this percentage (20%) derives in almost every measurement case, providing that the user sample is homogenous. Therefore we argue that regardless of the technique used to collect users opinions, some opinions will be based on false impressions. After isolating users opinions that had been based on false assumptions, we informed these users that their opinion was based on one or more wrong impressions. For example, in some cases of negative initial impressions we demonstrated the functionality that was considered as missing, or in cases of positive initial impressions we explained that features considered as a usability novelty by some users are rather a commodity in contemporary software. Consequently, we re-measured these users opinion using the same measurement technique and, to the extent possible, identical conditions to the former measurement. The right part of figure 4 illustrates the revised measurements. Although the results slightly vary from project to project, the overall conclusion is common in all measured projects. The positive opinions that had been based on an initial false impression were significantly revised and, although remained positive, they were distributed around a lower mean. On the other hand, the revisions of the negative opinions were minimum or none at all and the overall result remained very close to the former measurement. 5. CONCLUSION Based on both quantitative data such as those presented in figures 3 and 4, as well as qualitative data collected using mostly laboratory techniques, we draw the following conclusions with regard to the effect that first impressions on various quality factors have on users perception of quality. Extremely negative opinions are mostly related to users impressions about reliability and functionality, whilst extremely positive opinions are mostly related to efficiency and usability. Usability and functionality are the two factors with maximum effect on users perceived quality. Usability is based on current software use practices: if, based on these practices, a feature is expected but not found, this has a negative effect. On the other hand, a feature considered as a usability novelty is highly appreciated. 16

7 Although most users eventually focus on a small fraction of software s functionality, they expect to be able to locate all available functions easily. If certain functionalities are hard to locate, they may be easily perceived as missing and, although they may eventually located, the negative impression will remain. In a similar manner as impressionist painters perceived visual reality in terms of transient effects of light and color, users perceive software quality based on impressions related to usability and functionality. User s initial negative impressions, although may be based on false assumptions, are painted on their mind-canvas and are very hard to re-color. Therefore: Initial user impression about software quality is crucial. Positive impressions may be revised in time, but not significantly. Negative impressions are hard to change. ACKNOWLEDGEMENTS The results presented in this paper were collected from software projects in R. A. Computer Technology Institute (1997 to 2001) and in Software Quality Research Group, Hellenic Open University (2002 to 2007). I would like to thank all my colleagues in both organizations for assisting me in the data collection and analysis. Especially I would like to thank Professor Dimitris Christodoulakis director of Research Unit II in R. A. Computer Technology Institute for offering me the opportunity to contact the measurements and the guidance while analyzing the results ( ), my colleague in Hellenic Open University Professor Athanassios Skodras for the data from software projects under his supervision ( ), as well as one of my Professors in Computer Engineering and Informatics Department of Patras University ( ) Professor Athanassios Tsakalidis who being a painter himself inspire me to relate art with science. The work presented in this paper for the last 4 years is funded by the project: Support of Computer Science Studies at the Hellenic Open University ( , Greek Ministry of Education: Operational Programme Education and Initial Vocational Training ). REFERENCES Kaplan C., Clark R., and Tang V Secrets of Software Quality. McGraw Hill, New York. Krueger R.A. & Casey M.A Focus Groups: A practical Guide for Applied Research, 3 rd Edition. Sage Publications, Inc, Thousand Oaks, London. Morgan D.L Focus Groups as Qualitative Research, 2 nd Edition, Qualitative Research Methods Series. Sage Publications, Inc, Thousand Oaks, London. Nielsen J., Mack R. L Usability Inspection Methods. John Wiley & Sons, Inc., New York. Rubin J., Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. John Wiley & Sons, Inc, London. Stavrinoudis D., Xenos M., Peppas P., and Christodoulakis D Early estimation of users perception of software quality. Software Quality Journal. Vol. 13, pp Xenos M., and Christodoulakis D Measuring Perceived Software Quality. Information and Software Technology Journal, Vol. 39, Issue 6, pp

8 18