Winsome St John, Patricia Johnson Purpose: Scope: Findings: Conclusions:

Transcription

1 The Pros and Cons of Data Analysis Software for Qualitative Research Winsome St John, Patricia Johnson Clinical Scholarship Purpose: To explore the use of computer-based qualitative data analysis software packages. Scope: The advantages and capabilities of qualitative data analysis software are described and concerns about their effects on methods are discussed. Findings: Advantages of using qualitative data analysis software include being freed from manual and clerical tasks, saving time, being able to deal with large amounts of qualitative data, having increased flexibility, and having improved validity and auditability of qualitative research. Concerns include increasingly deterministic and rigid processes, privileging of coding, and retrieval methods; reification of data, increased pressure on researchers to focus on volume and breadth rather than on depth and meaning, time and energy spent learning to use computer packages, increased commercialism, and distraction from the real work of analysis. Conclusions: We recommend that researchers consider the capabilities of the package, their own computer literacy and knowledge of the package, or the time required to gain these skills, and the suitability of the package for their research. The intelligence and integrity that a researcher brings to the research process must also be brought to the choice and use of tools and analytical processes. Researchers should be as critical of the methodological approaches to using qualitative data analysis software as they are about the fit between research question, methods, and research design. JOURNAL OF NURSING SCHOLARSHIP, 2000; 32:4, SIGMA THETA TAU INTERNATIONAL. [Key words: qualitative research, computer applications, qualitative data analysis, methods] * * * Use of computer-based qualitative data analysis software (QDAS) has quietly revolutionized qualitative research (Fielding & Lee, 1996; Mangabeira, 1996). Many packages, with a range of capabilities, have been offered over the last two decades as qualitative researchers have sought more effective ways to manage and analyse their data. Computer programs devoted to organizing qualitative data for analysis have been available since 1984 (Tesch, 1991), when they were mainly developed by researchers computerizing their own analytical and management systems. Before that time, qualitative researchers adapted database managers and word processors to assist them in their analysis (Ross, 1994). More recently, the development and sale of QDAS packages have become increasingly subject to commercial considerations. Rather than unthinkingly using QDAS, researchers should pause and consider that technology is more than a tool. Technology requires researchers to reframe ideas about what can be done and how it is done. It also may have predetermined what is drawn to the researcher s attention. Although many see these attributes as positive features (Brown, 1996; Tesch, 1991; Richards & Richards, 1994; Ross, 1994), others have a sense of unease, believing that computer programs may entail data analysis strategies that are inconsistent with the methodological and theoretical foundations of qualitative research (Coffey, Holbrook, & Atkinson, 1996; Kelle, 1996; Seidel, 1991; Taft, 1993). Therefore, researchers should understand and reflect on the issues and methodological implications of using computers to assist in qualitative data analysis. This paper critically examines debates about the use of computer-assisted analysis in qualitative research. Qualitative Data Analysis Software (QDAS) Applications Many different software programs designed to assist qualitative data management are available. Commonly used programs in qualitative nursing research include: The Ethnograph, QUALPRO, NUD*IST, and MARTIN. Researchers using these software programs are able to retrieve Winsome St John, RN, RM, PhD, FRCNA, and Patricia Johnson, RN, Intensive Care Cert., MN (Hons), MRCNA; both at School of Nursing, Faculty of Nursing and Health, Griffith University, Queensland, Australia. Correspondence to Dr. St John, School of Nursing Faculty of Nursing and Health, Griffith University, PMB 50 Gold Coast Mail Centre, Queensland, Australia w.stjohn@mailbox.gu.edu.au Accepted for publication: January 14, Journal of Nursing Scholarship Fourth Quarter

2 and code data in a variety of ways, and some programs, such as NUD*IST, have the capability to assist exploration and testing of data and theory construction (Richards, 1998). This paper does not include discussion of the features of specific programs. For discussion and analysis of the capabilities of QDAS programs, readers are referred to Barry (1998), Dohan and Sanchez-Jankowski (1998), Fielding and Lee (1997), Lewins (1996), Miles and Weitzman (1996), Richards and Richards (1991), Russell and Gregory (1993) or Tak, Nield, and Becker (1999). Although computer software programs designed for qualitative research assist in the management and analysis of qualitative data, they do not define conceptual categories or themes, develop conceptual diagrams, write memos and journals, gain insight into phenomena, or develop theoretical understanding. Users of software programs can focus more fully on analysis because preparing, organising, and managing data are facilitated. Advantages of QDAS To a large group of qualitative researchers, QDAS has many eagerly grasped advantages. Qualitative researchers embracing computer technology no longer ask whether to use a QDAS package, but which one to use (Tesch, 1991). Using computer software for qualitative data analysis is well established, with organizations and Web-based information sites such as the Computer-Aided Qualitative Data Analysis Networking Project (CAQDAS) (Lewins, 1996) providing education, information, and advice about choice of computer applications. These researchers contend that computers are no more mechanical than paper, copying machines, typewriters, audio recorders, scissors, glue, and file cabinets (Tesch, 1991; Seidel, 1991). Advantages of using QDAS have been widely discussed in the literature. They include: an ability to deal with large amounts of qualitative data, reducing the amount of time needed for manual handling tasks, increased flexibility and thoroughness in handling data, providing for more rigorous analysis of data, and providing a more visible audit trail in data analysis. Convenience and Efficiency QDAS can enable researchers to save time performing tedious manual tasks, such as multiple copying, cutting and pasting, filing, and refiling. A computer can replace file cabinets. QDAS allows researchers to indicate the boundaries of text segments, attach information to text segments, file information, and search and assemble text segments with the same code. When writing the research report, material does not have to be retyped. Russell and Gregory (1993) pointed out that, although qualitative research is often a more lengthy process than is quantitative research, an effective data management software package can significantly help to expedite the process. Reducing the manual and clerical tasks associated with qualitative research allows researchers more time to attend to the real work of analysis. Flexibility and Thoroughness QDAS provides greater flexibility and complexity in handling data. It contains few limitations on the numbers of codes that can be generated and few limits to the number and size of transcripts that can be included in a research project. With QDAS, researchers have greater ability to change their coding systems, add notes, and define concepts as the research project progresses. Many software products allow for data to be coded in multiple ways, or for more than one code word to be used for a particularly rich piece of datum. Codes can be developed, merged, deleted, and moved as data are analysed. Data can be searched and ideas about the data can be linked. Most software products have ways of storing memos and journals, so that researchers can attach analytical writing and information about contexts of data as they develop their ideas. New products go beyond textual data, allowing users to link to pictures, include sounds, and develop graphical representation of models and ideas. Another characteristic that QDAS offers is the capability to support multiple researchers working on one project. Validity and Rigour QDAS can assist in supporting the rigour and validity of research because examination of data can be more complete and rigorous. An important feature of QDAS is the way it enables all data related to a topic to be examined. This ability contrasts with the human tendency of privileging parts that fit with one s own assumptions and world views. QDAS can enable researchers to examine their own assumptions and biases. Segments of data are not likely to be lost or overlooked, and all material coded in a particular way will be retrieved. A computer will not stop searching once it has found an example or vignette that will do (Coffey et al., 1996), but will continue searching to identify all instances of a code or search item in the data. Qualitative researchers have at times been accused of not making their analytical strategies explicit and only limited literature exists about qualitative data analysis processes. QDAS helps to make analysis processes more explicit. Part of the debate about QDAS (Kelle, 1997) perhaps has more to do with analytical strategies being more visible than about using QDAS itself. In making analyses visible, QDAS provides a basis for establishing credibility and validity by making available an audit trail that can be scrutinised. New Ways of Analysing Another challenge of QDAS is that some software developers argue that the capabilities of their packages go beyond assisting researchers with the manual and clerical tasks of research and project management, enabling them to deal will large volumes of data, examine complex relationships in the data, and engage in theory building. As far back as 1982, Sproull and Sproull were pointing out that qualitative researchers face a cruel trade-off between large amounts of rich qualitative data and the burdensome need to analyse it (p. 283). More recently, Bryman and Burgess (1994) described the amount of data in qualitative research 394 Fourth Quarter 2000 Journal of Nursing Scholarship

3 as voluminous, unstructured, and unwieldy (p. 216). Pfaffenberger (1988) suggested that, in the past, the volume of data generated in qualitative research restricted researchers to small data sets and simple analytical strategies. He argued that manual methods of qualitative research analysis can be likened to the more simple types of analyses that were carried out in quantitative research before the introduction of statistical packages. He posited that, by using computer-based technology and working quickly and conveniently with much larger units of data, qualitative researchers can now apply more sophisticated analytical techniques than were possible in the past. Use of QDAS might reveal patterns not noticed in earlier analyses (Lee & Fielding, 1991, p. 13). Many software packages enable examination of relationships or links in the data in ways that would not be humanly possible, by using hypertext links or Boolean searches of index systems. For example, many packages support automatic coding by enabling searches for words, patterns, or structures within the data, and by allowing selective searching for co-occurring codes and sequencing of codes. Codes can be cross-matched with demographic data and frequencies of instances can be counted. The capabilities of QDAS not only allow for more complex analysis, but began to blur traditional boundaries between qualitative and quantitative research. Kelle (1996) pointed out that computer programs may be used with traditional coding strategies for open-ended questionnaires, allowing development of quantitative variables from unstructured textual data. For example, the Q method provides a way to statistically measure subject factors in a given situation through analysing participants statements (Brown, 1996) and some packages enable data to imported or exported from quantitative packages such as SPSS. Concerns About the Use of QDAS Despite the increasingly widespread use of computer technology in qualitative research and the advantages delivered, a number of concerns are associated with an unquestioning and uncritical acceptance and utilisation of QDAS. Although little research has been carried out investigating the effects of the packages on research, the small body of existing research indicates that the particular tool used may affect analysis. Walker (1993) compared three programs and concluded that researchers need to pay attention to the fit between methods and the assistance provided by the QDAS. Researchers have expressed concern that using QDAS packages may result in: a focus on quantity instead of meaning, homogenisation of qualitative data analysis approaches, a privileging of coding and retrieval methods, distancing of the researcher from the data, inappropriate use of technology, time consumed in learning to use computer packages, pressures or expectations that all qualitative researchers will use them, and increased commercialism. A Focus on Quantity Instead of Meaning A major concern is that QDAS packages may create an obligation to obtain large amounts of data. Seidel (1991) questioned the ability to integrate, analyse, and meaningfully understand very large amounts of data, and also whether being pressured to collect large amounts of data would shape the analytical process. Moreover, he argued that attempting to manage large volumes of data would give rise to insufficient and insubstantial analysis, because the focus of analysis would be on quantity rather than meaning. A consequence of this approach is breadth rather than depth. Researchers may have so many transcripts that they may be unable to read them all. The basis for undertaking qualitative research is to explore complex thick data for meanings, conceptual understanding, and discourses. These understandings are not delivered by volume. The need to gather and analyse large amounts of data could be questioned, given the quest for richness and quality in data and substance and depth in analysis. Generating codes can be so easy in most QDAS that the researcher can almost drown in the number of codes that can be created. With so many codes, researchers can have problems recalling them all as they code and analyse data, and may not be able to see the conceptual forest for the coding trees. Homogenising Qualitative Data Analysis Approaches QDAS might structure the way researchers see data and conduct research and might even cause researchers to look at data in ways that are incongruent with some methodological approaches (Agar, 1991; Coffey et al., 1996). When using computers, data must be structured before being entered into the computer, making analysis rigid and inflexible (Tesch, 1991), possibly creating a situation in which software packages become a straight jacket (Holbrook & Butcher, 1996). Many programs assume data is textual. However, in addition to text and transcripts, qualitative data might include photographs, maps, brochures, films, audiotapes, videotapes, diagrams, organisational charts, documents, music, paintings, drawings, objects, and artifacts. The ability to neatly plug data into a computer raises the concern that researchers may design questions to fit the computer software available, resulting in QDAS driving the question and not vice versa. Agar (1991) expressed the concern that computers had the potential to shift from being aids to methods to being the method. Privileging Code and Retrieval Methods Most QDAS packages are based on an assumption that qualitative data analysis is based on coding and retrieval. Thus, QDAS may lead researchers to favour a code and retrieval method of data analysis. Many software packages have been developed by grounded-theory researchers who use coding and retrieval methods to engage in theory building. However, not all qualitative researchers want to engage in theory building. Although most qualitative researchers code their data, coding is not the only way to analyze data. Coding represents the breaking up of data into pieces. Although Journal of Nursing Scholarship Fourth Quarter

4 qualitative researchers engage in data reduction, care needs to be taken that these pieces of data do not become decontextualised. An overemphasis on coding and retrieval may lead researchers to miss the essence of meanings in data (Tallerico, 1991; Miles & Huberman, 1984). Distancing the Researcher from the Data Reduction of data can distance the researcher from the data, resulting in loss of meaning and context and creating sterile and dehumanised data the very critique that lead to the development of qualitative research. Sandelowski (1995) warned that researchers need to recognize the ways in which techniques and technology can undermine the holistic imperative and alter the aesthetic effect of qualitative research. A concern is that coding may reify the relationship between researcher and data (Seidel, 1991), with pieces of coded data becoming inflexible objects, losing richness and complexities such as voice tones, smells, feelings, sounds, or colour in the process. Language has words that may have many meanings. The meaning of words or phrases are derived from context, body language, and inflection, and a meaning may be implied without using actual words. Using technology in qualitative research may strip away the meanings with which inflection and body language imbue words. The codes may become objects manipulated by the researcher, with a life and meaning of their own, divorced from their contexts. Consequently, computer technology can distance the researcher from intimate interaction with their data. Countering these concerns, Pfaffenberger (1988) pointed out that because huge amounts of data are usually collected in qualitative research, the researcher is kept at arm s length from data no matter what analytical method is used. Most researchers code their data, with or without QDAS. Whatever the approach to analysis, researchers who manually code their data are still engaging in data reduction. Richards (1998) suggested that while closeness to data is important in qualitative research, the assumptions inherent in this notion require examination. She argued that while maintaining connection to the vitality of data is important, so is taking a wide-angle view and achieving analytical distance from data. QDAS as a Distraction Inappropriate use of QDAS or difficulties with computer technology is another concern. A computer s bells and whistles (Tallerico, 1992) can distract from engaging with and thinking about data. Seidel (1991) urged researchers not to become so infatuated with technology that it drives research; instead, it should serve research. Just as a writer who has used a sophisticated word processor may not necessarily produce good writing, activity and industriousness using QDAS cannot be mistaken for good analysis. Diligently sorting and filing data does not necessarily equate with valid and meaningful analysis. Many find that far from being a time saver, QDAS may be a time waster. Entry, coding, and management of data using a computer can be time consuming. A researcher many need to invest much time learning to use a computer package, invariably encountering computer-induced emotional and time-wasting traumas along the way. Just when researchers believe they have mastered a particular program, QDAS developers will produce an expensive new, improved version of their product that will take time and energy to master, even for the computer literate. Precious time will also be taken from interaction with data by the need to choose the most appropriate package for a particular research project. Researchers will often choose the package with which they are most familiar. Different QDAS packages have individual capabilities that may make them more or less appropriate for a particular research project. Many researchers do not have the knowledge to make informed choices from among the many programs available. This problem has led to the development of experts who focus on evaluating QDAS packages and providing advice to researchers (Lewins, 1996; Russell & Gregory, 1993). QDAS as an Expectation Availability of software programs to assist data analysis may create pressure on qualitative researchers to adopt them so as to legitimate their research findings, establish the kind of validity associated with quantitative research, and help compete for research funds traditionally given to quantitative projects (Richards & Richards, 1991; Sandelowski 1995). Sandelowski (1995) expressed the fear that: The very availability of computerized systems for qualitative research may impose the obligation to use them, often without regard to whether that use is mandated by the purposes of the research. (p. 208) If all qualitative research were to be conducted using QDAS, or worse still, with a limited number of software packages, then the products of research are in danger of becoming homogenous, deterministic, and limited. Commercialism The success of QDAS packages in an expanding market is accompanied by the threat of commercialism. Unbiased advice is usually the first casualty when money is at stake. In the future, researchers may not have the luxury of healthy academic critique of packages and debate about the merits and methodological effects of QDAS, as cash-strapped universities seek survival in an era of economic rationalism. Discussion and Conclusions Researchers should always have choices. To encourage diversity, creativity, new ideas, innovation, and progress in the development of qualitative research and methodology, researchers need to analyse each research project on the basis of purpose, question, and methods. The appropriate research tools should be selected according to the requirements of 396 Fourth Quarter 2000 Journal of Nursing Scholarship

5 Table 1. Considerations When Choosing a Qualitative Data Analysis Package Do I know the capabilities of the packages that are available? (If not, where can I get advice?) Do I have the resources to purchase appropriate hardware and software, and to prepare data for analysis? Do I know how to use the selected package? (If not, do I have access to education and help?) What are the advantages and disadvantages of this package for my research? What time and effort will be required if I use this package? What purpose will this package serve for this research project? Will this package handle the type of data I intend to collect? How do I intend to use the package? Will this package enable flexible handling of data for this project? Will this package enable me to interact with and reduce data in a way that is consistent with my methods? Will this package maintain my data in context? What processes will I use to ensure that I maintain methodological integrity while using this package? Have I reviewed, corrected, and interacted with my data before entering it into the computer for analysis? each project. Questioning assumptions and the way data are analysed may generate new methods, new ways of thinking about the world, new understandings and knowledge. Using a QDAS package should not be an automatic assumption for every qualitative research project, because they cannot accomplish every research purpose. If researchers choose to use a QDAS package, they should consider its effects on their research (Table 1). QDAS packages should be used in a manner appropriate to the particular methods chosen, and the researcher should remain in charge of the analysis. Healthy critique and debate about the effects of these software packages on methods of analysis should be encouraged. Tools should be reviewed and developed to meet the needs of different types of researchers and different methods. Those who have avoided computers may need to grapple with the pleasures and pains of using a computer. Current debates about the methodological implications of QDAS, provision of support, and comparative analysis of software packages are welcome developments. Communication links between developers and users of software packages should be maintained. QDAS developers should maintain their commendable openness to debate, new ideas, and the need to address the methodological and practical needs of researchers perhaps even providing an interface among different packages. Finally, the intellect and integrity that a researcher must bring to research should also be brought to the choice and use of tools and analytical processes. Researchers must be as critical of the methodological approaches to how QDAS is used as the fit among research question, methods, and research design. JNS References Agar, M. (1991). The right brain strikes back. N.G. Fielding, & R.M. Lee (Eds.), Using computers in qualitative research ( ). London: Sage. Barry C. (1988). Choosing qualitative data analysis software: Atlas/ti and NUD*IST compared. Sociological Research Online [On-line serial], 3(3), Available: Brown, S.R. (1996). Q methodology and qualitative research. Qualitative Health Research, 6(4), Bryman, A., & Burgess, R.G. (1994). Analysing qualitative data. London: Routledge. Coffey, A., Holbrook, B., & Atkinson, P. (1996). Qualitative data analysis: Technologies and representations. Sociological Research Online [On-line serial], 1(1), Available: Dohan, D. & Sanchez-Jankowski, M. (1998). Using computers to analyze ethnographic field data: Theoretical and practical considerations. Annual Review of Sociology, 24, Fielding, N.G., & Lee, R.M. (1996). Diffusion of a methodological innovation: CAQDAS in the UK. Current Sociology, 44(3), Fielding, N.G., & Lee, R.M. (1997). Applications of computer software in the sociological analysis of qualitative data. Bulletin de Methodologic Sociologique, 57, Holbrook, A., & Butcher, L. (1996). Uses of NUD*IST software in educational research: The literature, the hard questions and some specific research applications. ERA/AARE Conference, Singapore. Kelle, U. (1996). Computer-assisted qualitative data analysis in Germany. Current Sociology, 44(3), Kelle, U. (1997). Theory building in qualitative research and computer programs for the management of textual data. Sociological Research Online, 2(2), Lee, R.M., & Fielding, N.G. (1991). Computing for qualitative research: Options, problems and potential. N.G. Fielding & R.M. Lee (Eds.), Using computers in qualitative research (1-13). London: Sage. Lewins, A. (1996). The CAQDAS networking project: Multilevel support for the qualitative research community. Qualitative Health Research, 6(2), Mangabeira, W.C. (1996). CAQDAS and its diffusion across four countries: National specificities and common themes. Current Sociology, 44(3), Miles, M.B., & Huberman, A.M. (1984). Qualitative data analysis. Beverly Hills, CA: Sage. Miles, M.B., & Weitzman, E.A. (1996). The state of qualitative data analysis software: What do we need? Current Sociology, 44(3), Pfaffenberger, B. (1988). Microcomputer applications in qualitative research (Qualitative Research Methods No. 14). Newbury Park: Sage. Richards, L. (1998). Closeness to data: The changing goals of qualitative data handling. Qualitative Health Research, 8(3), Richards, L., & Richards, T. (1991). Computing in qualitative analysis: A healthy development? Qualitative Health Research, 1(2), Richards, T.J., & Richards, L. (1994). Using computers in qualitative research. N.K. Denzin & Y.S. Lincoln (Eds.), Handbook of qualitative research ( ). Thousand Oaks: Sage. Ross, B.A. (1994). Use of a database for managing qualitative research data. Computers in Nursing, 12(3), Russell, C.K., & Gregory, D.M. (1993). Issues for consideration when choosing a qualitative data management system. Journal of Advanced Nursing, 18, Sandelowski, M. (1995). On the aesthetics of qualitative research. Image: Journal of Nursing Scholarship, 27, Seidel, J. (1991). Method and madness in the application of computer technology to qualitative data analysis. N.G. Fielding & R.M. Lee (Eds.), Using computers in qualitative research ( ). London: Sage. Sproull, L.S., & Sproull, R.F. (1982). Managing and analyzing behavioral records: Explorations in nonnumeric data analysis. Human Organization, 41, Taft, L.B. (1993). Computer-assisted qualitative research. Research in Nursing and Health, 16, Tallerico, M. (1992). Computer technology for qualitative research: Hope and humbug. Journal of Educational Administration, 30(2), Tesch, R. (1991). Computer programs that assist in the analysis of qualitative data: An overview. Qualitative Health Research, 1(3), Walker, B.L. (1993). Computer analysis of qualitative data: A comparison of three packages. Qualitative Health Research, 3(1), Journal of Nursing Scholarship Fourth Quarter