The Value of Scientometric Studies: An Introduction to a Debate on IS as a Reference Discipline

Transcription

1 IS Research Perspective The Value of Scientometric Studies: An Introduction to a Debate on IS as a Reference Discipline Detmar Straub Former Senior Editor JAIS special section on IS Research Perspectives Overview This issue of JAIS is privileged to have two exciting and insightful articles on the perennial question of the impact of our discipline, the discipline of information systems. The articles are: 1. Wade et al.: Information Systems is Not a Reference Discipline (And What We Can Do About It) by Michael Wade, Markus Biehl, and Henry Kim 2. Grover et al.: A Citation Analysis of the Evolution and State of Information Systems within a Constellation of Reference Disciplines by Varun Grover, Ramakrishna Ayyagari, Rahul Gokhale, Jaejoo Lim, and John Coffey While the two sets of authors agree on certain, specific empirical results and a number of critical concepts, such as the basic point that the field is evolving into a mature discipline, they disagree on whether IS is influencing other fields. What is fascinating to scholars intrigued by the domain of scientometrics that is, the study of the scientific process itself is that the groups could have come to such dramatically different conclusions using large, similar datasets of roughly 70,000 articles. Scientometrics, Sociometrics, and Research-on-Research Before describing the process that led to the two papers found in this issue, I would like to express some thoughts on the value of this kind of work work that deals with fundamental questions of how scientific disciplines evolve. In the literature, this is often referred to as scientometrics, alternatively as socio-metrics, or, somewhat derogatorily, as research-on-research. I would like to present the argument that such work is important, and yet is not as highly valued as it should be in some quarters. At one point in the earlier history of MIS Quarterly, in fact, the editors decided not to continue to publish research-on-research articles. What was particularly ironic about this decision was that the editor-in-chief himself had a distinguished record of publishing this kind of work! Journal of the Association for Information Systems Vol. 7 No. 5, pp / May

2 This policy of MISQ seems to have lapsed with subsequent editors, allowing the publication of such seminal articles as Benbasat and Zmud s (2003) tour de force on the need for the IT artifact in our theoretical models and, more recently, Dennis, Vallacich, Fuller, and Schneider s (2006) investigation of promotion and tenure standards. But this does not gainsay that there is still a residual feeling among some of our colleagues that articles in this domain are not as legitimate as work in IS application areas. From a strictly normative standpoint, there is every reason for professional disciplines to have an inherent interest in understanding themselves better. Our scholarly endeavor should not draw artificial boundaries within the domain of information systems, and, without any persuasive logic or reasoning, cut itself off from topics that are clearly and intimately related to the practice of the IS profession, whether it be academic or practitioner. The famous Greek incantation to Know thyself is one viewpoint on this argument, but another is that the creation of knowledge and the dissemination of that knowledge should resonate with all researchers, whether we are talking about how a big four accounting firm operates or how the IS professoriate (or, strictly speaking, any professoriate) functions. There are numerous respected academics who have taken up the challenge of learning more about the nature of academic knowledge creation. In one niche area of this broad field, for example, scholars have addressed research questions about the choice of venues for academic knowledge dissemination and how it differs from that of practitioners. Using audience and methodology as criteria, Adler and Bartholomew (1992) classified knowledge sources into three broad dissemination types: (1) academic, (2) practitioner or professional, and (3) academic-practitioner. In making distinctions among these, Bagozzi and Phillips (1991) and Van Dyne, Cummings, and McLean (1995) argue that academics differ from practitioners in their focus on conceptual clarity. When concepts are relatively new, they lack stable definitional structures (Schwab, 1980), and thus, clear definitions formulated by academics can be a critically important contribution. This suggests that managers who value definitional clarity will need to seek out academic venues since little of this information will be found elsewhere. This is just one small example of eminent researchers who have engaged in studying the processes and goals of academic entities. It hardly needs to be shown that much of this work is practical, just as is much of the best applied IS application research (Lyytinen and King, 2004). But it can also explore highly theoretical questions, an instance of which would be how knowledge is managed by scholars. In effect, it is a question of knowledge management that is just as interesting as the issue of practitioners managing their knowledge bases. A list that names all the IS scholars who have engaged in such discussions of how knowledge is disseminated in information systems or how our field has grown and developed would be long indeed. The series of articles in JAIS and CAIS in the fall of 2003 and spring of 2004 that responded to the concepts underlying Benbasat and Zmud s IT artifact essay is one limited indication of how deeply valued this stream of work is. Much of this work talks about legitimacy in a field and how it is established, for example. This question is particularly profound. This specific list reads like a Who s Who of the most prominent people in the profession. In chronological order, it follows next in Table Journal of the Association for Information Systems Vol. 7 No. 5, pp / May 2006

3 Table 1. Scientometric Articles Dealing with the IT Artifact in CAIS and JAIS CAIS JAIS Alter 2003a Hirscheim and Klein 2003 Dufner 2003 Galliers 2003 Powers 2003 Robey 2003 Deans 2003 DeSanctis 2003 McCubbrey 2003 Ives, Parks, Porra, and Silva 2004 Guthrie 2003 Lyytinen and King 2004 Wu and Saunders 2003 Iivari 2003 Myers 2003 El Sawy 2003 Holland 2003 Alter 2003b Some of the objections one hears to scientometric research may stem from the relatively superficial nature of some studies in this vein, especially journal ranking-type of studies. Whereas such studies are eminently practical in providing empirical evidence for journal tier classification, they rarely go beyond the simplest research question to explore why journals are highly or lightly regarded. Nor do they always query differences between key groups, regional and otherwise, and why they view journal quality in such a different manner. Finally, they seldom investigate the serious issues that relate to methodological choices for determining the rankings (Chua et al., 2003). The current articles raise the level of discourse to the highest level. They pose interesting research questions that have to do with the nature of the scientific enterprise. In my opinion, they are examples of what we would hope to see in the future in all of our premier journals. The Review Process and Generation of the Critiques First, let me say that the papers were conceived and written completely independently of one another, and arrived at the journal close to the same time, essentially by chance. Also, the authors were not aware of each others work until quite recently. Each paper went through an independent, rigorous review. Each had three well qualified reviewers, that is, reviewers who were themselves familiar with scientometric studies and acceptable measurement and statistical qualities of such articles. The point of sharing insights on the reviewing process is to state unambiguously that the articles independently met the criteria for publication. Had one or both papers not been able to meet the high standards of a JAIS article, we would not now have this debate in front of us. Second, when it became clear to me in my role of senior editor that both papers were on the path to acceptance, I asked each set of authors if they would consider writing an additional, shorter article critiquing their counterparts work. I made it clear that this was Journal of the Association for Information Systems Vol. 7 No. 5, pp / May

4 to be a civil dialectic, although not constraining the authors from polite criticisms of approach, assumptions, or empirics. The result was two brief critiques: 1. Grover et al.: About Reference Disciplines and Reference Differences: A Critique of Wade et al. by Varun Grover, Ramakrishna Ayyagari, Rahul Gokhale, and Jaejoo Lim 2. Wade et al.: If the Tree of IS Knowledge Falls in a Forest, Will Anyone Hear?: A Commentary on Grover et al." by Michael Wade, Markus Biehl, and Henry Kim The authors met all my guidelines. They take issue with some conclusions of their counterparts, but are consistently respectful. I am not in any way suggesting that the authors would not have taken this tact on their own, without my explicit rules of engagement, but merely pointing out that not all such response-styled, critiquing articles assume this professional level of dialogue, as I believe they should. What is even more appealing to me is that they each surface the critical assumptions and research choices of the other. Choice of journal basket is one such element. In a JAIS article a few years ago, Chua et al. (2003) produced evidence about how dramatically results can change via different journal selections. In this case, the inclusion or removal of a journal from the designated list seems to significantly alter findings, which seems to explain, at least in part, why the authors come to such radically different conclusions. The critiques make it clear that to understand the intricacies of the debate, scholars need to consider the various assumptions and research choices and come to their own conclusions about the rectitude of the articles. Alternatively, it might be the case that readers see the virtues of each and conclude that more study is required and/or that the issue is unresolved. Irrespective of what might be the community s response to these contradictory findings, I want to thank both sets of authors for their diligence in responding to reviewer comments, which is perhaps expected, but also for their willingness to step out of these customary roles and respectfully criticize the other article in the interest of advancing knowledge. References Adler, N.J., and Bartholomew, S. "Academic and Professional Communities of Discourse: Generating Knowledge on Transnational Human Resource Management," Journal of International Business Studies (23:3) 1992, pp Alter, S. "18 Reasons Why IT-Reliant Work Systems Should Replace 'The IT Artifact' as the Core Subject Matter of the IS Field," CAIS (12:23, October) 2003a, pp Alter, S. "The IS Core XI: Sorting Out the Issues About the Core, Scope, and Identity of the IS Field," CAIS (12:Article 41, November) 2003b, pp Journal of the Association for Information Systems Vol. 7 No. 5, pp / May 2006

5 Bagozzi, R.P., Yi, Y., and Phillips, L.W. "Assessing Construct Validity in Organizational Research," Administrative Science Quarterly (36:3, September) 1991, pp Benbasat, I., and Zmud, R.W. "The Identity Crisis within the IS Discipline: Defining and Communicating the Discipline's Core Properties," MIS Quarterly (27:2, June) 2003, pp Chua, C., Cao, L., Cousins, K., and Straub, D.W. "Assessing Researcher Production in Information Systems," Journal of AIS (3:6, January) 2003, pp Deans, P.C. "The IS Core III: The Core Domain Debate and the International Business Discipline: A Comparison," CAIS (12:Article 33, November) 2003, pp Dennis, A.R., Valacich, J.S., Fuller, M.A., and Schneider, C. "Research Standards for Promotion and Tenure in Information Systems," MIS Quarterly (30:1, March) 2006, pp DeSanctis, G. "The Social Life of Information Systems Research: A Response to Benbasat and Zmud's Call for Returning to the IT Artifact," JAIS (4:16, December) 2003, pp Dufner, D. "The IS Core-I: Economic and Systems Engineering Approaches to IS Identity," CAIS (12:31, November) 2003, pp El Sawy, O. "The IS Core - IX: The 3 Faces of IS Identity: Connection, Immersion, and Fusion," CAIS (12:Article 39, November) 2003, pp Galliers, R.D. "Change as Crisis or Growth? Toward a Trans-disciplinary View of Information Systems as a Field of Study: A Response to Benbasat and Zmud's Call for Returning to the IT Artifact," JAIS (4:13, November) 2003, pp Guthrie, R.A. "The IS Core V: Defining the IS Core," CAIS (12:Article 35, November) 2003, pp Hirscheim, R., and Klein, H.K. "Crisis in the IS Field? A Critical Reflection on the State of the Discipline," JAIS (4:10, October) 2003, pp Holland, C. "The IS Core - X: Information Systems Research and Practice: IT Artifact or a Multidisciplinary Subject?" CAIS (12:Article 40, November) 2003, pp Iivari, J. "The IS Core VII: Towards Information Systems as a Science of Meta- Artifacts," CAIS (12:Article 37, November) 2003, pp Ives, B., Parks, M.S., Porra, J., and Silva, L. "Phylogeny and Power in the IS Domain: A Response to Benbasat and Zmud's Call for Returning to the IT Artifact," JAIS (5:3) 2004, pp Lyytinen, K., and King, J.L. "Nothing At The Center?: Academic Legitimacy in the Information Systems Field," JAIS (5:6, June) 2004, pp McCubbrey, D.J. "The IS Core - IV: IS Research: A Third Way," CAIS (12:Article 34, November) 2003, pp Myers, M.D. "The IS Core VIII: Defining the Core Properties of the IS Disciplines: Not Yet, Not Now," CAIS (12:Article 38, November) 2003, pp Powers, D.J. "The IS Core - II: The Maturing IS Discipline: Institutionalizing our Domain of Inquiry," CAIS (12:Article 32, November) 2003, pp Robey, D. "Identity, Legitimacy and the Dominant Research Paradigm: An Alternative Prescription for the IS Discipline: A Response to Benbasat and Zmud's Call for Returning to the IT Artifact," JAIS (4:15, December) 2003, pp Schwab, D.P. "Construct Validity in Organizational Behavior," Research in Organizational Behavior (2) 1980, pp Van Dyne, L., Cummings, L.L., and McLean Parks, J. "Extra Role Behaviors: In Pursuit of Construct and Definitional Clarity (A Bridge over Muddled Waters)," Research in Organizational Behavior (17) 1995, pp Wu, Y.A., and Saunders, C. "The IS Core VI: Further Along the Road to the IT Artifact," CAIS (12:Article 36, November) 2003, pp Journal of the Association for Information Systems Vol. 7 No. 5, pp / May

6 JAIS SENIOR EDITORS Soon Ang Nanyang Technological University, Singapore Varun Grover Clemson University, USA Elena Karahanna University of Georgia, USA JAIS EDITORIAL BOARD Ritu Agarwal University of Maryland Yolande E. Chan Queen's University, Canada Galletta Dennis Temple University, USA Ho Geun Lee Yonsei University, Korea Stuart E. Madnick Massachusetts Institute of Technology, USA Anne Massey Indiana University, USA B. Jeffrey Parsons Memorial University of Newfoundland, Canada Sudha Ram University of Arizona, USA Christina Soh Nanyang Technological University, Singapore Dov Te'eni Tel Aviv University, Israel Gillian Yeo Nanyang Business School, Singapore EDITOR Kalle Lyytinen Case Western Reserve University Izak Benbasat University of British Columbia, Canada Rudy Hirschheim Louisiana State University Robert Kauffman University of Minnesota, USA Michael Barrett University of Cambridge, UK Roger H. L. Chiang University of Cincinnati, USA Guy G. Gable Queensland University of Technology, Australia Jae-Nam Lee Korea University, Korea Ann Majchrzak University of Southern California, USA Emmanuel Monod Dauphine University, France Nava Pliskin Ben-Gurion University of the Negev, Israel Balasubramaniam Ramesh Georgia State University, USA Ananth Srinivasan University of Auckland, New Zealand Viswanath Venkatesh University of Arkansas, USA Youngjin Yoo Case Western Reserve University, USA ADMINISTRATIVE PERSONNEL Eph McLean AIS, Executive Director Georgia State University Cynthia Beath University of Texas at Austin, USA Wynne Chin University of Houston Matthew R. Jones University of Cambridge, UK Kai H. Lim City University of Hong Kong, Hong Kong Ryutaro Manabe Bunkyo University, Japan Eric Monteiro Norwegian University of Science and Technology Jan Pries-Heje Copenhagen Business School, Denmark Suzanne Rivard Ecole des Hautes Etudes Commerciales, Canada Kar Yan Tam Hong Kong University of Science and Technology Richard T. Watson University of Georgia, USA Kevin Zhu University of California at Irvine, USA J. Peter Tinsley Deputy Executive Director Association for Information Systems ISSN: Robert Fichman Boston College, USA Juhani Iivari University of Oulu, Finland Yair Wand University of British Columbia, Canada Anandhi S. Bharadwaj Emory University, USA Ellen Christiaanse University of Amsterdam Chidambaram Laku University of Oklahoma, USA Mats Lundeberg Stockholm School of Economics, Sweden Ji-Ye Mao Remnin University, China Jonathan Palmer University of William and Mary, USA Dewan Rajiv University of Rochester, USA Rajiv Sabherwal University of Missouri - St. Louis, USA Bernard C.Y. Tan National University of Singapore, Singapore Bruce Weber London Business School, UK Reagan Ramsower Publisher, JAIS Baylor University 246 Journal of the Association for Information Systems Vol. 7 No. 5, pp / May 2006