De staat van de sociale wetenschap en hoe die te meten. Paul Wouters and Thed van Leeuwen 27 September, 2012

Transcription

1 De staat van de sociale wetenschap en hoe die te meten Paul Wouters and Thed van Leeuwen 27 September, 2012

2 2

3 3

4 4

5 5

6 6

7 7

8 An example The Dutch architect Rem Koolhaas. Appointed as Professor at Harvard University. WoS covered publications: 75 papers Total number of citations: 3 H-index= 1 73 of the 75 publications are never cited The most used journal is Architecture d aujourd hui. All papers appearing in that journal, are cited in total 28 times.

9 9 Overview of the organization of Dutch research evaluation Standard Evaluation Protocol (SEP 2003, 2009) Association of Dutch Universities (VSNU) National Research Council (NWO) Royal Dutch Academy of Sciences (KNAW) Report Quality indicators for research in the Humanities (Committee on quality indicators for the humanities, November 2011). Report Towards a framework for the quality assessment of social science research (Committee on quality indicators for the social sciences, March 2013). Key issues that were addressed in both reports: How to deal with heterogeneity? [without standardizing it away] Publication cultures Societal relevance

10 10

11 Quality Aspects Assessment criteria Indicators Articles Monographs Scholarly publications Chapters in books Dissertations Other output Scholarly output Scholarly use of output Reviews Citations Other evidence of use Evidence of scholarly recognition Scholarly prizes Personal grants Other evidence of recognition

12 Quality Aspects Assessment criteria Indicators Articles in specialist publications Monographs for a wider public Societal publications Chapters in books for a wider public Other societal output Societal quality Societal use of output Projects in collaboration with civil-society actors Contract research Demonstrable civil-society effects Other evidence of use Evidence of societal recognition Societal prizes Other evidence of Societal recognition

13 Web of Science (WoS - CWTS) 30 million publications from 1981 onwards 500 million citations are linked to these source publications journal titles measured over the last 10 years. 48 million addresses of institutions and organizations

14 Disciplinary distribution of journals titles in the Natural, Life, Medical and Engineering sciences titles in the Social and Behavorial Sciences titles in the Humanities. However, these numbers are rapidly changing due to competition from other systems, such as Scopus, Google Scholar, etc.

15 Assessment of WoS Coverage Citing/Source Non- WoS WoS Non-Wos Journals?%?% Books Conference proceedings Cited/Target Non- WoS WoS Reports Etc.

16 Total WoS Database (2002) Citing/Source Non- WoS WoS 25% 75% Cited/Target Non- WoS WoS

17 Biochemistry & Molecular Biology Citing/Source Non- WoS WoS 8% 92% Cited/Target Non- WoS WoS

18 Humanities Citing/Source Non- WoS WoS 83% 17% Cited/Target Non- WoS WoS

19 Social & behavioral sciences 100% 90% References non-isi References ISI 80% 70% 60% 50% 40% 30% 20% 10% 0% ECONOMICS AND BUSINESS EDUCATIONAL SCIENCES MANAGEMENT POLITICAL AND PLANNING SCIENCE AND PUBLIC ADMINISTRATION PSYCHOLOGY SOCIAL AND SOCIOLOGY AND BEHAVIORAL ANTHROPOLOGY SCIENCES, INTERDISCIPLINARY

20 Difference between the internal registration system & representation WoS 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% (Bio)medicine Social sciences Philosophy History All Publications WoSPublications Law School of Management School of Economics Dominance university hospital in WoS realm extremely visible Law and Humanities disappear in WoS realm 20

21 Shift to SSCI journals DEN FRA GER NL SWE UK USA

22 Number of authors per paper: Three clusters of research fields AUTHORS/PAPER *PSYCHOL,MULTID *PSYCHOL,EXPER *ANTHROPOLOGY #ARCHAEOLOGY *EDUCAT RES *LINGUISTICS #LANG&LINGUIST #RELIGION #HISTORY #LITERATURE #PHILOSOPHY YEARS

23 Cited Half Life Humanities (A&HCI) Social Sciences (SSCI) 4 Informat & Library Sci 5 Theatre; Film Radio TV Environmental Stud 6 Political Sci; Educat Sci 7 Art Social Work 8 Archeology; Asian Stud Family Stud 9 Literature Clin Psychol; Anthropol 10 Philosophy; Linguistics Experimental Psychol 11 Religion; Music Communication 12 Economics; Devel Psych 13 History Law; Social Psychol 14 Lang & Linguist (A&HCI) Sociology

24 Phy Actual versus field normalized impact (CPP/FCSm) displayed against the output. CPP/FCSm Soc Eng Soc Med Hum Mat Phy Psy Med Phy Eng Che Psy Bio Che Psy Bio Med Bio Med Med Med Env Med Phy Large output can be combined with a relatively low impact TOTAL PUBLICATIONS

25 60 50 Med H-Index displayed against the output. 40 Med H-index Med Eng Soc Mat Hum Soc Phy Eng Psy Phy Bio Bio Med Med Med Che Psy Che Bio Phy Psy Med Env Phy Larger output is strongly correlated with a high H- Index value. High impact authors get low H- index values TOTAL PUBLICATIONS

26 26

27 Scholarly output Criteria Indicators Metis categories Scholarly publications Articles Monographs Chapters in books Dissertations Other output Journals Books Chapters Theses Scholarly use of output Reviews Citations Other evidence of use (WoS) Book Reviews WoS/Scopus/GS Citations Influencing other scholars Evidence of scholarly recognition Scholarly prizes Personal grants Other evidence of recognition Other Other Review committees, editorial boards, etc.

28 Criteria Societal quality Indicators Metis categories Societal publications Articles in specialist publications Monographs for a wider public Chapters in books for a wider public Other societal output Non scholarly journals Monographs for a wider public Chapters in books for a wider public Media appearances Societal use of output Projects in collaboration with civil-society actors Contract research Demonstrable civil-society effects Other evidence of use Other Reports Participation in advisory councils, or the public debate Media appearances Evidence of societal recognition Societal prizes Other evidence of Societal recognition Other Participation in advisory councils, or the public debate 28

29 Citation in two contexts Most citation theories based on communication system of science This is not identical to the social institution of evaluation in science Explaining the social life of citation indicators should be based on the latter Example: the black hole in informed peer review Puzzle: the combination of the two paradigms

30 Peter Dahler-Larsen The Evaluation Society Evaluations are not something that the individual can reject Evaluation as disembedded reflexive social practice Evaluation consists of: Evaluand Criteria Systematic methodology Purpose

31 Evaluations are liminal One often has the feeling that there should have been a clear-cut plan for the purpose and process of an evaluation, but this is often not the case. ( ) people realize too late that they had very different notions of plans for evaluation ( ) The purpose of the evaluation constitutes an ongoing controversy rather than a common logical starting point. (p. 15)

32 A good question If we have only imprecise or invalid indicators available, how can one be so sure that it is better to evaluate than not to evaluate?

33 Evaluation in organization theory The rational organization The learning organization The institutionalized organization Clear goals Predictable Learning cycle Positive feedback Ritualistic Legitimacy Power struggle

34 Evaluation in sociological theory Modernity Reflexive modernity The audit society Rationality Transparency Fragmentation Lack of use Evaluation culture Neo-rigorism Rationality strikes back

35 Evaluation Machines Primary function: make stuff auditable Mechanization of control degradation of work and trust? (performance paradox) Risks for evaluand and defensive responses What are their costs, direct and indirect? Microquality versus macroquality lock-in Goal displacement & strategic behaviour

36 Constitutive effects Limitations of conventional critiques (eg perverse or unintended effects ) Effects: Interpretative frames Content & priorities Social identities & relations (labelling) Spread over time and levels Not a deterministic process Democratic role of evaluations