Productivity versus citation impact: A study of persons, not just authors

Size: px

Start display at page:

Download "Productivity versus citation impact: A study of persons, not just authors"

Suzan Miller
5 years ago
Views:

1,4 1,2 1,8,6,4,2 Annual productivity Impact Up to 3 31-4 41-5 51-6 Over 6 Productivity versus citation impact: A study of persons, not just authors Lin Zhang North China University of Water

1 1,4 1,2 1,8,6,4,2 Annual productivity Impact Up to Over 6 Productivity versus citation impact: A study of persons, not just authors Lin Zhang North China University of Water Resources and Electric Power, Zhengzhou, China & Centre for R&D Monitoring (ECOOM), KU Leuven, Belgium Gunnar Sivertsen Nordic Institute for Studies in Innovation, Research and Education, Oslo, Norway

2 Outline 1. Introduction 2. Data 3. Methods 4. Results 5. Conclusions

3 Outline 1. Introduction 2. Data 3. Methods 4. Results 5. Conclusions

4 Collaboration on studying policy-related research questions with enriched bibliometric data Lin Zhang*, Ronald Rousseau, Gunnar Sivertsen. Science deserves to be judged by its contents, not by its wrapping: Revisiting Seglen s work on journal impact and research evaluation. Plos One, 217, 12(3), e17425.

5 Quantity versus quality: Research (policy) questions Does increased productivity only lead to repetition of normal science with mostly low impact papers? The Netherlands: The slow science movement; Productivity does not count any longer in the Standard Evaluation Protocol (SEP) UK, Portugal, Italy, Spain: Only a limited number of representative publications can be submitted for evaluation Or should productivity rather be stimulated? Higher chance of breakthroughs with repeated tries and more experience, e.g. Nobel laureates often have many more publications than normal researchers. A growth rate in publishing should be expected, given the growth rate in science and the need to finish projects with publications.

On the Relationship between Research Productivity and Impact. PLoS ONE 11(9): e16279.

6 Two papers presented at ISSI, Istanbul, 215, and published in PLoS ONE 216 Larivière, V., Costas, R. (216). How Many Is Too Many? On the Relationship between Research Productivity and Impact. PLoS ONE 11(9): e Sandström, U., van den Besselaar, P. (216). Quantity and/or Quality? The Importance of Publishing Many Papers. PLoS ONE 11(11): e

7 Productivity and citation impact is correlated

8 Opposing policy implications based on similar observations There are cumulative advantages in research (Merton). Hence, individuals are best assessed by qualitative peer review. Increased productivity in the research system is not a perverse effect of output oriented evaluation systems, but a positive development.

Our approach: To study authors as persons (The picture is from the doctoral forum at ISSI 215) The performances of researchers, as measured within Web of Science with bibliometric indicators after

9 Our approach: To study authors as persons (The picture is from the doctoral forum at ISSI 215) The performances of researchers, as measured within Web of Science with bibliometric indicators after disambiguation of author names, may be interrelated in a complex way with what roles researchers actually take in research, what position they have, what resources are available, and what they achieve in their careers.

10 Our aim: Better understanding of the conditions for individual productivity and citation impact What does bibliometric measurement of productivity mean at the individual level, taking social variables into account? What lessons for policy can be learnt by studying this?

11 Outline 1. Introduction 2. Data 3. Methods 4. Results 5. Conclusions

Matching authors/persons in three datasources An alternative to using WoS

by 17,75 authors from Norway s four largest universities 11,4 publications

gender, position, area of research for the same persons.

12 Matching authors/persons in three datasources An alternative to using WoS as the sole data source WoS is a subset of NSI 61,2 publications (25-216) by 17,75 authors from Norway s four largest universities 11,4 publications (25-216) by the same authors, identified as persons Social variables (age, gender, position, area of research for the same persons. Web of Science (NCR) The Norwegian Science Index (NSI) The Norwegian Research Personell Register

13 Publications per area of research and data source Only original peer-reviewed research publications are included WoS Non-WoS WoS % 1% 9% 8% 7% 6% 5% 4% 3% 2% 1% Arts & Humanities Life Sciences & Biomedicine Physical Sciences Social Sciences Technology %

14 Gender and area of research Female Male Female % 6% 5% 4% 4 3% % 1% Arts & Humanities Life Sciences & Biomedicine Physical Sciences Social Sciences Technology %

15 Age when publishing Up to Over 6

16 Latest position of the researchers Latest position PhD Post doc Assistant Professor Professor Technical staff 6.37% 31.15% 35.3% 13.38% 14.8%

17 Outline 1. Introduction 2. Data 3. Methods 4. Results 5. Conclusions

18 Measuring productivity: Whole counts Average annual productivity per author within publishing years 5 4,5 4 WoS only WoS & non-wos 3,5 3 2,5 2 1,5 1,5 Arts & Humanities Life Sciences & Biomedicine Physical Sciences Social Sciences Technology

19 Measuring productivity: Fractional counts Average annual productivity per professor within publishing years 1,4 1,2 WoS only WoS & non-wos 1,8,6,4,2 Arts & Humanities Life Sciences & Biomedicine Physical Sciences Social Sciences Technology

20 Measuring productivity: Square root of fractions Average annual productivity per professor within publishing years 2 1,8 1,6 WoS only WoS & non-wos 1,4 1,2 1,8,6,4,2 Arts & Humanities Life Sciences & Biomedicine Physical Sciences Social Sciences Technology

21 Average annual number of publications per active researcher, using different counting methods Number of researchers Whole counting Fraction Fractionroot Arts & Humanities Life Sciences & Biomedicine Physical Sciences Social Sciences Technology

22 Measuring productivity: Square root of fractions Average annual productivity per professor within publishing years 2 1,8 1,6 1,4 WoS only WoS & non-wos Our chosen method 1,2 1,8,6,4,2 Arts & Humanities Life Sciences & Biomedicine Physical Sciences Social Sciences Technology

23 Measuring citation impact 1. Field-normalized citations per publication (World average = 1.) 2. Per cent of the world s 1% most cited in the field 2 1,8 1,6 1,4 1,2 1,8,6,4,2 Examples: Averages per area of research Rel Cit 1 per cent 18% 16% 14% 12% 1% 8% 6% 4% 2% Arts & Humanities Life Sciences & Biomedicine Physical Sciences Social Sciences Technology %

24 Measuring citation impact 1. Field-normalized citations per publication (World average = 1,) 2. Per cent of the world s 1 per cent most cited in the field 2 1,8 1,6 1,4 1,2 1,8,6,4,2 Our chosen method Examples: Averages per area of research Rel Cit 1 per cent 18% 16% 14% 12% 1% 8% 6% 4% 2% Arts & Humanities Life Sciences & Biomedicine Physical Sciences Social Sciences Technology %

25 Independent variables Dimension Area of research Gender Age when publishing (B) Position Variables Arts & Humanities; Life Sciences & Biomedicine; Physical Sciences; Social Sciences; Technology Female; Male Up to 3; 31-4; 41-5; 51-6; 6 plus PhD; Post Doc; Assistant Professor; Professor

26 Outline 1. Introduction 2. Data 3. Methods 4. Results 5. Conclusions

27 Impact Moderate correlation Physical Sciences A B C D E Productivity class y = x R² =.615

28 Impact Low correlation 1.2 Technology y = -.112x R² = A B C D E Productivity class

29 Impact Low correlation Life Sciences & Biomedicine y = e -.21x R² = A B C D E Productivity class

30 Impact Low correlation 1.2 Arts & Humanities y = -.472x R² = A B C D E Productivity class

31 Impact No correlation 1.2 Social Sciences y =.52x R² = A B C D E Productivity class

32 Impact Summing up: a hammock 2 Arts & Humanities Life Sciences & Biomedicine Physical Sciences Social Sciences Technology A B C D E Productivity class

33 Average productivity and impact by area of research 1,4 Productivity Impact 1,2 1,8,6,4,2 Arts & Humanities Life Sciences & Biomedicine Physical Sciences Social Sciences Technology

34 Average productivity and impact by latest position 1,4 Productivity Impact 1,2 1,8,6,4,2 PhD Post doc Assistant Professor Professor

35 Average productivity and impact by average age of publishing 1,4 Productivity Impact 1,2 1,8,6,4,2 Up to Over 6

36 Average productivity and impact by gender 1,4 Productivity Impact 1,2 1,8,6,4,2 Women Men

37 Outline 1. Introduction 2. Data 3. Methods 4. Results 5. Conclusions

38 Tentative Conclusions Productivity and citation impact are only weakly correlated in our study: We add non-wos peer reviewed publications and get a result that differs from earlier studies We observe that there can also be visitors in research as authors of highly cited papers Hyperauthorship has an influence in the measurement Productivity within and outside of WoS depends on: The position reached in the career at the time of measurement The age when publishing Gender Area of research Taking these variables into account, citation impact seems less dependent on productivity

39 If the tentative conclusions hold, we have these policy implications Yes, also because the opportunies to be productive within WoS varies with social variables Productivity should of course be stimulated and supported, particularly among women and among researchers in early careers There are cumulative advantages in research (Merton). Hence, individuals are best assessed by qualitative peer review. (Larivière & Costas) Increased productivity in the research system is not a perverse effect of output oriented evaluation systems, but a positive development. (Sandström & van den Besselaer)

Data integration in Scandinavia

Data integration in Scandinavia Gunnar Sivertsen gunnar.sivertsen@nifu.no Nordic Institute for Studies in Innovation, Research and Education (NIFU) P.O. Box 2815 Tøyen, N-0608 Oslo, Norway Abstract Recent