Leiden Manifesto impact? Presented at American Evaluation Association Conference Atlanta, GA October 28, 2016

Transcription

1 Leiden Manifesto impact? Presented at American Evaluation Association Conference Atlanta, GA October 28, 2016

2 Diana Hicks, Paul Wouters, Ludo Waltman, Sarah de Rijcke, Ismael Rafols Nature, April 23, 2015, 520: , doi: /520429a

3 Volunteers translated into 11 languages Chinese Spanish Korean German Traditional Chinese Russian French Brazilian Portuguese Persian Catalan Basque

4

5 Metrics as of October 25, 2016 From ImpactStory: 52,135 visits, 59,723 page views to date on the Nature website 186 Google Scholar 74 or 46 Scopus (different views of the database) 44 Leiden citation matching algorithm in Web of Science 19 Web of Science

6 Winner of the 2016 Ziman award The Ziman award recognizes the most innovative cooperation in a venture to promote the public understanding of the social dimensions of science.

7 64 citations to LM classified Leiden manifesto as: citations Signifier of broader debate about use of bibliometric indicators 21 often connected to: critical of use of bibliometric indicators 16 Authority proposing best practice (usually 1 principle cited) 5 Critic of Journal Impact Factor 11 Reference for other findings stated in LM but originally published elsewhere Support for the use of bibliometrics in research evaluation 3 13 } 5 shared Source: Florian Beng, Leiden Manifesto Perception Project,

8 Impact metric timeline first 13 months after publication Tweets & blogs Citations

9 Independent review of the role of metrics in research assessment and management, commissioned by HEFCE in the UK. The concluding section, entitled Responsible Metrics states: Drawing on discussions over RRI, we propose the notion of responsible metrics as a way of framing appropriate uses of quantitative indicators in the governance, management and assessment of research. The notion of responsible metrics distils the essence of other important contributions to these debates, including the Leiden Manifesto and DORA. Responsible metrics can be understood in terms of a number of dimensions: Robustness: basing metrics on the best possible data in terms of accuracy and scope; Humility: recognising that quantitative evaluation should support but not supplant qualitative, expert assessment; Transparency: keeping data collection and analytical processes open and transparent, so that those being evaluated can test and verify the results; Diversity: accounting for variation by field, and using a range of indicators to reflect and support a plurality of research and researcher career paths across the system; Reflexivity: recognising and anticipating the systemic and potential effects of indicators, and updating them in response. As stated in the Leiden Manifesto research metrics can provide crucial information that would be difficult to gather or understand by means of individual expertise. But this quantitative information must not be allowed to morph from an instrument into the goal. (pp )

10 Fellow travelers often quoted together

11 Panama Evaluation of their largest research center shall use the Manifesto to define a set of indicators, different from the more traditional ones. Also defining quality standards to evaluate post graduate programs that we finance. We shall introduce some of the Manifesto guidelines to develop such standards. This will also take place towards the end of this year or early next year. Source: Carlos Aguirre Bastos, Asesor, Oficina de Planificación Secretaría Nacional de Ciencia, Tecnología e Innovación, Ciudad de Panamá, Panamá personal communication, October 2016

12 Portugal

13 GUIDING PRINCIPLES LETTER FOR THE FOUNDATION FOR SCIENCE AND TECHNOLOGY (...) The successful implementation of these guidelines calls for the guarantee of: stringent assessment procedures, always giving priority to peer review and based on thorough discussion of the scientific activity contents, complying with international standards and respecting clear and transparent rules recognized by the scientific community, including The San Francisco Declaration of 2012, the Recommendations of the Commission on Self - Professional Regulation in Science from the German DFG in September 2013 and The Leiden Manifesto of April (...)

14 29 August law decree regulating hiring of postdocs [(...) It is assumed, therefore, the recommendations in memos and relevant international documents - such as the European Charter for Researchers and the Code of Conduct for the Recruitment of Researchers referred to the Commission Recommendation of 11 March 2005; the Declaration of San Francisco of the American Society for Cell Biology, on the evaluation of research and development activity, December 2012; the Commission's Recommendations on Self-Regulation in Professional Science Deutsche Forschungsgemeinschaft, September 2013; and Leiden Manifesto on the use of metrics in scientific evaluation, April in order to consolidate among the Portuguese society the understanding that the content of scientific publications and their academic, scientific, social or economic ownership is much more important than the publication of metrics or its assessment depending on the entities that have published them.(...)] Unofficial translation by Silvana Roque de Oliveira, CHAM, FCSH-UNL, Portugal

15 Brazil Embrapa The Brazilian Agricultural Research Corporation (Empresa Brasileira de Pesquisa Agropecuária) is a state-owned research corporation affiliated with the Brazilian Ministry of Agriculture. Their mission is to "develop research, development and innovation solutions for the sustainability of agriculture, for the benefit of Brazilian society The Leiden Manifesto is important because its ideas are helping us to expand our vision in the evaluation of our 46 research centers. Wilson Corrêa da Fonseca, Secretariat of Management and Institutional Development of Embrapa, evaluation unit, October 2016 There are coincidences between the LM 10 principles and Embrapa evaluation processes, but LM is to narrow to encompass the full complexity of proposal evaluation in Embrapa because of the diversity of their research centers. (Penteado, Fonseca, Avila, 2016, O papel da pesquisa bibliométrica na avaliação de desempenho de instituições de pesquisa: o caso embrapa, DOI: / CONTECSI/PS-3656, p. 24)

16 Metrics should support but not substitute the work of experts (peer evaluation) Consider context and goals of research Data should be open, transparent and traceable, allowing audit Value local and regional impact Take into account field and practice specificities Revision and renewal whenever needed. Better have a battery than a single indicator

17

18 Can we implement the Leiden Manifesto principles in our daily work with research indicators? Report from the fifth meeting of the Danish Research Indicator Network (FIN) January 21st, 2016, at Copenhagen University Library - Frederiksberg Principle 1 Quantitative evaluation should support qualitative, expert assessment 2 Measure performance against the research missions of the institution, group or researcher Practioner reflection Evaluations can present both qualitative and quantitative analyses but sometimes the users only focus on the indicators. The qualitative results are typically only read if they are very concise. Often a mission is not available or could not be formulated. The user just asked for evaluations of everything or requested the traditional, known indicators (for example the h-index). A link to a mission is never made 3 Protect Excellence in locally relevant research Illustrate visually what we can measure and what we cannot measure for each evaluated unit. Countries with national research information systems have solved this problem.

19 Principle Practioner reflection 5 Allow those evaluated to verify data and analysis Often the tools we have access to are black boxes (for example Web of Science/Incites, Scopus/SciVal, Google Scholar), where often we must rely blindly on what we are served up. 7 Base assessment of individual researchers on a NA in Danish context qualitative judgement of their portfolio 8 Avoid misplaced concreteness and false precision The pressure is often on when users ask for single indicators. All agreed that this was unacceptable practice, and some resolved the issue by always delivering several indicators and attaching explanations and words of warning in delivered reports, even though these were not asked for. 9 Recognize the systemic effects of assessment and indicators 10 Scrutinize indicators regularly and update them Implementing new indicators and updating existing indicators/data require time and financial resources which are not always available. Sometimes the updating of indicators is done on the fly without a formal decision or supporting documentation.

20 Principle 4 Keep data collection and analytical processes open, transparent and simple Practioner reflection Complicated principle to follow in practice; open, transparent and simple can be interpreted in many ways. Simplicity is difficult to obtain when working with data. Data is not necessarily simple, since it in many cases consists of many calculations, before the data actually come into use. As an example of an indicator which at first seems quite easy to comprehend is the h-index (and thus popular), but the calculations behind the h-index may not be quite as simple, and easy to understand for all users. When we use data from Elsevier or Thomson Reuters it is at times unclear to us how data have been processed. Therefore already in the early stages of analyzing data we lose insight into the nature of data. And although we might be able to investigate it further, it is a very time-consuming task. Also data sets aren t static, and can change while we are doing our analyses. While working with data, which originate from external sources, can be a problem; it can also be an advantage. It was mentioned that using the name of the data set could even at times be a sign of quality, e.g. Leiden data. One way of complying with principle 4 in practice is applying information on data and its caveats to analyses. This is done as a rule according to the participants at the workshop both when doing longer term projects, but also on delivery of small datasets and figures for management purposes. It seems to be a sort of code of conduct, but it is also in some instances used to cover people s backs.

21 Principle 6 Account for variation by field in publication and citation practices Practioner reflection 3 strategies were discussed: 1) Use field normalized indicators, 2) Explain the difference between research fields, illustrate with data, allowing users to interpret the results, 3) Present a set of indicators and let those evaluated choose the indicators which best suit their purpose. None of the strategies were seen as straight forward. Especially field normalization. Field comparable relative numbers are attractive for many purposes but the calculation of the indicators must be carefully explained to the users of an evaluation, as well as the information included and how to interpret the results. New methods for field normalization are introduced quite often and this may affect the results of an evaluation more than the actual performance of those being analyzed. Number and definition of research fields debated is continually debated and changes.

22 A vision for the manifesto Parallel to the Vancouver Guidelines, a widely used definition of academic authorship Could the Leiden Manifesto have a similar status as principles for research metrics? To obtain this status two conditions must to be fulfilled: the manifesto must be relevant for all users and it must be maintained as a living document that is updated from time to time. Relevance for all users: Principle 3, Denmark and many other countries have CRISs where the coverage of the country s research publications is much broader than the coverage found in Web of Science. In Principle 6 a specific method is mentioned as the solution, namely Normalized indicators are required, and the most robust normalization method is based on percentiles. The report from the meeting in Denmark points to other possible solutions. Updating Over time, the use, methods and theories of research metrics change. Probably the Leiden Manifesto will have to be updated on a regular basis to reflect the state of art of research metrics. Would it be possible to establish an organization which has the responsibility for maintaining the Leiden Manifesto? The Vancouver Guideline has been updated 15 times since it was published in 1978 ( Marianne Gauffriau, Specialkonsulent Cand Scient Bibl. Det Kongelige Bibliotek The Royal Library

23 Important for impact Carpe diem timing is crucial, when the issue is of broader concern, write & publish. Would we have had the impact if we followed Metric Tide? Nature or hire an editor, give them authority, and limit yourself to 3,000 words Illustrate with examples 1 st thing the Nature editor told us; we submitted with principles and rationale Catchy name this was very deliberate Listicle genre of our time, there s a reason Buzzfeed is addicted to lists and Grand Challenges are most often lists Video Translations