Danish Centre for Studies in Research & Research Policy, Department of Political Science, Aarhus University, Aarhus, DK-8000, Denmark

Transcription

1 Research Evaluation, 25(3), 2016, doi: /reseval/rvv036 Advance Access Publication Date: 8 December 2015 Article What happens when national research funding is linked to differentiated publication counts? A comparison of the Australian and Norwegian publication-based funding models Jesper W. Schneider*, Kaare Aagaard and Carter W. Bloch Danish Centre for Studies in Research & Research Policy, Department of Political Science, Aarhus University, Aarhus, DK-8000, Denmark *Corresponding author. jws@ps.au.dk Abstract The experiences from Australia where undifferentiated publication counts were linked to funding of universities in 1993 is well known. Publication activity increased, but the largest increase was in lower-impact journals, leading to a general drop in overall citation impact for Australia. The experience from Australia has been a warning for what would most likely happen if funding were linked to publication activity. Nevertheless, in 2005, a performance-based model based on differentiated publication counts was implemented in Norway. The model was specifically developed to counter adverse effects like those identified in the Australian case. In the present article, we examine what happens at the aggregated level of publication and citation activity when funding is linked to differentiated publication counts. We examine developments in Norwegian publication activity, journal publication profiles, and citation impact. We also examine developments in publication activities at the individual level and developments in research and development resource inputs. We compare experiences in Australia to those in Norway. The results show that for the Norwegian case, overall publication activity goes up, impact remains stable, and there is no indication of a deliberate displacement of journal publication activities to the lowest-impact journals. Hence, we do not see the same patterns as in Australia. We conclude that the experience in Norway with differentiated publication counts linked to funding has been different from the experience in Australia with an undifferentiated model. This is an important observation because currently the Norwegian model is being or has been adopted in several European countries. Key words: performance-based funding; publication model; publication behaviour; perverse incentives. Introduction The main title of the present article is inspired by Butler (2004). Ina succession of papers, Butler demonstrated how Australian researchers presumably changed their collective publication behaviour in the early 1990s in response to a new national funding model, partly based on productivity measures undifferentiated by any measure of quality (Butler, 2002, 2003a,b, 2004). Annual collection of publication data for use in university funding began in 1993 and was fully incorporated in the funding model in 1995 (Butler 2002). Based on analyses of data from Institute for Scientific Information s (ISI) citation indices, Butler outlined that Australian publication output increased considerably, with the highest relative increases in lower-impact journals (Butler 2003a,b). For a consecutive number of years, this behaviour seemingly led to a general dropin overall citation impact. This effect was contrary to the intention of the implemented funding model, which was to reward quality. As it turned out, what seemed to be rewarded instead was quantity (Butler 2004). Butler s hypothesis is backed by a number of important observations supporting her causal claims. First, the trends of greater publication activity in lower-impact journals were only visible for the specific research sector included in the funding model, i.e. the university sector. Other research sectors not included in the funding model, such as hospitals and government research VC The Author Published by Oxford University Press. All rights reserved. For Permissions, please journals.permissions@oup.com 244

2 Research Evaluation, 2016, Vol. 25, No institutions, did not experience similar publication trends. Second, entries of new institutions into the university sector, as well as general increases in staff numbers in the university sector, were also rejected as potential explanations (Butler 2004). Third, within the university sector, the publication trends were present in all fields of research, including fields such as the humanities known to be less reliant on journal outlets. Finally, Butler (2004) seeks to corroborate her findings through comparison with experiences from the Spanish university system. In 1989, an individual research incentive system was implemented in Spain. Contrary to the Australian case where funding based on publication activity was distributed among research institutions, the Spanish system rewarded researchers directly with salary bonuses for publishing in prestigious journals, essentially the top one-third journals in the subject categories of the Journal Citation (Jiménez- Contreras, Moya Anegón and Lopez-Cozar 2003). Rewarding only selected publication behaviours in some sense makes the Spanish incentive system differentiated. In their longitudinal study of Spanish publication activity, Jiménez-Contreras, Moya Anegón, and Lopez- Cozar (2003) argue that several different causes successively have influenced Spanish research productivity, including the individual research incentive system implemented in Noticeably, causes such as growth in government investments were ruled out as a major explanation for the changes in publication activity. Despite the distinct differences between the two funding models, Butler finds supporting evidence in the Spanish case, arguing that [the work] demonstrates clearly that Spanish researchers have also responded to funding stimuli by increasing their output... (2004: 399). However, recently Osuna, Cruz-Castro and Sanz-Menédez (2011) have emphasized the overwhelming problems of making causal claims concerning potential direct impacts of research evaluation systems on collective behaviours and output. It is the well-known issue of demonstrating causality and treatment effects without an experimental intervention (e.g., Freedman 1991). Most interestingly, Osuna and colleagues conclude that [t]here is no clear evidence that the researchers increased their average productivity significantly in the years following the creation of the [research incentive system]... (Osuna, Cruz-Castro and Sanz- Menédez 2011: 588). Instead, the authors argue that their findings support the idea that publication growth is mainly a function of growth in the size of the Spanish research system, and to a lesser extent also an effect of maturation and historical processes of the system, as well as endogenous changes such as the expansion of international collaboration. According to Osuna and colleagues, the role of the research incentive system is a minor and indirect one, perhaps contributing to certain norm changes and maturation processes, but certainly not a direct cause and thus a main explanation for the overall growth in Spanish publication output. Nevertheless, due to Butler s hypothesis of adverse effects, the experience from Australia has been perceived as a warning of what would likely happen if funding were linked to publication activity. However, in 2003, a so-called quality reform of the higher education sector in Norway led to the construction of a performance-based model, where publication activity again was linked to funding. Similar to the Spanish case, the main political purpose of the model was to encourage more research activity (and thereby more publication activity) at universities and university colleges, and preferably also more international publication activity (Norwegian Association of Higher Education Institutions 2004). The model was implemented in 2005 and first used to distribute funds to universities and colleges in In 2008, the model was expanded with health care institutions as well as the public research institute sector. Obviously, the designers of the Norwegian model were well aware of the Australian case. As a consequence, a slightly more sophisticated model was developed (Schneider 2009; Sivertsen 2010). A key objective of the model was to encourage publication in international outlets (i.e., international journals and academic book publishers) and at the same time to counter perverse publication effects, where researchers seek to publish more but with less effort. Hence, a differentiated publication model was constructed, where publication outlets are ranked on two levels. The lowest-ranked first level comprises, in principle, all scholarly eligible publication outlets, where eligibility basically means a peer review process. The highest-ranked second level is an exclusive number of publication outlets, which are deemed to be leading in a field and preferably with an international audience. The number of Level 2 outlets is limited and may not account for more than one-fifth of all publications in a given field. The Norwegian publication model transforms actual publication counts into publication points by juxtaposing publication type with the level of the outlet and then weighting this number by the fraction of contributing authors from an institution, see Table 1. For example, a journal article published in a Level 2 outlet will receive 3 points; if the article has two authors from two different institutions, each institution will receive a total of 1.5 points due to fractioning. There is no limit to the fractioning. All research fields are included in the model, and publications basically have the same value before fractioning. Hence, a Level 1 journal article with one author is worth the same in physics and literature studies. Differences in field-specific publication activities are assumed to be neutralized partly due to the intrinsic point values and partly due to fractional counting. The annual sum of publication points for an institution is exchanged for funds, where the exchange rate is determined by the total number of publication points in the system in a given year. It is clear from Table 1 that the publication model gives special incentives to publish in Level 2 outlets, albeit Level 1 publishing is not disregarded and may also be a viable behaviour. Most importantly, for the present study, it is assumed that this differentiated point system will discourage researchers from speculating in easy publications in order to avoid a situation like the one in Australia, where only publication equivalents were counted. Consequently, the Norwegian and Australian publication models are fundamentally different when it comes to publication counts and thus incentives to publish. The Australian model did not differentiate between publication types and outlets, which according to Butler (2004) presumably have led to more activity but with less effort, resulting in lower national impact. The Norwegian model does differentiate between publication types and outlets; yet, so far, we have no solid evidence of the aggregated Table 1 The Norwegian publication model s point system Publication points Publication types Level 1 outlets Level 2 outlets Books 5 8 Book chapters and conference papers Journal articles (articles and reviews) 1 3 *Publications are categorized in three types and on two levels. Publication outlets such as journals and book publishers are manually ranked in one of two levels. Consequently, all publication types basically have two values, one for Level 1 and one for Level 2 outlets. Nomination and re-ranking of outlets are carried out annually in a number of specific subject committees. Publication outlets are only treated by one committee and thus have only one rank.

3 246 Research Evaluation, 2016, Vol. 25, No. 3 outcomes of this model. Is the experience for Norway the same as in Australia? The main research question we ask in this article is therefore: RQ: What happens at the aggregated national output and impact levels when institutional funding is linked to differentiated publication counts? This question is important, as it contributes to our understanding of performance-based funding in general and models based on publication activity in particular. Hitherto, we only have somewhat contradictory indications from the Australian and Spanish cases, which also differ in their design. While the Spanish research incentive system is targeted directly at individual researchers, both the Australian and the Norwegian models are targeted at the institutional level. While the Spanish system rewards specific publication behaviour at the individual level, it is very different from the meticulously constructed Norwegian publication model, where all activities are rewarded but some more than others due to the differentiated publication counts. It is therefore natural to make a comparison between the Australian and Norwegian cases, knowing that they both are targeted at the institutional level but differ in counting approaches. The Norwegian model redistributes approximately 2% of the annual funding among the institutions in the higher education sector (University and College sector (UH)-sector) (Aagaard, Bloch and Schneider 2015). In Australia, publication counts initially constituted 12.5% of a general formula to distribute approximately 220 million AUS$ (1996) as block grants among universities. Later, the percentage was adjusted to 10%, but the actual amount was increased so that individual publication values rose from approximately 700 AUS$ in 1996 to 3,300 AUS$ in 2002 (Butler 2003b). For comparison, the value of Norwegian publication points have been around 6,000 AUS$. A finally motivation for studying the potential effects of the Norwegian model is that it has recently been adopted in several European countries (Hicks 2012). Interestingly, this adoption has happened without addressing the political objectives with the model in the Norwegian context (i.e., more productivity and internationalization), and without empirical knowledge or evidence about the potential aggregated effects of the model in Norway. Consequently, based on longitudinal data, we examine the aggregated developments in publication patterns and impact for Norway and compare them to the developments in Australia. Focus is upon potential changes in collective behaviours around the time the models were implemented. The article is organized so that the next section presents data and methods used for the analyses, followed by the results section, which is split in three: first, we examine the developments in publication activity, then developments in citation impact, and finally the developments in journal publication profiles. In the final section, we discuss the main findings. Data and methods We examine the overall research question using aggregated time series of bibliometric data for Norway and selected benchmark countries (including Australia) and research and development (R&D) statistics for Norway, based on data from the NIFU R&D statistics bank, 1 supplemented by data and insights from interviews with heads of departments and faculties at Norwegian 1 Nordic institute for studies of innovation, research, and education (NIFU): universities and a survey among Norwegian researchers (Aagaard et al. 2014; Aagaard 2015; Aagaard, Bloch and Schneider 2015). Bibliometric data have been obtained from the publication database supporting the Norwegian funding model 2 and from Centre for Science and Technology Studies (CWTS) Web of Science (WoS) citation database.. The publication data from the model include both aggregated data at the institutional and field levels, as well as publication data at the individual level. Notice, we do not have full access to data on all individual researchers and their individual publication activity in the Norwegian system. However, we do have access to the individual publication activity for all researchers at the four main universities (University of Oslo, University of Bergen, University of Tromsø, and Norwegian University of Sciences and Technology), who have at least one publication in a given year in the period examined. 3 With these restricted data, we are able to address, at least indicatively, the important predicament whether increases in publication activity reflect that individual researchers publish more, or that more publishing researchers have come into the system, or whether researchers already in the system have become activated, or a combination of these different scenarios. Further, to supplement these time series, we include some insights from the surveys in the discussion to shed more light on individual behavioural issues. In order to document the development in overall publication activity, we use publication counts and publication points from the Norwegian documentation system. In order to document the international visibility of Norwegian journal publications, we use WoS data and compare Norway to developments in the database and for three benchmark countries: Australia, Sweden, and Denmark. Australia is chosen for obvious comparative reasons introduced in the Introduction. Denmark is chosen because an adoption of the Norwegian model was implemented there in Finally, Sweden is included because it is in many ways comparable to Norway, though Sweden has not implemented a publication-based funding model. In order to examine the developments in Norwegian impact, we use two basic indicators developed at CWTS, Leiden University: the mean normalized citation score (MNCS) and the proportion of publications among the 10% most highly cited in the database (PPtop10%). The MNCS is a field-normalized average citation score, and the PPtop10% is a non-parametric percentile indicator (see Waltman et al. 2012). Notice, in order to show developments in the indicators on the same scale, we divide the observed proportion of highly cited papers with the expected proportion of 10%. Hence, a value of 1 for the PPtop10% indicator corresponds to the expected proportion of 10%. All indicators are calculated using fractional counts, and only research articles and reviews are included. In order to examine potential changes in the overall journal publication behaviour, we use the mean normalized journal score 2 Norwegian Social Science Data Service (NSD)/Database for statistics on higher education (DBH): kategori_publiseringer.action.3 The individual publication data are provided by Cristin ( the research information system supporting the Norwegian model. Unfortunately, full data at the individual level for the period examined was only available for the main universities and not the university colleges.

4 Research Evaluation, 2016, Vol. 25, No (MNJS) indicator, which measures the impact of the journals in which Norwegian researchers have published (Waltman et al. 2012). This is somewhat different from Butler (2002, 2003a,b), as she examined changes in publication behaviour by dividing journals into quartiles based on their journal impact factor in respective subject categories and subsequently calculated the development in Australian shares of publications in the different quartiles. The interpretation of the MNJS indicator is the following: If a unit has an MNJS indicator of one, this means that on average the unit has published in journals that are cited equally frequently as would be expected based on their field. An MNJS indicator of, for example, two means that on average a unit has published in journals that are cited twice as frequently as would be expected based on their field citation activity. We present two analyses of journal publication behaviour: (1) an aggregated analysis on the country level, where we correlate the overall MNJS indicator with the two impact indicators; and (2) individual analyses for Norway and Australia, where we examine the development in journal publication behaviour according to different journal impact classes. Obviously, we focus on the years before and after the implementation of the funding models in Norway and Australia. In the second analysis, we disaggregate the journal publication behaviour by constructing five journal impact classes based on the MNJS (i.e., from A to E, where Class A contains the journals with the lowest MNJS, and Class E contains the highest indicator scores) and then examine the relative distribution of the country s 3-year overlapping publication outputs among these five classes over time. We do this relative to the total national output in the 3-year period and also relative to the total output in the WoS database for the 3-year period. The disaggregated journal publication behaviour analyses are, in principle, comparable to Butler s analyses, and we include Australia as a comparison because we expect to see the changes suggested by Butler. We are therefore able to examine whether such changes are also present in the Norwegian case. Notice, at the aggregate level of countries, MNJS journal publication profiles will most likely have strong correlations with a unit s MNCS score for its articles relative impact. Finally, contrary to Osuna, Cruz-Castro, and Sanz-Menédez (2011), we abstain from using statistical models in the present analysis for several reasons. Implicit in most disseminations of regression results are causal claims. Such claims are most often ill-founded with observational data (e.g., Leamer 1983; Lieberson 1985; Freedman 1991; Berk 2004). In conventional regression, one requires not just a nearly right model of how the data were generated, but good information justifying any claims that all causal variables are independently manipulable. In the absence of a nearly right model and one or more regressors whose values can be set independently of other regressors and the disturbances, causal inferences cannot make much sense (Berk 2010). The observational data available for the present analysis, and similar studies, are simply not suitable for such bold analyses. In general, R&D statistics are flawed and challenging to work with(see Wendt et al. 2012). Also, combing macro statistics such as general inputs and outputs of the research system and subsequently claiming effects in individual behaviour can easily lead to ecological fallacies and is in our view, not a viable strategy for inquiry. However, one premise for a causal claim is that a correlation is present, if absent such claims can be ruled out. We must accept the nature of the data at hand and rely on the distinct developments and correlations which they eventually reveal and treat the interpretations with caution. The next section presents the main results of the macro time series. Results In this section, we first address the development in the aggregated publication output from the Norwegian model as well as the development in international visibility in the WoS. Next, we address the developments in Norwegian citation impact, and finally, we examine the journal publication behaviour in relation to journal impact classes in order to examine potential adverse effects. Developments in publication activity First, we present the developments in publication counts and publication points from 2005 to 2012 in Fig. 1 below. Total publication points for the higher education sector in Norway have risen from 8,883 in 2005 to 15,190 in 2012, amounting to a 71% increase over the period. The trend in publication counts is similar, from 7,330 in 2005 to 14,057 in 2012, showing an even higher increase of 92%. Besides the considerable growth in publication counts 4 and points, it is also noticeable from Fig. 1 that publication points on Level 2 have remained very stable around 20% of the total annual number of points throughout the period. Fig. 2 below shows the relative growth of publications among the five main research areas. Again, we see quite similar growth rates. In 2005, there were four main universities in Norway, and their publication activity registered for the funding model corresponded approximately to 80% of the total. The subsequent 20% were distributed among the numerous university colleges. In 2012, four more institutions (previously university colleges) had gained university status. Thus, the Norwegian system now comprises eight universities, but they still constitute around 80% of the total output. Interestingly, the publication activity for the four old universities dropped from around 80% in 2005 to 68% in 2012, which reflects strong increases in publication activity among more recently established universities and among university colleges, many of which have previously had a much weaker focus on research activities (Aagaard, Bloch and Schneider 2015). These figures and their synchronous developments are a first indication of stability rather than adverse effects at the aggregate level. However, changes in numbers of and nominations to Level 2 outlets during the period may, to some extent, have influenced the share of Level 2 publications, potentially blurring fluctuations in publication behaviour. Unfortunately, we have not been able to account for that in this analysis. To examine international visibility, Figs 3 and 4 below show developments in the share of WoS publications for Norway and the three benchmark countries: Australia, Sweden, and Denmark. We show the developments in total shares of full and fractional counts in the WoS database from 1990 (Fig. 3) and the relative growth of WoS publications from 1990 based on full counts (Fig. 4). In 1990, Norway s share of WoS publications was 0.47%. This share rose to 0.60% in 2002, and in 2005, the first year of the model, the share was 0.66%. In the subsequent seven model-years, the share increased continuously and more steeply to 0.81% in Notice, this overall development is not unique for Norway. All benchmark countries seem to increase their share of WoS publications, at least until From then on, the developments differ. Australia s relative shares continue to rise, and the increase 4 Publication counts are here equivalents where articles, conference papers, and book chapters count as one, whereas books are counted as five publication equivalents.

5 248 Research Evaluation, 2016, Vol. 25, No. 3 Figure 1 Developments in publication counts and points; data from the Norwegian indicator s documentation system. Figure 2 Relative publication growth for five main research areas from 2005 to 2011 (notice, we do not have valid data for 2012). steepens from 2005 to The Danish relative shares are stable for a long period but begin to increase considerably around Finally, the Swedish relative shares decrease from 2001 to What is also noticeable from Fig. 3 is the widening gap between the shares of full and fractional counts for all countries, beginning in the mid-1990s. This is an indication of the increasing degree of international collaboration. For example, in 1990, 27% of the Norwegian publications indexed in WoS had at least one additional country address; in 2000, the corresponding figure was 44%, and in 2012, 59%.

6 Research Evaluation, 2016, Vol. 25, No Figure 3 Development and growth in total shares of WoS publications for Norway and benchmark countries (full and fractional counting at the country level). Figure 4 Development and relative growth of publications in the WoS database for Norway and benchmark countries (full counts). The increase in Norwegian publication activity in international journals indexed by WoS becomes more distinct if we examine the development as an index, where index 100 is the publication output in 1990, shown in Fig. 4. Overall, we see the same developments as in Fig. 3. All countries experience stable growth rates from 1990 until and rates above the general growth rate for the database. However, things change around In the following decade, all countries experience steeper growth rates, and the developments for Norway and Australia are distinct. In the Norwegian case, the growth index rises from 100 to 158 in the first decade from 1990 to The 4-year interval from 2001 to 2004 sees a further increase to 187. Finally, in the funding-model period, the publication activity growth index goes from 207 in 2005 to 350 in While this seems exceptional and striking in relation to the implementation of the funding model, it is important to notice that Australia also experiences a similar increase. Denmark, around 2009, also experiences a marked increase. Sweden generally follows the developments in the database. Notice, in this period, the WoS database saw a huge intake of so-called regional journals, especially from Asia, raising the total number of journals covered by WoS markedly (Testa, 2011). Consequently, all countries, except perhaps Sweden, experience a large increase in their international visibility from 2005 onwards in the WoS database. As Osuna, Cruz-Castro, and Sanz-Menéndez (2011) argue, it is difficult to make causal attributions to research evaluation systems, for example, claiming that a funding model targeted at institutions

7 250 Research Evaluation, 2016, Vol. 25, No. 3 causes general changes in collective publication behaviour visible in aggregated publication statistics. This may well be the case, but the challenging problem is to handle confounders and individual differences and thus to isolate effects. Matters are complicated further by mixing individual and aggregated levels of input and output data, often out of need because of scarcity of valid data. It is, for example, well-known, that on the aggregate level, input of resources into a science system, such as money and/or researchers, correlates well with rises in output measures such as publication counts (e.g., Wendt et al. 2012). In order to examine the potential effects of inputs into to the Norwegian university and university college science system, Fig. 5 below shows relative growths in the system when it comes to human resources in the public research sector (input), R&D investments (input), as well as the above presented growth rates for publication outputs: publication counts from the model, publication points from the model, and WoS publication counts. Index 100 is set to 2005, the implementation year of the model. Like in other Scandinavian countries, input of resources to the Norwegian science system has grown considerably in the last two decades. Nevertheless, growth rates in R&D investments are higher compared to human resources up until Here, both types of inputs experience a steeper rise in their growth rates. Full-time equivalents of researchers in the higher education sector increase by 31% from 2005 to 2012 and total higher education R&D investments increase by 22% from 2005 to 2011 (i.e., we do not have data for 2012). Fig. 5 also shows various publication growth rates. Obviously, the publication counts and points of the model begin in 2005, and the steep growth rates presented above are clearly visible. As indicated by the figure, the relative growth in publication points is smaller than the relative growth in the number of publications in the model. The gap actually widens, indicating that the average publication points are dropping from 2005 to The average publication point per publication drops from 1.23 in 2005 to 1.18 in 2012, most likely as a consequence of increased collaboration. While WoS publication growth rates are obviously correlated with the model publications, it is interesting to observe that Norway has continuously increased its visibility in WoS over the last two decades; yet, around 2005, the growth rates accelerate. The widening gap between the growth rates of model publications and WoS publications imply an increasing share of non-journal and non-wos publications in the model. Overall, it does seem as if input of economic and human resource factors from 2005 are correlated with general publication activity. But, especially, the discrepancy between the input of new researchers and the publication output may suggest that the publication activity within the system also has changed. Unfortunately, we do not have access to complete individual publication data in order to monitor all individual researchers publication activity during the period. We do, however, have access to the individual publication activity for all researchers at the four main universities who have at least one publication in a given year during the period examined, and by combing these data with relative developments in R&D statistics on the number of researchers for these four universities, and in general for all sectors, we may partially shed some light on potential changes in collective behaviours. Table 2 below shows the development in the average number of publications and publication points per publishing researcher at the four main Norwegian universities from 2005 to Also shown in the table is the percentage of Level 2 publications and the number of publishing researchers. The annual publication activity for publishing researchers has increased in the period, and what is especially noticeable is that the annual number of publishing researchers has gone up strongly so that annual average activity is calculated with very different denominators. We also see a small rise in the share of Level 2 publications but a decline in publication points. The decline in publication points is very likely a result of increased collaboration and therefore an effect of the fractionalization formula used in the model. The table thus indicates that the average individual publication activity has increased by around 23%, whereas the number of actively publishing researchers has increased by 73%. In comparison, Figure 5 Development in the growth of R&D inputs (total R&D investments and full-time equivalent researchers in the higher education sector) and publication outputs (WoS publications, model publications, and model publication points) for Norway (source for input statistics: NIFU R&D statistics bank).

8 Research Evaluation, 2016, Vol. 25, No Table 2 Developments in average publications and publication points (per publishing researcher), from 2005 to 2012 for the four main universities Publications Points Percentage Level 2 publications 23.2% 24.5% 25.3% 25.1% 24.5% 25.3% 25.0% 25.7% Number of publishing researchers 6,038 6,572 7,317 7,788 8,541 9,234 9,855 10,425 Figure 6 Developments in annual publication activities for a cohort of researchers at the four main universities that have published in 2004 and the number of R&D personnel in these four universities has increased by approximately 17% from 2005 to 2012, whereas the increase in R&D personnel for the higher education sector overall was around 21% (i.e., the R&D statistics are subject to some uncertainty). Nevertheless, these developments do suggest that, at least for the four universities, the increase in publication activity reflects to a large extent an increase in the share of researchers that actively publish in the academic channels covered by the model. The clear discrepancies between developments in publishing researchers and R&D personnel, however, also suggest that while new actively publishing researchers have come into the system and have contributed considerably to the general increase in publication activity, researchers already in the system have also to a lesser degree been activated and have commenced publishing. The question remains whether the publishing researchers already in the system have increased their individual publication activity. Fig. 6 below shows the developments in annual publication activities for the cohort of researchers at the four main universities which were actively publishing in 2004 and It is interesting that the average number of publications for this cohort have increased with almost two publications during the period. In fact, later cohorts show similar tendencies, although the average numbers are lower and their publication windows shorter. If we split the 2004/2005 cohort into quartiles according to their publication points in the beginning year and subsequently compare differences between the 2005 and 2012, we see that while all quartiles increase their activity in terms of number of publications, there is a clear bottom-up effect, where the increases are greatest at lower levels. The pattern is similar for publication points; however, average points actually decline for the highest quartile. These results suggest that the individual publication activity of publishing researchers may have increased somewhat during the period examined. However, while showing some indications, these calculations are far from sufficient to fully describe developments in publication volume at the individual level. In particular, the average number of publications and points are based on an increasing number of researchers, which complicates their interpretation. In addition, the fact that we do not have access to all researchers publication data limits the conclusions. Consequently, the overall trend seems to be a large increase in publication activity, which to some extent can be ascribed to both general developments in other countries, and to resource inputs, and potentially also to factors beyond these. One of them could be the publication-based funding model. In general, the restricted individual publication data suggest considerable new activity of researchers coming into the system, but also more publication activity among

9 252 Research Evaluation, 2016, Vol. 25, No. 3 researchers in the system. So, as in Australia, publication activity and international visibility have increased considerably for Norway in the specific period from the implementation of the model to However, while the growth rates are clearly steeper in this period, the trend was visible before the implementation. In the next subsection, we examine the developments in Norwegian citation impact in order to analyse whether experiences in Norway are comparable to those of Australia after the implementation of the publication-based funding model. Developments in citation impact Based on the above presented findings, it seems that the political intention behind the Norwegian funding model, i.e. more publication activity and more international visibility, has been achieved. But another important question remains: has the marked rise in publication activity also resulted in lower national impact for Norway as was the case in Australia? Fig. 7 presents three indicators that reflect this issue: the journal indicator MNJS and the two article-level indicators, MNCS and PPtop10%. We show indicators for Norway and the three benchmark countries and elaborate further on the journal publication behaviour in the next subsection. In order to smooth curves, we use 3- year overlapping publication windows. The development for Australia is shown in the upper-left panel of Fig. 7. Itisclearthatthetwoindicators,MNCSandMNJS,arecorrelated, as they follow each other closely. We see a general decline during the 1980s, with a low point in the years after the implementation of the publication-based funding model. The decline in the MNCS and MNJS are in line with Butler s findings (e.g., Butler 2003a). The figure also indicates that large decline in the proportion of highly cited articles from Australia in this period. Being a mean-based indicator, the MNCS is directly influenced by the PPtop10% indicator, which shows the unit s proportion of the most cited articles in the database. Fewer articles among the most highly cited will, other things being equal, affect the MNCS indicator negatively, and this seems to be the case in Australia. The Norwegian case is different (lower-left panel). Norway also experiences a general decline in the 1980s, with a low point in the early 1990s, but hereafter, we see a continuous rise in impact, peaking in So far, there is no indication that Norwegian impact has declined in the period where the funding model has been operating. There is accordingly no indication that the model has affected Norwegian impact negatively in any substantial way. However, it is also important to emphasize that there is no indication that Norwegian impact has been positively affected by the model. While impact has risen greatly over the 30 years examined, it has essentially been unchanged since the model s introduction in Nevertheless, the current Norwegian impact level would still be considered meagre in a West-European context (Nordforsk 2010). What seems to have increased the Norwegian impact are more highly cited publications and a continuous overall change in Norwegian publication behaviour changing towards publishing in Figure 7 Citation impact and relative journal publication profiles for Norway and benchmark countries measured by the MNCS, the normalized proportion of the 10% most cited articles (top10), and the MNJSs; all indicators are calculated based on fractional counts.

10 Research Evaluation, 2016, Vol. 25, No journals with relatively higher impact, although these impact levels are average or slightly above average for the specific fields the journals belong to according to WoS. But this development has mainly taken place before the model s implementation in Finally, Denmark and Sweden show very similar patterns (right panels), where both countries experience a high impact at the beginning of the period. Both countries experience a drop in the 1980s, where the Danish drop is most dramatic, but then the developments change. Denmark experiences a long period with a continuous increase in impact, with a noticeable increase in highly cited papers and a smaller increase MNJS scores. Sweden s impact has essentially been unchanged since the mid-1980s, though an increase in MNJS. By comparing the long-term trends for all four countries, similar patterns thus become visible: All countries experience a drop in impact in the 1980s, and all countries except Sweden see continuous increases from the mid-1990s onwards. The individual fluctuations are different though, and we cannot rule out that database effects are influencing the trends. The important issue here is, however, that we cannot see any trace of a general drop in the Norwegian national impact. Likewise, if we examine the development in the MNJS indicator, examining the overall journal publication behaviour of Norwegian researchers, we see that the indicator basically follows the same trajectory. Thus, on the aggregated level, there is no indication of more publication activity in lower-impact journals. What we cannot see from these figures is to what extent variation in publication activity among low-, medium-, and high-impact journals contributes to this stable MNJS indicator. In the next subsection, we therefore address this final issue and compare Norway to Australia when it comes to WoS journal publication profiles. Developments in the WoS journal publication profile As outlined in the data and methods section, we have disaggregated the journal publication behaviour by constructing five journal impact Classes, A E, based on the MNJS indicator, where Class A contains the journals with the lowest MNJS score, Class E contains the highest indicator scores, and Class C contains journals with average impact scores in their respective WoS fields. We examine the relative distribution of the country s 3-year overlapping publication outputs in these five classes over time. We do this relative to the total national output, as well as relative to the total output in the WoS database. Fig. 8 shows the 3-year cumulative output, relative to the total national output for Australia and Norway. Butler (2003a,b, 2004) found that Australian output generally increased after the implementation of the funding model for almost a decade, with the largest increases in the lowest two journal impact quartiles. Our approach is slightly different, and more in line with the citation impact calculations, although the results corroborate Butler s findings. When comparing relative output to the national total, we see that the largest output and also the largest relative increase in output are in impact Class B, i.e., journals with below average impact. Immediately after the implementation of the Australian model, the relative national output in this class increases to around one-third of the total output and remains at this level for well over a decade. When relative output begins to drop, citation impact begins to rise again. The Norwegian case is quite different. While impact Class B is also by far the largest output class, the relative output in this class continuously decreases over the years. At the time of the Norwegian model s implementation in 2005, output in Class B constitutes 28%, but it continues to drop in the subsequent years. In fact, it is surpassed by impact Class C as the class with the highest Figure 8 Comparison of the 3-year cumulative journal publication profiles for Norway and Australia relative to the total national output. Journal publication profiles are measured according to MNJS impact classes, where the value 1 corresponds to the average journal citation activity in a field.

11 254 Research Evaluation, 2016, Vol. 25, No. 3 relative output. But, we also see increases in the two highest impact Classes D and E, although these increases started before the implementation. They also seem to stagnate in the last couple of years. It is clear though that the experiences in Australia and Norway differ considerably, with the Australian increase in lower-impact classes than for Norway. Fig. 9 shows the 3-year cumulative journal publication profiles relative to the total output in the WoS database for the two countries. We see the same trends, though in a slightly different way. Fig. 9 corresponds to the trends presented by Butler (2003a). Here, we see that impact Class B is the largest in Australia and that it increases. But, we also see that almost all other impact classes also increase relative to the total output in the database after implementation of the Australian model. This is not the case for Norway, as there is basically no increase in Class A. Class B sees an immediate drop, although output begins to rise again in the last years examined. Interestingly, from the early 2000s, the share of Australian publications in the three highest impact Classes (C, D, and E) increases considerably, and correspondingly, Australian citation impact has also grown to its current level, which is the highest since 1980 (see Fig. 7). While the block grants in Australia continued to be distributed on the same basis, a number of new initiatives have shifted focus away from a purely quantity-driven allocation to quality issues. According to Butler (2010: 139), the shift of focus towards quality issues happened particularly after However, it is unclear what has instigated this remarkable rise in impact. Based on these comparative findings, we can conclude that developments in publication activity in Australia and Norway have been quite different. There is nothing in the presented trends that suggest a collective change in Norwegian publication behaviour towards publishing in lower-impact journals, with a subsequent effect of lower national citation impact. On the contrary, Norwegian collective journal publication activity seems to have shifted from lower- to average-impact journals, and citation impact has stabilized at its highest level in 30 years. But, it is also important to emphasize that these developments started well before the model was conceived and implemented. Discussion In this article, we have examined the question: What happens at the aggregated national output and impact levels when institutional funding is linked to differentiated publication counts? The question is interesting, as previous evidence was linked to undifferentiated publication counts in Australia. In the Australian case, publication activity rose considerably but adverse effects were detected, as researchers appeared to increase publishing in lower-impact journals, where publication presumably was easier, eventually resulting in a decline of national citation impact for Australia. In Norway, a funding model based on differentiated publication counts was implemented in The Norwegian model is inspired by the Australian case, and the shift from undifferentiated to differentiated counts is a deliberate choice to counter adverse effects. In this article, we have demonstrated that the publication activity and international visibility for Norway has been growing over a long period. We also demonstrate that the growth increases considerably after the implementation of the differentiated publication model. Figure 9 Comparison of the 3-year cumulative journal publication profiles for Norway and Australia relative to the total output in WoS. Journal publication profiles are measured according to MNJS impact classes, where the value 1 corresponds to the average journal citation activity in a field.

12 Research Evaluation, 2016, Vol. 25, No This increase in publication activity is similar to the Australian case. However, contrary to the Australian case, Norwegian citation impact does not decline after the implementation of the funding model. Instead, citation impact has been stable since the model s implementation, following steady growth in relative citation impact from the mid-1990s until the mid-2000s. There is therefore no indication at the aggregated level that Norwegian citation impact is in decline. This is a different experience compared to Australia, where impact eventually dropped in the years after the implementation of the model. A comparison of journal publication trends between the two countries around the time of the respective implementations of their funding models reveals that while both countries experienced a considerable growth in publication output, and the largest relative journal output for both countries are in the second-lowest journal impact Class (B), Australia s output in this class further increased after the implementation of their funding model, whereas Norwegian output in this class actually declined. In the Norwegian case, the increase takes place in impact Class C, which contains journals with average impact. Consequently, the answer to the research question: What happens at the aggregated national output and impact levels when institutional funding is linked to differentiated publication counts? is for the Norwegian case: (1) publication activity goes up, (2) impact remains stable, and (3) there is no indication of a shift of journal publication activities towards the lowest-impact journals in the WoS. Accordingly, we do not see the same patterns as in Australia. We find it conceivable that the differentiated Norwegian publication model has stimulated more research. However, we do not find any indication that the model has stimulated to higher impact. Impact has risen in the period prior to the funding model, but has been stable in the period since the model s implementation. It should also be pointed out that the Norwegian model was not explicitly designed to increase impact, but rather to avoid deterioration in impact. It is also important to emphasize that parallel to the implementation of the funding model, R&D investments, and to a somewhat lesser degree the number of public sector researchers, have also increased considerably. While the data are uncertain, some of the increase in publication activity is no doubt a result of the corresponding increase in resources. We have indications that more publishing researchers have come into the system, that nonpublishing researchers already in the system have commenced publishing, and finally, there also seems to be a slight increase in the general publication activity of publishing researchers from 2005 to It should be mentioned, that while there does not appear to be any indication of adverse effects in publication behaviour at the aggregated level, adverse effects at the local or field level cannot be ruled out. Interviews with heads of department and faculty deans as well as a survey among Norwegian researchers show that the model has been utilized to various degrees as an individual reward mechanism at several Norwegian research institutions (Aagaard et al. 2014). This practice, which is contrary to the recommendations of the designers of model, seems, in particular, to take place in departments where international journal publication hitherto has played a minor role, often within the Humanities and the Social Sciences (Aagaard 2015). The effects of these local implementations are to a large degree currently unknown, and should be examined in more detail in the future, as important differences may be found beneath the aggregated trends brought forward in this article. However, if we consider the aggregated picture, the interesting question in this context is: do the differentiated publication counts in Norway result in overall stability in impact and collective publication behaviour? However, we have no hard evidence to substantiate such a causal claim. What we can establish so far is that the experience in Norway, with a differentiated publication model linked to funding, is different from the experience in Australia, with an undifferentiated model. This is an important observation because currently the Norwegian model has been adopted in several European countries. In a few years, we may be able to substantiate these findings by examining the same research question for Denmark, where a version of the Norwegian publication model was implemented in Acknowledgements We gratefully acknowledge financial support from Aarhus University s Senior Management Strategic Funds for the project Contextualizing and Measuring Research Performance (CoRe). We also thank the three anonymous reviewers for their valuable comments and suggestions. References Aagaard, K. (2015) How Incentives Trickle Down: Local Use of a National Bibliometric Indicator System, Science and Public Policy, (2015) 42/5: Aagaard, K., Bloch, C. W., Schneider, J. W., Henriksen, D., Lauridsen, P. S. and Ryan, T. K. (2014) Evaluation of the Norwegian Publication Indicator (in Danish). Oslo: Norwegian Association of Higher Education Institutions. Aagaard, K., Bloch, C., and Schneider, J. W. (2015) Impacts of Performancebased Research Funding Systems: The Case of the Norwegian Publication Indicator, Research Evaluation, 24/2: Berk, R. A. (2004) Regression Analysis: A Constructive Critique. Newbury Park: Sage Publications. Berk, R. (2010) What You Can and Can t Properly Do with Regression, Journal of Quantitative Criminology, 26/4: Bloch, C. W., Sørensen, M. P., Graversen, E., Schneider, J. W., Kalpazidou Schmidt, E., Aagaard, K., and Mejlgaard, N. (2014) Developing a Methodology to Assess the Impact of Research Grant Funding A Mixed Methods Approach, Evaluation and Program Planning, 43: Butler, L. (2002) A List of Published Papers is no Measure of Value The Present System Rewards Quantity, Not Quality - But Hasty Changes Could be as Bad, Nature, 419/6910: 877. Butler, L. (2003a) Explaining Australia s Increased Share of ISI Publications The Effects of a Funding Formula Based on Publication Counts, Research Policy, 32/1: Butler, L. (2003b) Modifying Publication Practices in Response to Funding Formulas, Research Evaluation, 12/1: Butler, L. (2004) What happens when funding is linked to publication counts? In: Moed H. et al. (eds) Handbook of Quantitative Science and Technology Research, pp Netherlands: Kluwer Academic Publishers. Butler, L. (2010) Impacts of Performance-based Research Funding Systems: A Review of the Concerns and the Evidence, in Performance-based Funding for Public Research in Tertiary Education Institutions, pp Workshop Proceedings. Paris: OECD. Freedman, D. A. (1991) Statistical Models and Shoe Leather, Sociological Methodology, 21/1: Hicks, D. (2012) Performance-based University Research Funding Systems, Research Policy, 41/2: Jiménez-Contreras, E., Moya Anegón, F. D., and Lopez-Cozar, E. D. (2003) The Evolution of Research Activity in Spain The Impact of the National Commission for the Evaluation of Research Activity (CNEAI), Research Policy, 32/1: