European Commission 7th Framework Programme SP4 - Capacities Science in Society 2010 Grant Agreement:

Transcription

1 FP7 Grant Agreement Deliverable N and Title D2.5 Prtfli Mdel Disseminatin Level Wrk Package PU (public) WP2 Versin 1.0 Release Date 28 March 2014 Authr(s) Kayvan Kusha and Michael Thelwall Prject Website Eurpean Cmmissin 7th Framewrk Prgramme SP4 - Capacities Science in Sciety 2010 Grant Agreement:

2 Deliverable D2.5 Prtfli Mdel Althugh assessing the value f research is still a cmplex issue, citatin cunting has cmmnly been used as a practical methd t estimate the frmal impact f research. Hwever, there has been a lng debate n using biblimetric indicatrs and in particular citatin cunting fr research evaluatin. Because the main surces f citatin data are citatin indexes (e.g., Thmsn Reuters Web f Science and Elsevier s Scpus), which cntain a subset f the academic literature (e.g., mainly selected high impact English jurnals). In respnse, it has been suggested that citatin data frm wider types f infrmatin surces can help fr mnitring research impact, especially in sme scial sciences and many humanities fields where jurnal articles and their citatins are less imprtant than in the sciences. In particular, there are sme types f infrmal impacts f research that cannt be measured by citatin cunts. Fr instance, sme research might be frequently used in teaching, highly discussed in scientific cmmunities r frequently read by the public r by academics. T fill this gap, ACUMEN Prtfli suggests a web metrics mdel fr impact assessment f research as a supplement t, but nt replacement fr, traditinal biblimetric indicatrs by including new types f web evidence abut the value r use f research. The ACUMEN Prtfli web metrics mdel is based n previus research and ACUMEN research during the prject. The prpsed web metrics can prvide a wide range f types f impact evidence frm the web. The new metrics have been rated with a 5- stars system t reflect the quality f the evidence they prvide in as clear and practical way as pssible. This is designed t minimise the risk that the metrics are used unquestiningly. Hence, academics r reviewers can estimate scientific influence r value f research nt nly based n cnventinal citatin metrics but als based n mst high quality web metrics. Tw majr cmpnents f ACUMEN web metrics are (fr details see Guidelines fr Gd Evaluatin Practice with the ACUMEN Prtfli available at acumen.eu/prtfli): Mdeling web extracted citatins: This includes a series f methds and tls develped t extract citatins frm nline dcuments indexed by Ggle Bks, Ggle Schlar and cmmercial search engines (e.g., Ggle, Bing). This gives brader view f the impact f research based n citatins frm digitized bks and, nn- jurnal dcuments (e.g., nline theses, research/technical reprts, curse syllabuses and preprint repsitries) and PwerPint presentatins. Ggle Bks citatin: Cunting citatins frm bks can help schlarly impact assessment and that Ggle Bks (GB) is a useful surce f such citatin cunts, despite its lack f a public citatin index. T make it practical fr ACUMEN Prtfli t extract citatin metrics frm nline bks, we intrduced a methd t autmatically remve false and irrelevant matches frm GB citatin searches. The methd was evaluated by manual checking f sampled GB results and cmparing citatins t abut 14,500 mngraphs in the Thmsn Reuters Bk Citatin Index (BKCI) against autmatically extracted citatins frm GB acrss 24 subject areas. GB citatins were 103% t 137% as numerus as BKCI citatins in the humanities, except fr turism (72%) and linguistics (91%), 46% t 85% in scial sciences, but nly 8% t 53% in the sciences. In all cases, hwever, GB fund substantially mre citing bks than did BKCI, with BKCI's results cming predminantly frm jurnal articles. Result suggests that GB citatin mdel culd measure the different aspects f impact in ACUMEN Prtfli, filling the current gap fr assessing research utputs in bks based- fields, especially in arts and humanities. (see Publicatin 1, Kusha and Thelwall, in press 2014a). 1

3 Ggle Bks in Webmetric Analyst: As an exclusive sftware t meet the ACUMEN prject, a cmputer applicatin was designed and tested fr autmatic citatin extractin frm Ggle Bks. The prgramme is freely available and individuals, reviewers, institutins and universities acrss Eurpe and ther cuntries can dwnlad and use the applicatin fr mnitring their research perfrmance based n citatins frm millins f nline bks (see Publicatin 2, Thelwall and Kusha, Ggle Bks in Webmetric Analyst). Mdeling the web presence f Eurpean academics: In rder t investigate hw the EU researchers (including highly cited researchers) use web t disseminate research, we cnducted a large scale survey f and analysed hyperlinks frm academic web CVs. Disseminating Research with Web CV by Eurpean academics: Sme academic web CVs, including hmepages and publicatin lists, link t pen access (OA) papers, resurces, abstracts in publishers' websites, r academic discussins, helping t disseminate research. T assess hw cmmn such practices are and whether they vary by discipline, gender and cuntry, we cnducted a large scale survey f astrnmy and astrphysics, public health, envirnmental engineering, and philsphy acrss 15 Eurpean cuntries and analysed hyperlinks frm academic web CVs. Abut 60% f the 2,154 survey respnses reprted having a web CV r similar and there were differences between disciplines, genders and cuntries. A fllw- up utlink analysis f 2,700 web CVs fund that a third had at least ne utlink t an OA target, typically a public eprint archive r an individual self- archived file. This prprtin was cnsiderably higher in astrnmy (48%) and philsphy (37%) than in envirnmental engineering (29%) and public health (21%). There were als differences in linking t publishers' websites, resurces and discussins. Perhaps mst imprtantly, hwever, the amunt f linking t OA publicatins seems t be much lwer than allwed by publishers and jurnals, suggesting that many pprtunities fr disseminating full- text research nline are being missed, especially in disciplines withut established repsitries. Mrever, few academics seem t be expliting their CVs t link t discussins, resurces r article abstracts, which seems t be anther missed pprtunity fr research publicity (see Publicatin 3, Kusha and Thelwall, in press 2014b). Web presence f Eurpean highly cited researchers: The nline presence f abut 1,500 highly cited researchers wrking at Eurpean institutins shwed that abut 70% f them have a persnal website r ther web cntents, specially the scientists frm Denmark, Israel and the United Kingdm. Nevertheless, was results biased twards senir male researchers wrking in large cuntries (e.g., The United Kingdm and Germany). The mst frequent disciplines with high web presence were ecnmics, mathematics, cmputer sciences and space sciences, suggesting the success f pen access subject repsitries like RepEc, Arxiv r CiteSeerX (see Publicatin 4, Mas- Bleda & Aguill, 2013). A further study f 1,525 highly cited scientists wrking at Eurpean institutins als fund that 61% f them had a persnal website, althugh this was higher in scial sciences with 79% and lwer in health sciences with 49%. Webmetric analysis was carried ut fr the 892 scientists with either a persnal website r an nline list f publicatins fr which the crawler used wrked, shwing that 355 (40%) f them created at least ne utlink t pen access surces [OA repsitries, pdf, dc, dcx, rtf, ps, gz], being higher in hard sciences (59%) and scial sciences (58%) and engineering (45%) than life sciences (26%) and health sciences (19%). Disciplinary and gegraphical differences fr utlinking t OA research als were fund fr sampled highly cited 2

4 scientists (see Publicatin 5, Mas- Bleda, Thelwall, Kusha & Aguill, 2014). It was als investigated if these highly cited researchers had scial web presences and if these presences had a measurable impact. It was fund a very lw use f scial sites, althugh researchers having ne type f prfile were mre likely t have anther in many cases. Mst scial web prfiles had sme evidence f uptake, if nt impact, but the value f the indicatrs used is unclear (see Publicatin 6, Mas- Bleda, Thelwall, Kusha, & Aguill, 2013). Mdeling using scial web fr research cmmunicatin: Nwadays, researchers use infrmal schlarly channels t disseminate and publicise their research thrugh the web. T investigate this, we cnducted three studies based n 1) YuTube science vides cited in academic publicatins 2) ResearchGate statistics which is a scial netwrk site fr academics and 3) Academia.edu scial netwrk statistics. The rle f nline vides in research cmmunicatin: Althugh there is sme evidence that nline vides are increasingly used by academics fr infrmal schlarly cmmunicatin and teaching, the extent t which they are used in published academic research is unknwn. This article explres the extent t which YuTube vides are cited in academic publicatins and whether there are significant brad disciplinary differences in this practice. T investigate, we extracted the URL citatins t YuTube vides frm academic publicatins indexed by Scpus. A ttal f 1,808 Scpus publicatins cited at least ne YuTube vide and there was a steady upward grwth in citing nline vides within schlarly publicatins frm 2006 t 2011, with YuTube citatins being mst cmmn within arts and humanities (0.3%) and the scial sciences (0.2%). A cntent analysis f 551 YuTube vides cited by research articles indicated that in science (78%) and in medicine and health sciences (77%) ver three quarters f the cited vides had either direct scientific (e.g., labratry experiments) r scientific- related cntents (e.g., academic lectures r educatin), whereas in the arts and humanities abut 80% f the YuTube vides had art, culture r histry themes and in the scial sciences abut 63% f the vides were related t news, plitics, advertisements and dcumentaries. This shws bth disciplinary differences and the wide variety f innvative research cmmunicatin uses fund fr vides within the different subject areas (see publicatin 7, Kusha, Thelwall and Abdli, 2012). Mtivatins fr citing YuTube vides in the academic publicatins: This article explres what are the cmmn brad reasns fr using nline vides acrss majr subject areas. A cntextual analysis f citatins t YuTube vides based upn 207 sampled full- text publicatins indicated that in sciences 70% and in medicine abut half f the citatins were created t visual infrmatin (e.g., real- time scientific demnstratins and labratry experiments) as evidence t supprt r illustrate schlarly arguments. Hwever, in arts and humanities YuTube vides were mstly used fr citing artistic perfrmances r visual utputs (43%) such as music, dance r mvie parts and in scial sciences direct r indirect citatins t talks, speeches r lectures (45%) were cmmn mtivatins fr citing YuTube vides. This suggests that this new nn- standard academic medium can be helpful fr research cmmunicatin, althugh different fields cite nline vides fr different reasns in scientific literature (see Publicatin 8, Kusha and Thelwall, 2012). Academia.edu as infrmal surce f impact: Academic scial netwrk sites Academia.edu and ResearchGate and reference sharing sitesmendeley, Bibsnmy, 3

5 Zter, and CiteULike give schlars the ability t publicise their research utputs and cnnect t each ther. With millins f users, these are a significant additin t the schlarly cmmunicatin and academic infrmatin seeking ec- structure. There is thus a need t understand the rle that they play and the changes, if any, that they can make t the dynamics f academic careers. This article investigates attributes f philsphy schlars n Academia.edu, intrducing a median- based time- nrmalising methd t adjust fr time delays in jining the site. In cmparisn t students, faculty tend t attract mre prfile views but female philsphers did nt attract mre prfile views than did males, suggesting that academic capital drives philsphy uses f the site mre than friendship and netwrking. Secndary analyses f law, histry and cmputer science cnfirmed the faculty advantage (in terms f higher prfile views) except fr females in law and females in cmputer science. It als fund a female advantage fr bth faculty and students in law and cmputer science as well as fr histry students. Hence, Academia.edu verall seems t reflect a hybrid f schlarly nrms (the faculty advantage) and a female advantage that is suggestive f general scial netwrking nrms. Finally, traditinal biblimetric measures did nt crrelate with any Academia.edu metrics fr philsphers, perhaps because mre senir academics use the site less extensively r because f the range infrmal schlarly activities that cannt be measured by biblimetric methds (see Publicatin 9, Kusha and Thelwall, in press, 2014a). ResearchGate metrics fr research evaluatin: ResearchGate is a scial netwrk site fr academics t create their wn prfiles, list their publicatins and interact with each ther. Like Academia.edu, it prvides a new way fr schlars t disseminate their publicatins and hence ptentially changes the dynamics f infrmal schlarly cmmunicatin. This article assesses whether ResearchGate usage and publicatin data bradly reflect existing academic hierarchies and whether individual cuntries are set t benefit r lse ut frm the site. The results shw that rankings based n ResearchGate statistics crrelate mderately well with ther rankings f academic institutins, suggesting that ResearchGate use bradly reflects traditinal academic capital. Mrever, while Brazil, India and sme ther cuntries seem t be disprprtinately taking advantage f ResearchGate, academics in China, Suth Krea and Russia may be missing pprtunities t use ResearchGate t maximise the academic impact f their publicatins (see publicatin 10, Kusha and Thelwall, in press, 2014b). A theretical discussin n the impact f web- mediated schlarly cmmunicatin in ACUMEN: Quantitative research impact assessment has relied primarily upn analyses f citatins between published academic dcuments in citatin indexes (e.g., Scpus, Web f Science) which can nly cver a limited subset f the scientific literature. There are als ther academic activities and utputs that are essential t the prgress f science but rarely result in frmal publicatins such as teaching, schlarly discussins, science vides, presentatins which may smetimes cite research in ways that wuld be undetectable thrugh traditinal citatin indexes. Web surces are increasingly used in research and schlarly cmmunicatin and it may be pssible t extend biblimetric methds beynd cnventinal citatin indexes t the web. In a bk chapter we discusses the use f web metrics fr assessing the impact f academic research whether artifacts, articles, researchers r institutins and argues that web impact metrics culd ptentially 4

6 supplement cnventinal impact metrics by including new r unique types f surces f impact (e.g., presentatins, syllabi r digitised bks), emerging types f scientific utputs (e.g., nline vides r science blgs). We described different methds t capture web impact, including hyperlinks, web citatin, URL citatins and hybrid apprach (Web/URL citatin). These methds have been used t cllect web impact evidence by a web crawler r by queries t cmmercial search engines and have cmmnly been cmpared with cnventinal cunterparts (e.g., WS r citatins) t assess whether these web metrics are relevant fr research evaluatin. We reviewed previus attempts t extract and use frmal citatins frm web databases and digital libraries including CiteSeer, Ggle Schlar, Ggle Bk and reprts previus results abut using them as an alternative citatin impact metrics. It has been discussed that Ggle Schlar citatin metrics (citatins cunts, h- indexes, etc.) and Ggle Bks citatins frm a huge number f digitised bks can be used fr mnitring research perfrmance, especially in the scial sciences, arts and humanities, when traditinal citatin indexes are nt available r have insufficient cverage. We intrduced new types f web impact including citatins frm nline syllabi and curse reading lists which reflect the educatinal impact f research, dwnlad cunts f academic publicatins which may be an indicatr f reading and usage rather than frmal citatin cunts. We briefly discussed emerging scial web impact metrics r Altmetrics which can ptentially be used utside standard academic surces and indicatrs such as scial bkmarks, tweets r nline readership f scientific publicatins r views f nline academic vides. We cncluded that there are many ways in which research impact can be assessed using the web and the practical applicatins f web extracted metrics fr research assessment include calculating indicatrs fr bjects utside f traditinal citatin indexes, frm scientific publicatins t schlars and institutins. Hwever, web impact indicatrs suffer frm a generic lack f quality cntrl cmpared with schlarly citatins, and hence shuld be used cautiusly in research evaluatin (see publicatin 11, Kusha & Thelwall, 2014c). Publicatins assciated with D2.5 ACUMEN Prtfli Mdel 1. Kusha, K. & Thelwall, M. (in press, 2014a). An autmatic methd fr extracting citatins frm Ggle Bks. Jurnal f the American Sciety fr Infrmatin Science and Technlgy. Full- text available nline: 2. Thelwall and Kusha. (2013). Ggle Bks in Webmetric Analyst. Sftware. Available: 3. Kusha, K. & Thelwall, M. (in press, 2014b). Disseminating Research with Web CV Hyperlinks. Jurnal f the American Sciety fr Infrmatin Science and Technlgy. Full- text available nline: 4. Mas- Bleda, A., & Aguill, I. F. (2013). Can a persnal website be useful as an infrmatin surce t assess individual scientists? The case f Eurpean highly cited researchers. Scientmetrics, 96(1), Full- text available nline: Bleda%20%26%20Aguill%202013%20- %20Pstprint%20f%20EHC%20researchers%27%20persnal%20websites.pdf 5

7 5. Mas- Bleda, A., Thelwall, M., Kusha, K., & Aguill, I. F., (2014). Successful researchers publicizing research nline: An utlink analysis f Eurpean highly cited scientists' persnal websites, Jurnal f Dcumentatin, 70(1), Full- text available nline: preprint.pdf 6. Mas- Bleda, A., Thelwall, M., Kusha, K., & Aguill, I. F., (2013). Eurpean Highly Cited Scientists Presence in the Scial Web. In: 14 th Internatinal Sciety f Scientmetrics and Infrmetrics Cnference. Vienna, Austria, pp Full- text available nline: 7. Kusha, K., Thelwall, M. & Abdli, M. (2012). The rle f nline vides in research cmmunicatin: A cntent analysis f YuTube vides cited in academic publicatins, Jurnal f the American Sciety f Infrmatin Science and Technlgy, 63(9), Full- text available nline: in- preprint.pdf 8. Kusha, K. & Thelwall. M. (2012). Mtivatins fr Citing YuTube Vides in the Academic Publicatins: A Cntextual Analysis. 17th Internatinal Cnference n Science and Technlgy Indicatrs (STI), 5-8 September, 2012 in Mntreal, Quebec, Canada. Full- text available nline: 9. Thelwall, M. & Kusha, K. (in press, 2014a). Academia.edu: Scial netwrk r academic netwrk? Jurnal f the American Sciety fr Infrmatin Science and Technlgy. Full- text available nline: Thelwall, M. & Kusha, K. (in press, 2014b). ResearchGate: Disseminating, cmmunicating and measuring schlarship? Jurnal f the American Sciety fr Infrmatin Science and Technlgy. Full- text available nline: Kusha, K. & Thelwall, M. (2014c). Web Impact Metrics fr Research Assessment. In: B. Crnin & C.R. Sugimt, (Eds), Beynd Biblimetrics: Harnessing Multidimensinal Indicatrs f Schlarly Impact, MIT Press. ISBN: Summary f bk chapter available nline: 6