DEVELOPMENT OF SCIENTIFIC SOFTWARE: A SYSTEMATIC MAPPING, A BIBLIOMETRICS STUDY, AND A PAPER REPOSITORY

Transcription

1 International Journal of Software Engineering and Knowledge Engineering Vol. 23, No. 4 (2013) #.c World Scienti c Publishing Company DOI: /S DEVELOPMENT OF SCIENTIFIC SOFTWARE: A SYSTEMATIC MAPPING, A BIBLIOMETRICS STUDY, AND A PAPER REPOSITORY ROSHANAK FARHOODI prairiefyre Software Inc., Ottawa, Canada rfarhoodi@gmail.com VAHID GAROUSI Department of Electrical and Computer Engineering University of Calgary, Canada and Informatics Institute Middle East Technical University Ankara, Turkey vgarousi@ucalgary.ca DIETMAR PFAHL Institute of Computer Science, University of Tartu Estonia and Department of Electrical and Computer Engineering University of Calgary, Canada dpfahl@ucalgary.ca JONATHAN SILLITO Department of Computer Science University of Calgary, Canada sillito@ucalgary.ca Received 24 April 2012 Revised 17 October 2012 Accepted 9 January 2013 Scienti c and engineering research is heavily dependent on e ective development and use of software artifacts. Many of these artifacts are produced by the scientists themselves, rather than by trained software engineers. To address the challenges in this area, a research community often referred to as \Development of Scienti c Software" has emerged in the last few decades. As this research area has matured, there has been a sharp increase in the number of papers and results made available, and it has thus become important to summarize and provide an overview about those studies. Through a systematic mapping and bibliometrics study, we have reviewed 130 papers in this area. We present the results of our study in this paper. Also we have made the 463

2 464 R. Farhoodi et al. mapping data available on an online repository which is planned to be updated on a regular basis. The results of our study seem to suggest that many software engineering techniques and activities are being used in the development of scienti c software. However, there is still a need for further exploration of the usefulness of speci c software engineering techniques (e.g., regarding software maintenance, evolution, refactoring, re(v)-engineering, process and project management) in the scienti c context. It is hoped that this article will help (new) researchers get an overview of the research space and help them to understand the trends in the area. Keywords: Development of scienti c software; systematic mapping; bibliometrics study; paper repository. 1. Introduction New knowledge in science and engineering relies increasingly on results produced by scienti c software. To build high-quality software in a cost e ective manner for applications in science and engineering disciplines, scientists and the Software Engineering (SE) community have collaborated in the last few decades and a community mostly referred to as \Development of Scienti c Software" has emerged [63, 115]. In this context, the term \scienti c software" is referrers to software which is developed by scientists for scientists and which implements complex algorithms, e.g., to solve systems of mathematical equations or to provide simulations for physicists. Over the years, the SE practice and research has focused on techniques and concepts intended to be generally applicable to software development. However, SE best practices and research have only rarely been adopted by scienti c computing [131]. The motivation for the development of scienti c software is to conduct science. Unlike software engineers, scientists' focus is on performing science, not on writing software [132]. Also, the variation in di erent disciplines of science, maturity of the science, and motivation of scienti c software projects in uence development methods, and one would consequently expect large variations both across and within di erent disciplines. As the community and research area of scienti c software has matured, there has been a sharp increase in the number of papers and results made available, and it has thus become important to summarize and conduct a systematic classi cation of those studies. To address this need, we have conducted a systematic mapping study and have collected bibliometric and demographic data in this area. From an initial pool of about 500 publications, we systematically selected and classi ed (mapped) 130 publications. Regarding the type of research, two thirds of the set of 130 publications are \solution proposals", \experience papers", and \ eld studies". The most frequently mentioned scienti c discipline is \physics", and the most frequently used programming languages are \Fortran" and \Cþþ", with an emerging trend towards using more modern programming languages such as \Java" and \Python". The software development activities and techniques most frequently mentioned in our pool of selected publications are \development/coding", \architecture and design", and \testing/v&v/quality assurance". Based on an analysis of the bibliometric and demographic data (similar to previous studies such as [ ]), ten research questions regarding, for example, the

3 Development of Scienti c Software 465 frequency of relevant publications over time, the most prominent publication venues, the most cited publications, the most active researchers, and the geographical distribution of publications, have also been addressed in this study. Mapping studies and their results have several bene ts, for example, they can help new researchers in this area (e.g., new PhD students) gain a rst overview of their eld of research [137], and they can indicate directions for new research. We hope that the results of our study can provide a better understanding about the activities performed and the techniques used in the development of scienti c software. Our study also aims to identify current trends and directions in this eld, which are valuable to both researchers in software engineering (SE) and scientists who develop software. The other goal of this SM is to identify strengths and weaknesses of stateof-the-art in the area. The rest of this article is structured as follows. Background and related work are discussed in Sec. 2. Section 3 presents the goal of the study, the research questions and research method. The article selection process and details are presented in Sec. 4. Section 5 presents the classi cation scheme (map) of our systematic mapping. Results of the systematic mapping are described in Sec. 6. Section 7 provides the bibliometric data and demographics of the publications. Section 8 discusses threats to validity. Finally, Sec. 9 concludes the article and indicates directions of future research. 2. Background and Related Work We present a brief background on systematic mapping studies in software engineering in Sec Then in Sec. 2.2, we summarize the secondary studies in the area of development of scienti c software Background: Systematic mapping studies in software engineering As a research area matures, there is often a sharp increase in the number of papers and results made available, and it becomes important to summarize and provide an overview about those studies [138]. Many research elds (e.g., medical sciences) have speci c methodologies for such secondary studies, and they have been extensively used in for example evidence-based medicine. Budgen et al. [137] presented di erent bene ts of conducting mapping studies and usefulness of their results. For example, such studies can form a useful preliminary step for new PhD students in this area. The guideline paper by Petersen et al. [138] is a great resource on conducting systematic mapping (SM) studies in software engineering, which we used as a guide in designing our study. A software engineering systematic map is a de ned method to build a classi cation scheme and structure a software engineering eld of interest [138]. The analysis of results focuses on frequencies of publications for categories within the scheme. Thereby, the coverage of the research eld can be determined. Di erent facets of the scheme can also be combined to answer more speci c research questions.

4 466 R. Farhoodi et al. Similar to SMs, but on a more comprehensive scale, Systematic Literature Review (SLR) is another secondary study method that has gotten much attention lately in SE. Brie y, a systematic review (SLR) identi es existing primary studies, reviews them in-depth and describes used methodologies and achieved results related to a de ned subject matter topic. SLRs usually has more analytic depth than SMs. Often, SLRs include a SM as one of their elements. As the rst step in our e orts to review the existing literature on methods and techniques used in the development of scienti c software, the current article reports a SM (and not a SLR) Secondary studies in the area of development of scienti c software In total, we identi ed 130 relevant publications [1 130] for the period Within this set, only two recent publications [119, 139] were secondary studies. Howison et al. reviewed three publications and interviewed the authors and developers of the projects presented in the papers [119]. Based on the analysis of the selected publications and related interviews, the authors identi ed several modes of how the development and maintenance of software interleaves with the way research is conducted by scientists. These modes depend on the purpose of the software and the role it can play in the system of reputation incentives for scientists. The authors conclude that the development and maintenance of scienti c software may either be indirectly rewarded with academic credit if the software facilitates science publications, or it may be directly rewarded if the software can be released for academic credit on its own. The latter case, however, requires a certain degree of collaboration between scientists and a readiness of scienti c community to accept scienti c software as a research asset in its own right. Sletholt et al. presented a literature review of reported agile practices and their e ects in the development of scienti c software [139]. The authors identi ed ve publications reporting on experience with the use of agile practices in scienti c software projects. Based on the analysis of these papers, the authors conclude that there exists evidence that using agile practices can have positive e ects on test e ectiveness and e±ciency. 3. Research Method In the following, an overview of our research method and then the goal and research questions of our study are presented Overview This SM is carried out based on the guidelines provided by Petersen et al. [138], and Kitchenham and Charters [140]. In designing the methodology for this SM, methods from several other SMs such as [ ] were also incorporated.

5 Development of Scienti c Software 467 The protocol (method) that lies at the basis of this SM is outlined in Fig. 1, which distinguishes ve phases that are described in Secs All the steps of the research process were carried out manually. Data collection and statistical software tools (e.g., Microsoft Excel and Minitab) were helpful during the process Goal and research questions The approach we have used in our study is the Goal Question Metric (GQM) methodology [144]. Using the GQM's goal template [144], the goal of this study is to systematically review the state-of-the-art in utilizing software engineering (SE) techniques, and methods in the development of scienti c software, to classify the articles in this area, to explore the bibliometrics trends in this area, to identify the weaknesses and strengths, and to nd out the recent trends and directions in this eld from the point of view researchers and scientists in this area. In simpler terms, the goal is to understand existing scienti c software research area with regard to several attributes (e.g., type of papers, applications, and systems under analysis). We are also interested in understanding how the research has been changing over-time with regards to those research attributes. Based on the above goal, the following research questions are raised. To extract detailed information for each of the questions, each question is divided into subquestions. Metrics used to answer each of the questions are discussed in Sec. 5.. RQ 1 Systematic mapping: What is the research space of scienti c software development? This question aims at conducting a systematic mapping (classi cation) in the area. We raised the following sub-questions for RQ 1 which are similar to other SM studies [138]. RQ 1.1 Type of paper: What types of papers have been published in the area? This question will enable us to answer whether papers are technical in nature or propose opinions. RQ 1.2 Scienti c disciplines: What scienti c disciplines (e.g., chemistry, mathematics, and physics) have been covered more in the articles in this area? SS development is an interdisciplinary research area and by extracting the disciplines which are frequently demanding the development of software systems, more rigorous discipline-dependent practices and frameworks can be suggested to better t this demand. RQ 1.3 Attributes of the software systems studied: What are the attributes of the systems under study in the papers, such as LOC or programming language? RQ 1.4 SD/SE activities/techniques: What SD/SE activities/techniques are addressed/discussed in each paper?. RQ 2 Bibliometric data and demographics: What are the trend of publications over years, demographics of the authors/countries and also the

6 468 R. Farhoodi et al. bibliometrics data of the papers? Bibliometrics is a set of methods to quantitatively analyze scienti c literature. We raised the following sub-questions for RQ 2 which are similar to other bibliometric and demographic studies [ ]. RQ 2.1 Publication count by year: What is the annual publication count? RQ 2.2 Publication count by venue type: What is the annual publication count in di erent types of venues (conference, journals etc)? RQ 2.3 Citation count by year: What is the citation count for papers from di erent years? RQ 2.4 Top cited papers: What are the top cited papers in this area? RQ 2.5 Citation count by venue type: What is the citation count for di erent publication venue types? RQ 2.6 Citation rates of top venues: What is the citation count for most frequently used venues? RQ 2.7 Top venues: Which venues had the highest publication count? RQ 2.8 Top authors: Which authors have been most active? RQ 2.9 Author a±liation: How many authors belong to academia or industry? RQ 2.10 Publication count by country: Which countries have been more active? 4. Article Selection For the article selection phase of our SM, we explain the following steps in order:. Source selection and search keywords (Sec. 4.1).. Exclusion criteria (Sec. 4.2).. Inclusion criteria (Sec. 4.3).. Final pool of articles and the online repository (Sec. 4.4) Source selection and search keywords To nd relevant papers, we searched the following online article search engines: (1) IEEE Xplore, (2) ACM Digital Library, (3) Google Scholar, (4) Microsoft Academic Search, (5) CiteSeerX, and (6) Science Direct. In order to ensure that we are including as many relevant publications as possible in the pool of selected papers, we identi ed all potential search keywords regarding the focus of each of our research questions. First, we classi ed these keywords into two sets of search strings as show in Table 1. The rst set of search strings contained keywords related to scienti c software. The second set of search strings contained more software development related keywords. The strings from the rst group were combined with the strings from the second group to form a nal set of comprehensive search strings. We produced combined search strings via concatenation using the AND and OR operator.

7 Development of Scienti c Software 469 Table 1. Our two sets of search keywords. Set one \scienti c software or application or program" \computing software or application or program" \scienti c computing" \computational science" \scienti c software development" \scienti c computation" Set two \requirement" \requirement elicitation" \analysis" \design" \maintenance" \implementation" \programming" \testing" \veri cation", \validation" \architecture" \software engineering" \documentation" With these search strings we found 452 publications which we considered as our initial pool of potentially relevant publications. If multiple papers with the same (or similar) title by the same author(s) were found, the most recent one was included and the rest were excluded. This was especially applied for extended journal articles and their earlier conference versions. However, if titles of papers by same author(s) and also their contents were substantially di erent, we included all of them. Only publications written in English language and only publications which were electronically available were included in this pool. Note that we performed the literature search during the Summer (July until August) 2011 and thus all the publications available by the end of August 2011 were included in our pool. To keep our search and ltering process manageable, we decided to set year 1996 as the lower bound on publication date of papers, and thus our SM study reveals the trends in the last 15 years Exclusion of irrelevant publications The next step in our article selection process was to exclude those papers that were in our pool of 452 papers, but did not relate to the goals of our study. Speci cally, our focus is on papers that discuss the application of SE techniques, methods and tools in the development of scienti c software, as well as speci c conditions and issues observed in the development of scienti c software. For inclusion, we considered papers that validated their ndings empirically, and also those which were peer-reviewed. As a result we excluded other kinds of papers, such as papers that only describe a piece of scienti c software (e.g., a library or a tool to solve a speci c set of di erential equations). To apply this exclusion criteria to the initial pool, the rst three authors of this article inspected the publications in the initial pool and assigned a vote on a 4-point scale to each publication, with \4" indicating a strong opinion in favor of not excluding a publication, and \1" indicating a strong opinion in favor of excluding a publication. Thus, the maximum vote on a publication could be 12 marks. We decided to use a threshold of 8 marks for the decision on publication exclusion, i.e., publications with cumulative votes of less than 8 marks were excluded. The speci c threshold value of 8 marks was chosen since it represented a signi cant level of

8 470 R. Farhoodi et al. agreement. We reviewed the votes of the authors and threshold value of 6 or 7 did not seem to be good threshold values. To mark each paper we reviewed its title, abstract and keywords. If not enough information could be found in those sources, a more in-depth evaluation was conducted. Based on the results of the joint voting, the size of the pool of selected publications decreased from 452 to 118. We should note that, to increase the preciseness of our study and its results, we utilized systematic voting process among the four authors in the article selection, exclusion and inclusion phases Inclusion of additional publications To decrease the risk of missing relevant publications, similar to previous SM studies and SLRs, we searched the following potential publication venues manually:. Personal web pages of active researchers in the eld of interest.. References found in publications already in the pool.. Speci c venues. Speci c venues included journals and proceedings conferences and workshops in the area of scienti c software, for example: the Elsevier Journal on Advances in Engineering Software, IEEE Journal on Computing in Science & Engineering, Springer Journal of Scienti c Computing and International Workshop on Software Engineering for Computational Science and Engineering. All publications found in the additional venues that were not yet in the pool of selected publications but seemed to be candidates for inclusion were fed into the voting mechanism described in Sec If a candidate publication surpassed the threshold of 8 or more votes, we added the publication to the pool Final pool of articles and the online repository After the initial search and the follow-up analysis for exclusion of irrelevant and inclusion of additional publication, the pool of selected publications comprised was - nalized with 130 papers [1, 130] (refer to Sec for the full list of primary studies). The nal pool of selected publications has been published as an online repository using Google Docs, and is accessible publically online at URL [145]. We plan to update the online repository at least once each year and to add relevant material published in the future. The classi cations of each selected publication according to the classi cation scheme described in Sec. 5 are available in the online repository [145]. 5. Classi cation Scheme (Map) Iterative development of our classi cation scheme (map) is discussed in Secs. 5.1 and 5.2 presents our metrics and data extraction approach.

9 Development of Scienti c Software Iterative map development To develop our classi cation scheme (map), as it is shown in Fig. 1 (discussed in Sec. 3.1), we analyzed the papers in the pool and identi ed the initial list of attributes. We then used attribute generalization and iterative re nement to derive the le map. To increase the preciseness of our classi cation scheme, we utilized the \observer triangulation" method [146] in designing the classi cation scheme. As papers were identi ed as relevant to our research project, we recorded them in a shared spreadsheet (hosted in the online Google Documents system) to facilitate further analysis. The following information was recorded for each paper: (1) paper title, (3) authors, (2) publication venue (e.g., IEEE Software), (4) year of publication, (5) home country of authors, and (6) type of institution authors are a±liated with (governmental, academic, industry or a combination). With the relevant papers identi ed and recorded, our next goal was to categorize the papers in order to begin building a complete picture of the research that has been conducted in the scienti c software domain. Though we did not a priori develop a categorization scheme for this project, we were broadly interested in: (1) the methodological techniques reported on by the papers, (2) the scienti c software domain of the research, and (3) the software engineering activities and techniques covered in the research reported in the papers. We re ned these broad interests into a categorization scheme using an iterative approach that involved all four of the authors of this paper. The rst author of this IEEE Xplore ACM Digital Library Google Scholar Articles by browsing personal web pages Referenced articles Articles from specific venues Relevant articles found in databases (452) Application of exclusion criteria Filtered set (118) Application of inclusion criteria Final selection (130) Microsoft Academic Search CiteSeerX Science Direct Article selection (Section 4) Final Map Attribute Generalization and Iterative Refinement Initial Attributes Attribute Identification Classification Scheme/Map (Section 5) Systematic mapping (Section 6) Systematic mapping Systematic Mapping results RQ 1 RQ 2 Bibliometrics of the research space Bibliometrics and Demographic Analysis RQ 2 Demographics of the research space Trends, Bibliometrics and Demographics (Sec. 7) Fig. 1. Our research process.

10 472 R. Farhoodi et al. paper conducted an initial pass over the data, and based on (at least) the title, abstract and introduction of the papers created a set of initial categories and assigned papers to those categories. As a group we discussed and reviewed the results of this rst analytic pass and re ned the categorization. Next the rest of the researchers conducted a second pass over the data, to revisit the categorization. When the assignment of papers to categories could not be clearly determined just based on the title, abstract and introduction, more of the paper was considered. In this process both the categories and the assignment of papers to categories were further discussed and re ned. In the end, every paper was reviewed by at least two researchers. The rst set of categories in our scheme deal with the nature of the research reported on by the paper. These categories were in uenced by categories described by Petersen et al. [138] and our aim is to provide insights into the empirical foundation being developed by the body of research. The \research type" categories include:. Solution Proposal A paper in this category proposes a solution to a problem. The solution can be either novel or a signi cant extension of an existing technique. The potential bene ts and the applicability of the solution are shown only by a small example or a good line of argumentation.. Validation Research Involves the investigation of novel techniques or tools that have not yet been implemented in practice. Empirical techniques used are, for example, laboratory experiments.. Evaluation Research These studies go further than \ Validation Research" by implementing and evaluating techniques or tools in practice. These studies demonstrate how the technique is implemented in practice and explore the consequences of the implementation in terms of bene ts and drawbacks.. Philosophical Papers These papers sketch a new way of looking at existing things by structuring the eld in form of a taxonomy or conceptual framework.. Opinion Papers These papers express the personal opinion of the author(s) around whether a certain technique is good or bad, or how things should been done. They do not signi cantly rely on related work or research methodologies.. Experience Papers Experience papers explain how something has been done in practice, based on the personal experience of the author(s).. Field Study A study that aims to gather data about software engineering in real scienti c software environments. Examples data collection methods used include questionnaires, interviews, focus groups, and participant observation.. Secondary Study A study that uses data from secondary sources. Examples include systematic literature reviews, literature surveys, and systematic mappings.. Other A catchall category in the event that the work reported in a paper does not t into any of the above research types. The second set of categories in our scheme deal with the scienti c software domain under study, with the goal of comparing how the body of research is distributed

11 Development of Scienti c Software 473 across various disciplines. The following is the list the disciplines we identi ed in the papers we have reviewed:. Math. Physics. Biology. Chemistry. Medical. Bio-medical. Engineering. Geo (Earth) Science. High Performance Computing (HPC). General. Other The nal set of categories in our schema deal with the software engineering activities and tools covered in the research reported in the papers. We were interested in the programming language used in the scienti c software systems investigated (if any) and more especially in what SE activities have been well studied in the scienti c software context and which have been, comparatively, neglected. Through iterative re nement, we identi ed the following list of SE activities/techniques as the best category for this aspect of our mapping:. Requirements Engineering and Analysis. Architecture and Design. Development (Programming). Testing, Veri cation and Validation and Quality Activities. Maintenance and Evolution. Process/Project Management. Agile Development Practices. Con guration Management. Run-time Performance As an example, one of the papers in our pool is \The T experiments: Errors in scienti c software" [100] which has had the highest number of citations (83) as of this writing (April 2012) among all papers in our pool. This paper was categorized as follows:. Research type: categorized under evaluation research (empirical study) since they study presents comprehensive empirical study with details statistic, etc.. Scienti c discipline: engineering and geo- (earth) science. Software systems studied and their attributes: LOC: 5,233,639, number of systems: 123, programming languages: C and Fortran. SE activities/techniques covered/used in the paper: testing, V&V, and quality

12 474 R. Farhoodi et al. Table 2. Metrics extracted from the papers. RQs Metrics RQ 1 Systematic mapping RQ 1.1 Type of paper Research type RQ 1.2 Scienti c discipline Scienti c discipline discussed in paper RQ 1.3 Software systems studied LOC, number of systems, programming language RQ 1.4 SD/SE activities/ techniques SD/SE activities/techniques used in paper RQ 2 Bibliometric data RQ 2.1 Publication count publication year and demographics by year RQ 2.2 Publication count publication Venue by venue type RQ 2.3 Citation count Citation count, publication Venue by venue type RQ 2.4 Citation count Citation count, publication year by year RQ 2.5 Top venues publication Venue RQ 2.6 Citation rates of top Citation count, publication Venue venues RQ 2.7 Top authors Names of authors RQ 2.8 Author a±liation Author a±liation (university, government, industry) RQ 2.9 Publication count by country Countries of authors' a±liation By extracting the size of the software systems in the publications, a general overview on the scale of scienti c software in the publications can be achieved. This can motivate the development of speci c practices to t scienti c software development based on the context in which they are expected to be used, e.g., large-scale software systems need di erent considerations in their development process compared to mid/ small-scale systems Metrics and data extraction Based on our map, we were then able to de ne the metrics we need to extract to answer each of the RQs. The metrics/data we extracted from papers are listed in Table 2. To extract data, the papers in our pool were reviewed with the focus of each research question and the required information was extracted. To increase the preciseness of our classi cation scheme, we utilized the \observer triangulation" method [146] in designing the classi cation scheme and data extraction (mapping) phases. 6. Results of Systematic Mapping (RQ1) This section presents results related to our rst research question (RQ1). It aims at characterizing the research space related to the development of scienti c software

13 Development of Scienti c Software 475 and, thus, RQ1 contains the systematic mapping component of our study. The subquestions under RQ1 are described in the following one by one Type of paper (RQ 1.1) Figure 2 shows the classi cation of the 130 selected publications [1 130] according to the type of research they report. Exact paper references have also been provided in the right side. Since some publications deal with more than one system or application or type of study, in some cases publications were categorized in multiple categories, yielding a total of 148 di erent instances of research type classi cations. More than one third of all classi cations (55 papers) relate to presentations of solution proposals without further validation or evaluation. A similar number of classi cations (56 papers) relate to presentations of either the authors' experience with various approaches to the development of scienti c software (28 papers) or results of eld studies (28 papers) reporting such approaches applied by others in real-world development organizations. The remaining classi cations (33 papers) relate mostly to opinion papers (18 papers) or to reports of validation research (10 papers) and evaluation research (5 papers). The numbers of secondary studies (3 papers) and philosophical papers (only 1 paper) are very small. The distribution of publications with regards to type indicates that research about the development of scienti c software is not yet mature, since the number of publications that report about validation and evaluation of solution proposals is rather small. On the other hand, we found a substantial number of experience papers, eld studies, and secondary studies, indicating that some work has been done to better understand the speci cs of developing scienti c software. We assume that publications presenting solution proposals are to some degree based in an understanding of the speci c conditions and problems related to the development of scienti c software. Fig. 2. Frequency of research types mentioned in publications.

14 476 R. Farhoodi et al. Fig. 3. Trend of research types mentioned in publications over time. Figure 3 shows how the publication years of papers included in our study evolved per type during the past 15 years. The numbers indicate that interest in research about software development practices for scienti c software strongly increased beginning with year Scienti c disciplines of scienti c software (RQ 1.2) We categorized the publications with respect to their scienti c disciplines. 29 publications did not mention a particular scienti c discipline. From the 101 publications in which at least one discipline was mentioned, the distribution is shown in Fig. 4. Since several publications mentioned more than one discipline, the numbers in Fig. 5 add up to 139. By far, the most frequently researched scienti c discipline is physics (37 papers), followed by mathematics (20 papers), biology (14 papers) and highperformance computing (HPC) (14 papers) Software systems studied (RQ 1.3) Where a given primary study provided information about the software system(s) under study, we extracted the following three pieces of information: (1) Number of scienti c systems discussed in each paper, (2) Size in LOC, and (3) Programming languages used for development of the system(s). Characterizing the primary studies by the number and size of scienti c systems discussed in each paper will enable to assess their level of evidence and experience. For example, a given paper which quantitatively and qualitatively discusses the challenges of requirements engineering

15 Development of Scienti c Software 477 Fig. 4. Frequency of scienti c disciplines mentioned in publications. in the context of scienti c software with experience based on 100 large-scale systems will provide higher degree of evidence than another paper which reports similar challenges based on one system only Number of systems discussed Only 28 publications explicitly inform about speci c software scienti c systems they are discussing. In eleven cases, publications discuss more than one software system. Figure 5 shows whether selected publications discuss one or several systems. As can Fig. 5. Number of systems discussed in publications.

16 478 R. Farhoodi et al. Fig. 6. Frequency of programming languages mentioned in publications. be seen, the majority of publications (17) discuss exactly one system. One publication mentions 123 di erent systems from 73 di erent organizations, and two publications mention 7 di erent systems. When looking at these numbers, one needs to keep in mind, however, that some publications are referring to software systems that are also mentioned in other publications. Hence, Fig. 6 contains duplicates Programming languages used Of the 130 selected publications, 97 informed about the programming language that speci c systems used or for which speci c solution proposals were made or other kinds of analyses were conducted. In total, we found 128 occurrences where programming languages used in scienti c software were mentioned in the selected publications. Figure 6 shows the frequencies with which di erent programming languages were mentioned. According to this data, the majority of scienti c software systems were written in Fortran followed by Cþþ, C, Python, Java and Matlab. In the category \other", the languages Perl and Visual Basic (VB) were mentioned most frequently. We were interested in nding trends related to the usage of programming languages in scienti c software systems. Thus, we plotted the distribution of programming language occurrences in the selected publications over time. Although there are no clear trends, Fig. 7 indicates that both Fortran and Cþþ are mentioned throughout the whole period of consideration ( ), while programming languages such as Java, Matlab and Python (with one outlier in 1997) have only

17 Development of Scienti c Software 479 Fig. 7. Trend of programming languages mentioned in publications over years. been mentioned since The programming language C has not been mentioned during the last two years ( ) Size in LOC For those publications that mentioned the size of the software systems discussed, we calculated the average lines of code (LOC) per system. Figure 8 shows the average number of LOCs of systems mentioned per publication over time. We had to take the average values since many papers did not provide the exact LOC of all systems they had analyzed, but rather reported a LOC range, e.g., [50] reported analysis of 5 systems ranging between 100K 500K LOC. Thus, for example, if a publication mentions LOC data of two systems, then the average LOC data is presented in Fig. 8. More recent papers are more frequently reporting LOC data, and the average LOC data reported per publication is less than 1 million LOC. One publication reports on a system with more than 10 million LOC [79]. The system is called ATLAS, software developed for particle accelerator experiments at the Large Hadron Collider (LHC) located at CERN, Geneva, Switzerland. The system is written in Cþþ and Python. Since the same paper mentions another system of size 140,000 LOC, the related data point in Fig. 9 reports an average of 5,070,000 LOC. One publication reports about two sets of systems with a total of more than 5 million LOC [100]. For two programming languages, Fortran and C, many code packages were collected from many companies, adding up to 3,305,628 LOC Fortran code and

18 480 R. Farhoodi et al. Fig. 8. Tend of the size (in LOC) of the software systems under study over years. 1,928,011 LOC C code. The Fortran and C codes stem from 47 (for Fortran) and 26 (for C) di erent organizations, with package sizes ranging from 806 to 770,444 (for Fortran, total of 55 packages, average package size 60,102 LOC) and 361 to 431,655 (for C, total of 68 packages, average package size 28,353 LOC). Due to the large number of di erent systems, the average size shown in Fig. 9 is only 42,550 LOC (for both programming languages taken together). Fig. 9. Frequency of SD/SE activities/techniques mentioned in publications.

19 Development of Scienti c Software Software development/engineering activities or techniques (RQ 1.4) We also tried to understand which software development (SD) and software engineering (SE) activities and techniques were addressed in the selected publications. Figure 9 shows how often speci c SD/SE activities/techniques were mentioned in the 130 selected publications. Most of the publications discuss aspects related to architecture/design, development/coding and testing/veri cation/validation/ quality. A smaller share of publications mentions aspects related to requirements engineering and software maintenance/evolution. Generally, management activities are underrepresented as well as modern development techniques (i.e., agile development) and speci c topics such as software performance. In Ref. 63 the authors found that there is a weak correlation between the size of a scienti c software development project and the importance that is attributed to project and product management. Since most of the projects discussed in the publications contained in our repository are small, this might explain why management-related activities are underrepresented. What surprised us is that \performance" is only mentioned three times. One possible explanation for this might be that performance is so much at the core of scienti c software development that it is not explicitly mentioned. Note that, to keep the article size reasonable, we have grouped the SD/SE techniques discussed in the papers as shown in Fig. 9. Interested readers can access the online repository to read more details about the exact SD/SE techniques used/ proposed in each paper (e.g., the types of architecture and design techniques), as the authors have extracted and provided those details online. Fig. 10. Trend of top three SD/SE activities/techniques mentioned in publications over years.

20 482 R. Farhoodi et al. Figure 10 shows how the mentioning of the top three SD/SE activities/techniques evolved over time. The data seems to indicate that the proportions between the occurrences of the top three activities/techniques have not signi cantly changed in the period 1996 l Trends, Bibliometrics and Demographics (RQ 2) This section presents results related to our second research question (RQ2). It aims at characterizing demographics of the selected publications as well as bibliometric data. Bibliometrics comprises a set of methods to analyze scienti c and technological literature [147]. In particular, it utilizes quantitative analysis and statistics to describe patterns of publication within a given eld or body of literature. Bibliometric data such as the ranking of authors according to the number of their publication in the eld of scienti c software development can potentially be used by researchers and grad students as an entry point for literature search and establishing promising contacts and, perhaps, collaborations [56]. The answers to sub-questions under RQ2 are described in the following one by one Publication count by year (RQ 2.1) Figure 11 shows the distribution of the selected 130 publications over the period 1996 to The data clearly shows that the number of publications in the second half of the period is signi cantly higher than in the rst half. This indicates that the topic \development of scienti c software" has gained popularity and importance Publication count by venue type (RQ 2.2) Figure 12 shows the distribution of the 130 selected publications over types of publication venues. Almost one third of the selected publications (35) are journal articles. The majority of publications, however, are workshop papers (42) and conference papers (30). Smaller numbers of publications appeared in magazines and proceedings of symposia. Fig. 11. Frequency of publications over time.

21 Development of Scienti c Software 483 Fig. 12. Frequency of publication per venue type Citation analysis Citation count by year (RQ 2.3) Figure 13 shows the distribution of citations of the 130 publications over the period 1996 to The citation numbers were taken on January 25, 2012 with Google Scholar. There were two publications [19], and [45], which have been cited an astonishingly 3,712 and 448 times, respectively. They will be discussed in Sec In order to make the charts more readable, the data reported in Fig. 13 does not include these \outlier" points. Figure 14 shows the histogram of number of citations to 128 papers in the pool, excluding the two highly-cited publications [19, 45]. We can see that 26 articles have Fig. 13. Citation count of selected publications over time.

22 484 R. Farhoodi et al Frequency Number of citations (as of Jan ) 90 Fig. 14. Histogram of number of citations to 128 papers in the pool (excluding two highly-cited publications [19, 45]). had no citations. 11 and 24 articles were cited 1 and 2 times, respectively. The average number of citations to the pool articles by excluding and including the two highly-cited publications were 9.60 and 41.46, respectively Most cited papers (RQ 2.4) The list of top 10 cited papers are shown in Table 3 (ordered by number of citations). The mapping of each article based on our classi cation scheme (Sec. 5.1) is also shown. The 2004 paper [19] written by 25 authors is cited 3,712 times. This paper reports an open software development platform called Bioconductor ( org) for computational biology and bioinformatics. It is a collaboration among researchers from the Fred Hutchinson Cancer Research Center (located in Seattle, Washington, USA) as the core team, and researchers from various US and international institutions. The goals of the project include: fostering collaborative development and widespread use of innovative software, reducing barriers to entry into interdisciplinary scienti c research, and promoting the achievement of remote reproducibility of research results. The authors describe details of their aims and methods, identify current challenges, compare Bioconductor to other open bioinformatics projects, and provide working examples. Bioconductor has been operational since 2001 and since then it has become a prominent software project for computational biology and bioinformatics. The authors of [19] argue that the success of the project is due to many factors. These include the choice of R [148] as the main programming language, the adoption of standard practices of software design and a belief that the creation of software infrastructure is an important and essential

23 Development of Scienti c Software 485 Table 3. Table 3-top 10 cited papers. Article Number of citations Year published Research type Scienti c discipline Number/and Scale (LOC) of system(s) studied Programming language(s) SD/SE activity/technique(s) [19] Validation research, experience report [45] Solution proposal Physics (combustion, micro-tomography) [100] Empirical study 40 application areas (e.g., nuclear engineering, chemical engineering, civil engineering) Biology Not reported R Architecture, design, development, Prototyping capabilities, Designing by contract Not reported C, Fortran 77 Architecture, design 123/5,233,639 C, Fortran Testing/ V&V/Quality [53] Validation research General Not reported Cþþ, Fortran Development, performance [84] Solution proposal HPC Not reported Veri cation [108] Experience paper Mechanics (hyper-velocity vehicles) 1/500,000 Fortran, Ruby Process, project management [30] Solution proposal Math Not reported Cþþ Architecture and design [94] Experience paper, Physics 7/509,710 Not reported Quality engineering, project lessons learned management [93] Series of eld study Manufacturing, signal processing, product performance 5/317,800 Cþþ, Fortran, Matlab, Python, Perl Veri cation and validation [12] Field study Space Not reported Not reported Requirements, architecture, development, testing

24 486 R. Farhoodi et al. component of a successful project of this size. In terms of SD/SE activities and techniques, the authors explain the architecture and design of the frameworks and provide examples on the development, prototyping capabilities, and design by contract in Bioconductor. The 1999 paper [45] written by 8 authors is cited 448 times. It reports a work-inprogress e ort to develop a common component architecture for high-performance scienti c computing. It is a collaboration among several American governmental laboratories (e.g., Sandia and Oak Ridge National Laboratories) and universities (e.g., Indiana University and University of Utah). This research stems from growing recognition that the scienti c community needs to better manage the complexity of multidisciplinary simulations and better address scalable performance issues on parallel and distributed architectures. Driving forces are the need for fast connections among components that perform numerically intensive work and for parallel collective interactions among components that use multiple processes or threads. This paper focuses on the areas we believe are most crucial in this context, namely, an interface de nition language that supports scienti c abstractions for specifying component interfaces and a ports connection model for specifying component interactions. The third-highly cited paper (81 citations) is a 1997 paper [100] and has a single author. It is a statistical work in the context of scienti c software. As the author reports, between 1990 and 1994, with the help of a collaborator and the cooperation of many commercial software vendors, he conducted two large experiments to test whether science and engineering software really does, accurately, what it is supposed to do. Together, he called those investigations the T experiments. Experiment T1 measured defects without running programs. Speci cally, it gauged the formal consistency of several million lines of scienti c software written in C and Fortran 77, from many di erent industries and application areas, by a technique called static deep- ow analysis [149]. Experiment T2 measured, within one of these industrial application areas, the extent to which several independent implementations of the same algorithms acting on the same input data with the same parameters-gave the same results when the authors ran the programs. This article [100] was a result of analyzing a large code-base (in total 5,233,639 LOC) consisting of 3,305,628 LOC of Fortran 66/77 and 1,928,011 LOC in C, from 73 organizations. Experiment T1 showed that C and Fortran code, independent of the application area, are riddled with statically detectable faults [100]. For example, calling-sequence inconsistencies (also known as interface inconsistencies) occurred at the average rate of one in every 7 interfaces in the Fortran code they tested, and one in every 37 interfaces in C. The authors also found that Fortran components are typically 2.5 times as large as C components, but interestingly, also had 2.5 times as many arguments; and that roughly 30 percent of the Fortran code analyzed and 10 percent of the C codes would be deemed untestable. Experiment T2 was even more disturbing. Scientists like to think that their results are accurate to the precision of the arithmetic used. In this study, however, the level of agreement among the results

25 Development of Scienti c Software 487 Fig. 15. Citation count by venue type (excluding two highly-cited publications [19, 45]). from similarly implemented code gradually degenerated from 6 signi cant gures to 1 during the computation. The authors concluded that the reasons for this disagreement can be laid squarely at the door of software failure, as other possible causes are considered and rejected Citation count by venue type (RQ 2.5) Figure 15 shows the distribution of the aggregated citation count of the 130 selected publications over types of publication venues. Publications in journals and magazines have the highest total citation numbers, closely followed by workshop publications. This indicates that publications in magazines have the highest since the total number of publications for magazines is substantially smaller than for journals and workshops. This observation can also be made based on the data shown in Fig. 16 where the average citation rates per publication in each of the venue types are shown. The citation numbers were extracted on January 25, 2012 from the Google Scholar service. As mentioned above, two publications, [19] and [45], have been cited an astonishingly 3,712 and 448 times, respectively. In order to make the charts more readable, the data reported in Figs. 15 and 16 does not include these \outliers". Fig. 16. Average citation count by venue type (excluding two highly-cited publications [19, 45]).

26 488 R. Farhoodi et al. Fig. 17. Citation count by venue type (including two highly-cited publications [19, 45]). Figures 17 and 18 correspond to Figs. 15 and 16 with the only di erence that they include the two highly cited publications Citation rates of top venues (RQ 2.6) Figure 19 shows the distribution of citations over the period 1996 to 2011 for those venues with at least two publications from our selection of 130 publications. The ranking of the venues is according to the number of citations per publication. It should be noted that the publications with the highest citation numbers, [19] and [45], are not included in Fig. 19 because each has been published at venue at which no other publication of our selection of 130 publications has been published. Figure 19 indicates that the magazines IEEE Software and BMC Bioinformatics have the highest citation rates per publication and the highest total numbers of citations of all venues with at least two publications. Figure 20 shows the distributions of citations per venue type. Note that again only venues with at least two publications are included in each category and thus, again, the two publications with the highest citations are not included. Based on the data shown in Fig. 20, magazine publications have the highest impact in terms of citations, followed by journal publications. It is interesting to see that workshop papers Fig. 18. Average citation count by venue type (including two highly-cited publications [19, 45]).

27 Development of Scienti c Software 489 Fig. 19. Citation count of selected publications distributed over venues with at least two publications. have slightly higher impacts than conference papers. Note that the upper and lower boundaries of the boxes in the box-plots represent 75% and 25% quartiles, respectively. The line inside the boxes represents the 50% quartile (median), which is di erent from the average citation numbers (means) shown in Fig. 16. Fig. 20. Distribution of citation counts per venue type.

28 490 R. Farhoodi et al. Table 4. Top venues (with at least two papers in the pool). Venue name Acronym Number of papers from this venue in our pool International Workshop on Software Engineering for Computational SECSE 18 Science and Engineering International Workshop on Software Engineering for SEHPCA 17 High Performance Computing Applications IEEE Magazine on Computing in Science and Engineering CISE 9 International Conference on Software Engineering ICSE 9 IEEE Software IEEES 8 ACM Transactions on Mathematical Software TOMS 4 International Conference on Computational Science ICCS 4 BMC Bioinformatics BMCB 2 Working Conference on the Architecture of Scienti c Software CASS 2 Proceedings of the ACM Conference on Computer Supported CSCW 2 Cooperative Work Procedia Computer Science (Conference Proceedings) PCS 2 Scienti c Programming - Complexity in Scalable Computing SPCSC 2 Advances in Engineering Software AES 2 Journal of Computational Science JCS Top venues (RQ 2.7) Table 4 shows the distribution of selected publications over publication venues that attracted at least two publications. The most popular publication venues are the \International Workshop on Software Engineering for Computational Science and Engineering" (SECSE) and the \International Workshop on Software Engineering for High Performance Computing Applications" (SEHPCA). The SECSE workshop is linked to the \International Conference on Software Engineering" (ICSE) which also attracted a substantial number of the selected publications. The most frequently used magazines are \IEEE Computing in Science and Engineering" (CISE) and \IEEE Software" (IEEES) Top authors (RQ 2.8) Figure 21 shows the ranking of researchers who authored (or co-authored) two or more of the 130 selected publications. Diane Kelly, a±liated with the Royal Military College of Canada is the most active author with a total of ten publications [6, 61, 80, 83, 89, 91, 92, 98, 99, 103], followed by Judith Segal of Open University, UK, with seven papers [10, 12, 96, 110, 111, 125, 129]. Richard Kendall and his colleague Douglas E. Post, from the American Los Alamos National Laboratory, with six publications each [20, 81, 93, 94, 112, 114, 150] stand in the 3rd rank. It would be worthwhile to review the major research areas of the top researchers. We have summarized the main research areas of the top three-ranked authors in Table 5. It seems that among the top active areas of the top authors are software process, testing, quality, and project management in the context of scienti c software.

29 Development of Scienti c Software 491 Fig. 21. Most active researchers Author a±liations (RQ 2.9) Figure 22 shows the distribution of authors of the 130 selected papers over type of a±liation. We de ned four categories of a±liations: (1) governmental research groups such as NASA, (2) universities, (3) corporate research groups and laboratories in industry, and (4) collaborative work (joint research published by groups of

30 492 R. Farhoodi et al. Table 5. Main research areas of the top three-ranked authors. Rank Researcher Areas Paper references 1 Diane Kelly Testing [61, 80, 83, 89, 91, 92, 98, 99] Risk analysis [6] Evolution [103] 2 Judith Segal Testing, quality [10, 12, 96, 125] Process, project management [110, 111, 129] Design [10, 12] Requirements [10, 12] 3 (tie) Richard Kendall Process, project management [20, 81, 94, 150] Testing, quality [81, 93, 94] Design [20] Performance [114] 3 (tie) Douglas E. Post (a co-author of Richard Kendall) Same areas as Richard Kendall Process, project management [20, 81, 93, 94, 150] [112] Fig. 22. Author a±liations by type. authors a±liated with two or three di erent categories). The gure indicates that publications where all authors are a±liated with universities are the most frequent in our set of selected publications, followed by publications written by author groups having mixed a±liation categories. Figure 23 shows the distribution of types of author a±liations in the period 1996 to Generally, the proportions of a±liation types vary much from year to year but we couldn't nd any speci c pattern of change. It seems, however, that in the second half of the period, i.e., beginning with year 2004, the portion of publications produced by authors exclusively a±liated with universities is larger than in the rst half.