The Scholarly Work of Reliable and Well-Designed Mathematical Software

Transcription

1 1 The Scholarly Work of Reliable and Well-Designed Mathematical Software A personal perspective Tim Davis July 9, 2014

2 2 The Scholarly Work of Reliable and Well-Designed Mathematical Software Outline: What defines a scholarly work? Does mathematical software fit that definition? Why publish prototype-quality mathematical software? Why publish robust, reliable, and full-featured mathematical software? Troubling trends and a troubling status quo My recommendations for you and for SIAM

3 3 Definition of scholarly work Scholarly or artistic works, regardless of their form of expression: intended purpose is to disseminate the results of academic research, scholarly study, or artistic expression,... in any medium. When is a work scholarly (software or any other form)? Proposed metrics: 1 peer-reviewed 2 published in reputable academic journals 3 published in academic-quality books 4 topic of discussion at academic conferences 5 citation for academic society fellow

4 Peer-reviewed software ACM Transactions on Mathematical Software Founded in 1975 Purpose: As a scientific journal, ACM Transactions on Mathematical Software (TOMS) documents the theoretical underpinnings of numeric, symbolic, algebraic, and geometric computing applications. It focuses on analysis and construction of algorithms and programs, and the interaction of programs and architecture. Algorithms documented in TOMS are available as the Collected Algorithms of the ACM in print, on microfiche, on disk, and online. 942 Collected Algorithms of the ACM since 1975: peer-reviewed scholarly software I serve as an Associate Editor of this journal 4

5 5 ACM TOMS Editorial Charter The purpose of the ACM Transactions on Mathematical Software (TOMS) is to communicate important research results addressing the development, evaluation and use of mathematical software. In addition, TOMS publishes machine-readable computer software... In both research papers and software, TOMS seeks contributions of lasting value in which technical quality, relevance to significant computations, interest, and novelty are all high, and where presentation is effective.

6 6 ACM TOMS Algorithms Policy code is submitted, peer-reviewed, and published by the journal short codes are printed in hard copy journal all available electronically code novelty and quality is essential toms.acm.org/algpolicy.html The algorithm paper should give a brief description of what the software does and pertinent information on usage and maintenance. An ACM algorithm must be complete, portable, well documented, and well structured.

7 7 Scholarly software in academic books and journals Long history of academic software. For example: George Forsythe ( ), founder of computer science as an academic discipline consider mathematical software as scholarly work founder and head of Stanford s Computer Science Department President of ACM author of Computer solution of linear algebraic systems, Includes lengthy code printed in the book

8 8 George Forsythe: software as scholarly work Upon George Forsythe s death: It is generally agreed that he, more than any other man, is responsible for the rapid development of computer science in the world s colleges and universities. His foresight, combined with his untiring efforts to spread the gospel of computing, have had a significant and lasting impact; one might almost regard him as the Martin Luther of the Computer Reformation! He inaugurated a new area of scholarly work: refereeing and editing algorithms. (emphasis mine) by Donald Knuth, Communications of the ACM, 1972

9 9 George Forsythe: software as scholarly work Article by Knuth, continued: As an indication of his behind-the-scenes activities, here are some more excerpts from letters he wrote during January and February 1964: The program is really in poor style, and I m peeved with the referee for not saying so. You use a switch and a mass of goto s where straightforward ALGOL would use conditional expressions. You even goto here from the line above here!! [January 8, 1964] I am sorry that refereeing increases the time between submittal and publication, but I am confident that the net result of refereeing will be a large gain in the quality of our algorithms. [January 13, 1964] It is very hard to find matching begins and ends, which should be above each other or on the same line. [January 29, 1964]

10 SIAM Fellows with software contributions Marsha Berger, Courant: PDEs Bill Gear, NEC: ODEs, DAEs Jorge Moré, Argonne: optimization Nick Trefethen, Oxford: sci. computing Alan George, Waterloo: sparse matrices Cleve Moler, MathWorks: MATLAB Linda Petzold, UCSB: DASSL (DAEs) Uri Ascher, UBC: diff.eqns 10

11 SIAM Fellows with software contributions Iain Duff, Rutherford: sparse matrices Rande LeVeque, UWash.: hyperbolic PDEs Yousef Saad, UMinn: sparse matrices Danny Sorensen, Rice: ARPACK Bill Gropp, UIUC: MPI John Lewis, Boeing: sparse matrices Pavel Bochev, Sandia: finite element Sven Hammarling, NAG: LAPACK 11

12 SIAM Fellows with software contributions Bruce Hendrickson, Sandia: graph algo. Barry Smith, Argonne: PETSc Andrew Conn, IBM: optimization T.D. Texas A&M: sparse matrices Michael Saunders, Stanford: optimization Philip Gill, UCSD: optimization 12

13 Apply the metrics: Software as scholarly work Definition of scholarly work: purpose must be to disseminate results of academic research, scholarly study, or artistic expression Characteristics of scholarly work: 1 peer-reviewed ACM TOMS, peer-review of mathematical software 2 published in reputable academic journals ACM TOMS, SIAM Journals (as supplimentary appendices) 3 published in academic-quality books Many SIAM books include mathematical software or describe mathemaical software 4 topic of discussion at academic conferences this session, and many others 5 citation for academic society fellow For contributions to... software,... is not uncommon Thus: peer-reviewed software is a scholarly work 13

14 14 So why publish (prototype) mathematical software? useful to provide the code even as research prototype quality reproducibility of research results for a complex algorithm: not all details can fit in a paper Publish your code - it is good enough, Nick Barnes, Nature 467, 753 (2010). That the code is a little raw is one of the main reasons scientists give for not sharing it with others. Yet, software in all trades is written to be good enough for the job intended. So if your code is good enough to do the job, then it is good enough to release and releasing it will help your research and your field.

15 15 Randy LeVeque s thought-experiment LeVeque s thought-experiment: What if we treated proofs like we now treat code? The proof is too ugly to show anyone. I didn t work out the details. Giving out my proof would give away my secrets. My proof is valuable intellectual property. Proofs are too hard to review.... Top Ten Reasons To Not Share Your Code (and why you should anyway), April 2013, SIAM News

16 16 Randy LeVeque s answers (mine added) The code is ugly to show anyone: ugly code is better than no code. I didn t work out the details: the details you did work out are important to the scientific record. Giving out my code would give away my secrets: but as applied mathematicians we want others to rely on our methods. My code is valuable intellectual property. it s protected by copyright. Code is too hard to review. At ACM TOMS we ve been doing it for decades....

17 17 Troubling Trends Troubing Trends in Scientific Software Use, Science, Vol 340, May A survey of scientists in ecological modelling:... Many of these scientists rely on the fact that the software has appeared in a peer-reviewed article, recommendations, and personal opinion, as their reason for adopting software. This is scientifically misplaced, as the software code used to conduct the science is not formally peer-reviewed.

18 18 Troubling Trends, continued Recommendations, Moving Forward Education: Universities should produce scientists capable of instantiating science in code such that other scientists are able to peer-review code as they would other aspects of science. Formal training in statistics, computational methods, mathematics, and software engineering should be a core part of the science curriculum at undergraduate and research student levels. Scientific publication: Scientific software code needs to be not only published and made available but also peer-reviewed.

19 19 So why publish polished mathematical software? Typical path towards academic success: Create a new algorithm Write a paper Expect others to implement your algorithm An alternative path: Create a new algorithm and the software that implements it Write a paper and submit your code along for peer review Enable others use your algorithm and its software

20 20 How to publish polished mathematical software Code like a theorem/proof: clear, concise, complete, and well-written. Track down the bugs like you track down a flaw in a proof. Test suite: ensure all lines exercised and results verified. Publish the test suite and scaffolding-code Documentation: not an after-thought. Sample result: 3 bugs found in 15 years in 120K lines of code in x=a\b

21 Recommedations For the researcher: Create new algorithms. Embody them in polished code and submit them for peer-review. Clear path towards a career goal of high-impact research. For the industrial researcher: if a company allows you to publish your proof, why not your code? For SIAM Goal: enhance reproducibility of research Goal: enhance future research by fostering mathematical software tools Encourage code submission, even non-peer-reviewed prototype-quality. Allow for peer-review of high-quality software. 21

22 The Scholarly Work of Reliable and Well-Designed Mathematical Software Summary: Scholarly work: Purpose is to disseminate results of academic research. Mathematical software: Essential method for disseminating complex algorithms, can be peer-reviewed. Why publish prototype-quality mathematical software? Research reproducibility. Why publish robust, reliable, and full-featured mathematical software? Let others rely on your work and build on it. Troubling trends and a troubling status quo: Too much software hidden from scientific study. Needs to be included with journal publication (both reviewed and unreviewed). My recommendations for you and for SIAM: Write and publish high-quality peer-reviewed software. 22