Computational Reproducibility in Medical Research:

Transcription

1 Computational Reproducibility in Medical Research: Toward Open Code and Data Victoria Stodden School of Information Sciences University of Illinois at Urbana-Champaign R / Medicine Yale University September 8, 2018

2 Today: Technology is driving a reassessment of transparency 1. Big Data / Data Driven Discovery: high dimensional data, p >> n, 2. Computational Power: simulation of the complete evolution of a physical system, systematically varying parameters, 3. Deep intellectual contributions now encoded only in software. The software contains ideas that enable biology CSHL Keynote; Dr. Lior Pachter, UC Berkeley Stories from the Supplement from the Genome Informatics meeting 11/1/2013

3 The digital age in science Claim 1: Virtually all published discoveries today have a computational component. Claim 2: There is a mismatch between the traditional scientific process and computation, leading to reproducibility concerns.

4 Parsing Reproducibility Empirical Reproducibility Statistical Reproducibility Computational Reproducibility V. Stodden, IMS Bulletin (2013)

5 Empirical Reproducibility

6 Statistical Reproducibility False discovery, p-hacking (Simonsohn 2012), file drawer problem, overuse and mis-use of p-values, lack of multiple testing adjustments. Low power, poor experimental design, nonrandom sampling, Data preparation, treatment of outliers, re-combination of datasets, insufficient reporting/tracking practices, inappropriate tests or models, model misspecification, Model robustness to parameter changes and data perturbations,

7 Statistical Reproducibility In January 2014 Science enacted new manuscript submission requirements: a data-handling plan i.e. how outliers will be dealt with, sample size estimation for effect size, whether samples are treated randomly, whether experimenter blind to the conduct of the experiment. Also added statisticians to the Board of Reviewing Editors.

8 Computational Reproducibility An article about computational science in a scientific publication is not the scholarship itself, it is merely advertising of the scholarship. The actual scholarship is the complete... set of instructions [and data] which generated the figures. David Donoho,

9 INSIGHTS POLICY FORUM REPRODUCIBILITY Enhancing reproducibility for computational methods Data, code, and workflows should be available and cited By Victoria Stodden, 1 Marcia McNutt, 2 David H. Bailey, 3 Ewa Deelman, 4 Yolanda Gil, 4 Brooks Hanson, 5 Michael A. Heroux, 6 John P.A. Ioannidis, 7 Michela Taufer 8 Over the past two decades, computational methods have radically changed the ability of researchers from all areas of scholarship to process and analyze data and to simulate complex systems. But with these advances come challenges that are contributing to broader concerns over irreproducibility in the scholarly literature, among them the lack of transparency in disclosure of computational methods. Current reporting methods are often uneven, incomplete, and still evolving. We present a novel set of Reproducibility Enhancement Principles (REP) targeting disclosure challenges involving computation. These recommendations, which build upon more general proposals from the Transparency and Openness Promotion (TOP) guidelines (1) and recommendations for field data (2), emerged from workshop discussions among funding agencies, publishers and journal editors, industry participants, and researchers repreto understanding how computational results were derived and to reconciling any differences that might arise between independent replications (4). We thus focus on the ability to rerun the same computational steps on the same data the original authors used as a minimum dissemination standard (5, 6), which includes workflow information that explains what raw data and intermediate results are input to which computations (7). Access to the data and code that underlie discoveries can also enable downstream scientific contributions, such as meta-analyses, reuse, and other efforts that include results from multiple studies. RECOMMENDATIONS Share data, software, workflows, and details of the computational environment that generate published findings in open trusted repositories. The minimal components that enable independent regeneration of computational results are the data, the computational steps that produced the findings, and the workflow describing how to generate the results using the data and code, including parameter settings, random number seeds, make files, or Sufficient metadata should be provided for someone in the field to use the shared digital scholarly objects without resorting to contacting the original authors (i.e., bit.ly/2fvwjph). Software metadata should include, at a minimum, the title, authors, version, language, license, Uniform Resource Identifier/DOI, software description (including purpose, inputs, outputs, dependencies), and execution requirements. To enable credit for shared digital scholarly objects, citation should be standard practice. All data, code, and workflows, including software written by the authors, should be cited in the references section (10). We suggest that software citation include software version information and its unique identifier in addi- Access to the computational steps taken to process data and generate findings is as important as access to data themselves. Stodden, Victoria, et al. Enhancing reproducibility for computational methods. Science 354(6317) (2016)

10 7: Funding agencies should instigate new research programs and pilot studies. Reproducibility Enhancement Principles 1: To facilitate reproducibility, share the data, software, workflows, and details of the computational environment in open repositories. 2: To enable discoverability, persistent links should appear in the published article and include a permanent identifier for data, code, and digital artifacts upon which the results depend. 3: To enable credit for shared digital scholarly objects, citation should be standard practice. 4: To facilitate reuse, adequately document digital scholarly artifacts. 5: Journals should conduct a Reproducibility Check as part of the publication process and enact the TOP Standards at level 2 or 3. 6: Use Open Licensing when publishing digital scholarly objects.

11

12 The LifeCycle of Data Science as a Framework

13 Lifecycle of Data Berman et al., Realizing the Potential of Data Science, CACM, April 2018

14 Lifecycle of Data Science Framework to incorporate data science contributions from different fields, Explicit emphasis on re-use and reproducibility, Explicit emphasis on computational tools (e.g. Kubernetes), hardware (e.g. Google Edge TPUs) and software (e.g. Jupyter Notebooks) Surfaces ethics (human subjects, privacy), social context (interpretations of bias ), scholarly communication and reproducible research.

15 Lifecycle of Data Science: An Abstraction the study of data science ethics, documentation and metadata creation, best practices, policy; the science of data science application level experimental design data generation and collection data exploration and hypothesis generation data cleaning and organization feature selection and data preparation model building and statistical inference simulation and cross-validation visualization publication and artifact preservation / archiving infrastructure level notebooks and workflow software database structures workflow software and preregistration tools data management tools notebooks, workflow software; containerization tools notebooks, inference languages notebooks notebooks, visualization software workflow software, artifact linking tools system level hardware, cloud computing infrastructure, systems and system management, data structures, storage

16 Progress on computational reproducibility is enabled through coordination by a variety of stakeholders. Scientific Societies Funders (policy) Publishers (TOP guidelines) Regulatory Bodies (OSTP Memos) Researchers (processes) The Public/Press Universities/institutions (hiring/promotion) Universities/libraries (empowering w/tools, support)

17

18 Does artifact access on demand work? February 11, 2011: All data necessary to understand, assess, and extend the conclusions of the manuscript must be available to any reader of Science. All computer codes involved in the creation or analysis of data must also be available to any reader of Science. After publication, all reasonable requests for data and materials must be fulfilled... Survey of publications in Science Magazine from Feb 11, 2011 to June 29, 2012 inclusive. Obtained a random sample of 204 scientific articles with computational findings. Asked for the data and code! Stodden et al., Journal Policy for Computational Reproducibility, PNAS, March 2018

19 Response Count % of Total No response bounced Impossible to share Refusal to share Contact to another person Asks for reasons Unfulfilled promise to follow up Direct back to SOM Shared data and code % 2% 2% 7% 11% 11% 3% 3% 36% Total % 24 articles provided direct access to code/data.

20 Replicating Computational Findings We deemed 56 of the 89 articles for which we had data and code potentially reproducible We chose a random sample of 22 from these 56 to replicate

21 Computational Replication Rates We were able to obtain data and code from the authors of 89 articles in our sample of 204, overall artifact recovery rate estimate: 44% with 95% confidence interval [0.36, 0.50] Of the 56 potentially reproducible articles, we randomly choose 22 to attempt replication, and all but one provided enough information that we were able to reproduce their computational findings. overall computational reproducibility estimate: 26% with 95% confidence interval [0.20, 0.32]

22

23

24