HTT project: High Throughput Truthing. FY2019 Critical Path Proposal PI: Brandon D. Gallas

Transcription

1 HTT project: High Throughput Truthing FY2019 Critical Path Proposal PI: Brandon D. Gallas

2 Full Project Title High throughput truthing of microscope slides to validate artificial intelligence algorithms analyzing digital scans of same slides: data (images + annotations) as an FDA qualified medical device development tool (MDDT) Aim/Deliverable: Qualify an MDDT dataset Available to any algorithm developer Use to evaluate algorithm performance Use in submission to FDA 2

3 Why is the FDA doing this? help encourage more developers to translate advances into clinically actionable tools to benefit patients Scott Gottlieb, Commissioner FDA, Transforming FDA s Approach to Digital Health. Have you heard of Artificial Intelligence? Deep Learning/Machine Learning? Big Data/Data Mining? [Insert Buzz Word Here] Algorithms are here. More are coming. Huge investments being made Work appearing in scientific journals and popular press Big and small companies Computer Science Departments and Hospitals Submissions of algorithms in radiology have increased dramatically Expecting submissions in pathology More tasks = more algorithms (new frontier) Opportunity to prepare new pathways and new mechanisms for new stakeholders Many without regulatory experience Many without huge budgets Many without access to patient images and pathologists 3

4 Why is the FDA doing this? help encourage more developers to translate advances into clinically actionable tools to benefit patients Scott Gottlieb, Commissioner FDA, Transforming FDA s Approach to Digital Health. MDDT Medical Device Development Tool Impact New pathway. New mechanism. Reduce burden to sponsors Use MDDT data in the submission Replace 40 pages of a submission with, Using the MDDT dataset, our algorithm performance is Reduce burden to FDA Approve data once to support multiple sponsors New stakeholders. Not just industry Pathologists, Academia, Health Providers Associations, Societies, Colleges (CAP, USCAP, ASCP) Create an example for stakeholders to follow. Build consensus. Build tools. Disseminate. High throughput data collection tools and protocols Standardize annotation formats for humans and algorithms Statistical methods and software for algorithm performance evaluation Improve submissions. Enable interoperability. Give pathologists ownership and confidence. What the algorithms should do The validation process Improve clinical prac ce Shift oversight to community. Reduce FDA workload 4

5 Biomarker to be investigated TILs in breast cancer: Tumor infiltrating lymphocytes Prognostic for survival Key collaborator Roberto Salgado, MD Chair of international working group 140+ pathologists Publishing on clinical validity of prognostic performance and clinical use 5

6 Project Plan What needs to get done Develop algorithm use case and datacollection protocol Source slides: IRBs and agreements Prepare infrastructure Prepare data collection tools Coordinate and execute data collection events Develop statistical analysis methods and tools Propose data as MDDT to FDA/CDRH MDDT program We have experts and experience working group Pathologists and departments Program directors and project leads The Cancer Imaging Archive QuIP/caMicroscope eedap Clinical trial and algorithm experts Statisticians from FDA and collaborators FDA/CDRH/OSEL MDDT program office lead Laura Coombs Senior Director of the American College of Radiology (ACR) Data Science Institute (DSI) 6

7 High throughput Truthing Go where thousands of pathologists go Annual Society Meetings Data collection event (Demonstration) Baltimore Local destination. Low cost. TILs: Tumor Infiltrating Lymphocytes Pathologists evaluated regions of interest: Stromal TILs density task TIL marking task Microscope and digital modes 26 pathologists; 13 collection hours total. 7

8 Planned data collection events March 16, 2019 USCAP Annual Meeting National Harbor, MD June 2019 Data collection event at FDA Silver Spring, MD July 2 5, 2019 Digital pathology workshop Leeds, UK September 7, 2019 European Congress of Pathology Nice, France September 11, 2019 ASCP Annual Meeting Phoenix, AZ Tentatively supported (in part) by Stony Brook University (Joel Saltz) No cost to FDA Tentatively supported by Northern Pathology Imaging Cooperative (Darren Treanor) Tentatively supported by ECP Digital Pathology Workshop (Jeroen van der Laak) Funding not yet secured 8

9 Planned data collection events March 16, 2019 USCAP Annual Meeting National Harbor, MD June 2019 Data collection event at FDA Silver Spring, MD July 2 5, 2019 Digital pathology workshop Leeds, UK September 7, 2019 European Congress of Pathology Nice, France September 11, 2019 ASCP Annual Meeting Phoenix, AZ Tentatively supported (in part) by Stony Brook University (Joel Saltz) No cost to FDA Crowdsource pathologists through the web! Compliment data collection Tentatively on the supported microscope. by Northern Pathology Imaging Cooperative (Darren Treanor) Compare data across modalities. Tentatively supported by ECP Digital Pathology Workshop (Jeroen van der Laak) Funding not yet secured 9

10 Training Readers, CME We plan to include training with data collection events Recruitment incentive Training expected for clinical use Reduce reader variability Training materials exist already ( Offer feedback to participants? 11/30/

11 Planned data collection events: Logistics Collaborators need/want a CRADA Cooperative Research and Development Agreement Establish ground rules for materials and data sharing Allow exchange of money to support activities Requires IRB Started discussion with Research in Human Subjects Committee at FDA Started discussion with EEP Follow MDIC consortium model for CRADA? 11

12 Key Technologies TCIA: The Cancer Imaging Archive Image repository that allows management and controlled dissemination of the images Allows humans to interact with the images Allows algorithms to interact with the images (cloud computing) Program managers and project leads are collaborators camicroscope: Web Based Annotation and Visualization Platform for Digitized Whole Slide Images (WSI s) Hosted on TCIA Developers are collaborators eedap: evaluation environment for digital and analog pathology Allow pathologists to use the microscope Register glass slide and WSI: evaluate same locations on WSI or microscope Developers are FDA scientists 12

13 camicroscope: Web Based Annotation and Visualization Platform for (WSI s) Study Preparation (webbased): 1 3 pathologists Specify regions of interest (ROIs) Evaluate density of TILs Mark individual TILs (in subrois) Also involve algorithms Targeted sampling of ROIs: Spectrum of TIL densities Study Execution eedap microscope mode camicroscope digital mode 13

14 eedap: Evaluation Environment for Digital and Analog Pathology Monitor, Computer, motorized stage with joystick, microscope with mounted camera, reticle in eyepiece Microscope Mode Register glass slide and WSI Camera image of glass slide WSI Patch Allow pathologists to create annotations using microscope scanner agnostic annotations. Then map annotations to *any* WSI 14

15 Collaborators: Pathologists Roberto Salgado: TILs in Breast Cancer Working Group > 140 pathologists Darren Treanor: Leeds Teaching Hospitals, NHS > 45 pathologists 70 in region Yukako Yagi: Memorial Sloan Kettering > 100 pathologists Joel Saltz & Raj Gupta: Stony Brook University > 40 pathologists John Tomaszewski: University of Buffalo, Jacobs School of Medicine and Biomedical Sciences > 25 pathologists plus residents and fellows Jim Crawford: Hofstra/Northwell > 100 pathologists Provided letters of support Pathologists needed to: develop TILs clinical use case develop datacollection protocol prepare study sets recruit study participants BDG invited to working group meeting Dec. 6 Engage that group 15

16 Collaborators: Slides Sebastian Klein: Else Kroner Forschungskolleg Cologne Everyone provided letters of support Anant Madabhushi: Case Western Reserve University Joel Saltz and Raj Gupta: Stony Brook University FDA IRB under preparation Darren Treanor: Leeds Teaching Hospitals, NHS 16

17 Collaborators: Infrastructure Ashish Sharma (Emory) and Joel Saltz (Stony Brook University) Project leads of camicroscope, installed on The Cancer Imaging Archive (TCIA) Supported by NCI Center for Biomedical Informatics & Information Technology (CBIIT) Supported by NCI Informatics Technology for Cancer Research (ITCR) John Freymann: NCI Cancer Imaging Program Technical manager of The Cancer Imaging Archive Everyone provided letters of support Collaborators are offering: a home for the digital images and annotations (during and after data collection) Support coding datacollection workflows 17

18 Collaborators: Algorithm Support Anant Madabushi: Case Western Reserve University Corredor2018_Clin Cancer Res_accepted: Spatial architecture and arrangement of tumor infiltrating lymphocytes for predicting likelihood of recurrence in early stage nonsmall cell lung cancer. Project lead of HistoQC, open source quality control tool for digital pathology slides Joel Saltz: Stony Brook University Saltz2018_Cell Rep_v23p181: Spatial Organization and Molecular Correlation of Tumor Infiltrating Lymphocytes Using Deep Learning on Pathology Images Jeroen van der Laak, Radboud University Collaborators have published on algorithms for evaluating TILs Existing algorithms will help in the study design 18