Evidence for Effectiveness Developing a standards framework for digital health innovations Digitally empowering people to manage their health and care October 2018
The issue NHS England programmes Apps library - DAQs ITP/ITT Digital Diabetes programme Test Beds Innovation Landscape Review Literature review, workshops, interviews Clear requirement for consistent evidence standards Cycle of Pilots Little at scale implementation Emerging trust in digital health 2
The Problem The Goals How to demonstrate the impact of a digital health tool with appropriate evidence. Manufacturers are not clear what effectiveness and economic evidence they need to produce Commissioners are unclear what effectiveness and economic evidence they should be looking for A cycle of pilots and trials in the innovation space with little focus at scale implementation of such tools and value for money to the health and social care system. A trusted and respected source of advice on what evidence to produce; allowing innovators to produce better evidence, faster and more cost-effective, and in turn allow the NHS to commission, deploy and scale clinically and cost-effective digital health tools that meet demand. Related standards on how to produce evidence of effectiveness, economic impact and compliance with other standards. 3
Project Scope Type of digital health tools In Scope Digital health tools that are patient, citizen or clinician facing This will include mobile and web apps, wearables, digital therapeutics, VR/AR technology Apps that enable innovative value propositions and use of medical devices and diagnostics. Out of Scope Health IT systems True AI based digital tools the subject of a targeted initiative Conventional medical software required for the standard operation of a medical device or diagnostic. Type of evidence End goal of the advice Objective of the work Evidence to demonstrate clinical effectiveness Evidence to demonstrate behaviour change Evidence to demonstrate value for money and economic benefit To be commissioned or recommended by the NHS or its related health and care systems (e.g community clinics) and including social care Where applicable, to turn this advice into standards for future assessment to serve as a benchmark reference for innovators and commissioners To make easily accessible advice on how to generate evidence and why it matters Evidence of clinical safety Evidence that support security, data processing, technical stability Overseas market, lifestyle and wellbeing market To design the overall assessment framework for digital tools e.g. who will assess effectiveness and value for money in the system and when To review commissioning frameworks To design or build better infrastructure for 4 digital health research (NIHR)
Project Scope - what the project will and will not deliver Supporting evidence of effectiveness and economic impact Informing the assessment framework and process Evaluating individual tools 5
Project Milestones for developing Economic Standards July 18 Setting up project team and confirming scope of project Identify existing classification economic evaluation frameworks and establishing links with key stakeholders August 18 Developing a shared view of the conceptual approach to the economic standards. Sept 18 Host workshops with commissioners and innovators and develop content for digital tool Sept 18 Internal workshops with key advisors and project team Continue testing framework using and develop case studies that illustrate levels of evidence in the framework Oct - Nov 18 Workshops with wider set of stakeholders, innovators, academics, commissioners, ALBs, other key groups Dec - Jan 19 Development and launch of the support tool and standards Confirm arrangements for longer term maintenance 6
THE WIDER issue: innovation landscape review
A CODE OF CONDUCT principles and guidelines https://www.gov.uk/government/publications/code-of-conduct-for-data-driven-health-and-care-technology Real world feedback loop based on connected informative data sets 1 Idea 2 Standards 3 Evidence 4 Evaluation 5 Payment 6 Adoption Idea generation Does it meet the Standards? Evidence generation to show effectiveness Evaluation of ideas Payment and Commissioning Adoption by end users IDEA Code of Conduct For DHIs and Intelligent Algorithms ADOPTION, REGULATION AND EVIDENCE Digital Transformation Innovation landscape Review, Data led Policy, Value of Data HTA / Regulation / DAQs Commissioning, commercial and uptake WHAT THE CODE OF CONDUCT OFFERS 1. A clear set of principles and behaviours that creates a transparent and accountable environment for the development and deployment of trusted digital health innovations and intelligent algorithms within health and care. 2. Enable industry, academia, commissioners and individuals to easily identify the principles and guidelines of developing safe and effective products for the NHS. 3. Create a trusted set of products and innovations and a novel environment in health and care whereby we learn to base our policies on real world data and evidence thereby creating a unique framework to inform regulation. 8
Initial code of conduct for data-driven technologies Principle 9: Show evidence of effectiveness for the intended use You should provide evidence of how effective your product or innovation is for its intended use. If you are unable to show evidence, you should draw a plan that addresses the minimum required level of evidence given the functions performed by your technology...when building or developing the technology, consider what function the product delivers, and this will guide what sort of evidence generation plan should be put in place. The standards will also help the intended evaluation body review the package of evidence provided to them and help them outline gaps in the information they received from innovators Give feedback: https://www.gov.uk/government/publications/code-of-conduct-for-datadriven-health-and-care-technology 9
Who is involved? Project strategy & oversight Framework & Content Lead Comms & Integration Lead Evaluation Lead Innovation Lead AHSNs Lead Indra Joshi, NHS England Alexia Tonnel, Mark Salmon NICE Neelam Patel, MedCity Felix Greaves PHE Sam Roberts NHS England Axel Heitmeuller ICP, DH.L Executive Project Sponsors: Juliet Bauer, Chief Digital Officer, NHS England Simon Eccles, Chief Clinical Information Officer for Health and Care Gillian Leng Deputy Chief Executive, NICE www.medcityhq.com/evidence-for-effectiveness/ 10
The framework in outline Standards for clinical effectiveness evidence - based on functional classification Standards for economic effectiveness evidence - based on budget impact Educational/signposting resources, Case studies 11
DRAFT Clinical effectiveness standards Functional classification Evidence standards 12
Tier 2 evidence requirements DHTs that provide information, simple monitoring or communication functions only Evidence categories Must have Minimum evidence standard Could have Best practice standard Technical standards The DHT must have passed assessment using the NHS Digital DAQ tool n/a Content Be able to demonstrate that the information contained in the DHT is valid (aligned to best available sources e.g. NICE guidance, relevant professional organisations or reputable patient organisations, and appropriate for the target population), accurate up to date Evidence of a documented information review and update cycle during the lifetime of the DHT Evidence of endorsement, accreditation or recommendation by NICE, NHSE, relevant professional organisation or reputable patient organisation Evidence that the information content has been validated though independent accreditation/certification such as The Information Standard or HONcode certification Clinical engagement in the design process Be able to demonstrate that there was relevant clinical input in the design, development and testing of the DHT Relevance Evidence to demonstrate that the information contained in the DHT is relevant to the UK health and care system Evidence to demonstrate the DHT has been successfully piloted in the UK health and social care system Ongoing data collection n/a Commitment to collect and report to commissioners on usage and outcomes (using non-patient identifiable information) to show ongoing value 13
Economic Impact Standards Concept Budget impact of the DHT* to health and social care sector Economic evidence standards *DHT = Digital Health Tool 14
Conceptual framework Calculate the budget impact of the DHT* Classify the budget impact (England) of the DHT into one of four categories Economic evidence standards described for each budget impact category * Using publicly-available tool which is part of the framework 15
Budget impact categories Budget impact category* Annual net cost for England s Health and Social Care System Cost saving Less than 0 Cost neutral Less than 1 million Low cost Between 1 million and 15 million High cost Greater than 15 million *Categories based on existing NICE resource impact categories for England. Assign budget impact category based on the maximum annual net cost. 16
Economic evidence standards National budget impact category Cost saving Economic evidence standard for individual DHTs Output(s) Cost neutral Budget impact calculation Budget impact figure ( ) Low cost Budget impact and cost consequence analysis Budget impact, costs and consequences (such as health and other related outcomes) in natural units High cost Budget impact and cost effectiveness analysis Budget impact, costs and health outcomes summarised in a single nonmonetary measure e.g. QALY to provide an incremental costeffectiveness ratio (ICER) 17
Early Adoption Feedback, concerns? Ideas on adopting framework into programmes Feedback and Iteration 18