Phase 1 US Compliance Report

Transcription

1 Implementation of Regulatory Information Submission Standards (IRISS) ectd Tool Interoperability Group (ETIG) ectd Tool Interoperability and Compliance Study 3 (ETICS 3) ETICS 15 April 2011

2 Implementation of Regulatory Information Submission Standards (IRISS) ectd Tool Interoperability Group (ETIG) ectd Tool Interoperability and Compliance Study 3 (ETICS 3) Table of Contents Disclaimer... 3 About IRISS... 3 ETICS Background... 4 Introduction... 5 Sample Submission Methodology... 7 US ectd Severity High Validation Violation Submissions... 7 US ectd Severity Medium Validation Violation Submissions... 9 US ectd Severity Low Validation Violation Submissions Non-validation Error Submissions ICH Q36 Validation Violation Submissions Results Contact details List of Figures Figure 1: US Severity Level High Vendor Scores Viewer/Validators Figure 2: US Severity Level Medium Vendor Scores Viewer/Validators Figure 3: US Severity Level Low Vendor Scores Viewer/Validators Figure 4: US Non-validation Error Criteria Scores Viewer/Validators Figure 5: US Severity Level High Vendor Scores Viewers Figure 6: US Severity Level Medium Vendor Scores Viewers Figure 7: US Severity Level Low Vendor Scores Viewers Figure 8: US Non-validation Error Criteria Scores Viewers Figure 9: Vendor Tool Comparison of US Severity High Failure Percentage Figure 10: Vendor Tool Comparison of US Severity Medium Failure Percentage Figure 11: Vendor Tool Comparison of US Severity Low Failure Percentage Figure 12: Vendor Tool Comparison of US Non-validation Error Criteria Failure Percentage Copyright 2011 IRISS Forum Page 1

3 Implementation of Regulatory Information Submission Standards (IRISS) ectd Tool Interoperability Group (ETIG) ectd Tool Interoperability and Compliance Study 3 (ETICS 3) List of Tables Table 1: US Validation Criteria v1.0 points not tested... 5 Table 2: Validation criteria categories... 5 Table 3: US ectd Severity High Validation Violation Submissions... 7 Table 4: US ectd Severity Medium Validation Violation Submissions... 9 Table 5: US ectd Severity Low Validation Violation Submissions Table 6: Non-validation Error Submissions Table 7: ICH Q36 Validation Violation Submissions Table 8: Scoring Table Table 9: US Severity Level High Vendor Scores Viewer/Validators Table 10: US Severity Level Medium Vendor Scores Viewer/Validators Table 11: US Severity Level Low Vendor Scores Viewer/Validators Table 12: US Non-validation Error Criteria Scores Viewer/Validators Table 13: US Severity Level High Vendor Scores Viewers Table 14: US Severity Level Medium Vendor Scores Viewers Table 15: US Severity Level Low Vendor Scores Viewers Table 16: US Non-validation Error Criteria Scores Viewers Table 17: Analysis by US Validation Category Copyright 2011 IRISS Forum Page 2

4 Implementation of Regulatory Information Submission Standards (IRISS) ectd Tool Interoperability Group (ETIG) ectd Tool Interoperability and Compliance Study 3 (ETICS 3) Disclaimer Tests were independently designed & conducted collaboratively with a diverse group of community stakeholders from agencies, vendors and industry against available validation criteria. The study design is without regard to currently available software tools or known functionality that may, intentionally or unintentionally cause bias to a single tool. The analyses of results were conducted and the reports written by volunteers from the pharmaceutical industry and CROs. ETICS III has an expanded scope vs. ETICS I/II and while we have compared analyses, you should be aware that there are new tests and new tools being tested. These comparisons were completed to allow an understanding of any trends in interpretation and adherence to available validation criteria over time. Tool vendors who participated should not use their comparative results in promotional materials or public statements. It is acceptable to inform your customers or make public statements of support for the ETICS project or to indicate that your company participated. The IRISS ETIG ETICS III project is not a contest. We appreciate your support and participation as we continue to improve and expand on the ideals that created ETICS. We further hope you find these results valuable to your organization for internal discussions on interpretation of validation criteria and planning of software releases that provide value to your customers. We have released the full set of data in the event you find it useful to benchmark your own product, and appreciate your consumption of these results exclusively for internal use only, in the spirit of which ETICS III is intended and out of respect for the continuation of future efforts serving the ectd community. About IRISS Implementation of Regulatory Information Submission Standards (IRISS) is a global, non-profit professional organization dedicated to the implementation and widespread, successful adoption of electronic regulatory submission standards around the world. IRISS aims to accelerate the adoption of electronic submission standards by making experience and expertise available to its membership through an operating model commonly known as crowdsourcing. Issues are broadcast to a wide community of domain experts and novice users Solutions and points to consider are offered through the membership community Novice users benefit from lessons of experienced early adopters Development organizations can benefit from listening to trends and issues in the community of users All member organizations benefit from a wider pool of talent than exists within their own organization Copyright 2011 IRISS Forum Page 3

5 Implementation of Regulatory Information Submission Standards (IRISS) ectd Tool Interoperability Group (ETIG) ectd Tool Interoperability and Compliance Study 3 (ETICS 3) IRISS-forum operates as an independent, neutral body free from regional or vendor solution bias where experts and novices gather, regardless of organizational affiliation to discuss, clarify and address pressing issues relevant to the successful adoption of a paperless regulatory environment for submissions. ETICS Background ETICS III is a project of the IRISS ectd Tools Interoperability Group (ETIG). ETIG was launched in October The group s main purpose is to discuss ectd building and viewing tools and the interoperability aspects between them. Membership is comprised of global regulators, vendors and industry affiliates who have an interest in making the ectd a more Common format. The IRISS ectd Tool Interoperability group is a forum for discussion and resolution of ectd interoperability issues. Discussion amongst the membership is based on ideas and concepts and is not tool or vendor specific. The group s Concept Paper provides further details. Increasingly there is need to exchange ectds between tools and hence interoperability remains important. Interoperability is achieved when the ectd message created by one tool and received by another tool meets the following criteria: EITHER the ectd is loadable / readable without significant errors OR the receiving tool correctly identifies and describes impact of errors but the ectd is still transferred with information integrity intact. Consistency of submission building, the adherence to ICH and EU specifications and the ability to detect non-compliance with these specifications are all important factors. A significant error is one which indicates; a loss of information integrity; failure to follow ICH ectd Q&A Question 36 rules (compliance); indicates failure to follow the US ectd Validation Criteria (March, 2008/version 1.0). To judge the significance of an error, issue or information, generated by an ectd validation tool, the error message itself must be clear and detailed. Copyright 2011 IRISS Forum Page 4

6 Implementation of Regulatory Information Submission Standards (IRISS) ectd Tool Interoperability Group (ETIG) ectd Tool Interoperability and Compliance Study 3 (ETICS 3) Introduction The ETICS 3 Phase 1 project (launched February 2010) has built on the previous projects to incorporate new validation rules to help vendors to identify areas where compliance can be further improved. 81 out of 83 US ectd Validation Criteria (March, 2008/version 1.0) were tested. Table 1 lists the validation criteria which were not incorporated. Table 1: US Validation Criteria v1.0 points not tested NO. GROUP SECTION VALIDATION CRITERION SOURCE SEVERITY 1187 File Checks File References Modified leaf element path references a sequence that has not been validated FDA Criteria v1.0 Medium ETICS 3 Phase 1 tested not only compliance against the ICH ectd Q&A 36 rules (Oct, 2009/version 1.18) and the US ectd Validation Criteria (March, 2008/version 1.0); but also included nonvalidation issues 1. The sample submissions were created to test each participating tool s ability to detect these error conditions. Participating tool vendors assessed these submissions and reported their findings. The ETICS team provided 118 dossiers, each with three sequences where the US and ICH validation criteria were challenged. You assessed these submissions and reported your findings. Table 2 displays the validation criteria and the number of submissions which challenge them. Table 2: Validation criteria categories Validation Criteria and US severity Number of Submissions US severity High failure 19 US severity Medium failure 48 US severity Low failure 27 Non-validation Criteria Error 24 1 Known issue which may be under discussion for potential change requests/clarification Copyright 2011 IRISS Forum Page 5

7 Implementation of Regulatory Information Submission Standards (IRISS) ectd Tool Interoperability Group (ETIG) ectd Tool Interoperability and Compliance Study 3 (ETICS 3) Submission US-2840 is considered by the ETICS team to be compliant with current ectd specifications and therefore should not have reported any warnings or errors. The following tables (Table 3, Table 4, Table 5, and Table 6 and Table 7) identify the ETICS 3 test cases. To allow you to compare the results while ensuring confidentiality the name of the vendor tool has been replaced with a letter. The tabulated results for each of the sample submissions are presented in Table 9, Table 10, Table 11, Table 12, Table 13, Table 14, Table 15, and Table 16. These results are also displayed as charts in Figure 1, Figure 3 Figure 3, Figure 4, Figure 5, Figure 6, Figure 6, Figure 7, and Figure 8. These results were further analysed by the US validation categories (Table 17, Figure 9, Figure 10, Figure 11 and Figure 11). Copyright 2011 IRISS Forum Page 6

8 Sample Submission Methodology Copyright 2011 IRISS Forum Page 7 To aid review we have organised the submissions by US Validation Priority, Non-validation issues and then ICH ectd Q&A Questions 36 Validation Criteria. US ectd Severity High Validation Violation Submissions HIGH: The error is a serious technical error which prevents the processing of the submission and will require resubmission. The submission is considered not received by FDA. Table 3: US ectd Severity High Validation Violation Submissions Submission US ICH Group Section Severity Validation Violation US M1 Application High Application Number can only contain numeric characters" Number Description="as integer datatype. The element 'application-number' US M1 Application Number US M1 Application Number US M1 Application Number US STF STF Cumulative Approach High High High High with value Application Number does not match the folder name Application Number must contain 6 characters" Description="violates pattern constraint of '[0-9][0-9][0-9] [0-9] [0-9] [0-9]'.The element 'application-number' with value Application Number must remain constant Cumulative STF files are not allowed US General XML Validation High Invalid XML file US M1 Related Sequence High Related Sequence Number must contain 4 characters" Description="violates pattern constraint of '[0-9][0-9][0-9] [0-9]'.The element 'related-sequence-number' with value

9 Submission US ICH Group Section Severity Validation Violation US M1 Related Sequence High Related submission Number (ed. Sequence number) can only contain numeric characters" Description="as integer datatype. The element 'related-sequence-number' with value US M1 Submission Number High Sequence Number does not match the folder name US M1 Submission Number US STF STF File References US M1 Submission Number US M1 Submission Number High High High High Sequence Number is not unique STF cannot reference another STF Submission Number (ed. Sequence number) can only contain numeric characters" Description="as integer datatype. Submission Number (ed. Sequence number) must contain 4 characters" Description="violates pattern constraint of ' [0-9] [0-9] [0-9] [0-9]'.The element 'sequence-number' with value US General DTD Version High Unsupported DTD version in backbone XML file US General DTD Version High Unsupported DTD version in regional XML file US STF STF DTD Version High Unsupported STF DTD version in leaf element US STF STF DTD Version High Unsupported STF DTD version in STF XML file US General XML Validation High XML file error US General XML Validation High Invalid XML file Copyright 2011 IRISS Forum Page 8

10 US ectd Severity Medium Validation Violation Submissions MEDIUM: The error may impact the reviewability of the submission but cannot be determined without further inspection by the review staff. The submission might be considered received by FDA. Table 4: US ectd Severity Medium Validation Violation Submissions Submission US ICH Group Section Severity Validation Violation US M1 Related Sequence Medium Circular relationship found in related sequences US STF STF File Rules Medium Document ID for STF must remain constant US File Checks File References Medium Extraneous leaf element path US File Checks File References Medium Extraneous modified leaf element path US File Checks File References Medium Leaf element path contains invalid characters: backslash(\), colon(:), asterisk(*), question mark(?), less than(<), greater than(>), pipe( ) US File Checks File References Medium Leaf element path contains invalid characters: backslash(\), colon(:), asterisk(*), question mark(?), less than(<), greater than(>), pipe( ) US File Checks File Rules Medium Invalid file extension US File Checks File Operation Medium Invalid leaf element operation US STF STF File Rules Medium Invalid STF file extension US STF STF File Rules Medium Invalid STF file name US File Checks File References Medium Leaf element path contains invalid characters: backslash(\), colon(:), asterisk(*), question mark(?), less than(<), greater than(>), pipe( ) US File Checks File References Medium Leaf element path contains invalid characters: backslash(\), colon(:), asterisk(*), question mark(?), less than(<), greater than(>), pipe( ) US File Checks File Operation Medium Leaf element already replaced or deleted Copyright 2011 IRISS Forum Page 9

11 Submission US ICH Group Section Severity Validation Violation US File Checks File References Medium Leaf element path contains invalid characters: backslash(\), colon(:), asterisk(*), question mark(?), less than(<), greater than(>), pipe( ) US STF STF File Rules Medium Multiple STF doc-content file tags US File Checks Orphaned File Medium No leaf element for file US General Node Extension Medium Node-extension does not have a title US General Node Extension Medium Node-extension exists US File Checks Orphaned File Medium Non-Required File Exists US General Attribute Values (Repeat Groups) Medium Omitted attribute value US General DTD Version Medium Omitted DTD version in backbone XML file US General DTD Version Medium Omitted DTD version in regional XML file US File Checks File References Medium Omitted leaf element modified file path US File Checks File References Medium Omitted leaf element path US File Checks File Rules Medium Omitted leaf element title US STF STF File Rules Medium Omitted STF doc-content file tag US STF STF DTD Version Medium Omitted STF DTD version in leaf element US STF STF DTD Version Medium Omitted STF DTD version in STF XML file US STF STF XML Validation Medium Preceding the element file-tag with the element property is recommended" Description="Element 'file-tag' is unexpected according to content model of parent element 'doc-content. Expecting: property. US STF STF File References Medium STF does not relate to any leaf elements US STF STF DTD Version Medium STF DTD version must match the XML file US STF STF File Medium STF relates to a non-existent leaf element References Copyright 2011 IRISS Forum Page 10

12 Submission US ICH Group Section Severity Validation Violation US General Attribute Values Medium The attribute appears multiple times (Repeat Groups) US File Checks File References Medium The leaf referenced in the modify-file element cannot be found. US STF STF XML Validation Medium The maximum occurrences of the category element is three" Description="Element 'category' is unexpected according to content model of parent element 'document identifier' US M1 Related Sequence Medium This submission type cannot have a child submission US M1 Related Sequence Medium This submission type cannot have a related sequence US M1 Related Sequence Medium This submission type requires a related sequence US STF STF XML Validation US STF STF XML Validation US STF STF XML Validation Medium Medium Medium Use of content-block is not recommended" Description="Element 'content-block' is unexpected according to content model of parent element 'document'. Use of the element title is not recommended" Description="Element 'title' is unexpected according to content model of parent element 'doc-content' Use of the property element is required. The xlink:href attribute of the doc-content element is not used" Description="The attribute '{ on the element 'doc-content' is not defined in the DTD/Schema. US File Checks File References Medium Leaf element path contains invalid characters: backslash(\), colon(:), asterisk(*), question mark(?), less than(<), greater than(>), pipe( ) US File Checks File References Medium Leaf element path contains invalid characters: backslash(\), colon(:), asterisk(*), question mark(?), less than(<), greater than(>), pipe( ) US ,6,8 File Checks File References Medium You have referenced a file in the hlink:xref attribute of a leaf and the system is unable to locate the file. Copyright 2011 IRISS Forum Page 11

13 Submission US ICH Group Section Severity Validation Violation US File Checks Orphaned Leaf Medium You have referenced the file(s) listed in the validation report from the us-regional.xml or index.xml files without providing the actual file(s). US STF STF XML Validation Medium You have submitted an stf-xxxxxxxx.xml file. When the STF XML file fails to load, having no XML elements in it, or no document-identifier (or study-identifier) XML element error 1969 will be present. US STF STF File Rules Medium Your leaf that points to the STF is in under an invalid heading area or invalid module. For example, STF can only be associated with certain headings under Modules 4 or 5. US STF STF XML Validation Medium You have submitted an stf-xxxxxxxx.xml file. When the STF XML file fails to load, having no XML elements in it, or no document-identifier (or study-identifier) XML element error 1969 will be present. Copyright 2011 IRISS Forum Page 12

14 US ectd Severity Low Validation Violation Submissions LOW: The error is a technical error which may or may not impact the reviewability or the integrity of the submission. The submission will likely be considered received by FDA. Table 5: US ectd Severity Low Validation Violation Submissions Submission US ICH Group Section Severity Validation Violation US File Checks File Rules Low Leaf element title contains an invalid character per ectd Specification US General Attribute Values (Repeat Groups) Low Attribute value contains an invalid character per ectd Specification US General Attribute Values (Repeat Groups) Low Attribute value leads or ends with a space per ectd Specification US General Checksum Low Checksum value must match the actual checksum US M1 Company Name Low Company name must remain constant US General Attribute Values (Repeat Groups) Low Extraneous attribute US File Checks File Rules Low File name contains invalid characters per ectd Specification US File Checks File Rules Low File name exceeds maximum length (64 characters) per ectd Specification US File Checks File Rules Low File size exceeds maximum (100 MB) per ectd Specification US General Checksum Low Invalid checksum format US General Checksum Low Invalid checksum type US M1 Submission Date Low Invalid date format per ectd Specification US File Checks File References Low Leaf element path length exceeds maximum (230characters) US File Checks File Rules Low Leaf element title contains an invalid character per ectd Specification Copyright 2011 IRISS Forum Page 13

15 Submission US ICH Group Section Severity Validation Violation US File Checks File Rules Low Leaf element title leads or ends with a space per ectd Specification US File Checks File References Low More than one version of the required file exists US M1 Related Sequence Low Multiple related sequences are not allowed US General Node Extension Low Node-extension title contains an invalid character US General Node Extension Low Node-extension title leads or ends with a space US General Checksum Low Omitted checksum value US M1 Submission Date Low Omitted date per ectd Specification US STF STF File Rules Low STF doc-content title and leaf element title do not match US General Attribute Values (Repeat Groups) Low Attribute value leads or ends with a space per ectd Specification US General Attribute Values (Repeat Groups) Low Attribute value contains an invalid character per ectd Specification US M1 Submission Date Low Your date or date format is invalid US General Node Extension Low Node-extension title contains an invalid character US STF STF File Rules Low Your STF doc-content title and leaf element title do not match; viewer will show the title of the leaf only. Copyright 2011 IRISS Forum Page 14

16 Non-validation Error Submissions These submissions are technically valid but contain know issues which can lead tools to: EITHER behave inconsistently, AND/OR report errors where there should not be errors, AND/OR display the content of the ectd incorrectly, AND/OR report errors which the ETICS group think are not errors. Table 6 discusses these issues and lists what the ETICS group expected outcome is. Table 6: Non-validation Error Submissions Submission US-2811 US-2818 US-2823 US-2839 US-2841 US-2842 US-2843 US-2844 Validation Violation Correct a study report location FDA SMART FORMS Invalid characters M4-5 Datasets - Datasets are not always submitted in a datasets folder as defined by guidance (m4/datasets, m5/datasets). This causes issues when migrating sequences between tools. Module 4 All study reports in Module 4 have attributes for "species, route of administration, duration" Alternately, no attributes provided where they are required for Module 4 toxicology reports. Module 4 Multiple values for STF attribute species. Module stf.xml appends first submitted stf.xml, not the most recently submitted stf.xml. This results in different displays in different viewing tools. Module 5 Case report forms located either in m537-crf-ipl or in the study folder. For sponsors submitting globally, it is more efficient to remove CRFs from one single folder than from each individual study folder, so we do not want viewing/validation errors for either option. Copyright 2011 IRISS Forum Page 15

17 Submission Validation Violation US-2870 Special Characters: e.g. copyright US-2871 SPL: check xml file / images files / proper naming. Tested with a zip file in the SPL folder. US-2877 STF Scenarios (e.g. same study in 2 different sections). US-2878 STF title changes with lifecycle. US-2884 The title of the leaf which points to the STF should match the study title. US-2909 M4-5 Datasets - Datasets are not always submitted in a datasets folder as defined by guidance (m4/datasets, m5/datasets). This causes issues when migrating sequences between tools. US-2910 US-2911 US-2912 US-2913 US-2914 US-2915 US-2916 US-2917 US-2918 Cross-application linking (leaf cross-reference or hyperlinks). Related sequence is expected but not present. Business example is an IND where an amendment does not have related sequence referencing back to the original-application. M1 Folder structure is variable (sometimes a flat hierarchy, sometimes many subfolders). This causes issues when migrating sequences between tools. M1 - Related sequence Sequence related to 2 or more sequences Module 3 Document granularity - Single document in 3.2.S, single document in 3.2.P, single document for 3.2.P.4 Excipients. Related sequence: related sequence should be to an amendment. This should produce an error. SPL: check xml file / images files / proper naming. FDA request: Test with zip file in SPL folder. SPL: check xml file / images files / proper naming. FDA request: Test with zip file in SPL folder. Module 4 All study reports in Module 4 have attributes for species, route of administration, duration. Alternatively, no attributes provided where they are required for Module 4 toxicology reports. Copyright 2011 IRISS Forum Page 16

18 ICH Q36 Validation Violation Submissions Adherence to the items listed in the ICH ectd Q&A Q36 is technically necessary to exchange ectd messages. This list is a summary of what already exists in the ICH ectd Specification 3.2. Table 7: ICH Q36 Validation Violation Submissions Submission US ICH Group Section Severity Validation Violation US File Checks File References Medium Extraneous modified leaf element path US File Checks Orphaned File Medium No leaf element for file US File Checks File References Medium Omitted leaf element modified file path US File Checks File References Medium Omitted leaf element path US File Checks File Rules Medium Omitted leaf element title US M1 Submission Number High Sequence Number does not match the folder name US ,6,8 File Checks File References Medium You have referenced a file in the hlink:xref attribute of a leaf and the system is unable to locate the file. US File Checks Orphaned Leaf Medium You have referenced the file(s) listed in the validation report from the us-regional.xml or index.xml files without providing the actual file(s). Copyright 2011 IRISS Forum Page 17

19 Results Your results were reviewed by the ETICS team and points awarded as detailed in Table 8. Table 8: Scoring Table Score Definition 2 Error Identified. Tool identified the error condition and declared the ICH/FDA validation error. 1 Error Identified but not the ICH/FDA Tool error code. Identified the error condition, but did not declare the ICH/FDA validation error code..5 Non Intentional or Partial Error Identified. Tool identified partial but error message to vague to accurately score. 0 No Error Identified. Tool did not identify an error. Table 9: US Severity Level High Vendor Scores Viewer/Validators Sample No. Tested rule Severity Tool D Tool E Tool F Tool G Tool H Tool I Tool J Tool K US High US High US High US High US High US High US High US High US High US High US High US High US High Copyright 2011 IRISS Forum Page 18

20 Sample No. Tested rule Severity Tool D Tool E Tool F Tool G Tool H Tool I Tool J Tool K US High US High US High US High US High US High # Identified w/ Code # Identified # Incomplete # None Identified Total Score Copyright 2011 IRISS Forum Page 19

21 Figure 1: US Severity Level High Vendor Scores Viewer/Validators Tool D Tool E Tool F Tool G Tool H Tool I Tool J Tool K Copyright 2011 IRISS Forum Page 20

22 Table 10: US Severity Level Medium Vendor Scores Viewer/Validators Sample No. Tested rule Severity Tool D Tool E Tool F Tool G Tool H Tool I Tool J Tool K US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium Copyright 2011 IRISS Forum Page 21

23 Sample No. Tested rule Severity Tool D Tool E Tool F Tool G Tool H Tool I Tool J Tool K US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium # Identified w/ Code # Identified # Incomplete # None Identified Total Score Copyright 2011 IRISS Forum Page 22

24 Figure 2: US Severity Level Medium Vendor Scores Viewer/Validators Tool D Tool E Tool F Tool G Tool H Tool I Tool J Tool K Copyright 2011 IRISS Forum Page 23

25 Table 11: US Severity Level Low Vendor Scores Viewer/Validators Sample No. Tested rule Severity Tool D Tool E Tool F Tool G Tool H Tool I Tool J Tool K US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low Copyright 2011 IRISS Forum Page 24

26 Sample No. Tested rule Severity Tool D Tool E Tool F Tool G Tool H Tool I Tool J Tool K # Identified w/ Code # Identified # Incomplete # None Identified Total Score Copyright 2011 IRISS Forum Page 25

27 Figure 3: US Severity Level Low Vendor Scores Viewer/Validators Tool D Tool E Tool F Tool G Tool H Tool I Tool J Tool K Copyright 2011 IRISS Forum Page 26

28 Table 12: US Non-validation Error Criteria Scores Viewer/Validators Sample No. Tested rule Tool D Tool E Tool F Tool G Tool H Tool I Tool J Tool K US-2811 a US-2818 a US-2823 a US-2839 a US-2841 a US-2842 a US-2843 a US-2844 a US-2870 a US-2871 a US-2877 a US-2878 a US-2884 a US-2909 a US-2910 a US-2911 a US-2912 a US-2913 a US-2914 a US-2915 a US-2916 a-14c US-2917 a-14b US-2918 a-24b # Identified w/ Code # Identified Copyright 2011 IRISS Forum Page 27

29 Sample No. Tested rule Tool D Tool E Tool F Tool G Tool H Tool I Tool J Tool K # Incomplete # None Identified Total Score Copyright 2011 IRISS Forum Page 28

30 Figure 4: US Non-validation Error Criteria Scores Viewer/Validators Tool D Tool E Tool F Tool G Tool H Tool I Tool J Tool K Copyright 2011 IRISS Forum Page 29

31 Table 13: US Severity Level High Vendor Scores Viewers Sample No. Tested rule Severity Tool A Tool B Tool C US High US High US High US High US High US High US High US High US High US High US High US High US High US High US High US High US High US High US High # Identified w/ Code # Identified # Incomplete # None Identified Total Score Copyright 2011 IRISS Forum Page 30

32 Figure 5: US Severity Level High Vendor Scores Viewers Tool A Tool B Tool C Copyright 2011 IRISS Forum Page 31

33 Table 14: US Severity Level Medium Vendor Scores Viewers Sample No. Tested rule Severity Tool A Tool B Tool C US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium Copyright 2011 IRISS Forum Page 32

34 Sample No. Tested rule Severity Tool A Tool B Tool C US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium US Medium # Identified w/ Code # Identified # Incomplete # None Identified Total Score Copyright 2011 IRISS Forum Page 33

35 Figure 6: US Severity Level Medium Vendor Scores Viewers Tool A Tool B Tool C Copyright 2011 IRISS Forum Page 34

36 Table 15: US Severity Level Low Vendor Scores Viewers Sample No. Tested rule Severity Tool A Tool B Tool C US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low US Low Copyright 2011 IRISS Forum Page 35

37 Sample No. Tested rule Severity Tool A Tool B Tool C # Identified w/ Code # Identified # Incomplete # None Identified Total Score Copyright 2011 IRISS Forum Page 36

38 Figure 7: US Severity Level Low Vendor Scores Viewers Tool A Tool B Tool C 0 Copyright 2011 IRISS Forum Page 37

39 Table 16: US Non-validation Error Criteria Scores Viewers Sample No. Tested rule Tool A Tool B Tool C US-2811 a US-2818 a US-2823 a US-2839 a US-2841 a US-2842 a US-2843 a US-2844 a US-2870 a US-2871 a US-2877 a US-2878 a US-2884 a US-2909 a US-2910 a US-2911 a US-2912 a US-2913 a US-2914 a US-2915 a US-2916 a-14c US-2917 a-14b US-2918 a-24b # Identified w/ Code # Identified # Incomplete Copyright 2011 IRISS Forum Page 38

40 Sample No. Tested rule Tool A Tool B Tool C # None Identified Total Score Figure 8: US Non-validation Error Criteria Scores Viewers Tool A Tool B Tool C Copyright 2011 IRISS Forum Page 39

41 Vendor Code Table 17: Analysis by US Validation Category Error Identified Severity Level High Severity Level Medium Severity Level Low Non-validation Error Criteria Overall Error Identified without ICH/FDA code Partial Error Identified No Error Identified Percentage Not Identified (Partial + No Error / Total) Error Identified Error Identified without ICH/FDA code Partial Error Identified No Error Identified Percentage Not Identified (Partial + No Error / Total) Error Identified A % % % % % B % % % % % C % % % % % D % % % % % E % % % % % F % % % % % G % % % % % H % % % % % I % % % % % J % % % % % K % % % % % Error Identified without ICH/FDA code Partial Error Identified No Error Identified Percentage Not Identified (Partial + No Error / Total) Error Identified Error Identified without ICH/FDA code Partial Error Identified No Error Identified Percentage Not Identified (Partial + No Error / Total) Scores (117max.) Percentage Score Copyright 2011 IRISS Forum Page 40

42 Figure 9: Vendor Tool Comparison of US Severity High Failure Percentage % 90.00% 80.00% 70.00% 60.00% 50.00% 40.00% 30.00% 20.00% 10.00% 0.00% A B C D E F G H I J K Percent 68.42% 94.74% 85.00% 84.21% 52.63% 21.05% 21.05% 5.26% 26.32% 0.00% 10.53% Copyright 2011 IRISS Forum Page 41

43 Figure 10: Vendor Tool Comparison of US Severity Medium Failure Percentage % 90.00% 80.00% 70.00% 60.00% 50.00% 40.00% 30.00% 20.00% 10.00% 0.00% A B C D E F G H I J K Percent 91.67% % 75.00% 95.83% 50.00% 43.75% 29.17% 8.33% 77.08% 4.17% 20.83% Copyright 2011 IRISS Forum Page 42

44 Figure 11: Vendor Tool Comparison of US Severity Low Failure Percentage % 90.00% 80.00% 70.00% 60.00% 50.00% 40.00% 30.00% 20.00% 10.00% 0.00% A B C D E F G H I J K Percent 92.59% % 77.78% 85.19% 40.74% 66.67% 25.93% 11.11% 55.56% 11.11% 29.63% Copyright 2011 IRISS Forum Page 43

45 Figure 12: Vendor Tool Comparison of US Non-validation Error Criteria Failure Percentage % 90.00% 80.00% 70.00% 60.00% 50.00% 40.00% 30.00% 20.00% 10.00% 0.00% A B C D E F G H I J K Percent % % % 95.65% 86.96% 78.26% 73.91% 65.22% % 21.74% 52.17% Copyright 2011 IRISS Forum Page 44

46 Implementation of Regulatory Information Submission Standards (IRISS) ectd Tool Interoperability Group (ETIG) ectd Tool Interoperability and Compliance Study 3 (ETICS 3) Contact details Following your review of this report, if you wish to further discuss your individual results or would like to discuss the IRISS ETIG ETICS 3 US Phase 1 in general then please request a meeting with the EU regional lead using the details below. etics-us@iriss-forum.org Copyright 2011 IRISS Forum Page 45