Phase 1 US Compliance Report - PDF Free Download

Implementation of Regulatory Information Submission Standards (IRISS) ectd Tool Interoperability Group (ETIG) ectd Tool Interoperability and Compliance Study 3 (ETICS 3) ETICS 15 April 2011

Implementation of Regulatory Information Submission Standards (IRISS) ectd Tool Interoperability Group (ETIG) ectd Tool Interoperability and Compliance Study 3 (ETICS 3) Table of Contents Disclaimer... 3 About IRISS... 3 ETICS Background... 4 Introduction... 5 Sample Submission Methodology... 7 US ectd Severity High Validation Violation Submissions... 7 US ectd Severity Medium Validation Violation Submissions... 9 US ectd Severity Low Validation Violation Submissions... 13 Non-validation Error Submissions... 15 ICH Q36 Validation Violation Submissions... 17 Results... 18 Contact details... 45 List of Figures Figure 1: US Severity Level High Vendor Scores Viewer/Validators... 20 Figure 2: US Severity Level Medium Vendor Scores Viewer/Validators... 23 Figure 3: US Severity Level Low Vendor Scores Viewer/Validators... 26 Figure 4: US Non-validation Error Criteria Scores Viewer/Validators... 29 Figure 5: US Severity Level High Vendor Scores Viewers... 31 Figure 6: US Severity Level Medium Vendor Scores Viewers... 34 Figure 7: US Severity Level Low Vendor Scores Viewers... 37 Figure 8: US Non-validation Error Criteria Scores Viewers... 39 Figure 9: Vendor Tool Comparison of US Severity High Failure Percentage... 41 Figure 10: Vendor Tool Comparison of US Severity Medium Failure Percentage... 42 Figure 11: Vendor Tool Comparison of US Severity Low Failure Percentage... 43 Figure 12: Vendor Tool Comparison of US Non-validation Error Criteria Failure Percentage... 44 Copyright 2011 IRISS Forum Page 1

Implementation of Regulatory Information Submission Standards (IRISS) ectd Tool Interoperability Group (ETIG) ectd Tool Interoperability and Compliance Study 3 (ETICS 3) List of Tables Table 1: US Validation Criteria v1.0 points not tested... 5 Table 2: Validation criteria categories... 5 Table 3: US ectd Severity High Validation Violation Submissions... 7 Table 4: US ectd Severity Medium Validation Violation Submissions... 9 Table 5: US ectd Severity Low Validation Violation Submissions... 13 Table 6: Non-validation Error Submissions... 15 Table 7: ICH Q36 Validation Violation Submissions... 17 Table 8: Scoring Table... 18 Table 9: US Severity Level High Vendor Scores Viewer/Validators... 18 Table 10: US Severity Level Medium Vendor Scores Viewer/Validators... 21 Table 11: US Severity Level Low Vendor Scores Viewer/Validators... 24 Table 12: US Non-validation Error Criteria Scores Viewer/Validators... 27 Table 13: US Severity Level High Vendor Scores Viewers... 30 Table 14: US Severity Level Medium Vendor Scores Viewers... 32 Table 15: US Severity Level Low Vendor Scores Viewers... 35 Table 16: US Non-validation Error Criteria Scores Viewers... 38 Table 17: Analysis by US Validation Category... 40 Copyright 2011 IRISS Forum Page 2

Implementation of Regulatory Information Submission Standards (IRISS) ectd Tool Interoperability Group (ETIG) ectd Tool Interoperability and Compliance Study 3 (ETICS 3) Disclaimer Tests were independently designed & conducted collaboratively with a diverse group of community stakeholders from agencies, vendors and industry against available validation criteria. The study design is without regard to currently available software tools or known functionality that may, intentionally or unintentionally cause bias to a single tool. The analyses of results were conducted and the reports written by volunteers from the pharmaceutical industry and CROs. ETICS III has an expanded scope vs. ETICS I/II and while we have compared analyses, you should be aware that there are new tests and new tools being tested. These comparisons were completed to allow an understanding of any trends in interpretation and adherence to available validation criteria over time. Tool vendors who participated should not use their comparative results in promotional materials or public statements. It is acceptable to inform your customers or make public statements of support for the ETICS project or to indicate that your company participated. The IRISS ETIG ETICS III project is not a contest. We appreciate your support and participation as we continue to improve and expand on the ideals that created ETICS. We further hope you find these results valuable to your organization for internal discussions on interpretation of validation criteria and planning of software releases that provide value to your customers. We have released the full set of data in the event you find it useful to benchmark your own product, and appreciate your consumption of these results exclusively for internal use only, in the spirit of which ETICS III is intended and out of respect for the continuation of future efforts serving the ectd community. About IRISS Implementation of Regulatory Information Submission Standards (IRISS) is a global, non-profit professional organization dedicated to the implementation and widespread, successful adoption of electronic regulatory submission standards around the world. IRISS aims to accelerate the adoption of electronic submission standards by making experience and expertise available to its membership through an operating model commonly known as crowdsourcing. Issues are broadcast to a wide community of domain experts and novice users Solutions and points to consider are offered through the membership community Novice users benefit from lessons of experienced early adopters Development organizations can benefit from listening to trends and issues in the community of users All member organizations benefit from a wider pool of talent than exists within their own organization Copyright 2011 IRISS Forum Page 3

Implementation of Regulatory Information Submission Standards (IRISS) ectd Tool Interoperability Group (ETIG) ectd Tool Interoperability and Compliance Study 3 (ETICS 3) IRISS-forum operates as an independent, neutral body free from regional or vendor solution bias where experts and novices gather, regardless of organizational affiliation to discuss, clarify and address pressing issues relevant to the successful adoption of a paperless regulatory environment for submissions. ETICS Background ETICS III is a project of the IRISS ectd Tools Interoperability Group (ETIG). ETIG was launched in October 2008. The group s main purpose is to discuss ectd building and viewing tools and the interoperability aspects between them. Membership is comprised of global regulators, vendors and industry affiliates who have an interest in making the ectd a more Common format. The IRISS ectd Tool Interoperability group is a forum for discussion and resolution of ectd interoperability issues. Discussion amongst the membership is based on ideas and concepts and is not tool or vendor specific. The group s Concept Paper provides further details. Increasingly there is need to exchange ectds between tools and hence interoperability remains important. Interoperability is achieved when the ectd message created by one tool and received by another tool meets the following criteria: EITHER the ectd is loadable / readable without significant errors OR the receiving tool correctly identifies and describes impact of errors but the ectd is still transferred with information integrity intact. Consistency of submission building, the adherence to ICH and EU specifications and the ability to detect non-compliance with these specifications are all important factors. A significant error is one which indicates; a loss of information integrity; failure to follow ICH ectd Q&A Question 36 rules (compliance); indicates failure to follow the US ectd Validation Criteria (March, 2008/version 1.0). To judge the significance of an error, issue or information, generated by an ectd validation tool, the error message itself must be clear and detailed. Copyright 2011 IRISS Forum Page 4

Implementation of Regulatory Information Submission Standards (IRISS) ectd Tool Interoperability Group (ETIG) ectd Tool Interoperability and Compliance Study 3 (ETICS 3) Introduction The ETICS 3 Phase 1 project (launched February 2010) has built on the previous projects to incorporate new validation rules to help vendors to identify areas where compliance can be further improved. 81 out of 83 US ectd Validation Criteria (March, 2008/version 1.0) were tested. Table 1 lists the validation criteria which were not incorporated. Table 1: US Validation Criteria v1.0 points not tested NO. GROUP SECTION VALIDATION CRITERION SOURCE SEVERITY 1187 File Checks File References Modified leaf element path references a sequence that has not been validated FDA Criteria v1.0 Medium ETICS 3 Phase 1 tested not only compliance against the ICH ectd Q&A 36 rules (Oct, 2009/version 1.18) and the US ectd Validation Criteria (March, 2008/version 1.0); but also included nonvalidation issues 1. The sample submissions were created to test each participating tool s ability to detect these error conditions. Participating tool vendors assessed these submissions and reported their findings. The ETICS team provided 118 dossiers, each with three sequences where the US and ICH validation criteria were challenged. You assessed these submissions and reported your findings. Table 2 displays the validation criteria and the number of submissions which challenge them. Table 2: Validation criteria categories Validation Criteria and US severity Number of Submissions US severity High failure 19 US severity Medium failure 48 US severity Low failure 27 Non-validation Criteria Error 24 1 Known issue which may be under discussion for potential change requests/clarification Copyright 2011 IRISS Forum Page 5

Implementation of Regulatory Information Submission Standards (IRISS) ectd Tool Interoperability Group (ETIG) ectd Tool Interoperability and Compliance Study 3 (ETICS 3) Submission US-2840 is considered by the ETICS team to be compliant with current ectd specifications and therefore should not have reported any warnings or errors. The following tables (Table 3, Table 4, Table 5, and Table 6 and Table 7) identify the ETICS 3 test cases. To allow you to compare the results while ensuring confidentiality the name of the vendor tool has been replaced with a letter. The tabulated results for each of the sample submissions are presented in Table 9, Table 10, Table 11, Table 12, Table 13, Table 14, Table 15, and Table 16. These results are also displayed as charts in Figure 1, Figure 3 Figure 3, Figure 4, Figure 5, Figure 6, Figure 6, Figure 7, and Figure 8. These results were further analysed by the US validation categories (Table 17, Figure 9, Figure 10, Figure 11 and Figure 11). Copyright 2011 IRISS Forum Page 6

Sample Submission Methodology Copyright 2011 IRISS Forum Page 7 To aid review we have organised the submissions by US Validation Priority, Non-validation issues and then ICH ectd Q&A Questions 36 Validation Criteria. US ectd Severity High Validation Violation Submissions HIGH: The error is a serious technical error which prevents the processing of the submission and will require resubmission. The submission is considered not received by FDA. Table 3: US ectd Severity High Validation Violation Submissions Submission US ICH Group Section Severity Validation Violation US-2802 3043 M1 Application High Application Number can only contain numeric characters" Number Description="as integer datatype. The element 'application-number' US-2803 1519 M1 Application Number US-2804 3036 M1 Application Number US-2805 1527 M1 Application Number US-2812 1731 STF STF Cumulative Approach High High High High with value Application Number does not match the folder name Application Number must contain 6 characters" Description="violates pattern constraint of '[0-9][0-9][0-9] [0-9] [0-9] [0-9]'.The element 'application-number' with value Application Number must remain constant Cumulative STF files are not allowed US-2831 1493 1 General XML Validation High Invalid XML file US-2866 3064 M1 Related Sequence High Related Sequence Number must contain 4 characters" Description="violates pattern constraint of '[0-9][0-9][0-9] [0-9]'.The element 'related-sequence-number' with value

Submission US ICH Group Section Severity Validation Violation US-2867 3071 M1 Related Sequence High Related submission Number (ed. Sequence number) can only contain numeric characters" Description="as integer datatype. The element 'related-sequence-number' with value US-2868 1714 19 M1 Submission Number High Sequence Number does not match the folder name US-2869 1697 M1 Submission Number US-2872 1799 STF STF File References US-2879 3057 M1 Submission Number US-2880 3050 M1 Submission Number High High High High Sequence Number is not unique STF cannot reference another STF Submission Number (ed. Sequence number) can only contain numeric characters" Description="as integer datatype. Submission Number (ed. Sequence number) must contain 4 characters" Description="violates pattern constraint of ' [0-9] [0-9] [0-9] [0-9]'.The element 'sequence-number' with value US-2889 1459 General DTD Version High Unsupported DTD version in backbone XML file US-2890 1463 General DTD Version High Unsupported DTD version in regional XML file US-2891 1782 STF STF DTD Version High Unsupported STF DTD version in leaf element US-2892 1786 STF STF DTD Version High Unsupported STF DTD version in STF XML file US-2899 1510 1 General XML Validation High XML file error US-2906 1493 1 General XML Validation High Invalid XML file Copyright 2011 IRISS Forum Page 8

US ectd Severity Medium Validation Violation Submissions MEDIUM: The error may impact the reviewability of the submission but cannot be determined without further inspection by the review staff. The submission might be considered received by FDA. Table 4: US ectd Severity Medium Validation Violation Submissions Submission US ICH Group Section Severity Validation Violation US-2809 1561 M1 Related Sequence Medium Circular relationship found in related sequences US-2813 1850 STF STF File Rules Medium Document ID for STF must remain constant US-2815 1051 File Checks File References Medium Extraneous leaf element path US-2816 1068 4 File Checks File References Medium Extraneous modified leaf element path US-2817 1102 File Checks File References Medium Leaf element path contains invalid characters: backslash(\), colon(:), asterisk(*), question mark(?), less than(<), greater than(>), pipe( ) US-2822 1102 File Checks File References Medium Leaf element path contains invalid characters: backslash(\), colon(:), asterisk(*), question mark(?), less than(<), greater than(>), pipe( ) US-2827 1255 File Checks File Rules Medium Invalid file extension US-2828 1034 File Checks File Operation Medium Invalid leaf element operation US-2829 1867 STF STF File Rules Medium Invalid STF file extension US-2830 1884 STF STF File Rules Medium Invalid STF file name US-2832 1102 File Checks File References Medium Leaf element path contains invalid characters: backslash(\), colon(:), asterisk(*), question mark(?), less than(<), greater than(>), pipe( ) US-2833 1102 File Checks File References Medium Leaf element path contains invalid characters: backslash(\), colon(:), asterisk(*), question mark(?), less than(<), greater than(>), pipe( ) US-2834 1017 File Checks File Operation Medium Leaf element already replaced or deleted Copyright 2011 IRISS Forum Page 9

Submission US ICH Group Section Severity Validation Violation US-2835 1102 File Checks File References Medium Leaf element path contains invalid characters: backslash(\), colon(:), asterisk(*), question mark(?), less than(<), greater than(>), pipe( ) US-2847 1935 STF STF File Rules Medium Multiple STF doc-content file tags US-2848 1306 13 File Checks Orphaned File Medium No leaf element for file US-2849 1478 General Node Extension Medium Node-extension does not have a title US-2850 1476 General Node Extension Medium Node-extension exists US-2853 1314 File Checks Orphaned File Medium Non-Required File Exists US-2854 1357 General Attribute Values (Repeat Groups) Medium Omitted attribute value US-2857 1442 General DTD Version Medium Omitted DTD version in backbone XML file US-2858 1445 General DTD Version Medium Omitted DTD version in regional XML file US-2859 1170 4 File Checks File References Medium Omitted leaf element modified file path US-2860 1136 13 File Checks File References Medium Omitted leaf element path US-2861 1289 20 File Checks File Rules Medium Omitted leaf element title US-2862 1918 STF STF File Rules Medium Omitted STF doc-content file tag US-2863 1748 STF STF DTD Version Medium Omitted STF DTD version in leaf element US-2864 1752 STF STF DTD Version Medium Omitted STF DTD version in STF XML file US-2865 3015 STF STF XML Validation Medium Preceding the element file-tag with the element property is recommended" Description="Element 'file-tag' is unexpected according to content model of parent element 'doc-content. Expecting: property. US-2874 1816 STF STF File References Medium STF does not relate to any leaf elements US-2875 1765 STF STF DTD Version Medium STF DTD version must match the XML file US-2876 1833 STF STF File Medium STF relates to a non-existent leaf element References Copyright 2011 IRISS Forum Page 10

Submission US ICH Group Section Severity Validation Violation US-2881 1362 General Attribute Values Medium The attribute appears multiple times (Repeat Groups) US-2882 1153 File Checks File References Medium The leaf referenced in the modify-file element cannot be found. US-2883 3022 STF STF XML Validation Medium The maximum occurrences of the category element is three" Description="Element 'category' is unexpected according to content model of parent element 'document identifier' US-2885 1595 M1 Related Sequence Medium This submission type cannot have a child submission US-2886 1612 M1 Related Sequence Medium This submission type cannot have a related sequence US-2887 1629 M1 Related Sequence Medium This submission type requires a related sequence US-2893 3029 STF STF XML Validation US-2894 3001 STF STF XML Validation US-2895 3008 STF STF XML Validation Medium Medium Medium Use of content-block is not recommended" Description="Element 'content-block' is unexpected according to content model of parent element 'document'. Use of the element title is not recommended" Description="Element 'title' is unexpected according to content model of parent element 'doc-content' Use of the property element is required. The xlink:href attribute of the doc-content element is not used" Description="The attribute '{http://www.w3c.org/1999/xlink}href' on the element 'doc-content' is not defined in the DTD/Schema. US-2897 1102 File Checks File References Medium Leaf element path contains invalid characters: backslash(\), colon(:), asterisk(*), question mark(?), less than(<), greater than(>), pipe( ) US-2898 1102 File Checks File References Medium Leaf element path contains invalid characters: backslash(\), colon(:), asterisk(*), question mark(?), less than(<), greater than(>), pipe( ) US-2900 1119 2,6,8 File Checks File References Medium You have referenced a file in the hlink:xref attribute of a leaf and the system is unable to locate the file. Copyright 2011 IRISS Forum Page 11

Submission US ICH Group Section Severity Validation Violation US-2901 1323 12 File Checks Orphaned Leaf Medium You have referenced the file(s) listed in the validation report from the us-regional.xml or index.xml files without providing the actual file(s). US-2902 1969 STF STF XML Validation Medium You have submitted an stf-xxxxxxxx.xml file. When the STF XML file fails to load, having no XML elements in it, or no document-identifier (or study-identifier) XML element error 1969 will be present. US-2904 1901 STF STF File Rules Medium Your leaf that points to the STF is in under an invalid heading area or invalid module. For example, STF can only be associated with certain headings under Modules 4 or 5. US-2908 1969 STF STF XML Validation Medium You have submitted an stf-xxxxxxxx.xml file. When the STF XML file fails to load, having no XML elements in it, or no document-identifier (or study-identifier) XML element error 1969 will be present. Copyright 2011 IRISS Forum Page 12

US ectd Severity Low Validation Violation Submissions LOW: The error is a technical error which may or may not impact the reviewability or the integrity of the submission. The submission will likely be considered received by FDA. Table 5: US ectd Severity Low Validation Violation Submissions Submission US ICH Group Section Severity Validation Violation US-2801 1272 File Checks File Rules Low Leaf element title contains an invalid character per ectd Specification US-2806 1340 General Attribute Values (Repeat Groups) Low Attribute value contains an invalid character per ectd Specification US-2807 1344 General Attribute Values (Repeat Groups) Low Attribute value leads or ends with a space per ectd Specification US-2808 1374 11 General Checksum Low Checksum value must match the actual checksum US-2810 1544 M1 Company Name Low Company name must remain constant US-2814 1368 General Attribute Values (Repeat Groups) Low Extraneous attribute US-2819 1204 File Checks File Rules Low File name contains invalid characters per ectd Specification US-2820 1221 File Checks File Rules Low File name exceeds maximum length (64 characters) per ectd Specification US-2821 1238 17 File Checks File Rules Low File size exceeds maximum (100 MB) per ectd Specification US-2824 1391 11 General Checksum Low Invalid checksum format US-2825 1408 11 General Checksum Low Invalid checksum type US-2826 1663 M1 Submission Date Low Invalid date format per ectd Specification US-2836 1085 File Checks File References Low Leaf element path length exceeds maximum (230characters) US-2837 1272 File Checks File Rules Low Leaf element title contains an invalid character per ectd Specification Copyright 2011 IRISS Forum Page 13

Submission US ICH Group Section Severity Validation Violation US-2838 1276 File Checks File Rules Low Leaf element title leads or ends with a space per ectd Specification US-2845 1128 File Checks File References Low More than one version of the required file exists US-2846 1578 M1 Related Sequence Low Multiple related sequences are not allowed US-2851 1480 General Node Extension Low Node-extension title contains an invalid character US-2852 1482 General Node Extension Low Node-extension title leads or ends with a space US-2855 1425 11 General Checksum Low Omitted checksum value US-2856 1680 M1 Submission Date Low Omitted date per ectd Specification US-2873 1952 STF STF File Rules Low STF doc-content title and leaf element title do not match US-2888 1344 General Attribute Values (Repeat Groups) Low Attribute value leads or ends with a space per ectd Specification US-2896 1340 General Attribute Values (Repeat Groups) Low Attribute value contains an invalid character per ectd Specification US-2903 1646 M1 Submission Date Low Your date or date format is invalid US-2905 1480 General Node Extension Low Node-extension title contains an invalid character US-2907 1952 STF STF File Rules Low Your STF doc-content title and leaf element title do not match; viewer will show the title of the leaf only. Copyright 2011 IRISS Forum Page 14

Non-validation Error Submissions These submissions are technically valid but contain know issues which can lead tools to: EITHER behave inconsistently, AND/OR report errors where there should not be errors, AND/OR display the content of the ectd incorrectly, AND/OR report errors which the ETICS group think are not errors. Table 6 discusses these issues and lists what the ETICS group expected outcome is. Table 6: Non-validation Error Submissions Submission US-2811 US-2818 US-2823 US-2839 US-2841 US-2842 US-2843 US-2844 Validation Violation Correct a study report location FDA SMART FORMS Invalid characters M4-5 Datasets - Datasets are not always submitted in a datasets folder as defined by guidance (m4/datasets, m5/datasets). This causes issues when migrating sequences between tools. Module 4 All study reports in Module 4 have attributes for "species, route of administration, duration" Alternately, no attributes provided where they are required for Module 4 toxicology reports. Module 4 Multiple values for STF attribute species. Module 4-5 - stf.xml appends first submitted stf.xml, not the most recently submitted stf.xml. This results in different displays in different viewing tools. Module 5 Case report forms located either in m537-crf-ipl or in the study folder. For sponsors submitting globally, it is more efficient to remove CRFs from one single folder than from each individual study folder, so we do not want viewing/validation errors for either option. Copyright 2011 IRISS Forum Page 15

Submission Validation Violation US-2870 Special Characters: e.g. copyright US-2871 SPL: check xml file / images files / proper naming. Tested with a zip file in the SPL folder. US-2877 STF Scenarios (e.g. same study in 2 different sections). US-2878 STF title changes with lifecycle. US-2884 The title of the leaf which points to the STF should match the study title. US-2909 M4-5 Datasets - Datasets are not always submitted in a datasets folder as defined by guidance (m4/datasets, m5/datasets). This causes issues when migrating sequences between tools. US-2910 US-2911 US-2912 US-2913 US-2914 US-2915 US-2916 US-2917 US-2918 Cross-application linking (leaf cross-reference or hyperlinks). Related sequence is expected but not present. Business example is an IND where an amendment does not have related sequence referencing back to the original-application. M1 Folder structure is variable (sometimes a flat hierarchy, sometimes many subfolders). This causes issues when migrating sequences between tools. M1 - Related sequence Sequence related to 2 or more sequences Module 3 Document granularity - Single document in 3.2.S, single document in 3.2.P, single document for 3.2.P.4 Excipients. Related sequence: related sequence should be to an amendment. This should produce an error. SPL: check xml file / images files / proper naming. FDA request: Test with zip file in SPL folder. SPL: check xml file / images files / proper naming. FDA request: Test with zip file in SPL folder. Module 4 All study reports in Module 4 have attributes for species, route of administration, duration. Alternatively, no attributes provided where they are required for Module 4 toxicology reports. Copyright 2011 IRISS Forum Page 16

ICH Q36 Validation Violation Submissions Adherence to the items listed in the ICH ectd Q&A Q36 is technically necessary to exchange ectd messages. This list is a summary of what already exists in the ICH ectd Specification 3.2. Table 7: ICH Q36 Validation Violation Submissions Submission US ICH Group Section Severity Validation Violation US-2816 1068 4 File Checks File References Medium Extraneous modified leaf element path US-2848 1306 13 File Checks Orphaned File Medium No leaf element for file US-2859 1170 4 File Checks File References Medium Omitted leaf element modified file path US-2860 1136 13 File Checks File References Medium Omitted leaf element path US-2861 1289 20 File Checks File Rules Medium Omitted leaf element title US-2868 1714 19 M1 Submission Number High Sequence Number does not match the folder name US-2900 1119 2,6,8 File Checks File References Medium You have referenced a file in the hlink:xref attribute of a leaf and the system is unable to locate the file. US-2901 1323 12 File Checks Orphaned Leaf Medium You have referenced the file(s) listed in the validation report from the us-regional.xml or index.xml files without providing the actual file(s). Copyright 2011 IRISS Forum Page 17

Results Your results were reviewed by the ETICS team and points awarded as detailed in Table 8. Table 8: Scoring Table Score Definition 2 Error Identified. Tool identified the error condition and declared the ICH/FDA validation error. 1 Error Identified but not the ICH/FDA Tool error code. Identified the error condition, but did not declare the ICH/FDA validation error code..5 Non Intentional or Partial Error Identified. Tool identified partial but error message to vague to accurately score. 0 No Error Identified. Tool did not identify an error. Table 9: US Severity Level High Vendor Scores Viewer/Validators Sample No. Tested rule Severity Tool D Tool E Tool F Tool G Tool H Tool I Tool J Tool K US-2802 3043 High 0.5 0.5 1 1 2 2 1 1 US-2803 1519 High 0.5 0.5 1 1 2 2 1 1 US-2804 3036 High 0.5 0.5 2 2 2 2 2 1 US-2805 1527 High 0.5 0.5 1 1 2 2 2 1 US-2812 1731 High 0.5 0.5 0.5 0.5 0 0.5 1 0.5 US-2831 1493 High 1 2 1 1 2 1 2 1 US-2866 3064 High 0.5 2 2 1 2 2 2 1 US-2867 3071 High 0.5 2 2 1 2 2 2 1 US-2868 1714 High 0.5 2 2 1 2 2 2 1 US-2869 1697 High 0.5 2 1 1 2 1 1 1 US-2872 1799 High 0.5 0.5 1 0 2 2 2 1 US-2879 3057 High 0.5 2 0 1 1 0.5 2 1 US-2880 3050 High 0.5 0.5 1 1 1 0.5 2 1 Copyright 2011 IRISS Forum Page 18

Sample No. Tested rule Severity Tool D Tool E Tool F Tool G Tool H Tool I Tool J Tool K US-2889 1459 High 0.5 2 1 2 2 2 2 1 US-2890 1463 High 0.5 0.5 1 2 2 2 2 1 US-2891 1782 High 0.5 0.5 0.5 2 2 2 2 0.5 US-2892 1786 High 0.5 0.5 2 2 2 2 2 1 US-2899 1510 High 1 2 1 0 2 0.5 1 1 US-2906 1493 High 1 2 0 0 2 0.5 2 1 # Identified w/ Code 0 9 5 5 16 12 14 0 # Identified 3 0 10 10 2 2 5 17 # Incomplete 16 10 2 1 0 5 0 2 # None Identified 0 0 2 3 1 0 0 0 Total Score 11 23 21 20.5 34 28.5 33 18 Copyright 2011 IRISS Forum Page 19

Figure 1: US Severity Level High Vendor Scores Viewer/Validators 35 34 33 30 28.5 25 23 21 20.5 20 18 15 11 10 5 0 Tool D Tool E Tool F Tool G Tool H Tool I Tool J Tool K Copyright 2011 IRISS Forum Page 20

Table 10: US Severity Level Medium Vendor Scores Viewer/Validators Sample No. Tested rule Severity Tool D Tool E Tool F Tool G Tool H Tool I Tool J Tool K US-2809 1561 Medium 0.5 0.5 0.5 0.5 2 0.5 2 0.5 US-2813 1850 Medium 0.5 0.5 0.5 0.5 2 2 2 0.5 US-2815 1051 Medium 0 2 0.5 0 2 0.5 2 1 US-2816 1068 Medium 0 2 1 1 2 0.5 2 1 US-2817 1102 Medium 0.5 2 1 2 2 0.5 2 1 US-2822 1102 Medium 0.5 2 0.5 2 2 0.5 1 1 US-2827 1255 Medium 0.5 1 2 2 2 2 2 1 US-2828 1034 Medium 0 1 0.5 2 2 1 2 1 US-2829 1867 Medium 0.5 0.5 1 2 2 1 2 1 US-2830 1884 Medium 0.5 0.5 1 0 2 0.5 2 0.5 US-2832 1102 Medium 0.5 2 1 2 2 0.5 2 1 US-2833 1102 Medium 0.5 2 1 2 2 1 2 1 US-2834 1017 Medium 0.5 0.5 1 0 2 0.5 2 1 US-2835 1102 Medium 0.5 2 1 2 2 1 2 1 US-2847 1935 Medium 0.5 0.5 1 2 2 0.5 2 0.5 US-2848 1306 Medium 0.5 0.5 1 1 2 0.5 2 0.5 US-2849 1478 Medium 0.5 1 0.5 2 2 2 2 1 US-2850 1476 Medium 0.5 2 0.5 2 2 0.5 2 1 US-2853 1314 Medium 0.5 0.5 0.5 0 2 0.5 2 0.5 US-2854 1357 Medium 0.5 0.5 0.5 0 0 0.5 1 1 US-2857 1442 Medium 0.5 2 1 2 2 1 2 1 US-2858 1445 Medium 0.5 2 0.5 2 2 1 2 1 US-2859 1170 Medium 0.5 0.5 0.5 2 2 0.5 1 1 US-2860 1136 Medium 0.5 2 2 2 2 0.5 2 1 US-2861 1289 Medium 0.5 2 1 2 2 2 2 0.5 US-2862 1918 Medium 0.5 2 2 2 2 0.5 2 1 US-2863 1748 Medium 0.5 0.5 0.5 2 2 0.5 2 0.5 Copyright 2011 IRISS Forum Page 21

Sample No. Tested rule Severity Tool D Tool E Tool F Tool G Tool H Tool I Tool J Tool K US-2864 1752 Medium 0.5 2 1 2 2 0.5 2 1 US-2865 3015 Medium 0.5 2 0.5 0 0 0.5 1 0.5 US-2874 1816 Medium 0.5 0.5 1 0 2 0.5 1 1 US-2875 1765 Medium 0.5 2 1 2 2 1 2 1 US-2876 1833 Medium 0.5 2 1 0 2 0.5 2 1 US-2881 1362 Medium 1 0.5 0.5 1 1 0.5 2 1 US-2882 1153 Medium 0.5 0.5 1 1 2 0.5 2 1 US-2883 3022 Medium 0.5 0.5 0.5 1 2 0.5 2 1 US-2885 1595 Medium 0.5 0.5 0.5 0 1 0.5 1 1 US-2886 1612 Medium 0.5 0.5 0.5 0 0 0.5 2 1 US-2887 1629 Medium 0.5 0.5 0.5 0 2 0.5 2 1 US-2893 3029 Medium 0.5 2 0.5 1 2 0.5 2 0.5 US-2894 3001 Medium 0 0.5 0.5 1 1 0 0.5 1 US-2895 3008 Medium 0 0.5 1 1 1 0.5 0.5 1 US-2897 1102 Medium 0 2 1 2 2 0.5 2 1 US-2898 1102 Medium 1 0.5 1 0.5 1 0.5 2 1 US-2900 1119 Medium 0 2 1 1 2 0.5 1 1 US-2901 1323 Medium 0 2 2 1 2 0.5 2 1 US-2902 1969 Medium 0 0.5 1 1 2 0.5 2 1 US-2904 1901 Medium 0 0.5 0.5 2 0 0.5 2 1 US-2908 1969 Medium 0 0.5 1 1 2 0.5 2 1 # Identified w/ Code 0 21 4 22 39 4 39 0 # Identified 2 3 23 12 5 7 7 38 # Incomplete 35 24 21 3 0 36 2 10 # None Identified 11 0 0 11 4 1 0 0 Total Score 19.5 57 41.5 57.5 83 33 86 43 Copyright 2011 IRISS Forum Page 22

Figure 2: US Severity Level Medium Vendor Scores Viewer/Validators 90 80 83 86 70 60 57 57.5 50 40 41.5 33 43 30 20 19.5 10 0 Tool D Tool E Tool F Tool G Tool H Tool I Tool J Tool K Copyright 2011 IRISS Forum Page 23

Table 11: US Severity Level Low Vendor Scores Viewer/Validators Sample No. Tested rule Severity Tool D Tool E Tool F Tool G Tool H Tool I Tool J Tool K US-2801 1272 Low 1 2 0.5 0 1 1 2 1 US-2806 1340 Low 0.5 0.5 0.5 0.5 1 0.5 2 1 US-2807 1344 Low 0.5 0.5 0.5 0.5 2 0.5 2 1 US-2808 1374 Low 0.5 2 2 1 2 2 2 1 US-2810 1544 Low 0.5 0.5 0.5 0.5 2 0 2 0.5 US-2814 1368 Low 0.5 2 0.5 1 1 1 2 1 US-2819 1204 Low 0.5 2 2 2 2 2 2 0.5 US-2820 1221 Low 0.5 2 1 2 2 2 2 1 US-2821 1238 Low 0.5 2 0.5 2 2 2 2 1 US-2824 1391 Low 0.5 2 1 1 2 1 1 1 US-2825 1408 Low 0.5 2 2 1 2 2 2 1 US-2826 1663 Low 0.5 2 0.5 1 2 1 1 1 US-2836 1085 Low 0.5 2 2 2 1 2 2 1 US-2837 1272 Low 1 1 0.5 1 1 1 2 1 US-2838 1276 Low 0.5 0.5 0.5 2 2 0.5 0 0.5 US-2845 1128 Low 0 0 0 0 0 0 0 0 US-2846 1578 Low 0.5 0.5 1 2 2 0.5 2 0.5 US-2851 1480 Low 1 1 0.5 0 1 0.5 1 1 US-2852 1482 Low 0.5 1 0.5 2 2 0.5 1 1 US-2855 1425 Low 0.5 2 1 1 2 1 2 1 US-2856 1680 Low 0.5 2 0.5 2 2 0.5 2 0.5 US-2873 1952 Low 0.5 0.5 0.5 2 0 0.5 2 0.5 US-2888 1344 Low 0.5 1 0.5 1 2 0.5 0 1 US-2896 1340 Low 0.5 0.5 0.5 1 1 0.5 1 1 US-2903 1646 Low 0 0.5 2 1 2 0.5 2 1 US-2905 1480 Low 1 0.5 0 0 1 0.5 2 1 US-2907 1952 Low 0 0.5 0.5 2 0 0.5 2 0.5 Copyright 2011 IRISS Forum Page 24

Sample No. Tested rule Severity Tool D Tool E Tool F Tool G Tool H Tool I Tool J Tool K # Identified w/ Code 0 12 5 10 16 6 19 0 # Identified 4 4 4 10 8 6 5 19 # Incomplete 20 10 16 3 0 13 0 7 # None Identified 3 1 2 4 3 2 3 1 Total Score 14 33 22 31.5 40 24.5 43 22.5 Copyright 2011 IRISS Forum Page 25

Figure 3: US Severity Level Low Vendor Scores Viewer/Validators 45 40 40 43 35 33 31.5 30 25 22 24.5 22.5 20 15 14 10 5 0 Tool D Tool E Tool F Tool G Tool H Tool I Tool J Tool K Copyright 2011 IRISS Forum Page 26

Table 12: US Non-validation Error Criteria Scores Viewer/Validators Sample No. Tested rule Tool D Tool E Tool F Tool G Tool H Tool I Tool J Tool K US-2811 a-30 0.5 0.5 1 0.5 0 0.5 1 0.5 US-2818 a-6 0.5 1 1 0.5 0 0.5 1 1 US-2823 a-21 0.5 0.5 2 0 0 0.5 0 0.5 US-2839 a-28 0.5 0.5 0.5 0.5 0.5 0.5 0.5 1 US-2841 a-24 0.5 0.5 0.5 1 1 0.5 1 1 US-2842 a-17 0.5 0.5 0.5 0 1 0.5 1 0.5 US-2843 a-27 0.5 0.5 0.5 0 0 0.5 1 0.5 US-2844 a-23 0.5 1 0.5 0 0 0.5 1 1 US-2870 a-22 0.5 0.5 0.5 0 0 0.5 0 0.5 US-2871 a-14 0.5 0.5 0.5 1 1 0.5 1 1 US-2877 a-15 0.5 0.5 0.5 0 0 0.5 1 0.5 US-2878 a-20 0.5 0.5 0.5 0 0 0.5 0 0.5 US-2884 a-19 0.5 0.5 0.5 1 0 0.5 1 0.5 US-2909 a-28 0 0.5 0.5 0.5 0.5 0.5 1 1 US-2910 a-7 1 2 1 0 1 0.5 1 0.5 US-2911 a-9 0 0.5 1 1 1 0.5 1 1 US-2912 a-10 0 0.5 0.5 0 0 0.5 0 0.5 US-2913 a-11 0 0.5 0.5 1 1 0.5 1 1 US-2914 a-13 0 0.5 0.5 0 0 0.5 1 1 US-2915 a-12 0 0.5 0.5 0 1 0.5 1 1 US-2916 a-14c 0 0.5 0.5 0 0 0.5 1 0.5 US-2917 a-14b 0 0.5 0.5 0 1 0.5 1 0.5 US-2918 a-24b 0 0.5 0.5 1 0 0.5 1 1 # Identified w/ Code 0 1 1 0 0 0 0 0 # Identified 1 2 4 6 8 0 18 11 Copyright 2011 IRISS Forum Page 27

Sample No. Tested rule Tool D Tool E Tool F Tool G Tool H Tool I Tool J Tool K # Incomplete 13 20 18 4 2 23 1 12 # None Identified 9 0 0 13 13 0 4 0 Total Score 7.5 14 15 8 9 11.5 18.5 17 Copyright 2011 IRISS Forum Page 28

Figure 4: US Non-validation Error Criteria Scores Viewer/Validators 20 18 18.5 17 16 14 12 14 15 11.5 10 8 7.5 8 9 6 4 2 0 Tool D Tool E Tool F Tool G Tool H Tool I Tool J Tool K Copyright 2011 IRISS Forum Page 29

Table 13: US Severity Level High Vendor Scores Viewers Sample No. Tested rule Severity Tool A Tool B Tool C US-2802 3043 High 0 0 0 US-2803 1519 High 0 0 0 US-2804 3036 High 0 2 0 US-2805 1527 High 0 0.5 0 US-2812 1731 High 0 0 0 US-2831 1493 High 0.5 0 0 US-2866 3064 High 0 0 0 US-2867 3071 High 0 0 0 US-2868 1714 High 2 0 0 US-2869 1697 High 2 0 0 US-2872 1799 High 0 0 0 US-2879 3057 High 0 0 0 US-2880 3050 High 0 0 0 US-2889 1459 High 2 0 1 US-2890 1463 High 2 0 1 US-2891 1782 High 0 0 0 US-2892 1786 High 0 0 1 US-2899 1510 High 2 0 0.5 US-2906 1493 High 2 0 0 # Identified w/ Code 6 1 0 # Identified 0 0 3 # Incomplete 1 1 1 # None Identified 12 17 15 Total Score 12.5 2.5 3.5 Copyright 2011 IRISS Forum Page 30

Table 14: US Severity Level Medium Vendor Scores Viewers Sample No. Tested rule Severity Tool A Tool B Tool C US-2809 1561 Medium 0 0 0 US-2813 1850 Medium 0 0 0 US-2815 1051 Medium 0 0 0 US-2816 1068 Medium 0 0 0 US-2817 1102 Medium 0 0 0 US-2822 1102 Medium 0 0 0 US-2827 1255 Medium 0 0 0 US-2828 1034 Medium 2 0 1 US-2829 1867 Medium 0 0 0 US-2830 1884 Medium 0 0 0 US-2832 1102 Medium 0 0 0 US-2833 1102 Medium 0 0 0 US-2834 1017 Medium 0 0 0 US-2835 1102 Medium 0 0 0 US-2847 1935 Medium 0 0 0 US-2848 1306 Medium 0 0 0 US-2849 1478 Medium 0 0 0 US-2850 1476 Medium 0 0 0 US-2853 1314 Medium 0 0 0 US-2854 1357 Medium 0 0 0 US-2857 1442 Medium 2 0 1 US-2858 1445 Medium 2 0 1 US-2859 1170 Medium 0 0 1 US-2860 1136 Medium 0 0 0 US-2861 1289 Medium 0 0 0 US-2862 1918 Medium 0 0 0 US-2863 1748 Medium 0 0 0 Copyright 2011 IRISS Forum Page 32

Sample No. Tested rule Severity Tool A Tool B Tool C US-2864 1752 Medium 0 0 1 US-2865 3015 Medium 0 0 0 US-2874 1816 Medium 0 0 1 US-2875 1765 Medium 0 0 1 US-2876 1833 Medium 0 0 1 US-2881 1362 Medium 2 0 1 US-2882 1153 Medium 0 0 0 US-2883 3022 Medium 0 0 0 US-2885 1595 Medium 0 0 0 US-2886 1612 Medium 0 0 0 US-2887 1629 Medium 0 0 0 US-2893 3029 Medium 0 0 0 US-2894 3001 Medium 0 0 0 US-2895 3008 Medium 0 0 0 US-2897 1102 Medium 0 0 0 US-2898 1102 Medium 0.5 0 1 US-2900 1119 Medium 0 0 0 US-2901 1323 Medium 0 0 0 US-2902 1969 Medium 0 0 1 US-2904 1901 Medium 0 0 0 US-2908 1969 Medium 0 0 1 # Identified w/ Code 4 0 0 # Identified 0 0 12 # Incomplete 1 0 0 # None Identified 43 48 36 Total Score 8.5 0 12 Copyright 2011 IRISS Forum Page 33

Table 15: US Severity Level Low Vendor Scores Viewers Sample No. Tested rule Severity Tool A Tool B Tool C US-2801 1272 Low 0.5 0 1 US-2806 1340 Low 0.5 0 1 US-2807 1344 Low 0 0 0 US-2808 1374 Low 0 0 0 US-2810 1544 Low 0 0 0 US-2814 1368 Low 0.5 0 1 US-2819 1204 Low 0 0 0 US-2820 1221 Low 0 0 0 US-2821 1238 Low 0 0 0 US-2824 1391 Low 0.5 0 0 US-2825 1408 Low 2 0 0 US-2826 1663 Low 2 0 1 US-2836 1085 Low 0 0 0 US-2837 1272 Low 0.5 0 0 US-2838 1276 Low 0 0 0 US-2845 1128 Low 0 0 0 US-2846 1578 Low 0 0 0 US-2851 1480 Low 0.5 0 0 US-2852 1482 Low 0 0 0 US-2855 1425 Low 0.5 0 0 US-2856 1680 Low 0 0 0 US-2873 1952 Low 0 0 0 US-2888 1344 Low 0 0 0 US-2896 1340 Low 0.5 0 1 US-2903 1646 Low 0 0 0 US-2905 1480 Low 0.5 0 1 US-2907 1952 Low 0 0 0 Copyright 2011 IRISS Forum Page 35

Table 16: US Non-validation Error Criteria Scores Viewers Sample No. Tested rule Tool A Tool B Tool C US-2811 a-30 0 0 0 US-2818 a-6 0 0 0 US-2823 a-21 0 0 0 US-2839 a-28 0 0 0 US-2841 a-24 0 0 0 US-2842 a-17 0 0 0 US-2843 a-27 0 0 0 US-2844 a-23 0 0 0 US-2870 a-22 0 0 0 US-2871 a-14 0 0 0 US-2877 a-15 0 0 0 US-2878 a-20 0 0 0 US-2884 a-19 0 0 0 US-2909 a-28 0 0 0 US-2910 a-7 0 0 0 US-2911 a-9 0 0 0 US-2912 a-10 0 0 0 US-2913 a-11 0 0 0 US-2914 a-13 0 0 0 US-2915 a-12 0 0 0 US-2916 a-14c 0 0 0 US-2917 a-14b 0 0.5 0 US-2918 a-24b 0 0 0 # Identified w/ Code 0 0 0 # Identified 0 0 0 # Incomplete 0 1 0 Copyright 2011 IRISS Forum Page 38

Sample No. Tested rule Tool A Tool B Tool C # None Identified 23 22 23 Total Score 0 0.5 0 Figure 8: US Non-validation Error Criteria Scores Viewers 1 0.5 0.5 0 0 0 Tool A Tool B Tool C Copyright 2011 IRISS Forum Page 39

Vendor Code Table 17: Analysis by US Validation Category Error Identified Severity Level High Severity Level Medium Severity Level Low Non-validation Error Criteria Overall Error Identified without ICH/FDA code Partial Error Identified No Error Identified Percentage Not Identified (Partial + No Error / Total) Error Identified Error Identified without ICH/FDA code Partial Error Identified No Error Identified Percentage Not Identified (Partial + No Error / Total) Error Identified A 6 0 1 12 68.42% 4 0 1 43 91.67% 2 0 9 16 92.59% 0 0 0 23 100.00% 12 10.26% B 1 0 1 17 94.74% 0 0 0 48 100.00% 0 0 0 27 100.00% 0 0 1 22 100.00% 1 0.85% C 0 3 2 15 85.00% 0 12 0 36 75.00% 0 6 0 21 77.78% 0 0 0 23 100.00% 21 17.95% D 0 3 16 0 84.21% 0 2 35 11 95.83% 0 4 20 3 85.19% 0 1 13 9 95.65% 10 8.55% E 9 0 10 0 52.63% 21 3 24 0 50.00% 12 4 10 1 40.74% 1 2 20 0 86.96% 52 44.44% F 5 10 2 2 21.05% 4 23 21 0 43.75% 5 4 16 2 66.67% 1 4 18 0 78.26% 56 47.86% G 5 10 1 3 21.05% 22 12 3 11 29.17% 10 10 3 4 25.93% 0 6 4 13 73.91% 75 64.10% H 16 2 0 1 5.26% 39 5 0 4 8.33% 16 8 0 3 11.11% 0 8 2 13 65.22% 94 80.34% I 12 2 5 0 26.32% 4 7 36 1 77.08% 6 6 13 2 55.56% 0 0 23 0 100.00% 37 31.62% J 14 5 0 0 0.00% 39 7 2 0 4.17% 19 5 0 3 11.11% 0 18 1 4 21.74% 107 91.45% K 0 17 2 0 10.53% 0 38 10 0 20.83% 0 19 7 1 29.63% 0 11 12 0 52.17% 85 72.65% Error Identified without ICH/FDA code Partial Error Identified No Error Identified Percentage Not Identified (Partial + No Error / Total) Error Identified Error Identified without ICH/FDA code Partial Error Identified No Error Identified Percentage Not Identified (Partial + No Error / Total) Scores (117max.) Percentage Score Copyright 2011 IRISS Forum Page 40

Figure 9: Vendor Tool Comparison of US Severity High Failure Percentage 100.00% 90.00% 80.00% 70.00% 60.00% 50.00% 40.00% 30.00% 20.00% 10.00% 0.00% A B C D E F G H I J K Percent 68.42% 94.74% 85.00% 84.21% 52.63% 21.05% 21.05% 5.26% 26.32% 0.00% 10.53% Copyright 2011 IRISS Forum Page 41

Figure 10: Vendor Tool Comparison of US Severity Medium Failure Percentage 100.00% 90.00% 80.00% 70.00% 60.00% 50.00% 40.00% 30.00% 20.00% 10.00% 0.00% A B C D E F G H I J K Percent 91.67% 100.00% 75.00% 95.83% 50.00% 43.75% 29.17% 8.33% 77.08% 4.17% 20.83% Copyright 2011 IRISS Forum Page 42

Figure 11: Vendor Tool Comparison of US Severity Low Failure Percentage 100.00% 90.00% 80.00% 70.00% 60.00% 50.00% 40.00% 30.00% 20.00% 10.00% 0.00% A B C D E F G H I J K Percent 92.59% 100.00% 77.78% 85.19% 40.74% 66.67% 25.93% 11.11% 55.56% 11.11% 29.63% Copyright 2011 IRISS Forum Page 43

Figure 12: Vendor Tool Comparison of US Non-validation Error Criteria Failure Percentage 100.00% 90.00% 80.00% 70.00% 60.00% 50.00% 40.00% 30.00% 20.00% 10.00% 0.00% A B C D E F G H I J K Percent 100.00% 100.00% 100.00% 95.65% 86.96% 78.26% 73.91% 65.22% 100.00% 21.74% 52.17% Copyright 2011 IRISS Forum Page 44

Implementation of Regulatory Information Submission Standards (IRISS) ectd Tool Interoperability Group (ETIG) ectd Tool Interoperability and Compliance Study 3 (ETICS 3) Contact details Following your review of this report, if you wish to further discuss your individual results or would like to discuss the IRISS ETIG ETICS 3 US Phase 1 in general then please request a meeting with the EU regional lead using the details below. Email: etics-us@iriss-forum.org Copyright 2011 IRISS Forum Page 45