ERAU the FAA Research CEH Tools Qualification

Transcription

1 ERAU the FAA Research CEH Tools Qualification Contract DTFACT-07-C Dr. Andrew J. Kornecki, Dr. Brian Butka Embry Riddle Aeronautical University Dr. Janusz Zalewski Florida Gulf Coast University FAA SW&CEH Conference Denver, CO August 20, 2008 ERAU College of Engineering page 1

2 Outline Contract DTFACT-07-C A Study on Tool Qualification for Complex Electronic Hardware, Mar 15, 2007 May 14, 2009 o Introduction o CEH Tools and Development Process o Tool Qualification o Concerns and Possible Solutions o Conclusions and Future Work ERAU College of Engineering page 2

3 Introduction: CEH Tools Project ERAU/FGCU Project: A Study on Tool Qualification for Complex Electronic Hardware with an objective to produce report on the state of art in the CEH tool market and the programmable logic tool qualification issues Explore safety issues in the assessment and qualification of tools used in developing complex electronic hardware (CEH) for the aircraft Typical devices are programmable logic devices (PAL, PLA, GAL, FPGA, ASIC, etc.) used as components of programmable electronic hardware RTCA DO-254, Design Assurance Guidance for Airborne Electronic Hardware, Section 11.4, Tool Assessment and Qualification is the starting point ERAU College of Engineering page 3

4 Literature Study Perspectives: o A broad view of the issues in designing CEH (26) o A focus on safety issues in avionics applications related to safety critical aspects of CEH (6) o An industry practices in qualification of CEH tools and compliance with DO-254 standard (18) Literature review points: o Plan to develop and verify PLD programs in the same way as software programs o Plan the safety argument from the start, and build up evidence throughout development o Use mature tools, amenable to qualification and supported throughout the project life time o Investigate the use of formal notations and analysis techniques to increase verifiability o Do not use programmable logic hardware just to avoid developing safety-critical software ERAU College of Engineering page 4

5 What is a tool? RTCA DO-254/ED-80 provides guidance for design assurance of airborne electronic hardware defining design assurance, lifecycle, processes (planning, design, V&V, CM, assurance, certification liaison), and the lifecycle data No clear definition of a tool in the RTCA DO-254 or associated CAST papers; paraphrasing DO-178B: A computer program or a hardware device used to help develop, test, analyze, produce or modify hardware component, subsystem, system or its documentation A tool reduces, eliminates, or automates the objectives of the design or verification process ERAU College of Engineering page 5

6 RTCA DO-254 DO-254 distinguishes between two primary types of tools: o Design Tools - Tools whose output is part of hardware design and thus can introduce errors. For example, an ASIC router or a tool that creates a board or chip layout based on a schematic or other detailed requirement (used to generate the hardware item or the hardware design, thus an error in the tool could introduce an error in the hardware item) o Verification Tools - Tools used to ensure performance against predetermined standards or requirements. These tools do not introduce errors, but may fail to detect them. For example, an analog or digital circuit simulator or an automated test that measures actual circuit performance. (used to verify the hardware item, an error in the tool may cause the tool to fail to detect an error in the hardware item or hardware design) ERAU College of Engineering page 6

7 CEH Tools Issues ASIC/FPGA tools have a thick layer of abstraction between the user Verilog or VHDL input and the tools output: the tool interprets the input, synthesizes/optimizes the logic, creates net lists, and translates the net list into a hardware layout Synthesis includes typically the optimization of logic, timing, and power aspects - a black box, with design as an input and synthesized design as an output FPGA/CPLD vendors are interested in developing software required to take a design (schematic or HDL) into a form that can be used to program a circuit they offer cost-effective entry-level design environments (complete packages with design entry, simulators, libraries, and the back end) Tool industry is dynamic and hardware keeps evolving, the software developers for back-end tools have to attend to two primary activities, developing libraries for new EDA tools and simulators, while creating better fitters and routers for new hardware with more resources and more complex architectures Evolving interchange standards such as EDIF and Verilog/VHDL help standardize interfaces to CAD tools and simulators in form of EDIF-compatible library of design elements using Verilog or VHDL to implement the models necessary for simulation environment ERAU College of Engineering page 7

8 1. Identify the tool yes no 2. Identify the process the tool supports 3. Independent assessment? no 4. Tool is design A/B/C or verification A/B? DO-254/ED-80 Tools Assessment and Qualification Process yes yes 5. Relevant tool history? no 6. Establish qualification baseline and problem reporting 7. Basic tool qualification 10. Complete the process 9. Design tool qualification ERAU College of Engineering page 8 yes 8. Tool is design tool A/B? no

9 Is Tool Qualification Required? A company working on a design assurance level A project writes a test bench that runs on a simulator and produces a pass/fail output o The test bench automates the verification process and is therefore a tool and thus must be assessed and qualified Can relevant product service history get us out of tool qualification? o Many design tool suites have hundreds of users doing diverse designs o Relevant service history requires the tracking of problem (bug) reports and their resolution Traceability data for all functional failure paths It is rare that all of the necessary data is available ERAU College of Engineering page 9

10 Is Tool Qualification Required? (2) The other way to avoid tool qualification is if the tool outputs are independently assessed From DO-254: Independent assessment of a design tool s output that is generated in whole or in part by the tool may be established by the verification activities performed on the item, such as component, netlist or assembly. In this case, the integrity of the end item does not depend upon the correctness of the design tool output alone ERAU College of Engineering page 10

11 Is Tool Qualification Required? (3) The correctness of the design tool can be assessed by verification tools and in-system hardware verification tests that are run o Conventional debugging of the hardware using logic analyzers etc. will also assess the correctness of the design tool It seems that design tools would rarely need to be qualified since it is unlikely that the output of the design tool is used without being verified both in simulation and in hardware ERAU College of Engineering page 11

12 CEH: Logic Design Activities Design Entry: HDL, schematic entry, integration of IP cores Synthesis: translation HDL design definition into the logical or physical representation for specific hardware platform Implementation & Configuration: assignment of the logic created during design entry and synthesis into specific physical resources of the target device. Verification: design verification ranging from simulation to static timing analysis to equivalency checking via formal verification. Board Level Integration: compatibility of programmable logic design with the entire system. ERAU College of Engineering page 12

13 Generic Design Flow Graph Design Entry Behavioral Simulation Synthesis Time Analysis Other Verification Place & Route Power Analysis Time Simulation Programming Configuration Functional Simulation ERAU College of Engineering page 13

14 The Concern It is possible for errors to be introduced that occur while translating the simulated design into hardware Normal verification techniques will catch most of these errors during hardware verification But some circuits are designed to operate only if the hardware is malfunctioning Examples are: o Triply redundant hardware with voting circuits Used to mitigate failure in one path o Metastability detection circuits Can be used to detect single-event upsets due to alpha particle radiation These circuits are difficult to verify in working hardware ERAU College of Engineering page 14

15 Example: Triply Redundant Hardware The synthesis tool can optimize the design in an unexpected fashion In A In B Intended Implemented ERAU College of Engineering page 15

16 Example: Metastability Sensing Circuits It is possible to detect metastability by monitoring the Q and Qbar outputs of a single flip-flop Intended Implemented ERAU College of Engineering page 16

17 A Problem The redundancy and metastability problems occur when: o design optimizations are applied during synthesis, and o the tool would not generate explicit warnings that the optimization had occurred The circuits should not operate if the hardware is functioning normally, thus error like this may not be detected during verification of the hardware ERAU College of Engineering page 17

18 Possible Solutions Since the problems are caused by the synthesizer performing optimizations, we could turn off all synthesizer optimizations o It is a possible solution, but it could be difficult to meet timing and area constraints without optimizations An experienced designer would recognize these conditions ahead of time and configure the synthesizer appropriately o Possible but difficult to verify whether or not the designer has handled all possible areas of concern ERAU College of Engineering page 18

19 Possible Solutions (2) Catch the errors in hardware verification o Add additional test circuitry to allow verification of all circuits Difficult to anticipate early in the design so that it can be written into the requirements o Add additional verification tests to verify circuit operation If the additional circuitry is in place to detect single-event upsets due to alpha particle radiation we could add radiation susceptibility tests No current standard for alpha particle radiation testing. Should it be in DO-160 rather than handled in DO-254? ERAU College of Engineering page 19

20 Possible Solutions (3) The problems introduced by the synthesizer could be detected by running simulations using the gate level netlist with back annotated timing data o Running the full test suite on a gate level netlist is feasible but it will be very slow and expensive However DO-254 guidance does not require to do this: o Analyzing the implementation below the level of that specified by the designer, such as at the gate or transistor level, is not necessary. (DO-254, Appendix B page B-9) o The designer worked at the HDL level not the gate level ERAU College of Engineering page 20

21 Other Likely Problems At DAL level A and B elemental analysis is used to assure that all functional failure paths have been covered However, some failure mechanisms can exist which occur at conditions that may not be identified as functional failure paths Some failures are related to where within the FPGA the actual hardware is located during the place and route phase ERAU College of Engineering page 21

22 Non-Functional Failure Path Concerns Two problems can be caused if large numbers of devices within the FPGA connected to the same internal FPGA power supply switch at the same time Resistance in the internal FPGA power bussing can cause the voltage provided to circuits in a particular location to drop Inductance in the FPGA power bussing can create Simultaneous Switching Noise if many outputs switch simultaneously Both mechanisms can cause errors directly or change the circuit timing leading to errors elsewhere ERAU College of Engineering page 22

23 Non-Functional Failure Path Concerns (2) Much of the information needed to analyze these conditions is proprietary to the FPGA vendor Vendor supplied FPGA tools are the best at addressing these issues However our survey indicates that the majority of designs are implemented using third party tools such as Mentor Graphics and Synplicity ERAU College of Engineering page 23

24 Industry Survey - Methodology Questionnaire distributed during the 2007 FAA SW&CEH Conference, New Orleans, LA, July 24-26, 2007 on a dedicated CEH session attended by 54 individuals interested in the CEH and the application of DO-254 Posting the questionnaire on the web and linking from website Follow-up mailing to over 150 individuals engaged in development of the aviation software and hardware The questionnaire distributed internally in companies engaged in the design of programmable logic devices Collection of feedback at the Programmable Logic User Group meeting in Clearwater, FL, November 15, 2007 As a result we obtained a sample of 45 recorded responses ERAU College of Engineering page 24

25 Industry Survey - Qualification Only eight out of 45 respondents indicated that tool qualification was attempted: The responses were inconsistent and incomplete; here is not verified list: o Aldec level A o Synplify level C, level A o Questa level A o ModelSim level A o Actel Libero level A Two respondents listed several tools but without clear statement if they were actually qualified and without specifying the qualification level/type (qualification activities have been performed and documented but never reviewed or challenged by the FAA) ERAU College of Engineering page 25

26 Type of Devices Survey: Tool Landscape Vendors of Hardware Devices pla/pal/gal 24% fpga 32% xilinx 24% actel 20% atmel 6% soc 8% cpld 18% asic 18% quicklog 5% lattice 12% cypress 11% altera 22% Vendors of Software Tools Errors Encountered others 18% synopsis 19% cadence 8% aldec 9% mentor 27% synplicity 19% no 52% yes 48% ERAU College of Engineering page 26

27 Survey: Population Educational Background Type of Work SW 17% proj.mgt 25% tools 3% components 21% HW 83% verification 15% application 36% Years of Expertise Work on DO-254 Projects <3 years 12% 3-6 years 19% yes 33% >12 years 50% 7-12 years 19% no 67% ERAU College of Engineering page 27

28 Industry Survey Selected Comments We qualified a tool by comparing a logic analyzer trace from the actual system with a simulation from the tool (including propagations delays and functionality). The DO-254 guidance is not useful and does not include what should be done with the data. Most of the tools are widely in use and do not have a negative safety impact. Mainly the repeatability and verification of the design itself affects safety All companies are attempting qualification differently and we have not seen any increase in safety by doing a qualification. These tools are in use in thousands of designs in the world and the probability of a tool qualification effort finding an error is very low or negligible The tools are designed for rapid freeform development and not for requirements based methodical design. Fine for commercial applications but not really targeted to high reliability design. Qualification is basically demonstrating the pedigree of the tool itself, most vendors either do not keep requirements based design and verification information or that information is considered proprietary or trades secret. Thus independent assessment of tool output will continue to be the primary, and sometimes only, methodology available for qualification. As long as that fact doesn t change, there will be very little progress Not having specific guidelines, there is too much uncertainty on what needs to be done to assess or qualify the tools The issue is about the user competence about the task being performed - do they know how to appropriately apply the tool? Guidance is extremely vague on the objectives of qualification. The use of the word "tool" is somewhat misleading. How the tool is applied is sometimes more of a risk than the tool itself. E.g. if a verification environment consists of a simulation testbench running on Modelsim, the tool itself should be less of an issue for assessment/qualification than the user developed simulation testbench ERAU College of Engineering page 28

29 Observations Vendor supplied FPGA tools are best at identifying failures that require knowledge of the FPGA internals Vendor supplied design tools are not commonly used design tools Vendor supplied tools often cannot close timing on designs that are easily handled by third party tools Vendor supplied tools can take advantage of detailed knowledge of the hardware to implement unique features ERAU College of Engineering page 29

30 Summary For many designs normal the normal verification process activities are sufficient to independently assess the tool outputs But some design problems are difficult to assess and require extra effort to identify and address o Requires knowledgeable designers and DERs o The risk is people do not know what they do not know DAL level A FPGA design and verification requires an experienced team of design and verification engineers Gate level verification of the design appears to be necessary ERAU College of Engineering page 30

31 Moving Forward We are executing several test cases to investigate issues including: o Non Functional Failure Path issues such as Simultaneous Switching Noise o Comparing tool reported device timing to actual measured timing to evaluate design robustness o Un-programmed circuitry operation: observe whether or not un-programmed logic blocks switch during normal device operation and what are the implications with respect to power supply and signal integrity ERAU College of Engineering page 31

32 Questions? Questions, comments, suggestions? Andrew J. Kornecki Brian Butka Janusz Zalewski ERAU College of Engineering page 32