Introducing Functional Qualification

Transcription

1 Introducing Functional Qualification Certess Inc

2 Table of contents Introduction _ 3 Functional verification quality 4 Mutation based testing _ 7 Certitude: Leading functional qualification 8 Bibliography _ 10 Certess Inc

3 Introduction Functional verification is consuming more and more resources in IP design projects. As designs grow in complexity, designers have an increasing need to rely on a dedicated verification team to insure that systems fully meet their specifications. Verification engineers have at their disposal a set of dedicated tools and methodologies for verification automation and quality improvement. In spite of this, functional logic errors are the largest cause of re spins (43% of re spins according to Aart de Geus, Synopsys CEO, 2003). A key reason is that two important aspects of the quality of a verification environment cannot be analyzed or measured: its propagation and detection capabilities. Prior to functional qualification EDA tools were unable to assess how good simulation based functional verification was. Certess has created functional qualification, a new technology that can measure and help improve all aspects of functional verification quality for HDL simulation based functional verification. Certitude, the industry s first functional qualification tool, helps Certess' clients improve and measure the quality of their verification environment. This paper describes the fundamental aspects of functional verification that remain invisible to existing verification tools. It then introduces the origins and the main concepts of a technology that allows this gap to be closed: mutation based testing. It describes how Certess has used this technology to deliver functional qualification. Finally it describes how Certitude, Certess' leading tool, fills the functional verification gap and how it interacts with other verification tools. Certess Inc

4 Functional verification quality Dynamic functional verification is a specific field of competence, with specialized tools, methodologies and measures to manage the verification of increasingly complex sets of features and their interactions. From a project perspective, the main goal of functional verification is to get to market with acceptable quality within given time and resource constraints, while avoiding costly silicon re spins. But functional logic errors are the cause of 43% of re spins. At the start of design projects, once the system specifications are available, a functional testplan is written. From this testplan a verification environment is developed. This environment has to provide the design with the appropriate stimuli and check if the design's behavior matches what is expected. The verification environment is responsible for verifying that a design behaves as specified. Functional verification can thus be decomposed as follows (Figure 1): 1. A testplan defines the functionality to verify. 2. All functional portions of the RTL code specified in the testplan have to be stimulated (activation). 3. The design's response has to be observed (propagation). 4. The observed response has to match the expectations (detection). Figure 1: The four aspects of functional verification If we define a bug as some unexpected behavior of the design, the verification environment has to (1) plan which behavior must be verified, (2) activate this behavior, (3) propagate this Certess Inc

5 behavior to an observation point and (4) detect this behavior as not expected. Therefore, the quality of a functional verification environment is the quality of these four aspects combined. Having perfect activation does not help much if the detection is highly defective. Similarly, a potentially perfect detection will have nothing to detect if no stimulus is propagated. No traditional technology can provide objective information relating to propagation and detection. A simple example will illustrate this: Comment out the checking part of the verification environment and replace this with the equivalent of test=pass. Before Certitude this error could only be found by a manual review of the verification environment. There is a lack of adequate metrics to track the progress of verification. Prior to the introduction of functional qualification no tool provided accurate data to help engineers decide if the performed verification was sufficient. Indeed, when to stop verification is a key challenge of functional verification. This decision is still based on incomplete measures of the functional verification quality. How could a verification team know when to stop when there is no automated measurement and analysis of the propagation and detection part of verification? There is a new class of EDA tools that can solve these difficulties. These are functional qualification tools that use mutation based principles. Mutation based testing allows both improvement and debugging of the checking part of the verification environment and measurement of the verification progress. It measures the propagation and detection abilities of verification environments (Figure 3). Certess Inc

6 Functional Qualification Stimulus RTL Design VHDL/Verilog Ref Model/ checking Testplan Verification environment Figure 2: Functional Qualification Mutation based testing The microelectronic RTL design development chain needs functional qualification to analyze the propagation and detection capacities of verification environments, without which functional verification quality cannot be accurately assessed. The innovation Certess has introduced is the efficient use of mutation based technology. Mutation based testing research originated in 1971 in software research. The fathers of this computer science field are Richard J. Lipton and Richard A. DeMillo (see DeMillo et al, 1979). This technique aims to guide software testing towards the most effective test sets possible. A mutation is an artificial modification in the tested program, induced by a fault operator. Certitude uses the term fault to describe mutations for microelectronic systems. A mutation is a behavioral modification: it changes the behavior of the tested program. The test set is then modified in order to detect this behavior change. When the test set detects all the induced mutations (or kills the mutants in mutation based nomenclature), the test set is said to be mutation adequate. Several theoretical constructs and hypotheses have been defined to support mutation based testing. We call the program modified by a single behavioral change the mutated program. Certess Inc

7 If the [program] contains an error, it is likely that there is a mutant that can only be killed by a testcase that also detects this error (Offutt A.J., 1992) is one of the basic principles of mutation based testing. A test set that is mutation adequate is better at finding bugs than one which is not (Offutt A.J. and Untch R.H., 2000). So mutation based testing has two uses. It can: Assess/measure the effectiveness of a test set, i.e. how good is the test set at finding bugs? Help in the construction/improvement of an effective test set, i.e. what has to be modified in order to have a better test set? Significant research continues to concentrate on the identification of the most effective group of fault types (Mortensen M. and Alexander R.T., 2005). Research also focuses on techniques aimed at optimizing the performance of this testing methodology. Optimization techniques developed include selective mutation (Offut A.J. et all., 1993), randomly selected mutation (Acree A.T. et all., 1979) or constrained mutation (Mathur A.P., 1991). Certitude: Leading the way in functional qualification Using the principles of mutation based testing and the knowledge acquired through years of experimentation in this field and in microelectronic verification, Certess has created Certitude. This patent pending technology is used on numerous functional verification projects of large semiconductor and systems manufacturers. The generic fault model, adapted to microelectronic IP design verification, has been refined by Certess and tested in extreme situations, resulting in the Certitude fault model. Specific performance improvement algorithms have been developed and implemented in Certitude to increase performance when using mutation based methodologies for functional verification improvement and measurement. The basic principle of injecting faults into a design in order to check the quality of certain parts of the verification environment is known to verification engineers. Verifiers occasionally resort to this technique when they have a doubt about their test bench and there is no other way to obtain feedback. In this case of hand crafted mutation based testing, the checking is limited to a very specific area of the verification environment that concerns the verification engineer. Expanding this manual approach beyond a small piece of code would be impractical. By automating this operation, Certitude enables the use of mutation based analysis as an Certess Inc

8 objective and exhaustive way to analyze, measure and improve the quality of functional verification environments for complex IP blocks. Certitude provides detailed information on the propagation and detection capabilities of verification environments, identifying significant weaknesses and bugs that have gone unnoticed by classical coverage. The analysis of the faults that don't propagate or are not detected by the verification environment, points to weaknesses in the stimuli, the observability and the checkers. Certitude is complemented by methodologies that enable users to efficiently locate weaknesses and bugs in the verification environment, as well as accurate data to help correct them. Certitude is easy to use thanks to its HTML report that gives complete and flexible access to all results of the analysis (figure 4). The usability is enhanced by a TCL shell interface. The report in Figure 4 shows where faults have been injected in the HDL code, the status of these faults (not activated, or not propagated, or not detected...) and details about one specific fault. The original HDL code is presented with color highlighted links indicating where faults have been qualified by Certitude. Certess Inc

9 Figure 3. Example of Certitude HTML Report Certitude is tightly integrated with the main industry simulators: Mentor ModelSim, Cadence NC Sim and Synopsys VCS. It does not require modifications in the organization and execution of the user's existing verification environment. Certitude is fully compatible with up to date verification methodologies such as random based stimulus generation and PSL assertions. Certitude can be used with any verification environment. Easy to integrate in existing functional verification flows, Certitude functional qualification is the only technology that can assess and identify how good the verification environment is for activation, propagation and detection of errors. Certess Inc

10 Bibliography A.T. Acree, T.A. Budd, R.A. DeMillo, R.J. Lipton, and F.G. Sayward. Mutation Analysis. Technical Report GIT ICS 79/08, Georgia Institute of Technology, Atlanta GA, R.A. DeMillo, R.J. Lipton and F.J. Sayward. Hints on Test Data Selection: Help for the Practicing Programmer. IEEE Computer, 11(4): p , A.P. Mathur. Performance, Effectiveness and Reliability Issues in Software Testing. In 15th Annual International Computer Software and Applications Conference, p , Tokyo, Japan, M. Mortensen and R.T. Alexander. An Approach for Adequate Testing of AspectJ Programs Workshop on Testing Aspect Oriented Programs (held in conjunction with AOSD 2005), 2005 A.J. Offutt. Inverstigations of the Software Testing Coupling Effect. ACM Trans Soft Eng and Meth, Vol. 1, No 1, p. 5 20, A.J. Offutt, G. Rothermel and C. Zapf. An Experimental Evaluation of Selective Mutation. In 15th International Conference on Software Engineering, p , Baltimore, MD, A.J. Offutt and R.H. Untch. Mutation 2000: Uniting the Orthogonal. Mutation 2000: Mutation Testing in the Twentieth and the Twenty First Centuries, p , San Jose, CA, Certess Inc