Lies, Damned Lies and Hardware Verification. Mike Bartley, Test and Verification Solutions

Transcription

1 Lies, Damned Lies and Hardware Verification Mike Bartley, Test and Verification Solutions

2 Myth 1: Half of all chip developments require a re-spin, three quarters due to functional bugs Source? The 2004/2002 IC/ASIC Functional Verification Study by Collett International Research But it is misquoted! The percentages add to more than 100% Because the survey shows what was being fixed during the respin rather than the cause of the respin It actually shows that 71% of respins involve a fix to a functional bug rather than being caused by one

3 Myth 1: Half of all chip developments require a re-spin, three quarters due to functional bugs Issues Found on First Spin ICs/ASICs 90nm % 10% 20% 30% 40% 50% Functional Logic Error Analog Tuning Issue Signal Integrity Issue Clock Scheme Error Reliability Issue Mixed Signal Problem Too Much Power Has Path(s) Too Slow Has Path(s) Too Fast IR Drop Issues Firmware Error Other 20% 17% 14% 12% 11% 11% 10% 10% 7% 4% 3% Source: Aart de Geus, Chairman and CEO of Synopsys. Based on a survey of 2000 users by Synopsys 43% Fnal AMS 20% 21% 22% 23% 26% 17% Functional and AMS bugs are leading causes of silicon respins Source = Global IC (ASSP/ASIC) Service Management Report 2007, IBS Functional bugs ARE the main cause for re-spin But they do not cause 75% of re-spins

4 Myth 2: Verification takes 70% of the design cycle Source unknown An EDA sales person?? 2004 EE Times survey for IC 662 respondents to the chip design survey 47% design engineers, 28% engineering managers, 7% verification engineers, 5% CAD managers + others system-level design = 13% logic design = 20% functional verification = 22% synthesis = 8% IC place and route = 13% IC physical verification and analysis =11% analog/mixed-signal design = 13%.

5 Myth 2: Verification takes 70% of the design cycle Source: EE Times 2006 EDA Users Survey

6 Myth 2: Verification takes 70% of the design cycle Cost ($M) IC Design Costs Prototype Validation Physical Verification Architecture µm 0.13µm 90nm Silicon Feature Dimension Source: IBM/IBS, Inc.

7 Myth 2: Verification takes 70% of the design cycle Can we explain the 70% claim? Performing verification during 70% of the design cycle? If we consider just the RTL design phase then the survey suggests it is over 50% but still not up to 70%. What is verification? For example, does it include Debug? HW/SW co-verification? Prototyping? Is it just functional or does it include all verification? (incl physical) Is it just sometimes true? Does it just help sell more verification tools? What can we learn? For functional verification the consensus is about 30% - but it varies The trend is upwards!

8 Myth 3: The verification challenge is growing at a double exponential rate Source Various e.g. Is Methodology the Highway Out of Verification Hell? debated the following at DAC2005 many contend that the verification challenge is growing at a double exponential rate, Is verification related to design size? Does increased memory size increase verification effort? Is an 8 bit adder more complex to verify than a 4 bit adder? How many gates can we design a day? Verify a day? (note size may matter in formal)

9 Myth 3: The verification challenge is growing at a double exponential rate The verification challenge is more closely related to complexity But how do we measure complexity? The software world has quite a few metrics here! Function point analysis, cyclometric complexity What metrics do we have in hardware? Size of spec? KLOC? Number of coverage points? The way out of the problem Verification IP Verification methodologies to facilitate re-use Improved verification techniques

10 Myth 4: Pseudo-random simulation has killed off directed testing Coverage Source Vendors of pseudo-random simulation tools?? Comparing the two techniques How to determine pass/fail Pseudo-random test benches must be self-checking Directed can exploit that they know the expected result Some scenarios are hard/inefficient to Describe by coverage Hit in pseudo-random simulation Key Pseudo Directed Pseudo-random testing requires more simulations to hit the coverage Over time pseudo-random test generation is more efficient Pseudo-random hits many unexpected scenarios

11 Myth 4: Pseudo-random simulation has killed off directed testing Choosing between the two techniques Availability of an oracle Design complexity (again!) The economics of bugs (cheaper to fix the earlier they are found) The size of the state space Simulation times Technology Hardware/Software co-verification Legacy Inertia Reuse during Silicon bring-up Infrastructure Skills

12 Myth 5: Formal verification will eventually replace simulation Source Academia (and a few CAD companies) Comparing the two techniques Realism Cost People Tool Capacity Transparency and Confidence Continuous Results Incrementability and Regression Bottom-Up and Top-Down

13 Myth 5: Formal verification will eventually replace simulation Combining the two techniques Quite often there are separate formal and simulation teams This can create cost choices How do we combine the two into an effective solution? Given a proven property what simulations can I skip? Is this the right question? We need to be more tactical? Formal search from a complex scenario or previous bug Generate simulations from formal tools The tool vendors seem to be moving this way

14 Further Myths We don t need both functional coverage and structural coverage let s pick one They are complimentary use both Others assertion coverage, formal verification coverage metrics Our bug discovery rate has dropped we are good to ship Bug discovery rate can demonstrate the lack of maturity Be careful if using this metric to make a positive release decision It is a necessary but not a sufficient condition Our bug discovery rate has dropped AND we have high coverage we are good to ship How good is your test bench? stimulus, error propagation, detection Test bench qualification through mutation testing?

15 Further Myths Trust our design IP it is fully verified Audit your supplier! How mature is the IP? What defects have been detected? Overall verification strategy and its execution history What verification support is there with the IP? How many working chips using the IP? And non-working chips? Are my particular configuration options been proven in silicon? How many changes since the last successful chip? Verify the IP in the context of your chip? Is design re-use compounding the verification bottleneck? Can we get IP re-use without re-verification? Fight fire with fire Get a discount per bug found! Tell your supplier the cheque is in the post

16 Conclusion Beware sales people bearing statistics The relative verification effort IS increasing but not 70% in general Design IP re-use is not completely cost free the re-use cost is usually higher for the verification team Decisions, decisions (Directed vs. pseudo-random vs. formal) X (Block vs. Chip) Metrics-driven verification Verification teams already manage a wide variety of data structural coverage, functional coverage, bug statistics etc. one of the main skills is to interpret this data and make a recommendation for a release or tape-out decision the amount and diversity of the data will increase

17 Acknowledgements and Q&A Acknowledgements Shalom Bresticker, Intel Darren Galpin, Serrie Chapman and Tim Blackmore, Infineon Questions?