Interpretation von Software Qualitätsmetriken aus automatisierter statischer Analyse

Transcription

1 Interpretation von Software Qualitätsmetriken aus automatisierter statischer Analyse Institut für Computertechnik ICT Institute of Computer Technology Andreas Gerstinger IIR Konferenz Software Testen & Software Qualität , Holiday Inn Vienna South

2 I will try to answer the following questions Agenda Why a static analysis tool? What is internal and external quality? What are the requirements for a suitable tool? How can I find out which metrics are relevant? Which metrics really are relevant? Institut für Computertechnik 2

3 Motivation Software Maintenance is a major cost factor "80% of the lifetime cost of a piece of software goes to maintenance." (first sentence in Sun Java Code Conventions) streamlining maintenance is essential Practical experience also shows that most SW Engineering work is maintenance sometimes very, very, very old code... Institut für Computertechnik 3

4 Static Analysis is required in standards More Motivation mandated by several international standards (especially in the safety critical systems community) Static Anlaysis streamlines code reviews if performed before the review, it shows where most problems are expected Static Analysis useful for SWQM personell gives quick overview and enables a preliminary assessment, also for subcontracted code Institut für Computertechnik 4

5 Maintenance Efficiency Concerns all maintenance types corrective, perfective, adaptive, preventive Mainteance efforts consist of: reading code understanding code finding places where to edit bug tracking code analysis improvements Institut für Computertechnik 5

6 Internal vs. External Quality Static Analysis measures the internal quality of a software system it is a white-box analysis, opposed to black-box testing Input? Output Internal Quality often correlates with external quality Weak internal quality can cause bad external quality the system is perceived as having "bad quality" by the users! Institut für Computertechnik 6

7 Requirements for the Tool The objective was to reduce number of tools, and to ideally introduce one standard tool for the company The following requirements were established. The tool shall: support all relevant platforms (Win/Linux) support all relevant languages (C/C++/Java/C#) be easy to use have a GUI and command line interface be extensible with custom rules (to adapt to company coding guidelines) not force developer to change good code so the tool can run output human readable and automatically analysable calculate useful metrics automatically verify as many of the Coding Guidelines as possible Institut für Computertechnik 7

8 Which rules are good? What are the desired ranges for metrics? Which metric really influences maintainability? Problems There are essentially three methods to set limits: (A) Take limits from literature (CC<10, one return point, no goto,...) (B) Measure many projects and take mean as a guideline (C) Correlate metrics with qualified opinion of expert developers Method C selected Institut für Computertechnik 8

9 Outline of Method 1. Create a questionnaire with relevant questions regarding software quality and get answers from expert developers for various software packages they work with 2. Automatically measure potentially interesting metrics of the software packages 3. Correlate questionnaire responses with the measured metrics to find out which metric correlates with which property Institut für Computertechnik 9

10 Step 1: Questionnaire Questionnaire based on ISO9126 Quality Model external and internal quality functionality reliability usability efficiency maintainability portability suitability accuracy interoperability security functionality compliance maturity fault tolerance recoverability reliability compliance understandability learnability operability attractiveness usability compliance time behaviour resource utilisation efficiency compliance analysability changeability stability testability maintainability compliance adapatability installability co-existence replaceability portability compliance Institut für Computertechnik 10

11 Step 1: Questionnaire Consists of 12 questions, to be answered with Austrian school grades 1 best 5 worst Very easy to understand and quick to get responses Four questions about external quality, eight questions about internal quality Institut für Computertechnik 11

12 Step 1: Questionnaire Some questions regarding external quality Question Explanation Grade How would you rate the overall externally perceived quality of the system? How would you rate the reliability of the system? This includes all externally visible properties such as functionality, reliability, usability, efficiency,... Reliability is "the capability of the software product to maintain a specified level of performance when used under specified conditions". For your answer, consider unexpected crashes, the tolerance of the software to internal and external faults, the recoverability from faults, the perceived stability, and the number of faults during operation. Institut für Computertechnik 12

13 Step 1: Questionnaire Some questions regarding internal quality Question Explanation Grade How would you rate the overall internal quality of the software? This question is geared towards the overall maintainability and technical "beauty" of the internal quality. How would you rate the code readability based on code clarity? Is the code (not the comments) easy to read? Are variable/function/class names clear? Is the code well structured? Can you read the code "like a book"? Institut für Computertechnik 13

14 Step 2: Automated Static Analysis Analysis limited to C# Code All code in one directory, each module in one subdirectory Perl script running over all subdirectories Execution of static analysis tool Extraction of potential quality indicators from static analysis output files Process automated easy to add more code Institut für Computertechnik 14

15 Step 2: Automated Static Analysis Some Metrics that were measured: Average Cyclomatic Complexity (CC) per method Average Cyclomatic Complexity excluding methods with CC=1 Average Cyclomatic Complexity excluding methods with CC=1 and CC=2 Percentage of methods with Cyclomatic Complexity greater than 3 Percentage of methods with Cyclomatic Complexity greater than 5 Percentage of methods with Cyclomatic Complexity greater than 10 Percentage of methods with Cyclomatic Complexity greater than 15 Percentage of methods with Cyclomatic Complexity greater than 30 Average Class Size (in effective Lines of Code (elocs)) Average Method Size (elocs) Average number of public attributes per class Average number of comment lines per method Average number of return points per method Number of quality notices per KLOC (Kilo LOC) Number of severe quality notices per KLOC Number of quality notices pertaining to readability per KLOC Average file size (elocs) Ratio of comment lines to total lines Average interface complexity Institut für Computertechnik 15

16 Step 3: Correlation Correlation between questionnaire reponses and potential quality indicators was analysed XY Graphs to see correlation Trendlines added Strength of correlation was calculated Which metrics are relevant? Which are not? Let's look at some graphs! Institut für Computertechnik 16

17 Graph 1: Readability vs. Comment Density CommentLines / TotalLines Readability based on Comments (1-best, 5-worst) Institut für Computertechnik 17

18 Graph 2: Code Clarity vs. Quality Notices Severe Quality Notices / KLOC Code Clarity (1-best, 5-worst) Institut für Computertechnik 18

19 Graph 3: Code Clarity vs. Return Points Average number or return points Code Clarity (1-best, 5-worst) Institut für Computertechnik 19

20 Graph 4: Internal Quality vs. CC 6 5 Average Cyclomatic Complexity General Internal Quality (1-best, 5-worst) Institut für Computertechnik 20

21 Graph 5: Internal Quality vs. Method Size Average Method Size (elocs) General Internal Quality (1-best, 5-worst) Institut für Computertechnik 21

22 Graph 6: Internal Quality vs. File Size Average File Size (elocs) General Internal Quality (1-best, 5-worst) Institut für Computertechnik 22

23 Summary of Results Strongest correlation with perceived internal quality: Comment density Control Flow Anomalies No correlation with perceived internal quality: Cyclomatic Complexity Average Method Size Average File Size... Institut für Computertechnik 23

24 Conclusions Static Analysis is essential for software quality Tool must be easy to use, and must give immediate feedback to developers Correlation of metrics with perceived quality gives surprising results But: "Correlation does not imply causation!" There is still a lot of work to be done Thank you! Institut für Computertechnik 24