A Case History of Process Improvements at the NASA Software Engineering Laboratory. SEMATECH Technology Transfer A-TR

Transcription

1 A Case History of Process Improvements at the NASA Software Engineering Laboratory Technology Transfer A-TR

2 and the logo are registered service marks of, Inc. Ada is a registered trademark of the U.S. Department of Defense, Ada Joint Program Office. 1995, Inc.

3 A Case History of Process Improvements at the NASA Software Engineering Laboratory January 31, 1995 Abstract: This document describes the software improvement efforts of the Software Engineering Laboratory (SEL) of the National Aeronautics and Space Administration (NASA) Goddard Space Flight Center (GSFC). The SEL follows a structured improvement process of 1) understanding the current situation, 2) assessing the impact of a process changes, and 3) packaging and deploying the resulting process and lessons learned. The three-step process results in significant increases in software reuse and cost predictability, and impressive reductions in error rates, software development costs, and cost per line of code. This study shows that the SEL s approach can be modified for use by software developers in semiconductor manufacturing companies and suppliers. Keywords: Software Development, Software Reliability, Software Reuse, Manufacturing Systems, CASE, Object Oriented Systems Approvals: Bob Flegal, Program Manager Harvey Wohlwend, Project Manager Dan McGowan, Technical Information Transfer Team Leader

4

5 iii Table of Contents 1 EXECUTIVE SUMMARY INTRODUCTION THE NASA SOFTWARE ENGINEERING LABORATORY Background Overall SPI Results Process Improvement Approach Process Improvement Areas Explored Baselining SEL Software Performance Examples of Improvement Initiatives The Cleanroom Software Engineering (CSE) Method Object Oriented Technology (OOT) OVERALL MEASURED IMPROVEMENTS RESULTING FROM SPI IMPLICATIONS FOR IMPROVEMENT IN SEMICONDUCTOR FAB EQUIPMENT SOFTWARE CONCLUSIONS REFERENCES GLOSSARY OF TERMS...25

6 iv List of Figures Figure 1 NASA SEL Organization...3 Figure 2 Internal Organization of SEL...4 Figure 3 The SEL SPI Method...6 Figure 4 Areas of Improvement Emphasis over the Years...8 Figure 5 Language Usage, Effort Distribution, and Error Characterizations ( )...10 Figure 6 Error Detection Data ( )...11 Figure 7 Software Growth Profile ( )...11 Figure 8 Cleanroom Process Model...13 Figure 9 Impact of the Cleanroom Experiments on Change Rate, Reliability and Productivity...14 Figure 10 Reuse Improvements Resulting from OOT...16 Figure 11 OOT Impacts On Three Multimission Simulator Projects...17 Figure 12 Development Error Rate Trend...18 Figure 13 Development Error Rate Improvements...19 Figure 14 Software Development Effort/Cost Improvements...19 Figure 15 Reuse Improvements...20

7 Acknowledgements The author is extremely grateful to Frank McGarry, Computer Sciences Corporation (formerly of NASA) for access to and permission to use case study data and other related information described in this report. v

8

9 1 1 EXECUTIVE SUMMARY This case study examines as a benchmark software improvement organization the Software Engineering Laboratory (SEL) at the National Aeronautics and Space Administration (NASA) Goddard Space Flight Center (GSFC). Since 1976, the SEL has dedicated itself to understanding and improving the process of developing, managing, and maintaining software supporting all elements of flight dynamics systems. The major features of its software improvement program are as follows: Dual product and process improvement emphasis Measurement focus Empirical experiments to test potential impacts Packaged transfer and deployment of practices and lessons learned The SEL process improvement method consists of three steps, each characterized by extensive measurement. The steps are: Understanding the current situation Assessing the impact of a process change Packaging and deploying the resultant process and lessons learned In the early years, SEL pursued specific smaller-grained techniques, but over the past eight to ten years, the laboratory has focused on larger, more significant process changes. These have included the use of Ada, Cleanroom Software Engineering Method, object-oriented technology (OOT), and computer-aided software engineering (CASE), with each targeted to specific aspects of software process enhancements. Two examples of specific improvement efforts (Cleanroom and OOT) are described to show the impact on software processes and products. The long-term results of the SEL s software process improvement (SPI) program are impressive, showing quantitative evidence of a gradually maturing process throughout the organization. Based on a recent comparison of two consecutive, four-year baseline periods over the last eight years, SEL's SPI program has achieved the following measured results: Error rates during development decreased 75% from 85 to 93. Average software development costs to support a mission decreased by 55%. Average reuse increased by 300%. Average cost to develop a line of code decreased by 10% from $ 0.32 to $ Costs have become more predictable. For its pioneering vision of developing a measurement-based SPI program and its continuing efforts to communicate its successes and failures to the software engineering community, the SEL in 1994 received the first Institute of Electrical and Electronics Engineers (IEEE) Computer Society Award for Software Process Achievement. Many software organizations within and outside NASA have adopted SEL process handbooks. More significantly, software organizations throughout the contractor and civil service communities have adopted SEL concepts; examples will be provided in this report. This case study also will present lessons learned; the benefits of initiating an SPI program; and encouragement for semiconductor manufacturing companies starting an SPI program.

10 2 2 INTRODUCTION s earlier case study examined the Space Shuttle s on-board software supplier as a benchmark development organization that produced highly reliable software within a context of embedded control (see A Case History of the Space Shuttle Onboard Systems Project, Technology Transfer # A-TR). This second case study aims to interest executives of equipment supplier companies in the software issue. It also identifies types of software practices that enable the production of high quality, highly reliable software. The authors chose NASA SEL for a case study because that organization: Was able to show specific, measured benefits from its SPI efforts, including improved operational reliability and quality, reduced defect density and cycle time, and improved project performance and predictability Had collected data and lessons learned that documented explicit behavior changes as a result of SPI activities Allowed timely access to its data, people, and projects Was willing to review and approve this report for dissemination throughout the community Was developing applications in real-time control systems whose technical and complexity issues were similar to those faced in semiconductor fab integration and process control Effectively communicated its improvement efforts and underlying motivations in terms that could be understood by semiconductor manufacturers and suppliers is seeking to study any semiconductor equipment supplier whose SPI program satisfies the above criteria, with the goal of publishing additional case studies.

11 3 3 THE NASA SOFTWARE ENGINEERING LABORATORY 3.1 Background Since 1976, the SEL has been dedicated to understanding and improving the process of developing, managing, and maintaining software supporting all elements of flight dynamics for GSFC flight projects. As illustrated in Figure 1, the SEL supports the Flight Dynamics department of Goddard s Mission Operations and Data Systems. NASA Goddard Space Flight Center ($2.4 Billion budget) Mission Operations & Data Systems ($500 Million budget) Flight Dynamics ($50 Million budget) Software Engineering Laboratory Figure 1 NASA SEL Organization The SEL currently includes over 300 staff members from three organizations: the GSFC, the University of Maryland, and Computer Sciences Corporation (CSC). Approximately 275 to 300 people are responsible for developing and maintaining flight dynamics systems, while three to six people maintain SEL's historical data, and 10 to 15 people directly support its SPI program. Figure 2 shows the key characteristics of each function.

12 4 Software Developers (NASA+CSC) Staff: Typical project size: KSLOC Active projects: 6-10 Project staff size: 5-25 Total projects: 120 ( 76-94) Process Analysts & Improvers (NASA+CSC+U.MD) Staff: Functions: set goals, run studies, research, analysis, refine SW process, report findings Products: 300 reports Staff: 3-6 Functions: maintain and assure historical data (SEL DB, Forms library, reports library) Database Support (NASA+CSC) Figure 2 Internal Organization of SEL This report focuses on the efforts of the process improvement staff and their impact on the developers through improved practices. The database is crucial to factually reporting improvement results. The SEL's dual emphasis on improving both software processes and products simultaneously is key to its successful results. 3.2 Overall SPI Results The long-term effects of the SEL s SPI program are impressive. Data shows a gradually maturing process within the organization as a whole from 1985 to 1993, including a 10% decrease (from 32 cents to 28 cents) in the cost of producing a new line of code and a 75% decrease in error rates. Based on a recent comparison of two consecutive, four-year baseline periods, this SPI program also has demonstrated the following results: Average software development costs to support a mission decreased by 55%. Average reuse increased by 300%. Ability to predict costs improved. For its pioneering vision in developing a measurement-based software process improvement program and continuing efforts to communicate its successes and failures to the software engineering community, the SEL in 1994 received the first IEEE Computer Society Award for

13 Software Process Achievement. This is significant because the criteria and selection process for this award provide a model of excellence that could benefit the semiconductor software industry [PAAC, 1994]. The award is given to a software professional or team responsible for significant, measured and sustained improvement to their organization's software process. Significant means that improvements must have a demonstrated impact on the organization's performance or the practice of software engineering. Sustained means that the effort must have resulted in a broad, documented improvement that impacts current and future projects. Measured means that improvements are supported by data that identifies before-and-after performance and demonstrates the connection between an improvement and measured results Process Improvement Approach The SEL is unique in combining a functioning software development organization with a scientific software engineering laboratory. University of Maryland computer scientists provide most of the empirical software engineering research expertise. SEL process analysts and improvers treat each software project partly as an experiment in which existing baseline models and practices are verified and updated, and evolving, mature processes are assessed and infused into the organization. Putting the software measurement program in place early is the key to factual impact reporting. All facets of the process and the products developed are continually measured in detail so that the understanding of software and the impacts of process change can be quantified. Since its inception, the SEL has applied this concept to over 100 development projects, with each one providing new insights into the software process and the impact on resulting products [Morusiewicz and Valett, 1992]. Figure 3 shows that SEL process improvement follows three steps understanding, assessing, and packaging with each characterized by extensive measurement.

14 6 Understanding Assessing Know your SW business Determine effective improvements What are my SW characteristics? What process(es) do I use? What are my goals? Packaging Will inspections help? Will formal methods improve reliability? Will CASE cut costs? Make improvements part of your business Update standards Refine training Tailor processes Figure 3 The SEL SPI Method The Understanding step involves baselining an organization s process and products. This includes all product characteristics (e.g., cost, reliability, software size, reuse levels, error classes, etc.) as well as the processes used. The Assessing step occurs when potential process improvements are incorporated into development projects and then evaluated. Quantified SEL experiences (e.g., most significant causes of errors) and clearly defined goals for the software (e.g., decreased error rates) drive the selection of candidate process changes. After these changes are chosen, training and experiment plans are provided. Then, the processes are applied to one or more production projects from which detailed measurements are taken. These data are used to quantify process changes and product impacts; the new process is assessed by comparing these measures with the continually evolving baseline. As a result of the analysis, processes are adopted, discarded, or tailored for ensuing efforts, depending on the observed impacts. The Packaging step entails the infusion of all identified improvements into the SEL process. This includes updating and tailoring standards, handbooks, training materials, and development support tools [Landis, McGarry and Waligora, 1990 & 1992]; [Doland, Pajerski and Waligora,1993]. The SEL approach uses process models and baselines to gain an in-depth understanding of project and environment characteristics. A proposed process is evaluated for study, applied experimentally to a project, compared to baselines and process models, and evaluated against the experiment's goals. Based on the experiment's conclusions, results are packaged and the process is tailored for improvement. It is then reapplied, reevaluated, and repackaged. The process improvement culture exists at both the project or experiment level (observing two or three projects) and at the overall organizational level (observing trends of numerous projects over many years).

15 The Quality Improvement Paradigm (QIP) is an effective experimental framework for conducting studies and an evolutionary concept for learning and improvement. The QIP has six steps: 1. Characterize the project and its environment. 2. Set quantifiable goals for successful project performance and improvement. 3. Choose appropriate process models, supporting methods, and tools for the project. 4. Execute processes, construct products, collect and validate prescribed data, and analyze data to provide real-time feedback for corrective action. 5. Analyze data to evaluate current practices, determine problems, record findings, and recommend future process improvements. 6. Package the experience in updated and refined models, and save the knowledge gained from this and earlier projects in an experience base for future projects. The QIP uses two additional techniques: the Goal/Question/Metric (GQM) paradigm for planning and goal setting, and the Experience Factory Organization for providing the experimental apparatus. By applying specific process enhancements to multiple, selected projects and using sustained measurement against extensive baseline information, SEL experimentation provides reliable insight into the effects of process on product Process Improvement Areas Explored The early years of the SEL emphasized building a clear understanding of the process and products within its environment. This led to the development of models, relations, and general characteristics of the software business in the SEL environment. Most of the experiments (process changes) consisted of the study of very specific, focused techniques (e.g., PDL, structure charts, reading techniques), but the major enhancements were the infusion of measurement and process improvement concepts and the realization of the significance of process as part of the software culture. These early activities led to a continuing measurement process, where reporting accurate software process and product information became a standard part of SEL projects. The flow of topics addressed by the SPI program over the years is seen in Figure 4. The topics appear from left to right in the chronological order addressed. The parenthetical entries after each topic indicates the number of projects attempted using that technology. The separation of the effort into the three categories (model building, studies, and packaging) at the right indicates that roughly one-third of the total SPI effort was expended in each category.

16 Distributed development CASE (2) Cleanroom (4) Ada (9) Model Building Cumulative Number of Experiments OOD (10) IV&V (3) Testing approaches (4) Design techniques (5) Structured techniques Defect analysis Effort analysis Experiments Comparative Studies 0 Packaging (polices, processess, etc.) Year Figure 4 Areas of Improvement Emphasis over the Years Efforts with more significant process changes became standard procedure in the mid-1980s. Over the past eight to ten years, the SEL has continued to adhere to the three-step software process improvement method, but now focuses on larger, more significant process changes. These have included the use of Ada, cleanroom methods, object-oriented technology (OOT), and computer-aided software engineering (CASE), with each emphasizing different aspects of software process enhancements. 3.5 Baselining SEL Software Performance The SEL database contains software measurement data from 105 NASA/GSFC development projects (each ranging from 10,000 source lines of code [10 KSLOC] to 500 KSLOC in FORTRAN and Ada). The database contains over 220,000 records covering the period from 1977 through 1994 (approximately 106 megabytes) residing on an Oracle database management system (DBMS). The SEL data is disseminated twice a year to Data Acquisition Center Service/Rome Air Development Center (DACS/RADC) and is used worldwide by academic, commercial, and government researchers.

17 9 Database information for each project includes the following: Project characteristics (e.g., dates, size [total and executable], reuse level) Process characteristics (e.g., language) Effort (e.g., per week, by activity) Errors/changes (e.g., per week, by activity) Errors/changes (e.g., counts, type, effort to find/fix) Project dynamics (e.g., code growth, tests run/passed) Data has been collected over the SEL s lifespan. As seen in Figures 5 through 7, performance data from 1985 to 1989 was developed into a performance baseline, and used to compare performance with the subsequent five-year period. Baseline data characterized the SEL development environment in the following areas: Language usage Effort distribution by lifecycle activity Error classes and origins Error detection rates by phase Typical code growth profile

18 10 Language Usage Effort Distribution C 15% Other 26% Design 23% Fortran 65% Ada 15% 5% other Test 30% Code 21% (85% writing 15% reading) Origin of Errors Classes of Errors Code 72% Previous change 9% Design 13% Reqts 6% Data 24% Logic/control 30% Interfaces 18% Initialization 15% Computation 13% Figure 5 Language Usage, Effort Distribution, and Error Characterizations ( )

19 11 5 x x Errors Per KSLOC 4 3 x x x x 2 1 x x x x x x x x x xx xx x Code Test System Test Acceptance Operations Phase Figure 6 Error Detection Data ( ) Percent of Total SLOC Percent of Schedule Figure 7 Software Growth Profile ( )

20 12 Other SEL baseline data from indicated the following: The code production rate was about 26 SLOC per person, per day Maintenance costs were between 8% and 12% per year Cost per change indicated that 88% were considered to be very easy Code reuse rate was 20% The SEL baseline undergoes continual evolution. Promising techniques are filtered into the development organization as general process improvements, and corresponding measures of the modified process (effort distribution, reliability, and cost) indicate the effect on the baseline. 3.6 Examples of Improvement Initiatives Two examples of specific improvement initiatives Cleanroom Software Engineering (CSE) and object-oriented technology (OOT) are provided below to depict the impact on software processes and products The Cleanroom Software Engineering (CSE) Method CSE principles reflect those of semiconductor manufacturing: create an environment and a process that prevent defects from getting into the product. As with hardware products, software defects are prevented from entering the software development process. (More information on cleanroom software engineering techniques in practice at SEL is in [Green, 1990 & 1991] and [Selby, et al., 1987].) The CSE methodology was invented at IBM in the 1970s (by Mills, Linger, Hevner, Witt and others) and migrated to other organizations in the 1980s. In the 1990s, CSE usage spread rapidly as the results of previous projects became known. One aim of CSE is to avoid dependence on costly defect-removal techniques (e.g., software testing) by writing software increments correctly the first time, then verifying their correctness before testing. The cleanroom process model shown in Figure 8 incorporates statistical quality certification of code increments as they accumulate in the system (stacked boxes indicate successive increments). See [Linger, 1994] for more on the cleanroom process model. In applying the cleanroom method, the SEL pursued two goals: analysis of the cleanroom process to characterize resource allocation from the project manager's viewpoint, and analysis of the cleanroom product to characterize defects from the customer's perspective.

21 13 Customer Requirements Specification Function Usage Incremental Development Planning Formal Design Correctness Verification functional spec. incremental development plan usage spec. Statistical Test Case Generation source code test cases Statistical Testing interfail times improvement feedback Quality Certification Model MTTF Estimates Figure 8 Cleanroom Process Model The CSE method [Mills, et al., 1987] is based on defect prevention, incremental development, and software reliability engineering. Defect prevention begins with user requirements; it is accomplished using box-structured decomposition with intended functions at each level and team-based reviews to verify that each decomposition step meets its intended function. Incremental development builds the final product from a sequence of operationally meaningful increments. Each successive increment builds on preceding increments, and each build is testable against a subset of the user's requirements. Software reliability engineering is based on statistical testing using profiles of expected customer usage. Statistical usage testing reveals the average time which a piece of software will execute before it fails [Poore and Mills, 1988]. The Cleanroom process was selected to improve the reliability of delivered software without increasing overall development cost. Significant process changes included using formal code inspections, applying the formal design concept of box structures, using rigorous testing approaches driven by statistical methods, and providing extended training in software engineering disciplines such as design by abstraction. In 1987, the first project was selected, the team trained, and experiment plan written. The project's development process and products were measured meticulously. Process impacts were observed at several levels, including a 40%

22 14 increase in design effort and a different distribution of code reading versus code writing activities. This first experiment resulted in a 35% gain in product reliability and a 50% boost in productivity compared to SEL baselines. However, since this first project was small (40,000 delivered lines of code), two additional projects were selected for study using a refined set of Cleanroom processes derived from the first project's experiences. These subsequent projects provided additional evidence that components of the Cleanroom process were effective in reducing error rates (55%) while maintaining productivity for smaller projects, but the larger project had a smaller reliability improvement (15%) with a 25% loss of productivity (Figure 9 below). As a result, key Cleanroom concepts, such as focused inspections and process training, have been infused into the standard SEL process, but other aspects of Cleanroom are undergoing further analysis until the cost differences can be more fully explained Changes per KSLOC Errors per KDLOC Productivity in DLOC per day SEL Baseline ( 82-84) 1st Cleanroom project 2nd Cleanroom project 3rd Cleanroom project Figure 9 Impact of the Cleanroom Experiments on Change Rate, Reliability and Productivity Two satellite-control projects a 20 KSLOC attitude determination subsystem and a 150 KSLOC flight dynamics system were the second and third Cleanroom projects undertaken at SEL. These systems had a combined testing error rate of 4.2 errors per KSLOC [Green and Pajerski, 1991]. Testing error rate represents the residual errors in the software after verification. The SEL Cleanroom process model has evolved to where it appears applicable to smaller projects (fewer than 50 KSLOC), but additional understanding and tailoring still are required for larger scale efforts. As a result, it was determined that key elements of the Cleanroom development method could be successfully applied in the SEL environment. Indications were seen of lower error rates, higher productivity, a more complete and consistent set of code comments, and a redistribution of developer effort. The model will evolve as more data from development projects is gathered. Measurement will provide baselines for comparison, identify areas of concern and improvement, and provide insight into the effects of process modifications. In this way, quantitative expectations can be set and achievement of goals can be evaluated.

23 SEL's investigation into the Cleanroom method illustrates that the evolutionary infusion of technology is not trivial and that process improvement depends on a structured approach involving understanding, assessment, and packaging. The Cleanroom method is being used in several organizations today. [Hausler et al., 1994] report on the results of 17 Cleanroom projects in IBM, Martin Marietta, NASA, and Ericsson Telecom. These 17 projects involve a total of nearly 1 million SLOC, and have reported a weighted average of 2.3 errors per KSLOC found in all testing, as measured from first execution of the code. High rate errors are responsible for nearly two-thirds of software failures reported, even though they comprise less than 3% of total errors. Statistical testing techniques find high rate errors first, and are effective at extending mean time to failure (MTTF). Cleanroom software engineering produces exponential improvement in MTTF growth [Cobb and Mills, 1990] Object Oriented Technology (OOT) The development of highly reusable software is an advantage of using OOT. SEL expected that OOT would greatly increase the overall level of software reuse, thus driving down costs and increasing product reliability. In addition, SEL expected OOT to be more intuitive than the traditional, structured development techniques in widespread use there, thus making the development process more efficient. Specific measures SEL used to assess the impact of OOT included the following: Effort expended per KSLOC Errors per KSLOC Project duration in months The most significant OOT process change considerations included the extensive training required, changing the design approach, specific abstraction techniques, object/class identification, and specifying generics for reuse. The SEL began training its personnel in OOT in January 1985, and began its first OOT project soon thereafter. From 1985 to 1992, OOT was applied in 11 projects in three application domains. During that time, SEL developers gained more understanding of which OOT concepts were most applicable to Flight Dynamics. The use of OOT affected software reuse, which in turn affected how software specifications were written, as components became available. Figure 10 depicts the history of code reuse history for the six most recent projects, indicating a 150% increase in reuse over the baseline.

24 % 80 Percent Reuse % 20 0 Early Baseline ( 86-88) Ada Projects since 89 Figure 10 Reuse Improvements Resulting from OOT SEL s subjective experience showed that OOT was not as intuitive as expected, since the organization had used functional decomposition successfully for over 15 years. However, the effect of OOT use has been quite striking in the specific domain of multimission simulators. Figure 11 shows comparative measures on three successive projects. Project 1 was developed to be reused; Projects 2 and 3 are the first two projects to reuse its architecture. Costs were reduced by a factor of three, change and error rates were reduced by a factor of ten, and project cycle time was cut roughly in half.

25 Changes per KSLOC Errors per KSLOC Effort in hours per KSLOC Project Duration (in weeks) 1st project (UARSTELS) 2nd project (EUVETELS) 3rd project (SAMPEXTS) Figure 11 OOT Impacts On Three Multimission Simulator Projects However, an attempt to reuse this architecture for a different class of projects resulted in difficulties adapting the code and unsatisfactory run-time performance. SEL concluded that OOT, when coupled with the practices of domain analysis, enables high reuse across a range of applications within a given environment. The developer also must be concerned with run-time efficiency. OOT was the first technology covering the entire development life cycle in the Flight Dynamics Division, and thus heavily influenced process evolution at SEL. For more information see [Stark, 1992].

26 18 4 OVERALL MEASURED IMPROVEMENTS RESULTING FROM SPI The overall impact of SEL's SPI program has accelerated in the last eight years. As mentioned in section 3.2, SEL's improvement results include the following: Error rates during development decreased 75% from 1985 to Average software development cost to support a mission decreased 55%. Average reuse increased by 300% during the past four years. Average cost to develop a line of code decreased by 10% from $0.32 to $0.28. Costs have become more predictable. As the improvement process has evolved and matured, the SEL's product baselines of reliability and cost also have been measured and periodically reassessed. By grouping sets of projects with similar operational functionality into representative time periods, SEL baselines were evaluated and compared to measure overall process impacts. As seen in Figure 12, the decrease in overall software errors (per KSLOC) in the specific application domain of Attitude Systems has decreased significantly from 1977 to 1993 for 47 projects. Error rates have declined from a high of about eight errors per KSLOC to a current level approaching one error per KSLOC. 10 Average error rate trend over 47 Attitude System Projects 8 Errors 6 per developed KSLOC Year Figure 12 Development Error Rate Trend

27 As seen in Figures 13, 14, and 15, significant improvement is shown by comparing measures from the most recent performance baseline ( ) with the previous performance baseline ( ) in the areas of development error rates, effort reduction, and reuse improvements High = 8.9 Errors Per KSLOC (developed) 6 4 Avg. = High = 2.39 Low = 1.7 Avg. = 1.1 Low =.23 Baseline ( 85-89) New Baseline ( 90-93) Figure 13 Development Error Rate Improvements 800 High = 755 Effort (Staff Months) Avg. = 490 Low = 357 High = Avg. = 210 Low = 98 0 Baseline ( 85-89) New Baseline ( 90-93) Figure 14 Software Development Effort/Cost Improvements (Includes total software costs to support flight dynamics projects)

28 Ada 90% 80 Avg. = 79% Percent Reuse % Avg. = 18% Fortran 6 similar systems Ada 2 similar systems 32% 61% Fortran 3 similar systems 5 similar systems Baseline ( 85-89) New Baseline ( 90-93) Figure 15 Reuse Improvements In summary, the major characteristics of the SEL SPI Program include A dual product and process improvement emphasis Measurement focus Empirical experiments to test potential impacts Packaged transfer and deployment of practices and lessons learned 4.1 Process Transfer and Deployment SEL process handbooks [e.g., Landis, et al., 1990] have been adopted by many software organizations both within and outside NASA. More significantly, SEL concepts have been accepted by other software organizations throughout the contractor and civil service communities. Examples include the following: Hughes Applied Information Systems has adopted the SEL process improvement concept and is using SEL personnel to guide the process program for the EOS Core System (currently one of GSFC's largest development efforts, totaling $760M over 20 years). CSC has expanded the application of SEL concepts beyond the Flight Dynamics Division to cover the entire SEAS contract (valued at $1 billion over 10 years). On the NASA-wide level, a software engineering program modeled on the SEL's concepts was defined in Participation to date in this program includes GSFC, Marshall, Langley, Johnson, and JPL under overall NASA headquarters management. The program's specific plans include establishing a NASA-wide software process improvement program based on concepts of

29 the SEL. The first phase of the program, baselining NASA-wide software, is ongoing and scheduled for completion in IMPLICATIONS FOR IMPROVEMENT IN SEMICONDUCTOR FAB EQUIPMENT SOFTWARE How does this case study apply to the area of semiconductor fab software? One lesson is that software quality can and should carry the same priority as schedule and cost considerations, and is clearly a focus of SPI initiatives. Another is that software measurements (metrics) are essential to tracking quality improvements. A simple, frequently used measure of software quality is delivered defect density (number of defects per unit of size), which is related to the distribution of reported software problems. Another measure is the MTBF caused by software defects. The average trend over several projects, products, or releases within an organization reveals whether process improvements are paying off in product improvements. These measures can be related directly to key equipment performance measures, such as MWBI or yield. This kind of information should be discussed routinely between customers and equipment supplier senior management who are serious about achieving increased fab operational reliability with software improvements. This case study identifies the value of software improvement for companies committed to such action. However, it appears that significant culture changes must occur in member company fab units and in most SEMI/ suppliers for software development and delivery to be considered as important as hardware development and delivery. The authors submit that most executives and high-level managers of equipment companies are oriented to hardware by education and experience, and thus may not understand software well enough to fully appreciate the drawbacks of shipping defective code. For example, virtually all managers would delay shipping a piece of equipment if its serial interface connector were missing or if the limit stops on its robotic arm were not installed. However, these same managers may ship software containing a nonfunctional serial communication driver or defective motioncontrol software that permits the robotic arm to overtravel and break a wafer. This may occur because many managers don't understand software problems or because they are unaware that such software defects exist perhaps because inadequate in-house testing failed to detect them. Many companies in other business sectors are reporting success and significant benefits from their SPI programs. Return on investment (ROI) has been calculated at 7:1 in several cases [Herbsleb, 1993] [Krasner, 1994], with a lag of approximately two years between investment and observable benefits. The length of lag usually depends on organization size, type, and scope of improvements attempted. These successes show up in better performance and better products. Existing data suggests good reason for a company to start a well-focused, well-designed SPI program. The SPI Project has described an approach for establishing an SPI program in semiconductor equipment companies ( SPI Guidelines for Improving Software: Release 3.0, Technology Transfer # A-ENG) and has tested that approach with six SEMI/ members. Improving the basic understanding and visibility of software practices in small semiconductor equipment companies is achievable within a few months (Krasner, 1994b). This case study suggests examples of areas in which improvements that can be made.

30 22 6 CONCLUSIONS Software functionality and quality are major issues for fab decision-makers and executives of equipment supplier companies. Recent reports of software-related equipment failure in fabs [e.g., Maxion, 1994] indicate that the situation is critical. Current equipment control software appears generally deficient in such areas as handling of exceptions to normal operation, fault tolerance characteristics, documentation, design for maintainability, design of user interfaces, and reliability because of excessive software defects. VLSI Research, Inc. recently named the ten best semiconductor process equipment supplier companies for 1994 [Steele, 1994], based on responses to its annual customer satisfaction survey. Survey data revealed that overall rankings generally were lower in 1994 than in Across the board, customers continue to criticize the quality and support of control software. As in prior years, software support is the lowest-rated item among all semiconductor industry segments, indicating a major opportunity for large-scale improvement. A recent analysis by a member company on the value of software improvement within a typical fab showed that a 20% improvement in software would save up to $1.4 million per fab per year. This was a very conservative estimate based on software outages occurring less than 3% of production time. However, the authors have heard of software-related fab outages occurring up to 50% of the time. Obviously, potential savings would be much higher in such cases. For more information about the SPI Project, please contact: Harvey Wohlwend 2706 Montopolis Drive Austin, TX Phone: , Fax: harvey.wohlwend@sematech.org

31 23 7 REFERENCES Basili, V. (1985): Quantitative Evaluation of Software Engineering Methodology, Technical Report TR-1519, CS Dept., University of Maryland, College Park, MD, July Basili, V. and R. Selby (1987): Comparing the Effectivness of Software Testing Strategies, IEEE Trans. Software Eng., Dec. 1987, pp Basili, V. and H. Rombach (1988): The TAME Project: Towards Improvement-Oriented Software Environments, IEEE Trans. Software Eng., June 1988, pp Basili, V. (1989): Software Development: A Paradigm for the Future, Proc. Compsac, IEEE CS Press, Los Alamitos, Calif., Basili, V. and Green, S. (1994): Software Process Evolution at the NASA SEL, IEEE Software, Vol. 11, No. 4, July Cobb, R. and Mills, H. (1990): Engineering Software Under Statistical Quality Control, IEEE Software, November 1990, pp Condon, S., M. Regardie and S. Waligora (1993): Cost and Schedule Estimation Study Report, SEL , November Doland, J. R., Pajerski and S. Waligora (1993): The Software Engineering Laboratory Training Program, SEL-93-TP1, September Green, S. et al. (1990): The Cleanroom Case Study in the Software Engineering Laboratory: Project Description and Early Analysis, SEL , March Green, S. (1991): Software Engineering Laboratory (SEL) Cleanroom Process Model, Technical Report SEL , Software Engineering Laboratory, Greenbelt, MD., Hausler, P., Linger, R., and Trammel, C. (1994): Adopting Cleanroom Software Engineering With a Phased Approach, IBM Systems Journal, March Herbsleb, J. et al. Benefits of CMM-Based Software Process Improvement: Initial Results, CMU/SEI-94-TR-13. Software Engineering Institute, Carnegie Mellon University, Pittsburgh, PA, August Krasner, H. (1994): The Payoff for Software Process Improvement: What It Is and How to Get It, IEEE Technical Committee on Software Engineering, Software Process Newsletter, September Krasner, H. (1994b): A Case History of the Space Shuttle Onboard Systems Project, Technology Transfer # A-TR, October Krasner, H. and Ziehe, T. (1994): The Software Process Improvement Project: An Industry Specific Maturity Initiative, in Software Quality Matters, Vol. 2, No. 3, Fall 1994, University of Texas - Software Quality Institute, Austin, TX, September Landis, L., F. E. McGarry, and S. Waligora (1990): Manager's Handbook for Software Development (Revision 1), SEL , November 1990.

32 24 Landis, L. et al. (1992): Recommended Approach to Software Development: Revision 3, Technical Report SEL , Software Engineering Laboratory, Greenbelt, MD, June Linger, R. (1994): Cleanroom Process Model, IEEE Software, Vol 11, No. 2, pp , March Maxion, R. (1994): Preliminary Report on Software Failures, SEMI/ distribution of ETAB Report, May 12, McDermott, T., et al. (1990): Gamma Ray Observatory Dynamics Simulator in Ada (GRODY) Experiment Summary, SEL , September Mills, H., M. Dyer, and R. Linger (1987): Cleanroom Software Engineering, IEEE Software, Sept. 1987, pp Morusiewicz, L. and J. Valett (1992): Annotated Bibliography of Software Engineering Laboratory Literature, SEL , November PAAC (1994): Process Achievement Award Coordinator, Carnegie Mellon University, Software Engineering Institute, Pittsburgh, PA 15213, , Poore, J. and Mills, H. (1988): Bringing Software Under Statistical Quality Control, Quality Progress, Nov. 1988, pp SPI Guidelines for Improving Software, Release 3.0, Technology Transfer # A-ENG, October 31, Selby, R. Jr., V. Basili, and T. Baker (1987): Cleanroom Software Development: An Empirical Evaluation, IEEE Trans. Software Eng., Sept 1987, pp Stark, M. (1992): Impacts of Object-Oriented Technologies: Seven Years of SEL Studies, in the Proceedings of the NASA SEL Software Engineering Workshop, Technical Report SEL , December, Steele, L. (1994): Ten-Best Process Equipment Companies in 1994, VLSI Research Inc., San Jose, CA.

33 25 8 GLOSSARY OF TERMS AGSS Attitude Ground Support System; KSLOC systems; provides all attitude ground support for a specific satellite mission. CM Configuration Management CMU SEI CMM Carnegie Mellon University, Software Engineering Institute, Capability Maturity Model DACS/RADC Data Acquisition Center Service/Rome Air Development Center Defect The flaw in the software or related artifact that causes a failure. DLOC Developed SLOC. DLOC = 100% new SLOC + (Reuse factor * reused SLOC) where reuse factor = 20% for FORTRAN code, and 30% for Ada code. Effort + Total staff hours required to develop software including technical, management, and service hours. Error Rate Errors per KDLOC (errors per 1000 lines of code after being adjusted for reuse) Errors Errors reported from the end of unit testing until system delivery (unless otherwise noted). FAB Semiconductor Fabrication Plant Failure A software failure is any performance of the software that deviates from the stated requirements, regardless of how significant (project specific definition used). Improvement Percent (Old Value - New Value)/Old Value KDLOC 1000 DLOC KSLOC 1000 Source Lines of Code Mission The work and products associated with the operational support of a scientific satellite. MTBF Mean Time Between Failures MWBI Mean Wafers Between Interupts Reused SLOC Verbatim reused SLOC + slightly modified SLOC SLOC (Source Lines of Code) = Total lines of code including comments, blanks, and executable code. Measured by counting carriage returns. SPI Software Process Improvement Staff Month 156 staff hours TQM Total Quality Management

34 26

35

36 Technology Transfer 2706 Montopolis Drive Austin, TX