Experiences in developing and applying a software engineering technology testbed

Transcription

1 Empir Software Eng (2009) 14: DOI /s INDUSTRY EXPERIENCE REPORT Experiences in developing and applying a software engineering technology testbed Alexander Lam Barry Boehm Published online: 11 November 2008 # Springer Science + Business Media, LLC 2008 Editor: Audris Mockus Abstract A major problem in empirical software engineering is to determine or ensure comparability across multiple sources of empirical data. This paper summarizes experiences in developing and applying a software engineering technology testbed. The testbed was designed to ensure comparability of empirical data used to evaluate alternative software engineering technologies, and to accelerate the technology maturation and transition into project use. The requirements for such software engineering technology testbeds include not only the specifications and code, but also the package of instrumentation, scenario drivers, seeded defects, experimentation guidelines, and comparative effort and defect data needed to facilitate technology evaluation experiments. The requirements and architecture to build a particular software engineering technology testbed to help NASA evaluate its investments in software dependability research and technology have been developed and applied to evaluate a wide range of technologies. The technologies evaluated came from the fields of architecture, testing, state-model checking, and operational envelopes. This paper will present for the first time the requirements and architecture of the software engineering technology testbed. The results of the technology evaluations will be analyzed from a point of view of how researchers benefitted from using the SETT. The researchers just reported how their technology performed in their original findings. The testbed evaluation showed (1) that certain technologies were complementary and cost-effective to apply; (2) that the testbed was cost-effective to use by researchers within a well-specified domain of applicability; (3) that collaboration in testbed use by researchers and the practitioners resulted comparable empirical data and in actions to accelerate technology maturity and transition into project use, as shown in the AcmeStudio evaluation; and (4) that the software engineering technology testbed s requirements and architecture were suitable for evaluating technologies and accelerating their maturation and transition into project use. A. Lam (*) : B. Boehm University of Southern California, Los Angeles, CA 90089, USA alexankl@usc.edu B. Boehm boehm@usc.edu

2 580 Empir Software Eng (2009) 14: Keywords Testbed. Software maturity. Software adoption. Technology evaluation. Technology transition 1 Introduction A major problem in empirical software engineering is to determine or ensure comparability across multiple sources of empirical data. One approach is to provide a number of parameters to characterize each data source, and to use only data from sources comparable to one s decision situation. However, this may leave users with a relatively small set of relevant data upon which to base decisions. The U.S. National Aeronautics and Space Administration (NASA) recently faced a particularly strong challenge in this area. It placed a high priority on research and technology to improve software dependability for its space missions, and wished not only to compare the abilities of research products to improve software dependability, but also to accelerate the maturation and adoption of those research products. When an organization like NASA has to send its software to space, the software has to be dependable. If the software performs the wrong function, the mission could end, which would be a huge loss to the organization. However, developing defect-free software is a complex problem. There are many technologies available to help software engineers identify defects, but choosing the right technology can be difficult. There are many questions to be asked of the technology such as How does one know if the technology does as it states, Is the technology mature enough for use, and Will the technology work on my system? Furthermore, technology researchers face their own problems in getting their technology adopted. For dependability researchers, they have to prove that their technology can help decrease defects in the software system, that their technology is cost-effective with respect to alternative technologies, and that the technology will work for the end user in a real-mission situation. This paper reports on experiences in developing for NASA a full-service, mission-relevant, dependability technology-oriented software engineering technology testbed (SETT) to help researchers mature technologies and to help practitioners evaluate alternative software engineering technologies. The paper also addresses the use of the SETT in accelerating technology maturation and adoption, and provides the requirements and architecture for a SETT. For this paper, the primary dimension of dependability examined will be the number and types of defects in a software system representative of NASA planetary exploration vehicles. The structure of this paper is as follows. Section 2 explains the obstacles for technology adoption and how SETTs address the obstacles. Section 3 explains the requirements and architecture of a software engineering technology testbed which have not been discussed before in our prior publications. Section 4 details how researchers would configure a SETT to evaluate a technology. Section 5 provides the results of using an instance of the software engineering technology testbed to evaluate several technologies and the limitations of that instance. A more detailed analysis of the evaluation results is presented in this paper that has not been discussed before in prior publications of the SCRover work. Section 6 provides the benefits researchers received from using the SETT s features. Finally, conclusions are presented in Section 7. 2 Motivation Redwine and Riddle state that it takes about years to mature a technology to the point it gets used by practitioners. They provide several critical factors that explain why technology maturation can take so long (Redwine and Riddle 1985). This section will

3 Empir Software Eng (2009) 14: outline those critical factors and indicate how a software engineering technology testbed can address them in order to increase the speed of technology maturation and adoption. One of the critical factors in getting users to use new technologies is that there is little relevant data on prior experiences demonstrating positive feedback on a technology. This lack of an experience base causes software engineers to be hesitant to use new technologies. One requirement for a software engineering technology testbed would be to have an experience base of prior experiences, both positive and negative, for each technology, providing software engineers an indication of how well the technology worked on a representative software system. The information going into the experience base would contain information such as, but not limited to, the effectiveness of the technology to finding defects, what type of defects it found, training time to learn the technology, and a description of the technology. By analyzing the information, a software engineer would be able to gauge how well the technology will work on their project and evaluate alternative software engineering technologies. At times, a practitioner may not know if two or more technologies are complimentary or not. The technologies may find the same set of defects. With an experience base, practitioners can decide if two or more technologies are complimentary or not. In addition, researchers who use the testbed to evaluate their technology would be able to add their experiences/results to the experience base for practitioners to view. Another critical factor according to Redwine and Riddle is conceptual integrity. By using the software engineering technology testbed, researchers will be able to demonstrate that the technology is well developed by applying the technology on a representative software system and being able to find (seeded) detects. If the technology is unable to find the seeded defects or significant additional defects in the representative system, then the researcher will need to develop/mature the technology some more before being used by the technical community. A third challenge to technology adoption is showing a clear recognition of need for the researchers (dependability) technology. By applying their dependability technology to the software engineering technology testbed, researchers can demonstrate to practitioners how well their technology can identify certain classes of defects in a system, thereby proving to practitioners why they should use the dependability technology in their software development. The fourth critical factor outlined by Redwine and Riddle is tuneability. The researcher needs to show that a technology can be tuned to fit the user s needs. By adapting the technology to the software engineering technology testbed, researchers are able to demonstrate what activities will need to be done to tune a technology to work on an organization s software systems. Finally, Redwine and Riddle mentioned lack of training for the new technology as an impediment for technology transition. In the experience base that was previously mentioned, the experience base should also collect and maintain training information such as how the technology was applied to the testbed, how long it took to apply the technology, and how much training was involved before using the technology. 2.1 A Look at Other Testbeds Before NASA funded the Dependability Computing Program (HDCP) to determine how to evaluate a dependability technology, there were no established full-service software engineering testbeds in existence that could help an organization fully evaluate dependability technologies. A few testbeds existed, such as the Robocup (2007)

4 582 Empir Software Eng (2009) 14: competition and the DETER testbed (Benzel and et al. 2007) that could evaluate technologies, but these testbeds have their disadvantages. RoboCup is a head-to-head soccer competition between two research teams to determine who the winner is. The team that scores the most goals wins. The disadvantages to using RoboCup as a SETT are numerous. First, it can be difficult to judge how technologies compare against each other. In its soccer competition, it uses a playoff type format to determine the winner of the competition. As (Stone 2003) indicates, this can be an erroneous way to judge the value of one s research. Perhaps the technology was not suited for soccer, but it will work better in other fields of application. Likewise, the winner of the competition may not be well-suited to work in the practitioners domain. Another limitation of RoboCup is that the problem scope of the competition is limited to just a few applications: soccer, search and rescue missions, and dance challenges. With its limited scope, this could deter many practitioners from using the technology as they would be unsure how the technology would fit with their needs, i.e. it is not clear the technology can be tuned to the users needs (afterall, how many companies are building soccer-playing robots?). Finally, RoboCup doesn t provide the full-service software engineering support a SETT does. Only a simulator and very little code are provided. No specifications or other type of support are provided. Thus, an architecture researcher would have a difficult time using RoboCup to evaluate architecture models since none are provided. The cyber-defense Technology Experimental Research (DETER) testbed provides researchers support in evaluating computer/information security technologies. While the DETER testbed has many advantages such as providing a platform for repeatable experiments, a repository of complete experiments, and tools to help researchers gather and visualize their test data, there are a few disadvantages to the testbed. First, the DETER testbed is built primarily to evaluate computer security technologies. Other technologies would find it difficult to use the testbed. While the DETER testbed provides many great support facilities for the researcher, it doesn t provide the full-service software engineering support a SETT does. For example, specifications as detailed as the SETT specifies are not provided by the DETER testbed. Also, it is unclear how a practitioner would be able to search the DETER repository to identify a technology that would meet their development needs. If the practitioner does find a technology, no data is provided to indicate how difficult it will be to adapt the technology. From the literature review, it seems few testbeds provide specifications that would aid in a software engineering technology evaluation. In addition, no experience base is usually provided for practitioners to determine how one technology fared against several others and to indicate how difficult it will be to learn/adapt the technology. With the SETT, practitioners will be able to use the many functionalities of the SETT that will be discussed in Section 3 to make a much more informed choice as to which technology is best suited for them. 3 SETT Domain Model, Requirements and Architecture In order to generate the set of requirements and architecture for a SETT, the Domain- Specific Software Architecture (DSSA) Approach was used. DSSA is a process and infrastructure that supports the development of a Domain Model, Reference Requirements, and Reference Architecture for a family of applications within a particular problem domain. (Tracz 1995; Mettala and Graham 1992).

5 Empir Software Eng (2009) 14: Domain Model The domain model is composed of the customers needs statements, scenarios, domain dictionary, context diagrams, entity/relationship diagrams, data flow models, state transition models, and object models. For brevity, only the scenarios will be shown. To generate the scenarios, interviews were conducted with various technology researchers and NASA software engineers to collect their needs statements, which were then used to generate user scenarios as shown in Fig. 1 and Fig. 2. As summarized in Fig. 1, if a technology researcher has developed an idea for a new technology, the researcher will begin by exploring the experience base for technologies similar to the idea, to make sure the technology has not been developed yet (the researcher may also be researching the literature to ensure the technology has not been developed yet). If similar technologies already exist, the researcher will keep refining the idea. If no existing technology appears to provide the new technology s capabilities, then the researcher will develop the technology. Once finished, the researcher will configure the software engineering technology testbed to evaluate the new technology. Once the evaluation is complete, the researcher will prepare a technology evaluation report and submit the report to the experience base where practitioners looking for dependability help will view it. If the evaluation results are positive and a practitioner has been identified, the researcher may then proceed to collaborate with the practitioner to configure the technology for use on a live project. Once the configuration and application of the technology is performed on the live project, the experience base should be updated to reflect the new results. The evaluation report will be submitted to a testbed coordinator who is in charge of maintaining the testbed and its experience base. If the testbed evaluation results are negative, the researcher should update the experience base with the current results to indicate to practitioners that the technology Continue with research No candidates compare to the new idea Once technology is developed Explore Experience Base for similar technologies New Technology Idea Similar technology found; refine technology idea Technology Researchers Positive Results Configure testbed; Evaluate Technology Negative Results Collaborate with practitioner to configure technology for use on project. Apply, evaluate, and iterate technology Update technology needs and capabiltiies Prepare technology evaluation report, Compare evaluation with evaluations from similar technologies, and Update Experience Base Evaluate until Positive Results Found Refine Technology Fig. 1 Researchers user scenarios

6 584 Empir Software Eng (2009) 14: Practitioner Formulate/revise mission approach, dependability issues Unproven candidate(s) Explore Experience Base for relevant technology. Filter candidates Good candidate(s) Contact Researcher to perform evaluation on testbed Explore technology usage considerations on user project Researcher configures testbed and performs evaluation Prepare technology evaluation report and update Experience Base Collaborate with technology provider to configure technology for use. Apply and evaluate technology Fig. 2 Practitioners user scenarios may not be ready for use yet in this particular context/application and continue to refine the technology until the evaluation results are positive. The negative results could also indicate that the technology was not designed to work in the domain the testbed represents and that the technology would work best in another application/domain. As summarized in Fig. 2, for practitioners having dependability problems, they will first explore the experience base looking for technologies that fit their search criteria. The practitioner will read the technology evaluation reports to determine how well a technology performed on the testbed system or on past projects done for the organization. Once a technology has been identified as a promising candidate, the practitioner will explore using the technology on their project. This could involve collaborating with the technology provider to configure the technology for use. After applying the technology to their project, the practitioner will submit a technology evaluation report to the experience base detailing how well the technology worked for the project. If no proven candidates are identified, then the practitioner will contact the researcher of the technology and ask the researcher to perform an evaluation of the technology on the testbed. The researcher will configure the testbed, perform and evaluation, and update the experience base with the results of the evaluation that will be viewed by the practitioner.

7 Empir Software Eng (2009) 14: Requirements From the domain model, a list of functional and non-functional requirements was generated Functional Requirements F1 testbed should provide specifications and code for the system to be developed. F2 testbed should allow an evaluator to instrument the system in order to collect data during the experiment F3 testbed system should have a library of seeded defects (Mills 1972) that evaluators can use to seed defects into the specifications/code. F4 testbed should provide guidance to the evaluator on how to use the testbed F5 testbed should have a mechanism for evaluators to submit their technology evaluation results (including technology adoption data) for practitioners to review F6 evaluators should be able to run the missions/scenarios on a hardware platform or simulator to obtain or verify the technology evaluation results. Missions are various operational scenarios which the system can perform. An example scenario is for a rover to locate an object of interest. F7 testbed should provide a mission/scenario generator that will produce multiple missions/scenarios from which the evaluator can select from for their evaluation. This will allow the researcher to test their technology under various scenarios. F8 testbed should provide project data such as effort data as part of the testbed system. For example, effort data includes the time developers spent designing, developing, and testing the system. F9 testbed should have a user interface that will allow an evaluator to access the testbed Non-Functional Requirements NF1 the testbed, including mission applications specifications and code, should be representative of the systems the sponsoring organization develops and support a wide range of technologies. In addition, the specifications should follow good software engineering practices. NF2 specifications/code of the mission applications should be tailorable to meet the needs of multiple contributors/evaluators. NF3 testbed should be available for public use. NF4 testbed should have results that are combinable with other evaluators results and are representative to the end user. NF5 the defect library should be composed of actual defects incurred while implementing the mission applications. NF6 researchers should use a common (ideally low-cost) platform (i.e. computer equipment the code runs on) to evaluate their technologies allowing a fair comparison of technologies and allowing results to be combinable. NF7 the defect data classification used should be comparable to classifications used by the evaluators. In addition, evaluators should be able to classify the defects according to their own defect classification system.

8 586 Empir Software Eng (2009) 14: Architecture A brief description of each component in the testbed s architecture is given below along with why the component is important to the architecture (Fig. 3). Instrumentation an instrumentation class will help the evaluators collect data for their evaluation report. Without this feature, an evaluator will have to spend more time in figuring out how to collect the data they need. Seeded Defect Engine and Defect Pool seeded defects will help the evaluators estimate how well their technology finds defects and it prevents evaluators from building their technology to just pass a certain test or operational scenario. In addition, seeded defects give a measurement to evaluators on how well their technologies can find defects and what kind of defects the technology can and cannot find. Code a software system is needed to evaluate the technology on. Specifications specifications are needed if the organization wishes to test more than just technologies that work on code. Specifications provide the basis for the evaluation, along with the code. For example, architecture specifications are needed for architecture technology evaluations. Without specifications, evaluators whose technology requires specifications will not be able to use the testbed or it will increase the amount of time the evaluator conducts the evaluation since they will have to produce the specifications on their own. Thus, without specifications, the scope and usefulness of the testbed will be limited. Mission/Scenario Generator allows evaluators to do a more thorough evaluation of their technology since it will be operating under several scenarios and not just one. Also, prevents an evaluator from developing their technology to work under a specific circumstance. Fig. 3 SETT architecture

9 Empir Software Eng (2009) 14: Platform: Simulator a simulator is needed to do low-cost runs of the system. Running the system in simulation mode will help the evaluator acquire data and do runs of the system at a low cost. For those technologies that require code to be run, a simulator provides a free way to evaluate the system as many times as they want. Without a simulator, evaluators will have to buy hardware, which may not be possible due to limited budgets. Hardware platform hardware provided for evaluators to test their technology on an actual platform. Usually the evaluator would use the hardware after getting good results on the simulator and want to be sure the simulated data will match the actual data obtained from the hardware Manual and Experimentation Guidelines used to guide the evaluators in how to use the testbed and how to set up their experiments. Without this component, evaluators will spend more time to set up their experiment and increase the probability the experiment will be conducted incorrectly. Effort Data effort data that went into the development of the testbed. This data can be used by technologies dealing with cost-estimation. Technology Evaluation Report report indicating how well a technology worked on the testbed system. Includes such information as how many and the type of defects the technology found and technology adoption data such as how much effort it took to apply the technology to the testbed and where in the software lifecycle the technology was applied. Experience Base database of technology evaluation reports SETT users can search through. Without the experience base, practitioners will face a difficult time in finding and choosing the right technology to solve their dependability problems. 4 How to Configure a SETT Figure 1 provided an overview on how an evaluator would use a SETT but this section will go into more detail on how one would configure a testbed to evaluate a technology. To use the testbed, evaluators start by reading the experimentation guidelines that provide them a framework for conducting their evaluation and provide instructions on how to use the testbed. Next, they will define the type of defects the technology is expected to detect and what kind of data the instrumentation class should be collecting. These features will help gather the necessary data used to evaluate the performance of the technology. The next step is to define the appropriate operational scenario under which the technology will be evaluated. Then, based on the criteria defined, appropriate instrumentation and seeded defects are applied to the project artifacts associated with the selected operational scenario. Once the appropriate set of project specifications and code has been obtained, the evaluator will apply the technology to the set of project artifacts. Next, the technology is then executed. This step does not necessarily require code to be run. For example, if a new type of requirements peer reviews were being performed, then no code is necessary. After the execution of the technology, the evaluators use the data provided by the instrumentation to determine the percentage of seeded and unseeded defects of each type that were found. This enables an analysis of how well the technology performs in detecting, avoiding, or compensating for various classes of seeded and previously undiscovered defects, in comparison to alternative technologies. The data and the analysis are then stored in an experience base to be accessed by project managers interested in technology to increase the

10 588 Empir Software Eng (2009) 14: dependability of their delivered systems. Section 5 provides two examples of how evaluators used the software engineering technology testbeds. 5 SETT Instance SCRover and Technologies Evaluated Using the architecture outlined in Section 3, an instantiation of a SETT called SCRover was developed. The SCRover testbed provides an open, experimental framework representative of NASA planetary rovers that enables evaluators to determine the relative cost and effectiveness of a given software dependability technology on a NASA like project. The testbed contains software, supporting information such as specifications, metrics, instrumentation, seeded defects, and guidelines, a robotic platform (both real and simulated), and a development environment. SCRover represents a family of planetary rovers. However, due to export restrictions by the U.S. government and to ensure the testbed would be publicly available to most researchers, the SCRover team and the JPL team deemed it unwise for the USC team to develop an application that was so close to what NASA uses rovers for: looking for life on other planets. Thus, the SCRover and JPL teams decided a similar, but still NASA-representative mission, would be for SCRover to explore buildings for potential chemical hazards. Like its NASA counterparts, SCRover would still roam an unknown territory and use its cameras and sensors to locate objects of interest. To enhance its representativeness of NASA planetary rover software, SCRover was built using the Mission Data Systems (MDS) technology created by the NASA Jet Propulsion Laboratory (JPL) in The MDS goal has been to develop a set of closely matched tools and techniques to reduce development and debugging costs, promote reusability, and increase reliability throughout a project s lifecycle. The principal MDS products include a system engineering methodology called the State Analysis Process, a software framework, a goal-based operational methodology, and a cost estimation model based on COCOMO II (Dvorak et al 2000). Once the SCRover system was built, several technologies were applied to the testbed. A total of 6 technologies were evaluated on SCRover, but this paper will discuss three technologies evaluated, Mae (Roshandel et al. 2004a, b), AcmeStudio (Garlan et al. 2000), and ROPE (Fickas 2004). 5.1 Mae and AcmeStudio Mae According to Dr. Roshandel, the Mae technology serves as a step between the UML diagrams generated by developers and the implemented system. Mae is an extensible architectural evolution environment developed on top of xadl 2.0 that provides functionality for capturing, evolving, and analyzing functional architectural specification (Roshandel et al. 2004a, b; Boehm et al. 2004). Mae models the system components and its behaviors. Once the model is complete and inputted into the Mae tool, the tool reports any mismatches in how the component is used and its intended usage. For example, Mae can detect signature defects like if a component is calling a function that has not been implemented yet or if a component is passing the wrong parameters in a function call. In addition, Mae is able to detect errors in the pre- and post-condition of a function call. For example, a component is making a function call, but the pre-condition for calling that function has not been met yet. In using Mae, developers can find these types of defects faster and better than using traditional methods such as peer review.

11 Empir Software Eng (2009) 14: Since the Mae evaluation was done at USC with the help of the SCRover developers, there was no need for Dr. Roshandel to deploy the SCRover system on her own. She used the same robotic and simulator platforms the SCRover developers used. For the evaluation, Dr. Roshandel used many features of the testbed. First, she read the SCRover manuals (and consulted with the SCRover development team) to determine how the experiment should be set up and what activities she needed to do before the evaluation could start. Next, she defined the type of defects she was looking for and the instrumentation class was modified to collect the data she needed. The third step involved deciding which scenario she would perform her analysis under. The wall-following scenario was chosen. Next, she took the architecture specifications of the system and translated the provided UML models to xadl models. Roshandel used the class and sequence diagrams in determining the architectural configuration of SCRover in terms of components and connectors, and their interaction. The architecture specifications that were translated contained 38 seeded defects. The defects ranged from grammar defects to defects that could potentially cause harmful behaviors. Some of the defects were architectural in nature while others were conceptual. A subset of the architectural defects concerned functional behaviors that Mae captured, while other defects were behaviors not identified by Mae. Re-seeding these defects into the Mae models helped identify the defects that Mae can and cannot detect. The defects Mae cannot detect is valuable in identifying complementary technologies necessary to detect additional classes of architectural defects. Results Roshandel inputted the xadl models (with the seeded defects) into the Mae tool that performed the architecture analysis. The analysis provided by Mae revealed several inconsistencies in the SCRover architecture as dictated by the MDS Framework rules. The tool identified 15 of the seeded defects plus an additional 6 defects that were undetected before. According to Dr. Roshandel, these 6 additional defects concerned inconsistencies in the architecture specification. Specifically, the Mae tool detected inconsistent specification of interfaces and behaviors among interacting components, which could result in harmful system interactions. Figure 4 summarizes the number of seeded defects (left column) against the defects found by the Mae tool. These defects were classified under a categorization schema similar to Orthogonal Defect Classification (Chillarege et al 1992). Afterwards, the SCRover system was run under the wall-following scenario. While the SCRover rover was following various wall configurations, the data that she defined in step 1 was collected with the instrumentation class. With the data, Dr. Roshandel was able to perform an analysis of how many times each defect (found by the Mae tool) occurred in the system, which she then used to determine how much more reliable the SCRover system would have been if the Mae tool was used in the architecture phase of the life-cycle. Costs The Mae evaluation on SCRover took approximately 160 h. Most of the time was spent in translating the UML models into xadl models. However, this time is small when compared to how long it would have taken the Mae team to build a testbed with a NASArepresentative rover. The SCRover team spent approximately 6 months to build the testbed. AcmeStudio A similar experiment was conducted with Dr. David Garlan s AcmeStudio tool (Garlan et al. 2000), which is also designed to find defects in software architecture models. According to Garlan, AcmeStudio determines whether a system s architectural specifications

12 590 Empir Software Eng (2009) 14: Fig. 4 Mae defect detection yield by type 25 Mae Defect Detection Yield by Type Number of Defects Interface Class/Obj Logic/Alg Ambiguity Data Values Other Inconsistency 0 Number of Seeded Defects Number of Defects Mae Detected comply with the relationships and constraints imposed by the architectural style as chosen by the developers. In the SCRover testbed, AcmeStudio would evaluate whether the architecture models provided by the testbed follow the architecture rules as set by the MDS Framework. The AcmeStudio researchers believe that using AcmeStudio will result in developers finding more architecture violation defects than using traditional methods such as peer review. Results In the AcmeStudio experiment, only 18 of the original seeded defects were used. The reason is only the architectural type defects were analyzed. The results of the evaluation are summarized in Fig. 5. Peer Review was used to find the set of original Source: (Roshandel, Schmerl, et al., 2004) Fig. 5 Mae/AcmeStudio/peer review results

13 Empir Software Eng (2009) 14: seeded defects and the defects are classified by their architectural type. The figure indicates what type of defects each method can find. The results verified that Mae and AcmeStudio find different types of defects and are indeed complementary technologies that can be used jointly to find a greater number of defects. From the set of defects analyzed in Fig. 5, AcmeStudio is better at finding architecture mismatches than peer reviews. Likewise, from looking at Figs. 4 and 5, Mae is better at finding defects in the pre- and post-conditions, as well as inconsistency defects, than peer reviews. Costs The AcmeStudio evaluation on SCRover took approximately 120 h. Most of the time was spent in translating the UML models into Acme models. As in Mae, this time is small when compared to how long it would have taken the AcmeStudio team to build a testbed. Further analysis can be done from the Mae/AcmeStudio/Peer results to demonstrate to practitioners how using all three technologies in combination can benefit them in finding a greater pool of defects as shown in Fig. 6. For this study, the set of 32 defects showninfig.5 as well as the 20 seeded defects the Mae and AcmeStudio technologies couldn t find are used. If only peer reviews were used to identify defects, only 38 defects would be found leaving at least 14 more in the architecture specifications. However, if the Mae technology were applied after Peer Reviews, then an additional 6 more would be found. Likewise, if the AcmeStudio technology were applied after Mae and Peer Reviews, then an additional 8 more defects would be found. On the other hand, if AcmeStudio and Mae technologies were used, but not Peer Reviews, then the two technologies would find 32 of the defects in the architecture specifications, leaving at least 20 more defects in the specifications. Many of the 20 defects that AcmeStudio and Mae didn t find were not specifically architectural in nature (e.g. logic and algorithm defects) demonstrating the importance of Peer Reviews. 60 Architecture Defects Found vs. Effort Architecture Defects Found Peer Review first Mae second Acme second Acme third Peer Review third 10 Mae first Effort (Hours) Fig. 6 Architecture defects found vs. effort

14 592 Empir Software Eng (2009) 14: Implications for JPL Project use of Mae, AcmeStudio, and SCRover Testbed Results The results of the Mae and AcmeStudio experiments with the SCRover testbed were of considerable interest to JPL-MDS personnel, who had been experimenting individually with their capabilities. The complementarity of their defect identification and avoidance capabilities, the relatively low level of effort in developing and analyzing the specifications, and the prospect of combining the two toolsets opened up new prospects for using ADLs to supplement MDS s current state-oriented architectural approach. Potential benefits included stronger defect avoidance, detection, and diagnosis; stronger compositional modeling of MDS components and connectors; and an overall strong return-on-investment (ROI) potential of software architecture modeling and analysis compared to that of traditional but expensive engineering review processes. Individually, Dr. Garlan and Dr. Roshandel were also able to use their respective SCRover experience to determine if the tool was useful to a practitioner and what improvements needed to be made to make the tool more useful, thus maturing the tool to where it is suitable for industry use. Furthermore, they were able to use the SCRover experience to effectively demonstrate to the JPL-MDS personnel how effective their respective research was on the MDS technology. The SCRover experience provided the JPL-MDS personnel more confidence in using the tools and working with the researchers to advance their tools and accelerate their usage in industry. 5.3 ROPE The Reasoning about Operational Envelopes (ROPE) project s purpose is to determine the operational envelope of the environment for a given technology during runtime. The operational envelope is defined as the environment or conditions in which the system will work dependably. However, if the system moves outside the envelope, this indicates there are dangers in the environment and that the system cannot be expected to work dependably. Unfortunately, no software system can be built to meet all possible environmental conditions. Ideally, the system s envelope should be identified before placing the system in usage. The ROPE technology will help define what the system s operational envelope is and indicate what the defects of the system are. The defects will be described as environmental conditions in which the system will fail. (Fickas et al. 2004). Unlike Dr. Roshandel, the ROPE researchers conducted their evaluation at the University of Oregon and not at USC. Thus, this required the ROPE team to deploy the SCRover system on their computers and own robotic hardware. The ROPE team first obtained a copy of the whole SCRover testbed system, including all the specifications and code. The researchers then followed the instructions to deploy the SCRover code. However, changes to the code were made first. Since the ROPE team had a different rover and simulator that they wanted to use instead, the ROPE team swapped the provided device drivers with their own robotic device drivers. Once the changes were made, the ROPE team was successively able to run SCRover on their platforms. Afterwards, the ROPE team conducted their evaluation in a similar way as Dr. Roshandel did. This series of activities also demonstrated that the SCRover testbed is portable by being able to run on a different rover. Results After the port, the first activity the ROPE team did was defining the data they wanted to collect and configuring the platform and instrumentation class to collect the data. Then, the team picked the wall-following scenario to evaluate and looked at the SCRover specifications to develop their dependability model. Afterwards, the ROPE team ran the wall-following scenario and was able to detect defects in the environment that would cause the rover to fail its mission.

15 Empir Software Eng (2009) 14: One of the seeded defects the ROPE team discovered was that if two walls converged to a small angle, and if the rover was following one of the walls, the rover would eventually get stuck in the small angle. With the SCRover results, Dr. Fickas was able to demonstrate to the JPL-MDS team the advantages of using ROPE on a rover project. Costs The ROPE team spent 80 h to configure the testbed system to work on their own rover and to run the experiment. 5.4 Synergy between Technology Evaluations Even though a wide range of technologies was applied to SCRover, each researcher used the evaluation process outlined in Section 4. Each researcher was able to configure the testbed to their specific needs (such as what defects they were looking for) and used only artifacts that they needed for the evaluation. In the cases of the Mae and ROPE evaluations, both researcher teams used all the components of the SCRover testbed, save for the project effort data. Both teams used the manuals and guidelines to determine how to conduct the evaluation, used the scenario/mission generator to select which scenario they would use in the experiment, applied their respective technology to the set of project artifacts (including code and specifications), used seeded defects to determine how well their technologies performed, used the platforms to execute the code and collect instrumentation data, and finally use the technology evaluation reports to compare their technologies with similar ones. Since the teams were not doing research in software costs, they did not need to use the project effort data collected by the SCRover team. However, effort data to determine how long their evaluation took compared to other technologies were used by the teams, but that effort data were stored in the technology evaluation reports. The cost of doing each evaluation was one to two person-months for each researcher, most of which was in one-time learning and initial setup effort, thus indicating that using testbeds can be a cost-effective method for evaluating technologies. 5.5 SCRover Limitations While SCRover is representative of NASA s planetary rovers, not every researcher will be able to use the SCRover testbed. In some cases, the SCRover testbed may not have enough capabilities or requirements to satisfy the researcher s technology needs. However, SCRover does provide a representative instance of how software systems at NASA are developed since the JPL-MDS methodology was followed. All project artifacts that MDS required are part of the SCRover testbed. These artifacts span the software lifecycle from inception to delivery, allowing for a wide range of software engineering technologies to be evaluated. Furthermore, while the number of missions SCRover performs is small compared to a Mars planetary rover, the missions that SCRover does execute are part of the missions that a Mars planetary rover performs as well. In fact, the first mission that SCRover performs is the same mission the JPL- MDS group developed in their Mars rover prototype. Both SCRover and Mars planetary rovers do automated obstacle avoidance, battery power management and replanning, and use their cameras to search for and detect objects of interest. Thus, while SCRover may be less complex in terms of capabilities than a Mars planetary rover, it is complete to provide researchers a model of how software for NASA is developed. An example of where SCRover couldn t be used by researchers involved the STAR technology (Roshandel et al. 2006). STAR is a technology that analyzes state diagrams to

16 594 Empir Software Eng (2009) 14: estimate the reliability of a system. However, it was soon discovered that SCRover did not have enough states in its system to perform a solid evaluation of the STAR technology. STAR needed approximately 30 states while SCRover provided about ten states. The following diagrams illustrate various dimensions of the SCRover testbed that will help researchers decide whether or not they can use the SCRover testbed for their technology evaluation. Figure 7 provides an overview of the SCRover testbed while Figs. 8 and 9 provide information for researchers evaluating technologies respectively in architecture and requirements engineering. For each technology family to be evaluated, similar diagrams can be created. The diagrams below are not meant to include all possible dimensions that the SCRover testbed can be measured. However, the diagrams provide enough information for a researcher to decide if the SCRover testbed is appropriate for them to use. Figure 7 outlines the overall project characteristics for the SCRover testbed. This will provide researchers an overview of how complex the SCRover system is. For this diagram, we consider a complex Mars rover as having ratings of along each axis. The Project Data axis is used to describe how much data was collected during the development of the SCRover system. The SCRover team collected much data on defects, effort spent in developing the system and SLOC count, data that the NASA JPL-MDS team collected as well. However, more data could have been collected such as daily builds data and effort to fix defects, which kept the team from scoring a rating. Counting the number of lines of code that was part of the MDS Framework, SCRover had over 300 k LOC (of which about 5 k was the SCRover adaptation code and the rest is the MDS Framework code), which is a fairly high number. The JPL-MDS team had between k for their Mars prototype rover. The number of Operational Capabilities was in the medium range when compared to a Mars rover. The SCRover team spent much time Project Data Scenarios 8 6 Medium 4 2 Low Medium Low 10k Low Medium 50k 300k Source Lines of Code Platforms Medium Low 4 Low 8 Low Medium Medium Operational Capabilities Specifications Fig. 7 Project overview

17 Empir Software Eng (2009) 14: Components 15 Scenarios Architecture Diagrams 8 6 Medium 4 2 Low Medium Low Medium Low 10 Low Medium Low 2 Low 4 Medium6 8 Medium States State Variables Fig. 8 Homeground for architecture technologies 80 Classes developing a high amount of specifications for the SCRover system. The SCRover team used a well-instrumented version of the Win Win Spiral model called Model-Based (System) Architecting and Software Engineering (MBASE) (Boehm and Port 2001; Boehm and USC Center for Systems and Software Engineering 2003) for system and software development. MBASE involves the concurrent development of the system s operational concept, prototypes, requirements, architecture, and lifecycle plans, plus a feasibility rationale ensuring that the artifact definitions are compatible, achievable, and satisfactory to the system s success-critical stakeholders. MBASE shares many aspects with the Rational Unified Process (RUP) (Kruchten 2001), including the use of the Unified Modeling Language (UML) (Booch et al. 1999) and the spiral model anchor point milestones (Boehm 1996). In addition, all specifications that the MDS Framework required were generated as well. The number of platforms SCRover can run on is three (two hardware and one simulator platforms), which places it in the medium high range. The JPL-MDS team planned on using MDS on several platforms, including one simulator. For their prototype, the MDS team tested on a simulator and two different hardware platforms as well. Finally, the last axis covers the number of scenarios/missions the rover could do. SCRover has three missions that placed it in the low-medium range while the Mars rover would have a higher number of missions it could do. However, since SCRover incorporated the MDS goal-driven scenario generator, some classes of additional scenarios can be straight forwardly added. Figure 8 is to be used by architecture researchers to help them decide what type of architecture analysis can be performed. The figure covers many dimensions that

18 596 Empir Software Eng (2009) 14: Medium Low Capability Requirements Low Medium Project Requirements Medium 10 Low Low Medium Level of Service Requirements Requirement Specifications Fig. 9 Homeground for requirement engineering technologies architecture researchers considers in their work, but by no means, does it cover every dimension a researcher may work on. As in Fig. 7, we consider a complex Mars rover as having ratings of along each axis. Components and State Variables are used in the context defined by the MDS Framework (Rinker 2002). States are defined as the states of the system. Classes are the number of object-oriented classes the SCRover team developed and Scenarios are the number of missions/scenarios SCRover could perform. For each of the five axes defined, SCRover falls in the medium range, as the rovers the JPL-MDS team built have a greater number of scenarios, components, states, state variables, and classes. However, the number of architecture diagrams the SCRover team defined is in the high range as the SCRover team produced similar architecture specifications the JPL-MDS team would produce. Figure 9 is to be used by requirements engineering researchers to give them an idea of what type of analysis they could perform. The figure covers many dimensions that many requirements researchers considers in their work, but by no means, does it cover every dimension a researcher may work on. As in Fig. 7, we consider a complex Mars rover as having ratings of along each axis. Capability Requirements are defined as capabilities the system can do. For example, the rover should be able to use its camera to detect objects of interest. Project Requirements are defined as constraints placed upon the design team: e.g., solution constraints on the way that the problem must be solved, such as a mandated technology. For example, requiring MDS Framework to be used on the SCRover system. Project Requirements also summarizes process-related considerations

19 Empir Software Eng (2009) 14: such as cost or schedule constraints. For example, developing the project by a specified date. Level of Service Requirements are defined as how well the system should perform a capability requirement. For example, the accuracy of reaching a target should have an error range of ±10% distance of the expected position of the target. (Boehm and USC Center for Systems and Software Engineering 2003) For each of the requirements, SCRover falls in the medium range, as the JPL-MDS Mars rovers would have a higher number of requirements. However, the amount of requirement specifications the SCRover team defined is in the high range as the SCRover team produced similar requirement specifications the JPL-MDS team would produce. 5.6 TSAFE Testbed The HDCP program also funded another group to develop a testbed to evaluate technologies. The Fraunhofer Center at the University of Maryland (FC-MD) developed the TSAFE testbed (Lindvall et. al. 2007) which includes many of the same architectural elements of a SETT. TSAFE provides a set of seeded defects or faults for technologies to find, a set of specifications to understand the TSAFE system, instrumentation to monitor the TSAFE system and its faults, and provided various test cases. The main difference between the TSAFE and the SCRover testbed is that SCRover is representative of a rover mission that the NASA-JPL group would develop while TSAFE was built for the US Air Traffic Control System. When USC and FC-MD developed their respective testbeds, both groups found that the features stated above were important to include in the testbed when evaluating dependability technologies. In addition, both groups discovered that providing a common platform for researchers to evaluate their technologies benefits both the practitioners and researchers. Researchers do not have to spend a lot of time developing their own testbeds and practitioners are able to use a common testbed to evaluate different technologies. 6 Benefits for Researchers This section provides a summary of the benefits the researchers obtained from using the software engineering technology testbeds components. Specifications and Code all researchers used the specifications and/or code to set up their evaluation. The specifications and code proved to be configurable as the researchers used the same set of artifacts. Without common specifications and/or code, a comparative evaluation would be impossible or at least more difficult to perform. Mission/Scenario Generator more experiments will have to be conducted to determine the relative effectiveness of having a mission generator in the testbed, but as with other scripting languages, its use would generally be more cost effective than manually preparing each test scenario. Instrumentation Roshandel (Mae) was able to use the instrumentation to quickly gather data for her performance analysis. For those researchers needing to collect data during the running of the code, the instrumentation class provided them a tool to quickly gather data. Seeded Defects and Defect Pool the seeded defect approach was effective in identifying the degree to which Mae could identify defects of various classes. However, after estimating nine likely remaining defects, we found that AcmeStudio alone discovered eight remaining defects, five of which were in categories (style usage, completeness) not in our defect categorization scheme. Thus it appears that the seeded defect technique s

20 598 Empir Software Eng (2009) 14: maximum likelihood estimate is better considered as a lower-bound estimate of the defects remaining in the categories constituting the current universe of defect sources. As an analogy, since the seeded defect technique derives from the use of fish tagging to estimate the total number of fish in a body of water, the technique can only estimate the number of fish catchable by the type of net used in catching tagged and untagged fish. There may be a number of smaller but significant fish (i.e., defects) swimming around undetected. Platform: Rover and Simulator Researchers were able to use the simulator and/or rover to execute the code and collect data for their performance analysis. Without a platform to run the code, the evaluation would be impossible for those researchers working with code or needing code to run. In addition, for those researchers using the simulator, it was a relatively low cost way for them to evaluate the technology. Experience Base/Technology Evaluation Results Roshandel and Garlan were able to use the evaluation results to indicate that their technologies were complementary and used it to demonstrate to JPL-MDS personnel that their technology can work on NASA software systems, leading to further usage of their technologies at JPL. Experimentation Guidelines Manuals researchers used the guidelines provided to learn how to conduct an experiment. The manuals were useful in teaching researchers about the MDS Framework and the SCRover testbed. Without the manual, researchers such as the ROPE team would have spent lots of time asking NASA questions about how the MDS technology worked. Project Data no researcher has used this testbed feature yet, but it is our belief that the data will be valuable for researchers developing technologies to estimate cost and schedule as it contains effort data in developing SCRover and defect data collected during the development phase. Ideally, the defect data would contain the time it took to fix the defect and how this impacted the schedule. Finally, one more benefit the researchers obtained from using SETTs is the early feedback on their technologies. Both Dr. Roshandel and the ROPE team indicated that by applying their technologies to the SCRover testbed first, the experience identified what improvements to their respective technologies were needed to be made first before they tried to convince NASA JPL users to adopt their respective technologies. The SCRover experience provided a better idea of how their technology should be applied to NASA-like systems, as opposed to the software systems they built themselves. 7 Conclusions From Section 2, Redwine and Riddle stated the following as difficulties for getting practitioners to adapt to new technologies (Redwine and Riddle 1985): No collection of prior experiences demonstrating positive feedback on a technology with the use of the SCRover testbed, researchers such as Dr. Roshandel and Garlan were able to provide practitioners (in this case, NASA) a positive experience with using their technology on a NASA-like representative system. In addition, the SCRover testbed helped demonstrate to NASA that the Mae and AcmeStudio technologies were indeed complimentary and could be used in combination to find a greater number of defects. Conceptual integrity by performing evaluations on SCRover, the researchers from Mae and ROPE were able to keep refining their technologies until they reached a point where they could demonstrate to NASA JPL that their technologies was well developed to work on their systems.

21 Empir Software Eng (2009) 14: Showing a clear recognition of need for the technology with the SCRover experience, researchers such as the AcmeStudio team and Dr. Roshandel were able to demonstrate to NASA JPL practitioners how well their technologies can detect certain classes of defects in a representative software system and how easy/difficult it was to apply the technology. Tuneability by applying their technologies on SCRover, researchers were able to indicate to NASA what kind of activities one would need to configure their technologies to work on a NASA-like system. Lack of training for the new technology with the SCRover experience, researchers were able to work with the NASA practitioners to identify what training needed to be provided as well as indicate technology adoption data such as, but not limited to, how long it took to apply the technology and how much training was involved before using the technology. In conclusion, this paper introduced the requirements, architecture, and concept of operation of a successfully-used software engineering technology testbed. The experiences of three technology evaluations on an instance of the SETT called SCRover were reported. The results and benefits each researcher obtained from using SCRover were presented, as well as how a practitioner can interpret the data obtained from the evaluations. This paper also included several charts that define the current domain of applicability of the testbed. And as a bottom line, the SCRover testbed provided a working example of how SETTs and their ability to provide users with comparable empirical data can overcome the challenges of technology adoption and maturation in order to increase the speed of the technology maturation and adoption process. Acknowledgements This work was supported by NASA-HDCP contracts to CMU, JPL, and USC. It also benefited from support from the JPL-MDS team and the NSF HDC programs that includes Dr. Roshanak Roshandel, Dr. Steve Fickas and his graduate students, Dr. David Garlan and the AcmeStudio team, Dr. Gupta, Dr. Helmy, Ganesha Bhaskara, and Dr. Carolyn Talcott. In addition, I d like to acknowledge the USC graduate students who helped in developing SCRover. References Benzel T, Braden R, Kim D, Neuman C, Joseph A, Ostrenga R et al Design, Deployment, and Use of the Deter Testbed. Proceedings of the DETER Community Workshop on Cyber Security Experimentation and Test, August Boehm B (1996) Anchoring the software process. IEEE Softw 73 82, (July). doi: / Boehm B, Port D Balancing discipline and flexibility with the Spiral Model and MBASE. Crosstalk, December 2001, pp ( Boehm, B. and USC Center for Software Engineering (2003) Guidelines for Model-Based (System) Architecting and Software Engineering. ( Boehm B, Bhuta J, Garlan D, Gradman E, Huang L, Lam A et al (2004) Using testbeds to accelerate technology maturity and transition: The SCRover Experience. ACM-IEEE International Symposium on Empirical Software Engineering, August, pp Booch G, Rumbaugh J, Jacobson I (1999) The unified modeling language user guide. Addison Wesley, Reading Chillarege R, Bhandari IS, Chaar JK, Halliday MJ, Moebus DS, Ray BK et al (1992) Orthogonal Defect Classification A Concept for In-Process Measurements. IEEE Trans Softw Eng 18(11). doi: / Dvorak D, Rasmussen R, Reeves G, Sacks A (2000) Software architecture themes in JPL s Mission Data System. Proceedings of 2000 IEEE Aerospace Conference. Fickas S, Prideaux J, Fortier A ROPE: Reasoning about OPerational Envelopes. uoregon.edu/research/mds/

22 600 Empir Software Eng (2009) 14: Garlan D, Monroe RT, Wile D (2000). Acme: architectural description of component-based systems. In: Leavens GT, Sitaraman M (eds) Foundations of component-based systems. Cambridge University Press Kruchten P (2001) The rational unified process (2nd edn). Addison Wesley, Reading Lindvall M, Rus I, Donzelli P, Memon A, Zelkowitz M, Betin-Can A et al (2007) Experimenting with software testbeds for evaluating new technologies. Empir Softw Eng 12(4): doi: / s Mettala E, Graham M (1992) The domain-specific software architecture program. Technical Report CMU/ SEI-92-SR-9, CMU Software Engineering Institute Mills H (1972) On The Statistical Validation of Computer Programs. IBM Federal Systems Division Report Redwine S, Riddle W (1985) Software technology maturation. Proceedings of the 8th International Conference on Software Engineering (ICSE1985), pp Rinker G (2002) Mission Data Systems Architecture and Implementation Guidelines. Ground System Architectures Workshop (GSAW 2002). El Segundo, California RoboCup < Roshandel R, Schmerl B, Medvidovic N, Garlan D, Zhang D (2004a) Understanding tradeoffs among different architectural modeling approaches. Proceedings of the 4th Working IEEE/IFIP Conference on Software Architecture (WICSA 2004) Oslo, Norway Roshandel R, van der Hoek A, Mikic-Rakic M, Medvidovic N (2004b) Mae a system model and environment for managing architectural evolution. ACM Trans Softw Eng Methodol 11(2): doi: / Roshandel R, Banerjee S, Cheung L, Medvidovic N, Golubchik L (2006) Estimating software component reliability by leveraging architectural models. 28th International Conference on Software Engineering (ICSE 2006), Shanghai, China, May, pp Stone P (2003) Multiagent competition and research: lessons from RoboCup and TAC. RoboCup-2002: Robot Soccer World Cup VI. Springer Verlag, Berlin, pp Tracz W (1995) DSSA (Domain-Specific Software Architecture) pedagogical example. ACM SIGSOFT Softw Eng Notes 20(3):49 62 Dr. Alexander Lam received his PhD in Computer Science from the University of Southern California (USC) in His research interests include software engineering, in particular software processes.

23 Empir Software Eng (2009) 14: Dr. Barry Boehm TRW Professor of Software Engineering, Computer Science Department, USC Director, USC Center for Systems and Software Engineering B.A., Math, Harvard, 1957; Ph.D., Math, UCLA, Dr. Barry Boehm served within the U.S. Department of Defense (DoD) from 1989 to 1992 as director of the DARPA Information Science and Technology Office and as director of the DDRE Software and Computer Technology Office. He worked at TRW from 1973 to 1989, culminating as chief scientist of the Defense Systems Group, and at the Rand Corporation from 1959 to 1973, culminating as head of the Information Sciences Department. He entered the software field at General Dynamics in His current research interests involve recasting systems and software engineering into a value-based framework, including processes, methods, tools, and an underlying theory and process for value-based systems and software definition, architecting, development, validation, and evolution. His contributions to the field include the Constructive Cost Model (COCOMO) family of systems and software engineering estimation models, the Spiral Model and Incremental Commitment Model of the systems and software engineering process, and the Theory W (win-win) approach to systems and software management and requirements determination. He has received the ACM Distinguished Research Award in Software Engineering and the IEEE Harlan Mills Award, and an honorary ScD in Computer Science from the University of Massachusetts. He is a Fellow of the primary professional societies in computing (ACM), aerospace (AIAA), electronics (IEEE), and systems engineering (INCOSE), and a member of the U.S. National Academy of Engineering.