A Complexity Reliability Model

Transcription

1 20h Inernaional Symposium on Sofware Reliabiliy Engineering A Complexiy Reliabiliy Model orm Schneidewind 1 and Mike Hinchey 2 1 aval Posgraduae School, Monerey, CA, USA 2 Lero he Irish Sofware Engineering Research Cenre, Universiy of Limerick, Ireland ieeelife@yahoo.com, mike.hinchey@lero.ie Absrac A model of sofware and reliabiliy is developed. I uses an evoluionary process o ransiion from one sofware sysem o he nex, while merics are used o predic he reliabiliy for each sysem. Our approach is experimenal, using daa perinen o he ASA saellie sysems applicaion environmen. We do no use sophisicaed mahemaical models ha may have lile relevance for he applicaion environmen. Raher, we ailor our approach o he sofware characerisics of he sofware o yield imporan defec-relaed predicors of qualiy. Sysems are esed unil he sofware passes defec presence crieria and is released. Tesing crieria are based on defec coun, defec densiy, and esing efficiency predicions exceeding specified hresholds. In addiion, anoher ype of esing efficiency a direced graph represening he of he sofware and defecs embedded in he code is used o evaluae he efficiency of defec deecion in ASA saellie sysem sofware. Complexiy merics were found o be good predicors of defecs and esing efficiency in his evoluionary process. 1. Inroducion Sofware developmen can be hough of as he evoluion of absrac requiremens ino a concree sofware sysem. Developmen, achieved hrough a successive series of ransformaions, is inherenly an evoluionary process. Sofware evoluion is ofen subopimal, because requisie informaion, such as reliabiliy and, may be missing during he ransformaions. While some undersanding of sofware may be reasonably clear a a given ime, fuure dependencies may no be fully undersood or accessible. The clarificaions obained over ime make he sysem more concreely undersood, bu here may be loss of relevan informaion. Some informaion may be los due o failure o be fully acquained wih dependencies among various sofware arifacs [5]. As poined ou by [11], as sysems change hrough successive builds, he characerisics of he individual modules ha make up he sysem also change. Changes o sofware sysems are measured on hese aribue domains o provide prime indicaors of poenial problems inroduced by he changes. Esablishing a measuremen baseline permis he comparison of a sequence of successive sofware builds. We adop his concep by using he firs sysem in a series of sofware sysems as he baseline for relaing o reliabiliy, and from his baseline, evolve successive releases wih decreasing and increasing reliabiliy. Complexiy decreases and reliabiliy increases across releases because even hough here may be added funcionaliy in subsequen releases, his is counerbalanced by evoluionary improvemens in sofware developmen concurren wih increasing faul removal. As Lehman poins ou [9], i is beneficial o deermine he number of disinc addiions and changes o sysems and consiuen modules of he sysem per release in order o assess sysem volailiy. This can assis evoluion release planning in a number of ways, for example by poining o sysem areas ha are ripe for resrucuring because of high defec raes. Some auhors have suggesed ha if he IT indusry used sandardized and inerchangeable sofware componens, he problem of unreliable sysems would largely disappear [4]. Unforunaely, for one-of-a-kind space sysems ha are addressed for solving unique research problems, he COTS soluion will no work. In space sysems, reliabiliy and across sysems will show considerable variaion. Therefore, our analysis canno be limied o a single sysem; i mus address muliple sysems, as we describe laer. Some researchers have found a Pareo effec when analyzing sofware defecs: a small number of modules accoun for a large number of defecs [1]. We did no U.S. Governmen Work o Proeced by U.S. Copyrigh DOI /ISSRE

2 find his characerisic in he ASA JM1 saellie sysem sofware we examined. This sofware has no paern of defecs across modules. Thus, we canno focus on a few modules o deec and remove defecs. Insead, he approach o be described involves conducing experimens o see wheher a mapping can be developed beween changes in sofware and changes in reliabiliy. If his mapping can be achieved wih he desired degree of saisical significance, he approach would be judged a success [13]. This research builds on previous work in sofware reliabiliy process [14]. However, we now focus on produc reliabiliy predicors in conras o previous work ha emphasized he process of predicion. 2. Complexiy Reliabiliy Model Sofware is he degree o which sofware is difficul o analyze, undersand, or explain [2]. If sofware lacks srucure, i will be difficul o undersand, es, and operae, and herefore, has a direc bearing on reliabiliy. In order o idenify beneficial sofware evoluionary seps, as hey relae o reliabiliy and sofware, we develop our reliabiliy model wih he aim of reducing and hereby increasing reliabiliy. Sofware reliabiliy of a program is hypohesized as a mulivariae a scalar a es or operaional ime. Why sudy in relaion o reliabiliy? The answer is ha breeds bugs. The more complex he sofware, he harder i is o make i reliable [16]. Thus, building saisical models for esimaing failure-proneness of sysems can help sofware organizaions make early decisions on he qualiy of heir sysems. Such early esimaes can be used o help inform decisions on esing, refacoring, code inspecions, design rework, ec. This has been demonsraed by he efficacy of building failureproneness models, based on code merics, across he Microsof Windows operaing sysem [2]. The abiliy of such models o esimae failureproneness and provide feedback on merics helps guide he evoluion of he sofware o higherand- higher plaeaus of reliabiliy. Some sofware projecs, such as he ASA JM1 saellie sysem, have no recorded failure daa, bu have recorded defec coun as a ime. Thus, we model he defec deecion process by leing D () be defec coun a ime, in equaion (0.1): D () = f (C, )), (0.1) where C is a scalar of merics M = m 1,, m i,, m n, and m i = cyclomaic (CC) and edge coun (EC) of he direced graph of he program. The defec and merics daa were obained from sofware modules in several JM1 saellie sysems. Based on some researchers resuls, one may quesion why we use cyclomaic. According o [6], heir research validaes previously raised concerns abou he meric, using a new daa se. A simple ransformaion of he meric was invesigaed whereby cyclomaic divided by he size of he sysem in source saemens produced a densiy raio. This raio was demonsraed o be a useful predicor of sofware mainenance produciviy on a small sample of mainenance projecs. While his may be rue for mainenance produciviy on a small sample, we will show ha for hree safey criical sofware sysems, comprised of 88 modules, cyclomaic, along wih edge coun, provides accurae predicions of reliabiliy. Furhermore, we do no hink i wise o compue a densiy raio because cyclomaic may be highly correlaed wih number of source saemens (e.g., for he ASA daa). McCabe s cyclomaic measure [10] provides an indicaion of how difficul i is o undersand and es a program by calculaing he number of linearly independen pahs hrough a program. The measure also represens he oal number of decision poins in a program plus one. McCabe proposed ha modules wih a value exceeding 10 may be problemaic by being difficul o undersand [7]. Therefore, in Figure 1 we show ha cyclomaic, wih mean value much larger han 10, and large mean edge coun, provide imporan independen variables in predicing defec coun. The pracical effec of his resul is ha a reliabiliy engineer could develop predicor equaions for oher sofware and daa, using his approach, o predic defec coun. We also ried a predicion funcion ha included source lines of code (sloc) coun bu doing his produced a Mean Relaive Error, MRE = compared o MRE =.2631 when sloc was no included. The reason for his resul is ha sloc has high correlaion of and wih CC and EC, respecively. The reader migh wonder why boh cyclomaic and edge coun are included in he regression equaion when cyclomaic is a edge coun in he direced graph of a program. We wondered abou his oo and developed a regression equaion of defec coun solely as a funcion of edge coun. Sill, he MRE = of his equaion 2

3 is larger han he MRE =.2631, when boh cyclomaic and edge coun are included. The reason is ha defec coun is dependen on cyclomaic. Therefore, cyclomaic mus be included when predicing defec coun. Finally, we invesigaed node coun as he sole predicor of defec coun. Since node coun, C, is no available in he daa, we compued i in equaion (0.2) based on he definiion of cyclomaic dependency: C = (EC CC) + 1 (0.2) This approach did no work because using an equaion wih jus C as he independen variable produced he highes error (MRE = ) of any of he regression equaions. Thus, we seled on he regression equaion ha included cyclomaic and edge coun. 3. Produc Line Concep Applied o Complexiy Reliabiliy Model When dealing wih complex sysems, and in paricular sysems exhibiing any form of auonomy or auonomic properies, i is unrealisic o assume ha he sysem will be saic. Complex sysems evolve over ime, and he archiecure of an evolving sysem will change even a run ime, as he sysem implemens self-configuraion, self-adapaion, and mees he challenges of is environmen. An evolving sysem can be viewed as muliple versions of he same sysem. Tha is, as he sysem evolves i represens muliple insances of he same sysem, each wih is own variaions and specific changes. Tha is o say, an evolving sysem may be viewed as a produc line of sysems, where he core archiecure of he produc line is fixed (i.e., he subsanial par of he sysem ha does no change), and each version of he evolving sysem may be viewed as a paricular produc from he produc line [12]. This concep can be applied o our reliabiliy model by considering he core sysem, as he firs in a series of sysems, comprised of a se of modules, each sysem wih evolving ses of defecs and aribues. Our objecive, hen, is o model reliabiliy across he produc line, providing a holisic view of reliabiliy ha we find is rarely achieved in pracice. We consider he evoluion of sysems as progressing o he poin where a sysem has me he defec coun goal a a specified es ime and can be released for operaional usage. As long as his goal is no saisfied, he sofware sysems coninue o evolve, as he resul of coninuing esing. Complexiy merics are used o assess wheher defec coun is decreasing. Figure 2 provides an overview of he evoluion of ASA s JM1 saellie sofware as i progresses hrough defec deecion, esing, and defec removal. As he sysems ha comprise his sofware ransiion hrough hese process seps, hey are subjeced o defec densiy, defec coun, and esing efficiency checks ha will be described in deail laer. 4. Using Defec Densiy as a Meric of Sofware Evoluion Defec densiy DD (), ha is number of defecs divided by program size, presen a ime, is considered he de faco measure of sofware qualiy (and reliabiliy) [3]. While his is rue, we consider i o be also a measure of sofware evoluion because, as he sofware evolves based on he removal of defecs, defec densiy should asympoically approach zero. For he ASA sofware ha we evaluaed, we will consider.02 as he defec densiy goal (i.e., 1 defec per 50 source language saemens). This goal is judgmenal based on he range of defec densiies observed for his applicaion. Figure 3 is used o invesigae wheher he evoluion of sofware releases is achieving he defec densiy goal. To do his, we plo prediced DD () for hree sysems of modules. Sysem 1 is esed for = 22 and Sysems 2 and 3 are esed for = 22 and hen operaed for 8 ime inervals. We see ha none of he sysems consisenly achieve he goal of.02 defecs per source language saemen. Thus addiional inspecion and esing is necessary. 4.1 Cos of Tesing While we have no informaion on he acual cos of esing for he ASA applicaion, we can use a surrogae, n, he number of random ess required o achieve a probabiliy of failure bound P f, wih a confidence of, in equaion (0.3) [8]: log(1 ) n = [ ] 1 (0.3) log(1 P f ) The probabiliy of failure bound can be compued from he defec coun a ime is summed over m modules, as follows: 3

4 P f = D () m 1 D () (0.4) Equaions (0.3) and (0.4) are pu o use in Figure 4 ha shows ha he siuaion is sabilizing wih respec o number of ess, as addiional sysems are analyzed. Tha is, he number of ess for Sysem 2 has much less variabiliy han for Sysem 1, and Sysem 3 has much less variabiliy han Sysem 2. This is evidence of progress in he evoluion of releases, wih respec o operaing ime bu, more imporan, i will be necessary o see wheher he reliabiliy goal is me based on number of ess. This check is made in Figure 5, where if Sysem 3 is esed 250 imes, he prediced defec coun is less han one. If he reliabiliy goal, in erms of defec coun is less han one, he reliabiliy goal has been achieved. Wheher ASA could afford 250 ess can be assessed by predicing he number of defecs removed in he ess and wheher his would be a coseffecive process. This issue will be addressed nex by compuing he probabiliy of repairing defecs a ime, P r (), in equaion (0.5) [15]. nr () Probabiliy of defec repair: P r () =, (0.5) D () where nr () is he number of defecs repaired a ime. ow o obain anoher expression for P r (), we can reason ha i is he raio of cumulaive number of ess a ime, for deecing and repairing defecs, o he cumulaive number of ess over m modules, where i is a summaion index, in equaion (0.6): P r () = i 1 m i 1 n () n () (0.6) Then equaing equaions (0.5) and (0.6), we compue he expeced number of defecs repaired a ime in equaion (0.7): nr () = D ()* n () i 1 i 1 m n () (0.7) Once nr () is compued, we define esing efficiency e () a ime in equaion (0.8), which is also equal o he probabiliy of defec repair in equaion (0.5): e () = n r () / D () (0.8) Figure 6 indicaes success in esing and herefore success in achieving he reliabiliy goal because a es ime = 28, for Sysem 3, esing efficiency exceeds he limi of.90. A = 30, we expec ha all defecs would be repaired. 25 Series 1: acual D () Series 2: prediced D () = (.0123 * CC) + (.0137 * EC) CC = cyclomaic, mean = , EC = edge coun, mean = MRE =.2613 Series1 Series2 15 D () Figure 1. Defec Coun D () vs. Tes Time, ASA JM1 Sofware 4

5 5. Efficiency of Tesing Using Direced Graph of Program One of he imporan benefis of auomaically generaing es cases is freedom from bias. When called upon o consruc es cases, sofware developers usually suffer from biases ha resul in underexploraion of many esing regions. Knowledge of sysem inernals and operaional expecaions influence he design of es suies. Auomaed ools can augmen he esing process by providing es cases ha are no suscepible o he same biases as he human developers [17]. One major facor lef ou of his argumen is ha humans design he es ools and heir biases are injeced ino he ools! Therefore, we believe i is imporan o consider he inernals of sofware when developing es sraegies, and o do i in an unbiased fashion by exercising all pahs in ess, as described below. Previously, we had prediced number of ess based on probabiliy of defec repair ha, in urn, is based on prediced defec coun in equaion (0.1). While prediced defec coun uses program merics in is compuaion, here is srucural informaion of a program, like is direced graph, ha is no used. In order o assess esing efficiency, using program srucure, we developed a direced graph of module 30 of Sysem 3 in Figure 7. We picked his module because Figure 6 old us ha is esing efficiency, as compued in Equaion (0.8), is Thus, we were ineresed in comparing esing efficiency by wo mehods he second mehod based Figure 2. ASA Saellie Sofware JM1 Evoluionary Process Sysem 1 defec deecion predic D () as a merics predic DD () as a merics predic number of ess as a probabiliy of failure predic defec coun as a number of ess D () < limi DD () < limi e () > limi Sysem 2 release Sysem 1 defec deecion predic D () as a merics predic DD () as a merics predic number of ess as a probabiliy of failure predic defec coun as a number of ess Sysem 3 D () < limi DD () < limi e () > limi defec deecion predic D () as a merics predic DD () as a merics predic number of ess as a probabiliy of failure release Sysem 2 predic defec coun as a number of ess D () < limi DD () < limi e () > limi defec coun: D () defec densiy: DD () esing efficiciency: e () process nex sysem release Sysem 3 5

6 on efficiency of deecing defecs using pah esing in a direced graph. Figure 3. Evoluion of Prediced Defec Densiy DD () vs. Time, ASA JM1Sofware DD () (defecs per sloc) Series 2: DD (): es ime, 22 modules, Sysem 1 Series 4: DD (): es ime + operaional ime, 30 modules, Sysem 2 Series 1: DD () es ime + operaional ime, 30 modules, Sysem 3 Series 3: DD () limi =.0200 Series2 Series4 Series1 Series3 es ime operaional ime

7 Figure 4. Evoluion of Prediced umber of Random Tess n () vs. Time, ASA JM1 Sofware n Series1 Series2 Series Series 1: Sysem 1, 22 modules Series 2: Sysem 2, 30 modules, original defec coun predicion es ime operaional ime D (n) (.011 n) D (n) =.9143 e R 2 =.9615 w ih n = 250, D () < n Figure 5. Evoluion of Prediced Defec Coun D (n) vs. umber of Tess n () for Sysem 3 of ASA JM1 Sofware 7

8 Series 2: e () for Sysem 3, 30 modules Series 1: e () limi exceeds limi a = 28 e () Series2 Series Figure 6. Tesing Efficiency e () vs. Tes Time Edge Probabiliies: 1, 2:.333 1, 3:.333 1,4:.333 2, 4: , 4: , 2:.500 4, 3:.500 Figure 7. Direced Graph of ASA JM1 Module 30 in Sysem 3 sar node 1 example pah: 1, 2, 4, 2, 4 pah probabiliy =.333 * *.500 * = erminal node We documen he parameers of he direced graph of Figure 7 in Tables 1 and 2, where we assume ha edge probabiliies are equal and ha only a single ieraion is performed where loops occur in he program. We make hese assumpions because we do no have he code for he ASA JM1 sofware. We only have cyclomaic, edge coun, and defec coun per module. While hese assumpions obviously affec defec predicion for individual modules, here is no effec on a relaive basis across modules. Referencing Table 2, where he oal expeced number of defecs deeced = D () and he number of pah ess required o deec he defecs = 7, he pah esing efficiency is compued as follows: In oher words, for each defec presen in he sofware, we would expec o deec.5719 of hem wih pah esing. ow his is much less han he e () = from Figure 6 based on random esing. However, he laer does no accoun for he srucure of he sofware, and herefore, is accuracy may suffer relaive o he direced graph approach. Thus, we believe i is beer o do he analysis of Tables 1 and 2 and Figure 7, bu recognizing he greaer ime and effor involved. Tesing efficiency = D () / 7 = D (). 8

9 Table 1. Pahs and Pah Probabiliies CC =4, EC = 7, C =4 Pah Idenificaion Edge Sequence Pah Probabiliy 1 1, 2, , 3, , , 2, 4, 2, * *.500 * = , 3, 4, 3, * *.500 * = , 4, 2, *.500 * 1.000= , 4, 3, * *.500 * =.167 Table 2. Pahs, Edges, and Expeced umber of Defecs Deeced (one defec per edge assumed) Pah Idenificaion Pah Probabiliy umber of Expeced D () Edges *2 = *2 = *1 = *4 = *4 = *3 = *3 =.501 Toal 7 pah ess D () 8. Conclusions We found ha for he ASA saellie JM1 sofware, he merics, cyclomaic, and edge coun are accurae predicors of defec coun and defec densiy. However, we do no claim ha hese resuls would hold for all sofware because oher researchers have arrived a conrary resuls. For example, in [3], he auhors did no find evidence in evaluaing heir paricular sofware ha merics are good predicors of eiher faul-prone or failure-prone modules. In addiion, hey have reservaions abou using faul densiy as a predicive meric. Since defecs usually lead o fauls, he same assessmen could be made of defec densiy. In anoher depressing repor, i was concluded ha a major problem is he inabiliy of measures o predic differen sofware qualiy aribues accuraely, for example, reliabiliy, and mainainabiliy [18]. Bu wha is mean by accuraely? In our experience, over a number of ASA and DoD sofware projecs, a reliabiliy predicion accuracy of MRE = can be obained. While his may no be considered accurae, i is beer han no predicion! Therefore, all is no los because while resuls will vary dependen on he characerisics of sofware, i is imporan o adop a process, such as he one used in his research, o evaluae sofware reliabiliy. We also conclude ha using an evoluionary model of sofware reliabiliy and is advanageous because i provides he abiliy o ierae he predicors across successive sysems unil he desired reliabiliy is achieved and he sofware can be released. In addiion, several crierions for judging wheher sofware should be released should be used. Among hese are saisfying defec coun, defec densiy, and esing efficiency hresholds. Finally, we observed ha when evaluaing esing efficiency, i is imporan o use srucural informaion such as he direced graph of a program. Acknowledgmen This work was suppored in par by Science Foundaion Ireland gran 03/CE2/I303_1 o Lero he Irish Sofware Engineering Research Cenre ( References 1. C. Andersson and P. Runeson, A Replicaed Quaniaive Analysis of Faul Disribuions in Complex Sofware Sysems, IEEE Trans. on Sofware Engineering, 33(5): , May S.A. Bohner, An Era of Change-Toleran Sysems, IEEE Compuer, June E. Fenon,. Ohlsson, Quaniaive Analysis of Fauls and Failures in a Complex Sofware Sysem, IEEE Trans. on Sofware Engineering, 26(8): , Augus J.L. Fiadeiro, Designing for Sofware's Social Complexiy, Compuer, 40(1):34-39, January B. George, S.A. Bohner, R. Prieo-Diaz, Sofware Informaion Leaks: A Complexiy Perspecive, In Proc. 9

10 inh IEEE Inernaional Conference on Engineering Complex Compuer Sysems (ICECCS 04), pp , IEEE Compuer Sociey Press, G.K. Gill and C.F. Kemerer, Cyclomaic Complexiy Densiy and Sofware Mainenance Produciviy, IEEE Trans. on Sofware Engineering, 17(12): , December E. Gold, A.M. Mohan and P.J. Layzell, Spaial Complexiy Merics: An Invesigaion of Uiliy, IEEE Trans. on Sofware Engineering, 31(3): , March W. Jones and M.A. Vouk, Field Daa Analysis, In M.R. Lyu, ed., Handbook of Sofware Reliabiliy Engineering, IEEE Compuer Sociey Press and McGraw-Hill Book Company, M. M. Lehman, Rules and Tools for Sofware Evoluion Planning and Managemen, Inernaional Workshop on Feedback And Evoluion In Sofware And Business Processes, Imperial College, London, U.K., July, Revised and exended version in Annals of Sofware Engineering 11:15-44, ovember T. McCabe, A Sofware Complexiy Measure, IEEE Trans. on Sofware Engineering, 2(4): , April J.C. Munson and D.S. Werries, Measuring sofware evoluion, In Proc. Third Inernaional Sofware Merics Symposium (METRICS'96), J. Peña, M.G. Hinchey, M. Resinas, R. Serri and J.L. Rash, Designing and managing evolving sysems using a MAS produc line approach, Science of Compuer Programming 66, pp 71 86, Elsevier, F. Schneidewind, Sofware Evoluion and Feedback, In. Madhavji, ed., Requiremens Risk and Sofware Reliabiliy, pp , John Wiley & Sons, F. Schneidewind, Sofware reliabiliy engineering process,, Innovaions in Sysems and Sofware Engineering: a ASA Journal, 3(2): , Springer, Sepember Schneidewind, Sofware Defec Process and Produc Model for High Assurance Applicaions, AIAA Journal of Aerospace Compuing, Informaion, and Communicaions, Volume 4, March L. Sha, Using Simpliciy o Conrol Complexiy, IEEE Sofware, 18(4):20-28, July/Augus, A. Wakins, D. Bernd, K. Aebischer, J. Fisher and L. Johnson, Breeding Sofware Tes Cases for Complex Sysems, In Proceedings of he 37h Annual Hawaii Inernaional Conference on Sysem Sciences (HICSS'04), Track 9, C. Wohlin, Revisiing Measuremen of Sofware Complexiy, In Proc. Third Asia-Pacific Sofware Engineering Conference (APSEC'96),