On Operational Availability of a Large Software-Based Telecommunications System

Transcription

1 On Operatonal Avalablty of a Large Software-Based Telecommuncatons System Randy Cramp and Mladen A. Vouk Wendell Jones North Carolna State Unversty BNR Inc. Department of Computer Scence, Box 826 P.O. Box Ralegh, NC Research Trangle Park,NC 2779 Abstract Modern telecommuncatons systems are dependent on software for ther successful operaton. In many cases, the software operates on an establshed hardware platform and subsequent releases of the system dffer prmarly n the software component. Because of the ncreased dependence of the socety on advanced telecommuncatons systems, the relablty and avalablty of network swtchng elements are very mportant. Emprcal nformaton on the operatonal unavalablty due to total system outages of a large software-based telecommuncatons system s presented and dscussed. About 5% of these outages are reported as beng caused by software. The data s used to show that the unavalablty of ths system can be descrbed well usng a classcal two-state avalablty model. 1. Introducton The mportance of relablty and avalablty n the context of modern telecommuncatons systems cannot be overemphaszed. These systems, to a very large extent, depend on software for ther successful operaton. In many cases, the software operates on an establshed hardware platform and subsequent upgrades of the system dffer prmarly n the software component. Because of the socety's ncreased dependence on advanced telecommuncatons, Bellcore has establshed generc requrements for the performance evaluaton of telecommuncatons systems [Bellcore, 199]. In order to dagnose the root causes of a falure (or outage) n network swtchng elements "t s mportant to gather outage data for all servce falures, regardless of cause, and to classfy the data by cause of falure" [Bellcore, 1989]. Ths ncludes hardware causes, software causes, human error causes, and envronmental varables. In ths paper, we dscuss avalablty of a large telecommuncatons swtchng system. A typcal Other 26% 2% Procedural 6% Hardware Software 48% Fgure 1.1 Typcal dstrbuton of reported causes of system falures. dstrbuton of the reported causes of outages for the system s shown n Fgure 1.1. We see that although software causes appear to account for almost 5% of the outages, system avalablty s sgnfcantly affected by a varety of the other types of causes. Ths llustrates the mportance of addressng all causes of falures when modelng the avalablty of a practcal software-based system. Many models have been proposed for evaluatng avalablty of software-based systems (e.g., [Trved, 1975], [Costes et al., 1978], [Shooman, 1983], [Lapre et al., 199], [Lapre et al., 1991], and [Lapre and Kanoun, 1992]). Some ncorporate nteracton wth hardware and attempt to account for dfferent types of falure causes (e.g., [Lapre and Kanoun, 1992]). In the remander of ths secton, we provde some background nformaton concernng the system we studed and the avalablty measures we use throughout ths paper. In Secton 2, we dscuss emprcal unavalablty. In Secton 3, we examne system falure and repar rates, and n Secton 4, usng an example, we show how a very smple avalablty model can be used to descrbe and predct avalablty of the system studed. In order to protect propretary nformaton, we have removed numercal nformaton from the graph ordnates.

2 Smlarly, the unts have been removed from the abscssae. 1.1 System The system dscussed n ths paper has two prncpal swtchng products runnng smlar software, but dstngushed by ther hardware compostons. We refer to these separate systems (or products) as P1 and P2. Snce the software runnng on these systems generally changes more frequently than the hardware, we let a new software release represent the upgrade to a new system. A software release s normally nstalled at hundreds of stes. We dstngush among the releases by ther release numbers (e.g., R7, R8, etc.). Software lbrares for ths system exceed 1 mllon lnes of hgh level code. A typcal executable software load conssts of approxmately 7 mllon lnes of hgh level code of whch 1% s new or modfed code. As part of the software process and product qualty assurance actvtes a large amount of data s collected regularly on the development and performance of each product release. The data ncludes falure and fault nformaton resultng from the product testng and ts operatonal, or feld, phases. For example, n addton to the exact product verson and nstallaton nformaton for each ste, the database stores calendar dates of the outages, duraton of the outages, and cause classfcatons made by the feld offce (e.g., hardware, software, procedural, etc.). 1.2 "" System usage s central to the dea of relablty and avalablty. Therefore, t s very mportant to choose an approprate measure of usage. For example, system usage can be measured n nservce tme or calendar tme. We defne nservce tme as the cumulatve tme a gven software release has been executed over all stes usng that release. In contrast, calendar tme s the cumulatve tme (.e. weeks, months, etc.) that has elapsed snce the software was released. At dfferent calendar tmes dfferent software releases wll have been nstalled at a dfferent number of stes. Ths means that the usage ntensty of the system vares over calendar tme and changes accordng to the amount of tme the stes usng the release have been n servce. Therefore, from both hardware and software vewponts exposure to usage nservce tme s a more representatve measure of usage than calendar tme. However, calendar tme s a measure that n all lkelhood better reflects the percepton of users (.e., the telecommuncatons companes) snce calendar tme avalablty and degradaton of servces are very mportant from the customers' pont of vew. The number of stes that operate a partcular product release at any tme vares wth product type and release verson. Fgure 1.2 llustrates the number of stes n servce for the two dfferent product types. Fgure 1.3 compares the nservce tme wth calendar tme for dfferent releases of one partcular product type. We see that the "load" on the product software vares consderably among product platforms and releases. For example, n Fgure 1.3, we see that at 5 calendar tme unts release R1 has had less nservce tme than release R9 and R11. Number of Stes Release R Calendar Fgure 1.2 Calendar tme varaton n the number of stes operatng release R1. Ths needs to be taken nto account when detaled performance models of the system are consdered. For nstance, the larger the number of stes usng a product release wthn a gven calendar tme perod the larger s the rate of accumulaton of nservce tme durng that perod. Ths means that releases whch are nstalled on more stes provde a larger exposure sample wthn a shorter calendar tme perod and acheve larger nservce tme earler n the calendar lfe of a release. Ths n turn means that some of the characterstcs of the avalablty functons, such as ts transent perod, may end much sooner n the wdely deployed releases than n the case of releases that are nstalled at only a few stes. Inservce R11 5 Calendar P2 1 P1 8 1 P2 Stes Fgure 1.3 Inservce tme versus calendar tme for varous releases of P2. R9 R1 R8 15

3 1.3 Avalablty We use the general term of "falure" to represent a system outage. An outage s defned as a loss or degradaton of servce to a customer for a perod of tme (outage duraton). In general, outages can be caused by hardware or software falures, human errors, and envronmental varables (e.g., lghtnng, power falures, fre, etc.). A falure resultng n the loss of functonalty of the entre system s called a "total system outage" [Bellcore, 1989]. A total (telephone) system outage mples a "loss of orgnaton or termnaton capablty n all termnatons for a perod n excess of 3 seconds, or loss of all stable calls for a perod n excess of 3 seconds." Successful recovery from a total system outage occurs "when both of the followng apply: ) orgnaton and termnaton capablty n all termnatons s restored, and ) the system's engneered call handlng capacty has been restored" [Bellcore, 1989]. In ths paper, we are only concerned wth total system outages Instantaneous Avalablty Instantaneous avalablty s the probablty that the system wll be avalable at any random tme t durng ts lfe [Sandler, 1963]. We estmate "nstantaneous" avalablty n a perod as follows: ^A () = uptme n perod total nservce tme for perod (1.1) where the nservce tme s the total tme n the perod durng whch all swtches of a partcular type (.e., P1 or P2) operated a partcular software release (whether fully operatonal, partly degraded, or under repar), whle uptme s the total tme durng perod at whch the swtches were not n the "1% down" state (or total system outage state). Correspondngly, the nstantaneous unavalablty estmate s (1 - ^A()). Assocated wth ths measure are "nstantaneous" system falure, λ(), and recovery rates, µ(), whch are estmated as follows: ^λ () = number of outages n perod total uptme for perod ^µ () = number of outages n perod total downtme for perod (1.2) (1.3) where "nservce tme" for perod s the sum of the downtme and uptme n that perod Uptme Avalablty Snce the raw data are often "nosy", the data are usually presented after some form of smoothng has been appled. Ths gves rse to a famly of "smoothed" avalablty metrcs. In ths study, we frequently used both one-sded and symmetrcal movng average smoothng (e.g., 11- pont symmetrcal movng average). An extreme form of smoothng s provded by the uptme, or average, avalablty. Uptme avalablty s the proporton of tme n a specfed nterval [,T] that the system s avalable for use [Sandler, 1963]. We estmate uptme avalablty up to and ncludng perod as follows: uptme n perod x x=1 Ac() = total nservce tme for perod x x=1 (1.4) Total uptme and total nservce tme are cumulatve sums startng wth the frst observaton related to a partcular release. Uptme ncludes degraded servce. Assocated wth uptme avalablty are average system falure, ^λc(), and recovery rates, ^µc(), whch are estmated as follows: number of outages n perod x x=1 ^λc() = uptme n perod x x=1 number of outages n perod x x=1 ^µc() = downtme n perod x x=1 2. Emprcal Unavalablty (1.5) (1.6) Fgure 2.1 llustrates a typcal unavalablty observed for the system. In addton to the "nstantaneous" data, we show the nfluence of dfferent smoothng approaches. Each "raw" data pont on the graph corresponds to the data collected durng one calendar perod 1. Abrupt changes n nstantaneous unavalablty are smoothed by uptme averagng. Under the assumpton that product unavalablty becomes smaller wth tme, we would expect uptme unavalablty estmates to be generally conservatve. We found the 11-pont symmetrcal movng average smoothng useful for examnng trends n the nstantaneous unavalablty. 1 Note: In order to draw the "raw" data on the logarthmc scale, the data ponts (perods) where no falures were observed (.e. zero falure rate) were censored and only the adjacent non-zero falure rate ponts are shown.

4 Fgure 2.2 shows uptme unavalablty of P2 for releases R8 through R11 versus nservce tme. In all releases, we see that there s a perod mmedately after the product s made avalable to customers n whch there s consderable oscllaton n the observed unavalablty. We refer to the tme perod from the pont where the product s made avalable to the customers (.e., tme zero) to the pont where the nstabltes abate as the "transent regon." It corresponds to the transent part of the avalablty functon. The duraton of ths regon of nstablty depends on the product type and release. Once the nstablty has decayed, all releases exhbt farly smooth unavalablty decay curves whch n ths case can be approxmated by almost straght lnes (Note: the ordnate s drawn on logarthmc scale). Unavalablty Instantaneous Unavalablty 1 P2 Release R1 Uptme Unavalablty 2 Inservce Smoothed Instantaneous Unavalablty (11-pont, symmetrcal, movng average) Fgure 2.1 Illustraton of smoothng optons appled to release R1 of P2. Uptme Unavalablty Product P2 1 R8 R1 2 3 Inservce R9 4 3 R Fgure 2.2 Varablty of uptme unavalablty for dfferent P2 releases. Fgure 2.3 compares smoothed nstantaneous (11-pont symmetrc movng average) and uptme unavalablty of P1 and P2 for release R1 usng lnear ordnate. Notce the maxmum n the P2 unavalablty functons characterstc of relablty growth [Lapre and Kanoun, 1992]. Unavalablty 1 P2 (Uptme) P2 (11-pont symmetrcal smoothng) 2 Inservce Release R1 P1 (11-pont symmetrcal smoothng) 3 P1 (Uptme) Fgure 2.3 Smoothed and uptme unavalablty for Release R1. 3. Falure and Recovery Rates Two measures whch drectly nfluence the avalablty of a system are ts falure rate and ts feld repar rate (or swtch recovery rate). Fgure 3.1 shows P2 falure and recovery rates for release R11. Apart from the censored 2 "raw" data two other representatons are shown. In one the data are smoothed usng an 11-pont symmetrcal movng average. In the other, we show uptme data, or the cumulatve average of the data. In a system whch mproves wth feld usage we would expect a decreasng functon for falure rate wth nservce tme (mplyng fault or problem reducton and relablty growth). Ths was behavor s observed (see Fgures 3.1 and 3.2). Immedately after the product release date, there s consderable varaton n the falure rate. Later the falure rate reduces and stablzes. Falure rate s connected to both the operatonal usage profle and the process of problem resoluton and correcton. Recovery rate depends on the operatonal usage profle, the type of problem encountered, and the feld response to that problem (.e., the duraton of outages n ths case). If the falures encountered durng the operatonal phase of the release do not exhbt duratons whch would be preferentally longer or shorter at a pont (or perod) n the lfe-cycle, then we would expect the "nstantaneous" recovery rate to be a level functon wth nservce tme (wth, perhaps, some oscllatons n the early stages). Ths behavor was generally observed. In Fgure 3.1, we see that the smoothed recovery rate s approxmately constant wth age, although there are consderable oscllatons n the "raw" data. It s nterestng to note that the recovery rate s about 3 to 4 orders of magntude larger than the falure rate, and snce the recovery rate s approxmately constant, t would be expected that the avalablty would be governed prmarly 4 2 Zero valued data ponts are not shown n order to allow the use of logarthmc scale on the ordnate.

5 by the stochastc changes n the falure rate (e.g., [Lapre et al., 1991]). Fgures 3.2 and 3.3 depct the uptme falure and recovery rate curves for releases R7 through R11. Falure rate curves have smlar shapes and suggest an exponentally or logarthmcally decayng functon once past the transent regon. There s more varety among the recovery rate shapes. The transent perod n the recovery rate occurs on a longer nterval than the falure rate and suggests consderable varance n the duraton of falures n the transent regon. 4. Models The tme varyng nature of both the falure rate and repar rate ndcates that a full avalablty model whch would descrbe the system n ths study should be nonhomogeneous. In addton, the dstrbuton of outage causes as well as the possblty of operaton n degraded up states suggest that a detaled model should be manystate. In ths secton, we show that nonetheless a very smple two-state model may provde a reasonable descrpton of the system avalablty beyond the transent regon. Falure Rate vs Recovery Rate Outages per mnute Uptme Recovery Rate Recovery Rate 11-pt Smoothed Recovery Rate Uptme Falure Rate 11-pt Smoothed Falure Rate Falure Rate Recovery Falure Rate Rate Inservce Fgure 3.1 Feld recovery and falure rates for P2 release R11 computed usng cumulatve outages, downtme and uptme P2 Uptme Falure Rate Releases R7-R11 P2 Uptme Recovery Rate Releases R7 - R11 Outages per Mnute R1 R7 R11 R9 Outages per Mnute R8 R7 R1 R9 R11 R Inservce 4 Fgure 3.2 Uptme falure rates for P2 systems for releases R7 - R Inservce 4 Fgure 3.3 Uptme feld recovery rates for P2 systems under releases R7 - R

6 4.1 Homogeneous Two State Markov Models Consder a dscrete-tme stochastc process {Xn, n =1,2,...} where Xn s a random varable denotng the state of the process at tme n. Let the stochastc process be a Markov chan. Let there be two states whch we call state and state 1. In state, the system s operatonal, whle n state 1, the system s under repar. Let the condtonal probablty of falure n the perod t, t + dt be λdt (.e., probablty of transton to states 1 gven the system s n state ), and the condtonal probablty of completng repar n t, t + dt be µdt (.e., probablty of transton to state gven the system s n state 1). Then λ s falure rate and µ s repar and/or recovery rate, and the one-step state transton dagram s Runnng (1 λ) τ t λ τ t 1 t µ τ t Under Repar (1 µ) τ The one-step transton probabltes are P = 1 - λdt, P1 = λdt, P1 = µdt and P11 = 1- µdt, where Pj denotes transton probablty between state and state j. The onestep transton matrx s then: P = P = 1 - λdt P1 = λdt P1 = µdt P11 = 1- µdt (4.1) In general, n th step transton probablty, P n, (.e., probablty that the process startng n state wll be n state j after n transtons), s expressed by Chapman- Kolmogorov equatons (e.g., [Trved, 1982], [Musa et al., 1987]). Gven that the ntal system probablty vector s v(t=) = [P(), P1()], where P(t) denotes the probablty that the system s n state at tme t, then after n transtons (and at tme t) v(n,t=t) = v()p n (4.2) The probablty that the system that starts n state (.e., v() = [1,]) ends n state after n transtons and at tme t s the system avalablty A(t) = 1 x P n, and A(t) = 1 - A(t) s system unavalablty. Furthermore, assumng a Posson arrval process for falures, an exponental dstrbuton for repar tmes, and the same ntal probablty vector, we can set and solve dfferental equatons descrbng the homogeneous process (e.g., [Trved, 1982], [Shooman, 1983]): A(t) = P (t) = 1 - A(t) = P 1 (t) = µ λ+µ + λ λ+µ e-(λ+µ)t λ λ+µ - λ λ+µ e-(λ+µ)t (4.3) The measure of uptme avalablty can be formally defned as the expectaton From (4-3) we get µ A c (T) = λ+µ + A c (T) = 1 T Τ A(t)dt (4.4) λ (λ+µ) 2 T λ (λ+µ) 2 T e-(λ+µ)t (4.5) The system becomes ndependent of ts startng state after operatng for enough tme for the transent part of (4.3) and (4.5) to decay away. Ths steady-state avalablty of the system s A( ) = lmt{a(t = T )},.e., µ A( ) = λ+µ (4.6) 4.2 Nonhomogeneous Two-State Markov Models The two-state model dscussed above represents a system whch can be ether fully operatonal or completely offlne and under repar. However, not all realstc systems follow ths smple model. In fact, P1 and P2 not only have falure rates and repar rates whch vary wth tme, and can have dfferent down states (e.g., FCC reportable or not [FCC, 1992]), but they can also functon n more than one up state (.e., the system may reman operatonal but wth less than 1% functonalty for some falures). Thus, a many-state nonhomogeneous Markov model may be more approprate for descrbng the detals of these systems (e.g., [Trved, 1975], [Lapre and Kanoun, 1992], [Ibe and Wen, 1992]). Nevertheless, a classcal two-state model for avalablty of recoverable systems based on constant falure rate λ and constant recovery rate µ can be used to approxmate behavor of more complex nonhomogeneous systems such as P1 and P2. We llustrate ths through two approaches: one we call "steady-state" approxmaton, and another we call n-step approxmaton. The "steady-state" approxmaton s based on observatons made by Trved [Trved, 1975] and Shooman [Shooman, 1983]. They noted that once the system has been

7 operatonal for some tme, the steady-state equaton (4.6) may be used to approxmate the nstantaneous avalablty by assumng a pecewse-constant varaton of λ and µ n tme. Lettng ^λ(t) and ^µ(t) be estmates at tme t for λ and µ, respectvely, we can estmate nstantaneous avalablty as ^A(t) = ^µ(t) ^λ(t)+^µ(t) (4.7) The ^λ(t) and ^µ(t) approxmatons can be obtaned from the emprcal data: the former through applcaton of a relablty model, the latter s often assumed to be a constant (e.g., [Lapre et al., 1991]). The n-step transton approxmaton s a numercal approach. Chapman-Kolmogorov equatons are replaced by a seres of matrx multplcatons descrbng a seres of one-step transtons wth tme-varyng values for λ and µ. The λ and µ functons are obtaned from separate models and are recomputed and substtuted wth each ncrement, t, n tme. If the tme ncrement s small enough and the arthmetc s accurate enough the approxmaton provdes an acceptable descrpton of the actual system behavor even n the transent regon. That s P n (t+ t) P n-1 (t) Avalablty follows from (4.2). 4.3 Example 1 - ^λ(t) t ^λ(t) t ^µ(t) t 1- ^µ(t) t (4.8) Ths example provdes a smple llustraton of the applcaton of the above approxmatons. Consder the release R11 for product P2. Fgure 3.1 shows the P2 falure and repar rates for the same release. From Fgure 3.1, we see that the uptme recovery rate s approxmately constant once suffcent nservce tme has passed. There s more varaton n the nstantaneous rate and the 11- pont smoothed rate. We make the smplfyng assumpton that the recovery rate s constant and choose t to be the average of the perod beng consdered (.e., t s the uptme recovery rate of the sample pont wth the largest nservce tme). Fgure 4.1 shows the unavalablty profle of P2 for R11. Fgure 4.2 shows the correspondng nstantaneous falure rate and a model ft. From earler work by Jones (e.g., [Jones, 1991], [Jones, 1992]) we know that the Logarthmc Posson executon tme (LPET) model ([Musa and Okumoto, 1984], [Musa et al., 1987]) provdes a good descrptve, as well as predctve, model for the falure rate of the P1 and P2 systems. Therefore, we used the LPET model to model the nstantaneous falure rate. The least squares ft s shown n Fgure 4.2. Unavalablty Instantaneous Unavalablty 1 Smoothed Instantaneous Unavalablty (11 pont, symmetrcal, movng average) 2 3 Inservce Uptme Unavalablty Fgure 4.1 Unavalablty data for. Rate (outages/mnute) Falure 1 Instantaneous Falure Rate LPET ft Inservce Fgure 4.2 LPET least squares ft to falure rate. The LPET falure rate equaton wth the parameters obtaned from the ft and the constant repar rate were used to compute both the n-step and the "steady-state" approxmatons for nstantaneous unavalablty. The tme step ( t) used n these n-step approxmatons was.1. These nstantaneous unavalablty approxmatons, along wth the emprcal unavalablty, are shown n Fgure 4.3. We see that both approxmatons practcally concde once past the transent perod. However, both approxmatons le below uptme unavalablty, shown n the fgure as the thck sold lne, because they model nstantaneous unavalablty whch tends to be less conservatve than the uptme unavalablty. 5 6

8 Instantaneous unavalablty Uptme unavalablty (as upper bound) n-step approxmaton (.1) "steady-state" approxmaton Unavalablty Constant Repar Rate LPET Ft to Instantaneous Falure Rate Inservce Fgure 4.3 Unavalablty fttng usng LPET. 5 6 Unavalablty Uptme unavalablty computed usng "steady-state" approxmaton based on LPET ft to falure data up to "Cut-off pont" Cut-off pont Emprcal Uptme unavalablty Instantaneous unavalablty Instantaneous unavalablty computed usng "steady-state" approxmaton based on LPET ft to falure data up to "Cut-off pont" Inservce Fgure 4.4 Unavalablty fttng usng LPET wth data up to "cut-off pont" only. It s nterestng to note that the n-step approxmaton shows a maxmum typcal of systems that experence relablty growth. In contrast, the "steady-state" approxmaton does not exhbt ths mode. In practce, models would be used to predct future unavalablty of a system. Of course, only the data up to the pont from whch the predcton s beng made would be avalable. We refer to the tme pont at whch the predcton s made as the "cut-off pont". The predcton

9 would dffer from the true value dependng on how well the model descrbes the system. In Fgure 4.4, we show both nstantaneous and uptme unavalablty approxmatons for release R11 of product P2 based on the average recovery rate at the "cut-off pont" and the LPET falure ft to ponts from the begnnng of the release's operatonal phase up to the "cut-off pont". The uptme unavalablty approxmaton was calculated usng the dscrete form of equaton (4.4). We see that the approxmaton for uptme unavalablty follows the emprcal uptme unavalablty qute well. Smlar results have been obtaned for other releases usng varous "cut-off ponts". 5. Summary We have presented emprcal data on unavalablty of a large software-based telecommuncatons system due to total system outages. The total system outages were found to occur for a number of reasons wth software as the major cause. Dfferent releases of the system exhbt smlar falure and recovery rate profles. The system recovery rate appears to be approxmately constant beyond the transent regon, whle the falure rate s a decreasng functon of the nservce tme. A complete unavalablty model for ths system needs to ncorporate tme-dependent parameters, as well as more than one operatonal state, and more than one falure state to account for software and other types of causes and dfferent classes of falure duratons. However, usng an example, we demonstrated that t s possble to reasonably approxmate the emprcal behavor of the system unavalablty due to total system outages usng a smple two-state Markov model. We are currently nvestgatng unavalablty of the system due to ndvdual causes, such as software, procedural, hardware, etc. We wll use the results to evaluate a many-state model of the system that accounts for degraded system operaton, dfferent falure causes, and dfferent classes of falure duratons. 6. References [Bellcore, 1989]. Network Swtchng Element Outage Performance Montorng Procedures - SR-TSY- 963, Issue 1, Aprl [Bellcore, 199]. Relablty and Qualty Measurements for Telecommuncatons Systems (RQMS) - TR-TSY- 929, Issue 1, June 199. [Costes et al., 1978]. A. Costes, C. Landrault, and J.C. Lapre, "Relablty and Avalablty Models for Mantaned Systems Featurng Hardware Falures and Desgn Faults," IEEE Transactons on Computers, vol. C-27, June 1978, pp [FCC, 1992]. Federal Communcatons Commsson, "Notfcaton by Common Carrers of Servce Dsruptons," 47 CFR Part 63, Federal Regster, Vol. 57 (44), March 5, 1992, pp [Ibe and Wen, 1992]. O.C. Ibe and A.S. Wen, "Avalablty of Systems wth Partally Observable Falures," IEEE Transactons on Relablty, IEEE, vol. 41, no. 1, March 1992, pp [Jones, 1991]. W.J. Jones, "Relablty Models for Large Software Systems n Industry," Proceedngs Frst Internatonal Symposum on Software Relablty Engneerng, pp [Jones, 1992]. W.J. Jones, "Relablty of Telecommuncatons Software: Assessng Senstvty of Least Squares Relablty Estmates," To appear n Proceedngs of TRICOM '92, Feb [Lapre et al., 199]. J.C. Lapre, C. Beounes, M. Kaanche, and K. Kanoun, "The Transformaton Approach to the Modelng and Evaluaton of the Relablty and Avalablty Growth," Proc. 2th IEEE Int. Symp. on Fault Tolerant Computng: (FTCS 2), IEEE, pp [Lapre et al., 1991]. J.C. Lapre, K. Kanoun, C. Beounes, and M. Kaanche, "The KAT (Knowledge-Acton- Transformaton) Approach to the Modelng and Evaluaton of Relablty and Avalablty Growth," IEEE Transactons on Software Engneerng, IEEE, vol. 18, no. 4, Aprl 1991, pp [Lapre and Kanoun, 1992]. J.C. Lapre, and K. Kanoun, "X- Ware Relablty and Avalablty Modelng," IEEE Transactons on Software Engneerng, IEEE, vol. 18, no. 2, Feb. 1992, pp [Musa and Okumoto, 1984]. John D. Musa and Kazuhra Okumoto, "A Logarthmc Posson Executon Model for Software Relablty Measurement," Proceedngs Seventh Internatonal Conference on Software Engneerng, Orlando, 1984, pp [Musa et al., 1987]. John D. Musa, Anthony Iannno, and Kazuhra Okumoto, Software Relablty: Measurement, Predcton, Applcaton, McGraw-Hll, New York, [Sandler, 1963]. Gerald H. Sandler, Systems Relablty Engneerng, Prentce-Hall, Englewood Clffs, N.J., [Shooman, 1983]. M.L. Shooman, Software Engneerng, McGraw-Hll, New York, [Trved, 1975]. A.K. Trved, "Computer Software Relablty: Many-State Markov Modelng Technques," Ph.D. Dssertaton, Polytechnc Insttute of Brooklyn, June, [Trved, 1982]. K. S. Trved, Probablty & Statstcs wth Relablty, Queung, and Computer Scence Applcatons, Prentce-Hall, Englewood Clffs, N.J., 1982.