Prediction Capabilities of Vulnerability Discovery Models

Transcription

1 Predictio Capabilities of Vulerability Discovery Models Omar H. Alhazmi, Colorado State Uiversity Yashwat K. Malaiya, Ph. D., Colorado State Uiversity Key Words: Security holes, Operatig Systems, Vulerability, Itrusios, Quatitative models. SUMMARY & CONCLUSIONS Quatitative approaches for software security are eeded for effective testig, maiteace ad risk assessmet of software systems. Vulerabilities that are preset i a operatig system after its release represet a great risk. Vulerability Discovery Models (VDMs) have bee proposed to model vulerability discovery ad have has bee fitted to vulerability data agaist caledar time. The models have bee show to fit very well. I this paper, we ivestigate the predictio capabilities that these models offer by evaluatig accuracy of predictios made with partial data. We examie both the recetly proposed logistic model ad a ew liear model. I additio to VDMs, we cosider static approaches to estimatig some of the major attributes of the vulerability discovery process, presetig a static approach to estimatig the iitial values of oe of the VDM s parameters. We also suggest the use of costraits for parameter estimatio durig curve-fittig. Here we develop computatioal approaches for early applicatios of the models ad examie the predictive capability of the models. We use data from Widows 98, Widows 000 ad Red Hat Liux 7.1. We examie the impact of usig a specific costrait whe the parameters of the logistic model are estimated Plots for the predictio error are give. The results demostrate that the predictio error is sigificatly less whe a costrait based o past observatios is added. It is observed that the liear model may yield acceptable projectios for systems for which vulerability discovery has ot yet reached saturatio. The results also suggest that it may be possible to improve the predictio capability by combiig static ad dyamic approaches, or by combig differet models. 1. INTRODUCTION Security vulerabilities i operatig systems, servers ad other software have emerged as a major threat to systems with iteret coectivity. A Software Vulerability is defied as a defect which eables a attacker to bypass security measures (Ref. 1) or as a weakess i the security system that might be exploited to cause loss or harm (Ref ). A umber of vulerabilities are discovered after a operatig system or server is released. The umber of udiscovered vulerabilities ad their discovery rates are amog the major factors that cotribute to the risk. Vulerabilities are a subset of software defects. For geeral software defects, several quatitative methods that use static metrics or software reliability growth models (SRGMS) are available. Because vulerabilities are defects, we ca expect to be able to develop some aalogous methods for vulerabilities. Recetly there have bee attempts to develop models for describig the discovery of vulerabilities termed vulerability discovery models (VDMs) (Refs. 3-6). Such models ca be applied to assess the security risks associated with a system. Software developers ca itegrate such models i the developmet process to allocate testig ad patch developmet resources i a cost effective maer. Oe way to evaluate the applicability of a model would be to see how well it fits the available data (Ref. 7). However a maager may ofte be iterested i makig projectios whe oly part of the data is available. I the software reliability egieerig field, several researchers have ivestigated the predictive capability of a umber of reliability growth models (Refs. 8, 9). Geerally, the two of the measures of iterest ofte are average magitude of error, ad the average bias which measures the tedecy of the model to overestimate or uderestimate the umber of vulerabilities (Ref. 9). We will examie the predictio capability of vulerability discovery models usig a similar two-compoet predictability measure. The capabilities of the VDMs will be examied by usig actual data from major software systems ad plottig the error i estimated values. This will allow us to compare differet schemes, idetify procedures that are likely to keep the error small, ad assess the accuracy of projectios. The available data for several major operatig systems suggest that vulerability discovery teds to maifest differet tred durig differet phases. We cosider the questio of how we ca obtai prelimiary estimates of the parameters of the models i the early phases i presece of such shifts. This requires iterpretatio of the model parameters. The models proposed by Alhazmi ad Malaiya demostrate very good fit with actual data (Ref. 4). The authors used chi-square (χ ) goodess of fit tests o several data sets ad showed the fit to be sigificat. They also showed that some of the major attributes such as the umber of vulerabilities ad vulerability desity fall withi a rage for differet systems whe ormalized (Ref. 7). However, a good fit does ot ecessarily imply good predictive capability. Because treds chage, a model that fits well i the early stage may o loger fit whe additioal data is obtaied. Here we evaluate two models usig partial data to examie the predictable capabilities of the models. We will the examie

2 the accuracy of the projectio by matchig the estimatio usig later data. The model parameters are related to the critical poits ad other observable attributes of the discovery process. We will discuss the sigificace of these attributes. No-liear regressio ofte requires iitial estimates of model parameter to esure that the fit is globally optimal. With limited iitial data, the parameter values may be ustable. A uderstadig of the model parameters will assist i iterpretig the values computed. Some comparable work has bee doe with Software Reliability Growth Models (SRGMs) i (Ref. 10), i which the authors have iterpreted some of the parameters of the SRGMs. I the ext sectio we discuss the two models examied. We ext itroduce the measures used for evaluatio ad the approaches used. Fially, we preset ad discuss the results obtaied.. VULNERABILITY DISCOVERY MODELS We cosider two models: Alhazmi-Malaiya Logistic model (AML) ad Liear Vulerability Discovery model (LVD). The secod model may be cosidered to be a simple approximatio. Alhazmi-Malaiya Logistic model (AML): This model was proposed by Alhazmi ad Malaiya i (Ref. 4); AML has show good fit o a rage of operatig systems vulerability data reported i (Refs. 11,1). Below, Figure 1 below shows a example of the fitted model, together with actual data for cumulative umber of vulerabilities Ω for Widows 95. As this figure shows, the vulerability discovery rate dω/dt icreases at the begiig, reaches a steady rate ad the starts to declie. The cumulative umber of vulerabilities thus shows a icreasig rate at the begiig as the system starts attractig a icreasig share of the istalled base. After some time, a steady rate of vulerability fidig yields a liear curve. Evetually, as the vulerability discovery rate starts droppig, there is saturatio due both to reduced attetio ad a smaller pool of remaiig vulerabilities. The model presumes that the rate of chage of the cumulative umber of vulerabilities dω/dt is govered by two factors, as give i Equatio 1 below. The first factor (AΩ) icreases with the time eeded to take ito accout the risig share of the istalled base. The secod factor (B-Ω) declies as the umber of remaiig udetected vulerabilities declies. While it is possible to obtai a more complex model, this model provides a good fit to the data (Ref. 4). Let us assume that the vulerability discovery rate is give by the differetial equatio:- dω AΩ( B Ω) dt (1) where Ω is the cumulative umber of vulerabilities, t is the caledar time, ad iitially t0. A ad B are empirical costats determied from the recorded data. Cumulative Vulerabilities Ju-96 Dec-96 Widows 95 Actual data The Model Ju-97 Dec-97 Ju-98 Dec-98 Ju-99 Dec-99 Ju-00 Dec-00 Ju-01 Dec-01 Ju-0 Dec-0 Ju-03 Dec-03 Figure 1 Alhazmi-Malaiya Logistic (AML) model fitted to Widows 95 vulerability data set. By solvig the differetial equatio we obtai B Ω ( t ) () ABt BCe + 1 where C is the itegratio costat itroduced while solvig Equatio 1. It is thus a three-parameter model give by the logistic fuctio. I Equatio, as t approaches ifiity, Ω approaches B. Thus, the parameter B represets the total umber of accumulated vulerabilities that will evetually be foud. It should be oted that the saturatio phase may ot be see i a OS which has ot bee preset for a sufficietly log time. Also, if the iitial adaptatio is quick due to better prior publicity, i some cases the early learig phase (whe the slope rises gradually) may ot be sigificat. This may cause the available data to appear largely liear, ad the secod model cosidered later may apply. A o-liear regressio may require a iitial estimate of the parameter values. A iitial estimate of B may be obtaied by otig the size of the software ad usig the typical vulerability desity values of similar software. The curve beds at two trasitio poits. To idetify the trasitio poits we take the derivatives of Equatio with respect to time t. 3 ABt d Ω AB Ce (3) ABt dt ( BCe + 1) From Equatio 3, the highest vulerability discovery rate occurs at the midpoit of Figure at time

3 T m The secod derivative is: 1 l BC AB 5 ( ABt) d Ω A C B e ABt dt ( BCe + 1) 3 4 ABt A B Ce ABt ( BCe + 1) The secod derivative exhibits a maximum ad a miimum. Thus two trasitio poits i Figure are obtaied by equatig the third derivative to zero. The trasitio poits occur at times t equal to Cumulative Vulerabilities 3 l BC T 1 AB Trasitio poit 1 ad Alhazmi-Malaiya Logistic Model T Caledar Time + 3 l BC. AB Mid poit Trasitio poit Figure AML model with the trasitio poits ad mid poit show We ca demostrate that the maximum slope is give by AB /4, which occurs at Ω B/. It ca be show that the two trasitio poits are.63/ab time period apart. Therefore, the duratio of the liear phase decreases as AB grows ad icreases as AB drops. If we regard B, the total umber of vulerabilities to be costat, we coclude that the parameter A cotrols the duratio of the liear phase. A estimate of the duratio betwee the two trasitio poits ca be obtaied from the data from prior software systems. We will use this mathematical result later to costrai the rage of the regressio parameters values. Parameter C impacts the locatio of the first trasitio poit. As we ca observe from Equatio, the ifluece of parameter C weakes as t icreases. Liear Vulerability Discovery model (LVD): Here we propose a ew liear model for compariso. Available data for some systems (Refs. 4, 7) demostrates the early learig (4) phase is sometimes egligible. This may be due to the similarity betwee the ew release ad its prior versio which results i a shorter learig phase for the testers to become familiar with the ew release. Furthermore, it has bee observed that at the saturatio phase, the vulerabilities discovered i the ext versio iclude vulerabilities shared with the modeled versio. This ca prolog the liear tred eve though a icreasig market share is take by the ewer versio. The liear model (LVD) is a approximatio of the AML model ad is expected to perform well o software systems that are relatively ew. We wat to determie whether a liear model ca be a reasoable approximatio i some cases. The liear model is give by a liear equatio, Ω ( t ) p + qt (5) where q is the slope ad p is a costat factor. The liear model may fit the cases i which much of the data is liear (for example i Figure 3) ad largely falls withi the two trasitio poits show i Figure. It ca thus be see as a approximatio of the AML model. There are other proposed VDMs that eed to be examied similarly i future (Ref. 6). Cumulative Number of Vulerabilities Widows NT Time (Moths) Vulerabulities LVD AML Figure 3 (AML) ad (LVD) models fitted to Widows NT 4.0 vulerability data set. 3. PREDICTABILITY MEASURES We evaluate predictability by examiig the accuracy of predictio usig early partial data. We apply the model uder cosideratio for umber of times (t 1, t, t 3,, t ) with equal caledar time iterval periods betwee estimatios. For each t i, the partial data at t i is fitted to the model at the best fit usig regressio aalysis to determie the best values for the parameters. The parameters are used to plot the complete predictio curve to obtai the estimated umber of vulerabilities (Ω i ). These estimated umbers of vulerabilities (Ω 1, Ω, Ω 3,, Ω ) are compared with the actual umber of vulerabilities (Ω) to determie the ormalized estimatio error (Ω i -Ω)/Ω. We the take the

4 average of the ormalized error magitude values to obtai the measure average error (AE). Average bias (AB) is similarly obtaied whe the sig of the error is also cosidered (Ref. 9). The average error (AE) ad average bias (AB) are defied as: 1 Ωi Ω AE (6) i 1 Ω 1 Ωi Ω AB (7) i 1 Ω where Ω is the actual umber of vulerabilities while Ω i is the umber of vulerabilities predicted at time t i. A alterative approach is to evaluate the ext step predictio is that the data up to t i is used to estimate the value at t i+1 (Refs. 13, 14). However, i cases where models are eeded to make loger term projectios, predictive capabilities for a variable period are also eeded, ad AE ad AB are more appropriate measures. Next, the estimatio approaches will be previewed ad the predictive capabilities of these approaches will be evaluated. 4. PREDICTING THE NUMBER OF VULNERABILITIES We examie three approaches for estimatig the umber of vulerabilities. These approaches are the static approach which uses the program size as the static metric. The dyamic approach uses the vulerability discovery models (VDMs). The third approach combies the dyamic capabilities with the static estimatio to filter out the extreme estimatios. The Static Approach: This approach requires the kowledge of the vulerability desity of comparable software systems. Vulerability desity (Ref. 7) is defied as the umber of vulerabilities per thousad lies of code. The kowledge of vulerability desity of at least oe previous software system is a requiremet of the static approach. Such previous software systems should preferably have bee developed i a comparable maer-i.e., developed by a similar team i the same orgaizatio. The assumig that the vulerability desity is close to the older system, we will have a error rage that ca be predetermied for addressig factors such as better developmet practices, more thorough testig before release, fractio of code that is access-related, etc. The Dyamic Approaches: We have used two differet implemetatios of the AML model i additio to the LVD model. 1) Ucostraied AML: This approach does ot place ay prior costraits o the parameters; rather the AML model give by Equatio is directly applied. The three parameters are estimated usig iterative computatio utilizig a least squares fit. ) AML costraied (AML-C): We chose to apply the costrait that the duratio betwee the two trasitio poits is obliged to be withi a certai limit. The duratio has bee show to be.63/ab for the AML model. Hece, we ca limit.63/ab betwee some miimum ad maximum values. The choice of the miimum ad maximum values determied by the values form the previous software systems. This costrait assumes that the trasitio poits are withi the time-frame of the expected lifetime of the software, thus eforcig the S- shape ad aticipatig the two trasitio poits. 3) Liear (LVD): The liear LVD model give by Equatio 5. The parameters values are determied simply by liear regressio aalysis. The Combied Dyamic/Static Approach: The Dyamic vulerability discovery estimatio approaches (AML), (AML- C) ad (LVD) are applied to estimate the total umber of vulerabilities. However, we ca also use the static approach to estimate ceilig ad floor values whe the dyamic model is applied ad some extreme estimatio occurs due to some bumps that appears early i the vulerabilities datasets. That would require the use of vulerability desity data from prior similar systems. I AML ad AML-C the static estimatio was used to obtai the bouds of the parameter B. 5. TESTING THE ACCURACY OF ESTIMATION We have applied the approaches metioed above to estimate the umber of vulerabilities i three systems: Widows 98, Widows 000 ad Liux 7.1. Because the use of static methods ad costraits requires the use of prior data, operatig systems such as Widows 95 ad Widows NT 4.0 caot be used due to lack of data from applicable previous versios. Table 1 illustrates the use of prior data for the three systems. The vulerabilities data is curret up to April 005 (Refs. 7, 1, 13). The static approach will oly be able to give estimate oly for the fial poit as a whole figure but ot estimates for differet poits of time. From Table 1 we ote that the vulerability desities for Widows 98 ad Liux 7.1 do lie withi ±5% of the value of their respective predecessors. However the vulerability desity for Widows 000 is outside of the ±5% of the value of its predecessor Widows NT. This may be due to the fact that Widows 000 cotais a larger fractio of o-access related code compared with NT. The Figures 3-5 show the ormalized error i estimatios at differet times for the three systems. The time scale is

5 give as a percetage of the total time period cosidered. As expected, projectios made usig AML or LVD closer to the ed time exhibit less error. The shaded regio i these figures shows where ceilig ad floor values would be if Ω was estimated usig the static approach assumig the defect desity to be withi ±5% of the predecessor. Table 1 Estimatig the umber of vulerabilities usig the static approach Software Systems Widows 98 Widows 000 RH Liux 7.1 Comparable V D (Wi95) (Wi NT 4.0) (RHL 6.) Code Size Est. Ω With 5±% 45 to 75 Actual Ω to to The plot for AML error for Red Had Liux 7.1 (Figure 5) demostrates that the fitted model first shows uderestimatio util about 45% of the time, followed by overestimatio. At about 65% of the time, the projectio stabilizes ad is reasoably accurate. Use of costraits durig the parameter value search ca cut out such swigs. For Widows 98 (Figure 3) AML geerally uderestimates while AML-C overestimates. For Widows 000, both AML ad AML-C uderestimate. I all cases the error reduces sigificatly as we approach last part of the life time. Error i Estimatio 60% Widows 98 50% 40% 30% 0% 10% 0% -10% 30% 40% 50% 60% 70% 80% 90% 100% -0% -30% -40% -50% -60% Percetage of Elapsed Caledar Time Static Estimatio AML AML-C LVD versios ted to have shorter lifetime tha Widows. The figures show improvemets i the accuracy of estimatios for the AML model. The improvemet is quite sigificat over the earlier estimatios as the impact of extreme projectios is miimized by applyig the two costraits. Table shows the average error (AE) ad the average bias (AB) give by Equatios 6 ad 7 have bee calculated for the three data sets. We ote the sigificat improvemet i both AE ad AB whe the costrait is used i AML. Error i Estimatio 190% 170% 150% 130% 110% 90% 70% 50% 30% 10% -10% -30% -50% -70% 5% 30% 35% 40% 45% Static Estimatio Widows % 55% 60% 65% 70% 75% 80% Percetage of time elapsed caledar time 85% 90% AML AML-C LVD Figure 4 Estimatio error values for Widows 000 Error i Estimatio 100% 80% 60% 40% 0% -40% -60% -80% -100% Static Estimatio Liux Red Hat 7.1 0% -0% 5% 35% 45% 55% 65% 75% 85% 95% Percetage of Elapsed Caledar Time 95% AML AML-C LVD Figure 5 Estimatio error values for Red Hat Liux 7.1 Table The Average Error (AE) ad Average Bias (AB) for the estimatios usig dyamic models 100% Figure 3 Estimatio error values for Widows 98 The first costrait o B assumed a 5% rage (Table 1). This costrait had to be dropped for Widows 000. The secod costrait o Widows 98 ad Widows 000 was chose to be 4 (.63/AB) 48. However, for Red Hat Liux 7.1 the costrait was 18 (.63/AB) 36 sice Liux Approach Software System AML with AML Liear (LVD) costraits AE AB AE AB AE AB Widows % -43.8% 0.1% -0.1% 9.1% 9.1% Widows % -9.8% 16.8% -15.5% 4.6% 4.6% Red Hat Liux % -5.3% 15.4% 6.3% 19.% -4.0%

6 It is iterestig to ote that LVD actually does a good job for Widows 000 ad RH Liux 7.1. This is because a oticeable saturatio i vulerability discovery has ot yet set i for these two systems. From Figures 3-5 we ca see that a combiatio of static ad dyamic approaches ca improve the predictio capability. The estimator has more flexibility i estimatig the umber of vulerabilities i a system. Moreover, the estimator ca have more cofidece i the estimatio results if more tha oe approach is used. REFRENCES 1. E. E. Schultz Jr., D. S. Brow, ad T. A. Logstaff, Respodig to Computer Security Icidets, Lawrece Livermore Natioal Laboratory, ftp://ftp.cert.df.de/pub/docs/csir/ihg.ps.gz, July C. P. Pfleeger, Security i Computig, Pretice-Hall, R. J. Aderso, Security i Opes versus Closed Systems - The Dace of Boltzma, Coase ad Moore, Ope Source Software: Ecoomics, Law ad Policy, Toulouse, Frace, Jue O. H. Alhazmi ad Y. K. Malaiya, Quatitative vulerability assessmet of systems software, Proceedigs of 51 st Aual Reliability ad Maitaiability Symposium, Alexadria,VA, Jauary 005, pp E. Rescola, Is fidig security holes a good idea? Security ad Privacy, IEEE Security ad Privacy, Vol. 03, No. 1, Jauary-February 005 pp O. H. Alhazmi ad Y. K. Malaiya, Quatitative vulerability assessmet of systems software, to appear i Proc. Iteratioal Symposium o Software Reliability Egieerig, Chicago, IL, November O. H. Alhazmi, Y. K. Malaiya, I. J. Ray, Security Vulerabilities i Software Systems: A Quatitative Perspective, to appear i the Proceedigs of the 19th Aual IFIP WG 11.3 Workig Coferece o Data ad Applicatios Security, Storrs, CT, August J. D. Musa, K. Okumoto, A Logarithmic Poisso Executio Time Model for Software Reliability Measuremets, Proceedigs of 7 th Iteratioal Coferece o Software Egieerig, Silver Sprig, MD, March 1984,pp Y. K. Malaiya, N. Karuaithi, ad P. Verma. Predictability of software reliability models, IEEE Trasactios o Reliability, Vol. 41, No. 4, pp , December Y. K. Malaiya ad J. A. Deto. What do software reliability parameters represet? I Proc. Iteratioal Symposium o Software Reliability Egieerig, pages , Albuquerque, NM, November ICAT Metabase, April The MITRE Corporatio, April A. A. Abdel-Ghaly, P.Y. Cha, ad B. Littlewood, Evaluatio of Competig Software Reliability Predictios, IEEE Trasactios o Reliability, Vol. SE-1, No. 9, September 1986, pp S. Brocklehurst, P.Y. Cha, B. Littlewood ad J. Sell, Recalibratig Software Reliability Models, IEEE Trasactios o Software Egieerig, Vol. 16, No. 9, April pp BIOGRAPHIES Omar H. Alhazmi Computer Sciece Departmet Colorado State Uiversity Fort Collis, CO USA. omar@cs.colostate.edu Omar Alhazmi is curretly a Ph.D. studet at Colorado State Uiversity sice Jauary 00. He received his Master's degree i computer sciece from Villaova Uiversity i 001. Yashwat K. Malaiya,Ph.D Computer Sciece Departmet Colorado State Uiversity Fort Collis, CO USA. malaiya@cs.colostate.edu Yashwat K. Malaiya is a Professor i Computer Sciece Departmet at Colorado State Uiversity. He received MS i Physics from Sagar Uiversity, MScTech i Electroics from BITS Pilai ad PhD i Electrical Egieerig from Utah State Uiversity. He has published widely i the areas of fault modelig, software ad hardware reliability, testig ad testable desig sice He has also bee a cosultat to idustry. He was the Geeral Chair of 003 IEEE Iteratioal Symposium o Software Reliability Egieerig. He is a seior member of IEEE.