IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 58, NO. 2, MAY Weiping Tan, Brian J. Sauser, and Jose Emmanuel Ramirez-Marquez

Transcription

1 IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 58, NO. 2, MAY Analyzing Component Importance in Multifunction Multicapability Systems Developmental Maturity Assessment Weiping Tan, Brian J. Sauser, and Jose Emmanuel Ramirez-Marquez Abstract Recently, a system maturity scale, i.e., system readiness level (SRL), was proposed to measure the maturity that a system achieves during development. However, this SRL assesses maturity for systems from only a single function and capability perspective, and is unable to assess maturity for multifunction multicapability (MFMC) systems. With ever increasing systems that provide multiple functions and capabilities, it is challenging for managers to properly allocate resources to ensure the achievement of critical capabilities and functions. In order to prioritize the allocation of resources for component development, it is common to perform importance analysis during system development and maintenance. Therefore, this paper first enhances the original SRL definition for maturity assessment at the capability, function, and system levels, and then, proposes the use of component importance analysis for the identification of the most important components for system development. This paper approaches component importance by introducing three important measures with respects to three main factors: technology readiness level/integration readiness level, developing cost, and developing effort. The paper uses an illustrative example of a MFMC system to show the proposed methodology and enhanced definitions and concludes with a discussion of the added value and future work. Index Terms Component importance, development of technology management strategies, developmental decisions under risk and uncertainty, project planning, resource management, technology management, technology selection. I. INTRODUCTION AFUNDAMENTAL principle of our understanding of systems is that the whole is greater than the sum of the individual parts, and individual parts play different roles through which the whole is formed, and there are consequences for not understanding the dynamics of each part. Thus, to assume that any one part or component is equally important to the function of a system would be to ignore such a fundamental system s principle. In the engineering and management of systems, identifying, prioritizing, and ranking components with respect to Manuscript received October 6, 2009; revised May 1, 2010, and July 16, 2010; accepted August 15, Date of publication September 30, 2010; date of current version April 20, This work was supported by the Naval Postgraduate School Acquisition Research Program under Grant N Review of this manuscript was arranged by Department Editor J. Sarkis. The authors are with the Systems Development and Maturity Laboratory, Stevens Institute of Technology, Hoboken, NJ USA ( wtan@stevens.edu; bsauser@stevens.edu; jmarquez@stevens.edu). Color versions of one or more of the figures in this paper are available online at Digital Object Identifier /TEM their importance is the key to understanding the importance of any one component to another and is notable for tradeoffs among cost, schedule, and performance parameters. One of the first to describe this necessity for the management of systems development was Birnbaum [1], who introduced a quantitative definition of component importance. Since then, the area of component importance measures (IM) has received significant attention, and its utility has gained more and more recognition. Hwang [2] specified that an importance index is for the measurement of the relative importance of a component with respect to other components in a system, and thus, it can be used to rank the contributions of components or basic events to the system s performance [3]. By its definition, IM can quantify the criticality of a particular component within a system, used as a tool for evaluating and ranking the impact of individual components, to determine the most important component with respect to the overall system performance, identify system weaknesses, and prioritize system performance improvement activities [4], [5]. Zio et al. [6] further emphasizes the great practical aid provided by IMs in that they allow system designers and managers to track system bottlenecks and also provide guidelines for effective actions when determining system improvements. Although it has been studied for decades, the research of component importance analysis has largely been restricted to system reliability. Very limited work has been published in what component importance means to developmental maturity and how we may better manage the system developmental lifecycle. In this paper, developmental maturity is the characterization of the development status of a technology/integration/system that can be quantified to determine the corresponding readiness. Likewise, there is a need for quantitative analysis and new models that can facilitate making informed decisions regarding the allocation of resources at key milestones throughout the system development lifecycle [7]. Being able to determine the importance of any one component over another for effective developmental maturity decisions is still dependent on the creation and use of effective metrics, scales, or indexes. One of the most noted and documented component maturity metrics is the prescriptive metric of technology readiness level (TRL). TRL has been used across many U.S. government agencies to assess the developmental maturity status of evolving technologies before incorporating them into a system or subsystem [8]. However, as has been documented [8] [13], TRL does not take into account the componentlevel concerns involving integration, interoperability, and sustainability, which are critical to understanding system maturity. To address this problem, Gove [9] proposed the integration /$ IEEE

2 276 IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 58, NO. 2, MAY 2011 readiness level (IRL) to measure the maturity of the integration points among system components, and thus, enables the consistent comparison of the maturity of different integration points. While the two basic elements technology and integration have their corresponding metrics (i.e., TRL and IRL) to quantify their maturity, Ramirez-Marquez and Sauser [8] explained that they are insufficient for evaluating the developmental state, potential, or risk of a system. Therefore, Sauser et al. [14] introduced a system readiness level (SRL) that incorporates the current TRL and IRL scales for managing system development and making effective and efficient decisions. This metric was later described in more detail by Sauser et al. [8], [12]. Likewise, these readiness levels have been described for use in making engineering management decisions about relative importance to a system s development [8], [11], [12], [14]. While the application of SRL has shown success in terms of increasing the insight of the decision maker into the system maturity and allowing for preemptive actions to be taken to mitigate potential developmental issues [15] [19], there are issues that the application of SRL does not yet address, which are: 1) developmental maturity assessment from a capability or function perspective; and 2) numerical impact of the components development maturity on system maturity that would enable the comparison of components importance. The current SRL approach assumes that a system performs a single function with single capability, and thus, every component is treated equally while calculating the system SRL. However, as Kim et al. [20] specified a technical system usually comprises a number of subsystems and components that are interconnected in such a way that the system is able to perform multiple required functions. The systems to which the SRL application has been applied so far tend to have multiple functions. Moreover, in order to ensure the success of the development of a system, even for a specific function, it is common to have one or several backup capabilities. For example, in a system studied by Forbes et al. [19], which will be further analyzed in Section II, there are fundamentally two functions mine detection and mine neutralization to be performed, and there are four capabilities in the first function, and two capabilities in the second. Recognizing that more and more systems are providing multiple functions with multiple capabilities, Volkert [21] suggested using the SRL approach to gain insights into the development of the constituent capabilities of a system, and further expressed that program managers need better methodologies to ensure the satisfaction of critical capabilities during systems development. Component importance analysis, by its history of application, has the potential to assist managers to accomplish this need. Such analysis can determine the component importance in terms of the impact of component maturity on the maturity of the whole system. In general, a system becomes more ready to deploy through the maturing of its components (i.e., technologies and integrations); therefore, how to allocate available resources to mature the system and not just its components independently poses a challenge to program managers. Combining the component importance analysis and the SRL approach presents a method to answer questions regarding component importance, such as, which component as it is being matured or potentially being matured, has the most potential to increase the system s developmental maturity? Which component(s) should the budget be spent on such that the system s developmental maturity can be advanced to a certain level? However, for component importance analysis to be performed for the development of multifunction multicapability (MFMC) systems, the premise is that a suitable metric can be applied to evaluate the developmental maturity of such systems. Unfortunately, the current definition of SRL lacks the ability to compound this, and needs to be enhanced to take into account the operational capabilities and functions. Therefore, in order to address the problem of identifying the most important components at a static point in the MFMC systems maturity/development cycle, so the available resources can be optimally allocated and the developmental risks can be effectively mitigated, this paper proposes a methodology for component importance analysis through the measurement of importance along three dimensions with respects to: 1) technology and integration maturity, i.e., TRL/IRL; 2) developmental cost (in dollars); and 3) developmental effort (in labor hours). Through the comparison of the impacts of components development on system maturity with respects to these three dimensions, the most important components can be identified. For such analysis to be performed, the current SRL definition will be enhanced with the consideration of functions and capabilities to enable the developmental maturity assessment of MFMC systems. This paper is organized as follows: Section II reviews the state-of-art development and application of the SRL addresses its incapability to accommodate system capabilities and functions, and thus, proposes an enhanced definition of SRL. Section III proposes three component IMs with respects to TRL/IRL, developmental cost, and effort, respectively, based on the review of historical development of component importance analysis. Section IV presents an illustrative example of applying the enhanced SRL definition and three IMs, and investigates the implication of these IMs to systems engineering management. Finally, Section V concludes the paper with summary and future work on this on-going topic. II. SYSTEM DEVELOPMENTAL MATURITY ASSESSMENT The prescriptive metric TRL is a systematic metric/measurement that supports assessment of the developmental maturity of individual technologies and the consistent comparison of maturity between different types of technologies. It has been used as an assessment of the maturity of evolving technologies prior to incorporating them into a system or subsystem within many U.S. government agencies and their support contractors. While its application has been critical in indicating progress and has aided dramatically in keeping numerous programs on track since its first introduction in early 1990s, TRL is unable to address the integration issues, which are unavoidable from a system s perspective [8], [9], [11] [13]. In order to address the legacy constraints with TRL of not specifically addressing integration, the metric of IRL was proposed [9], [11]. IRL is a systematic analysis of the interfacing of compatible

3 TAN et al.: ANALYZING COMPONENT IMPORTANCE IN MFMC SYSTEMS DEVELOPMENTAL MATURITY ASSESSMENT 277 interactions for various technologies and the consistent comparison of the developmental maturity between integration points. IRL provides a quantitative indicator of the maturity of the integration between two technologies, and a direction for improving the integration. While TRL has been designed to assess the risk associated with developing technologies, IRL is intended to assess the risk of integration [10]. Noting that technologies and the integrations among them are the two basic elements to the construction of a system, the metric of SRL is defined as the function of TRLs of the technologies and IRLs of the integrations [14]. It is one of the first instances that a single readiness level metric has been defined based on the combination of other existing readiness levels, and thus, providing a new way to combine readiness level metrics. Other efforts have shown correlations between readiness levels, but not quantitatively combining individual readiness levels to determine a single readiness level metric. One aspect of its importance is that it can give credibility to the quantitative combination of readiness levels and opens the potential to further expand SRL by incorporating other readiness level metrics, such as manufacturing readiness level and software readiness level [12]. Mathematically, the current procedure for calculating the SRL is as follows (assuming n technologies within the system). 1) Normalize the [0, 9] scale original TRLs and IRLs into [0,1] scale, and denote them by matrices TRL 1 TRL 2 Normalize TRL= TRL = TRL... 9 TRL n TRL 1 TRL 2 = (1)... TRL n IRL 11 IRL IRL 1n IRL 21 IRL IRL 2n Normalize IRL = IRL n1 IRL n2... IRL nn IRL 11 IRL IRL 1n IRL = IRL IRL 21 IRL IRL 2n = IRL n1 IRL n2... IRL nn where IRL ij = IRL ji. When there is no integration between two technologies, an original IRL value of 0 is assigned; for integration of a technology to itself, an IRL value of 9 is used, i.e., original IRL ii =9. 2) ITRL matrix is the product of TRL and IRL matrices (2) ITRL = Norm IRL TRL (3) i.e., ITRL 1 1/m ITRL 2... = 0 1/m ITRL n /m n IRL 11 IRL IRL 1n TRL 1 IRL 21 IRL IRL 2n TRL 2... = IRL n1 IRL n2... IRL nn TRL n (IRL 11TRL 1 + IRL 12TRL IRL 1nTRL n)/m 1 (IRL 21TRL 1 + IRL 22TRL IRL 2nTRL n)/m 2... (IRL n1trl 1 + IRL n2trl IRL nntrl n)/m n where m i is the number of integrations of technology i with itself and all other technologies, and Norm is to normalize the ITRL i from [0, m i ] scale to [0, 1] scale for consistency, thus Norm = diag[1/m 1,1/m 2,...,1/m n ]. 3) System SRL is the average of all ITRLs n i=1 ITRL i. SRL = ITRL 1 + ITRL ITRL n = n n (4) For a more detailed description of calculating and applying thesrl,see[12]. After the introduction of SRL, there have been many attempts to complement and expand it in various ways. Ramirez-Marquez and Sauser [8] proposed an optimization model to maximize the SRL scale based on resource allocation to provide a decision support approach that enhances managerial capabilities in the system development lifecycle. Recently, Magnaye et al. [22] proposed another constrained optimization model to identify which critical technology elements and integration links can be matured to which levels at a particular time such that the development costs are minimized, while a targeted SRL value is attained on schedule. In order to mitigate the inherent subjectivity in the estimation of prescriptive metrics, Tan et al. [23], [24] proposed a probabilistic method for performing readiness level estimation. As a means of utilizing SRL into a customer-based decision method, Fomin et al. [25] presented a means of incorporating SRL into an industry standard technique, i.e., the house of quality to correlate customer requirements to engineering capabilities and ultimately provide a method for requirements tracking and resource allocation. Cueller and Sauser [26] described how SRL could be used as a decision metric in corporate portfolio management of advanced capabilities. Currently, a systems lifecycle maturity management approach called system earned readiness management (SERM) is under development in the System Development and Maturity Laboratory (SDML) at Stevens Institute of Technology [27] that uses SRL as a project performance monitoring tool for managing system s developmental maturation. Tetlay and John [28] have argued the difference between system readiness and system maturity and how this may influence the development of a capability readiness. In this paper, we distinguish these two terms as: the readiness of a

4 278 IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 58, NO. 2, MAY 2011 TABLE I EXAMPLE OF A TWO-FUNCTION SIX-CAPABILITY SYSTEM system, technology, or integration implies how ready it is to be deployed on a numeric scale, and maturity is the characterization of development that is quantified to determine the readiness. Thus, readiness is a scale, and maturity is the definition of each level in the scale. While the SRL approach is being verified and validated in conjunction with system developers from Department of Defense (DoD), Lockheed-Martin, National Aeronautics and Space Administration (NASA), and Northrop Grumman [8], the original SRL definition is restricted to a system with single function and capability. In practice, a system evolves with time from a single capability or a specific function to a more complicated one in order to afford multiple functions, and to ensure the operational performance of a function by having several backup capabilities. These systems are often required to be open to further integration of other mission packages in order to satisfy future requirements for a to-be-defined service [29]. For example, in a system studied by Forbes et al. [19], there are basically two functions mine detection and mine neutralization to be performed. There are four capabilities in the first function, and two capabilities in the second, as shown in Table I (green shaded for the first function and yellow shaded for the second), these are: 1) bottom Mapping and change detection; 2) shallow and littoral water mine detection; 3) bottom and volume mine detection-i; 4) bottom and volume mine detection-ii; 5) contact mine neutralization; and 6) influence mine neutralization, i.e., there are six capabilities that are realized by six threads of components, as listed in Table I. With an ever-increasing complexity of the systems that are being developed and realized, multiple functions and capabilities are common for the development of most systems. As a result, managers require a metric that enables the assessment of MFMC system developmental maturity to manage the potential risks [21], and since exploited the necessity to enhance the previous SRL definition to accommodate such a reality. This paper intends to address this concern by enhancing the SRL approach to take into account the system functions and capabilities, which allows for the opportunity to field initial capabilities/functions earlier, while adding others later. Ramirez-Marquez and Sauser [8] have extensively explained the rationale behind the SRL. Taking into consideration the notions of function and capability in a system, this paper proposes a hierarchical SRL (see Fig. 1), where the SRL is defined at three different levels: capability-based SRL (SRL_C), function-based

5 TAN et al.: ANALYZING COMPONENT IMPORTANCE IN MFMC SYSTEMS DEVELOPMENTAL MATURITY ASSESSMENT 279 Fig. 1. SRL hierarchy. SRL (SRL_F), and the whole system-based SRL (composite SRL). The capability-based SRL calculates the SRL for a particular capability thread that includes a set of components to enable an intended capability. Based on the calculation of SRL_C s, the function-based SRL addresses the SRL for a specific function that encompasses one or several capability threads. The composite SRL indicates the SRL for the whole system, which includes multiple functions with multiple capabilities. Here, we adopt the rationale from Ramirez-Marquez and Sauser [8] for calculating the SRL, but restrict it to the capability level. As described, they elaborated that one would be interested in addressing the following considerations in the system development lifecycle (adopted with minor modification to accommodate capability SRL). 1) Quantifying how a specific technology is being integrated with every other technology to develop the system. Note that this quantifier should be a function of both the integration of a technology with every other technology that it has to be integrated with (as dictated by the system architecture) and the maturity of the different technologies, i.e., for each technology, this metric should be a function of both TRLs and IRLs. Thus, for technology fk(i), one can view this metric (ITRL fk(i) ) as subsystem measurement of this technology integrates within the system. In a mathematical representation: ITRL fk(i) = f(trl fk(j),irl fk(i)(j) ). 2) Based on such a metric (ITRL fk(i) ), SRL_C should provide a capability level measurement of readiness. Note that this new metric should be a function of the different ITRLs of each technology or in mathematical representation: SRL_C fk = f(itrl fk(1), ITRL fk(2),..., ITRL fk(nfk )) under the assumption that the capability contains n fk technologies. With consideration of capability and function in a system, this paper enhances it with the following considerations: 3) Given that the capability SRL SRL_C fk, the function SRL SRL_F f is to provide a function level measurement of readiness. Since there are multiple capabilities to back up a specific function, this metric should be a function of the different SRLs of each capability or in mathematical representation: SRL_F f = f(srl_c f 1, SRL_C f 2,..., SRL C fkf ) with the assumption that the function contains K f capabilities. 4) Based on the calculation of capability and function SRL, the system composite SRL is to provide a holistic picture of the system by enabling system level measurement of readiness. Since there are multiple functions with multiple capabilities to be performed by a composite system, this metric should take into account all functions and capabilities or in mathematical representation: composite SRL = f(srl_c fk ), where f = 1,..., r and k = 1,..., K f with the assumption that the system contains r functions and r f =1 K f capabilities. A. Enhanced SRL Procedure Mathematically, the enhanced procedure to calculate SRL is defined as follows. 1) System Definition: Assume that a system includes a total of n technologies and let T denote the technology set: T = {TRL i,i=1, 2,...,n}.

6 280 IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 58, NO. 2, MAY 2011 The system includes r functions and let F denote the function set: F = {F f,f =1, 2,...,r}. Within each function F f, there are K f capabilities: F f = {C fk,k =1, 2,...,K f }. Within a set C fk, there are n fk technologies and integrations among these technologies: C fk = {TRL fk(i),irl fk(i)(j),i,j =1, 2,..., n fk }. Finally, m fk(i) is the number of integrations of technology T fk(i) with itself and all other technologies within capability C fk. 2) SRL Calculation Procedure: 1) Normalize the [0, 9] scale original TRLs and IRLs into [0,1] scale TRLs and IRLs by dividing each of them by 9, denote them by matrices, as shown (5) and (6), at the bottom of this page, where IRL fk(i)(j) = IRL fk(j)(i). When there is no integration between two technologies, an original IRL value of 0 is assigned; for integration of a technology to itself, an original IRL value of 9 is used, i.e., original IRL fk(i)(i) =9. 2) ITRL_C fk matrix is the product of TRL fk and IRL fk matrices (note: those formatted in Italic and bold denote matrices) ITRL C fk = Norm fk TRL fk IRL fk (7) i.e., as shown (ITRL C fk ), at the bottom of the next page, where m fk(i) is the number of integrations of technology TRL fk(i) with itself and all other technologies within capability C fk, and Norm fk is to normalize the SRL_C fk(i) from [0, m fk(i) ] scale to [0, 1] scale for consistency. Thus, matrix Norm fk = diag[ 1/m fk(1), 1/m fk(2),..., 1/m fk(nfk ) ]. 3) SRL_C fk denotes the SRL for capability C fk. It is defined as the average of the all the normalized technologies ITRL values, which is given by SRL C fk = ITRL C fk(1) + ITRL C fk(2) ITRL C fk(n fk ) n fk n fk i=1 = ITRL C fk(i). (8) n fk 4) SRL_F f is the SRL for function F f. With consideration 3) in mind, although there are multiple capabilities to ensure the same function, the maximum of these capability SRLs represents the readiness of that function and is defined as follows: SRL F f =Max(SRL C fk ), k =1, 2,...,K f. (9) SRL_F matrix includes all the function SRLs, and is denoted by SRL F 1 SRL F 2 SRL F =.... (10) SRL F r 5) Finally, the composite SRL for the whole system is the average of all capability SRLs to address consideration 4) as shown (11), at the bottom of the next page. 6) Besides, according to the previous definition of T, then T is also the union of the technologies of all capabilities, thus, we have T = {TRL fk(i),f =1, 2,...,r; k =1, 2,...,K f ; i =1, 2,...,n fk }. (12) Note that although a specific technology could be involved in different capabilities and functions, the number of technologies in set T should equal the number of technologies in the system, which is n. 3) Example to Illustrate the SRL Calculation: In order to illustrate the calculation of the enhanced SRL definition, we demonstrate it by a simple example, whose context diagram is depicted in Fig. 2. As shown in Fig. 2, this system includes five technologies and four integrations: two functions F 1 and F 2, and three capabilities C 11, C 12, and C 21, each capability utilizes three technologies. TRL fk =[TRL] n fk 1 = TRL fk(1) TRL fk(2)... Normalize TRL fk = TRL fk 9 = TRL fk(1) TRL fk(2)... (5) TRL fk(nfk ) TRL fk(n fk ) IRL fk(1)(1) IRL fk(1)(2)... IRL fk(1)(nfk ) IRL fk(2)(1) IRL fk(2)(2)... IRL fk(2)(n IRL fk =[IRL] n fk n fk = fk ) Normalize IRL fk = IRL fk 9 = IRL fk(nfk )(1) IRL fk(nfk )(2)... IRL fk(nfk )(n fk ) IRL fk(1)(1) IRL fk(1)(2)... IRL fk(1)(n fk ) IRL fk(2)(1) IRL fk(2)(2)... IRL fk(2)(n fk ) (6) IRL fk(n fk )(1) IRL fk(n fk )(2)... IRL fk(n fk )(n fk )

7 TAN et al.: ANALYZING COMPONENT IMPORTANCE IN MFMC SYSTEMS DEVELOPMENTAL MATURITY ASSESSMENT 281 Fig. 2. System with two distinct functions, three unique capabilities using the same five technologies. First, let us calculate CSRL 11 TRL TRL 11 = TRL 2 = 8 Normalize TRL 11 = 0.89 TRL IRL 11 IRL 12 IRL IRL 11 = IRL 21 IRL 22 IRL 23 = IRL 31 IRL 32 IRL Normalize IRL 11 = ITRL C fk(1) ITRL C fk(2) ITRL C fk =... Since there is no integration between Technologies 1 and 3, the integration IRL 13 = 0 1/ ITRL C 11 = 0 1/ / = SRL C 11 = ITRL C ITRL C ITRL C = = ITRL C fk(nfk ) 1/m fk(1) /m fk(2)... 0 = /m fk(n fk ) IRL fk(1)(1) IRL fk(1)(2)... IRL fk(1)(n fk ) IRL fk(2)(1) IRL fk(2)(2)... IRL fk(2)(n fk ) TRL fk(1) TRL fk(2)... = IRL fk(n fk )(1) IRL fk(n fk )(2)... IRL fk(n fk )(n fk ) TRL fk(n fk ) (IRL fk(1)(1) TRL fk(1) + IRL fk(1)(2) TRL fk(2) + + IRL fk(1)(n fk ) TRL fk(n fk ))/m fk(1) (IRL fk(2)(1) TRL fk(1) + IRL fk(2)(2) TRL fk(2) + + IRL fk(2)(n fk ) TRL fk(n fk ) )/m fk(2)... (IRL fk(n fk )(1) TRL fk(1) + IRL fk(n fk )(2) TRL fk(2) + + IRL fk(n fk )(n fk ) TRL fk(n fk ) )/m fk(n fk ). Composite SRL = ( K 1 k=1 SRL C 1k )+( K 2 k=1 SRL C 2k )+ +( K r k=1 SRL C r K f rk) f =1 k=1 = SRL C fk K 1 + K K r r f =1 K. (11) f

8 282 IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 58, NO. 2, MAY 2011 TABLE II SYSTEM READINESS LEVELS Following the same way, we can get SRL_C 12 = 0.62 and SRL_C 21 = Then, SRLF 1 =Max(SRL C 11,SRL C 12 ) = Max(0.44, 0.62) = 0.62 SRL F 2 =Max(SRL C 21 )=Max(0.54) = Finally, the composite SRL for the whole system is as follows: Composite SRL = (SRL C 11 + SRL C 12 )+SRL C = = B. Discussion of the Enhanced SRL Definition The new SRL definition is enhanced based on the previous one to accommodate its application to MFMC systems, and can also be used to assess the developmental maturity of a system providing single function with single capability. Actually, when the numbers of functions and capabilities equal to one (r = 1 and K f = 1), the new definition is exactly the same as earlier; therefore, it is compatible with the previous one. It should be pointed out that even though there could be various ways to expand the previous SRL definition so as to apply it to MFMC systems, the proposed procedure is based on our understanding of the relationships among the three levels: capability, function, and system levels. These relationships are also explained in the calculation procedure. As the previous definition, the enhanced SRL is undergoing verification, validation, and calibration with its application. Ramirez-Marquez and Sauser [8] have pointed out that calculating a SRL without articulated correlation to qualitative systems engineering practices would make it difficult to determine the gained value in understanding its implication on the developmental lifecycle, i.e., without such correlation, SRL would just be a metric that describes how developmental maturity evolves over time. Earlier, Sauser et al. [14] used documented qualitative data to calculate the SRL for four systems in-development and correlate the SRL of these systems to four standard systems engineering lifecycles (i.e., Typical High-Technology System, ISO 15288, DoD, and NASA). Among the cases, there were systems development successes and failures, levels of abstraction, and views in retrospect. The authors of this paper have been subsequently performing further verification and validation of the SRL approach with other cases in conjunction with system developers from DoD, Lockheed-Martin, NASA, and Northrop Grumman. This has resulted in a modified SRL definition from what was originally described in [8] and [12], as depicted in Table II. III. COMPONENT IM FOR SYSTEM DEVELOPMENTAL MATURITY ASSESSMENT Historically, the development of component importance analysis has been dominantly reflected in the research area of system reliability. In general, component importance tells the degree of impact that a component has on the system for the achievement of a specific capability and/or function, and can be evaluated along two dimensions: structural and reliability [2], [30]. The structural dimension refers to the location of the component in the system, while the reliability dimension refers to the reliability of the physical unit installed at the specific location. When components with distinct reliabilities can be arbitrarily assigned to several positions in a system, the structural aspect should be considered in building that system since, presumably, designers and developers want to allocate the more reliable components

9 TAN et al.: ANALYZING COMPONENT IMPORTANCE IN MFMC SYSTEMS DEVELOPMENTAL MATURITY ASSESSMENT 283 to the more important positions. The reliability is relevant when the components are already installed in the system, but there is a desire to improve system reliability via improving the component reliability. In 1969, Birnbaum [1] introduced the Birnbaum s IM to investigate the sensitivity of the system unreliability to changes in the failure probability of a component by calculating the partial derivative of the system unreliability with respect to the failure probability of the component. Given that the system is failed at a time, Gandini [31] defined the criticality importance as the probability that a component has caused system failure; and Fussell and Veseley [32] suggested another IM as the conditional probability that a cut set containing the component has failed. Barlow and Proschan [33] defined a component IM as the conditional probability that a component causes the system to fail during a time interval, given that the system has failed in that interval. Risk achievement worth (RAW) measures the increase in system unreliability assuming the worst case of the failure of the component, i.e., when the unreliability of the component is one. Conversely, risk reduction worth (RRW) measures the decrease in the risk or the system unreliability when the unreliability of the component is zero, i.e., the component works perfectly [34]. Later on, Hwang [35] mathematically showed how the various IMs relate based on the survey on them, and rationally claimed that the more important components deserve more maintainable attention and more reliable units. While traditional reliability theory has focused on binary systems (e.g., function perfectly or fail thoroughly), some researchers have given attention to multistate systems, whose components may operate in a degraded manner causing the system to provide service at less than full capacity [4]. Thus, for a multistate system with multistate components (MSMC), some component IMs have been proposed. For example, Levitin and Lisnianski [36] proposed importance and sensitivity measures for multistate systems with binary capacitated components, through which the IMs are obtained through the universal generating function. Zio and Podofillini [37] extended the RAW, RRW, Fussell Vesely, and Birnbaum s IM for MSMC. While other researchers focused MSMC IMs on investigating how a particular status of a set of states affects multistate system reliability, Ramirez-Marquez and Coit [4] devoted research efforts to IMs that are involved in measuring how a specific component affects the multistate system reliability. Although the majority of the measures of importance that have been developed are strictly for coherent analysis (a coherent system is one, where each component of the system is relevant, and the structure function is increasing), Andrews and Beeson [38] specified the necessity to expand the analysis to noncoherent systems, to which they extended the application of Birnbaum s IM by considering the probabilistic measure of component importance. Since Birnbaum s measure of component importance is the milestone and the core of other measures, their extension to noncoherent systems made possible the derivation of other measures. Their succeeding research has extended four commonly used measures of importance to noncoherent systems and demonstrated how to select appropriate measures for component importance analysis [39]. Just as the types of component IMs have been diverse, so have the application of these IMs. Although plenty of literature discusses IMs in general, some others explore IMs with specific application. Arguing traditional component reliability importance indexes were only for single-input-single-output (SISO) systems, and thus, were unable to capture the general function, Hiber and Bertling [40] presented three component reliability importance indexes for application to electrical networks. Zhang et al. [41] spreaded the component importance analysis to the all-digital protection systems and select two suitable measures for their specific application. Hengcheng et al. [42] pointed out that conventional fault tree analysis techniques have the limitations in terms of accuracy and efficiency for calculating the component importance in the aviation system, and proposed to use the binary decision diagram method to improve the efficiency and precision. While the topic of component importance has been extensively developed with regard to system reliability, it is void in the area of system maturity assessment due to its very recent emergence. Smith [43] points out that the TRLs do not account for the criticality of a component to the system as a whole and cannot provide any means to deal with how the relative contributions of the various aspects of maturity vary throughout the lifecycle of a system, and thus, exploited the necessity to perform research regarding these issues. Given that the utility of SRL for system developmental maturity assessment, system reliability and system maturity share many commonalities, such as 1) being scaled within the interval (0, 1); 2) being calculated from the figure-of-merit of components; and 3) involving component selection or component improvement to upgrade the corresponding figure-of-merit of the system. Hereby, this paper proposes to extend the application of importance analysis to the area of system developmental maturity assessment to facilitate decision makers on components selection for maturing systems. In general, component analysis in system maturity assessment will help to 1) identify the critical component to system maturity, i.e., which component (a technology or an integration) has the largest impact on system maturity; 2) select the technology that can mature the SRL more when there are competing technologies; 3) prioritize component development based on constrained resource availability; and 4) tradeoff between system capabilities/functions with a given developmental budget. Since system performance, developmental cost, and effort are focal points of project and engineering management, it is insightful to take into account these factors while performing importance analysis. Thus, this paper executes component importance analysis in two aspects with respect to: 1) the component maturity attributes, i.e., TRL or IRL; and 2) the developmental resource, i.e., developmental cost and effort (in terms of labor hour). It should be pointed out that development cost and effort somehow overlap each other. Although it would be more meaningfully to have orthogonal factors from a mathematical perspective, this paper considers the importance factors from a practitioner view for different types of systems, i.e., the second factor of developmental cost is especially for

10 284 IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 58, NO. 2, MAY 2011 budget-oriented systems (e.g., systems with long development periods, but much emphasis on limited budget), the third factor of developmental effort is mainly for schedule-oriented systems (e.g., first-to-market systems). Therefore, in this part, we introduce three types of component IMs with respect to the developmental maturity of the components (TRL/IRL), development cost, and development effort (in terms of labor hour) to mature components. Each IM involves, directly or indirectly, change of component maturity from its original value to a target value. The change of component maturity in this paper may result from two actions: 1) further application of resources on the development of a specific component makes the component more mature; or 2) replacement of one technology by another, which enables the same functionality. Since the change of components maturity will cause change on the system maturity, this paper proposes to compare the resultant SRLs under different cases. Using the original components status as a baseline, within each case, the SRL will be calculated with the assumption that only one of the components will be changed from its original maturity to a target value, while the others remain on their original values. Within one scenario, the calculation will continue until each component has been changed once, and then, the component importance can be ranked based on the calculation of the SRLs. Therefore, the system/function/capability SRL is dependent on the constituent technology TRLs and integration IRLs, and the IM values are dependent on the degree of impact the corresponding components have on the SRL. The detailed definitions of three IMs are described separately in the following. Note that in this part, whenever SRL is mentioned, it can be the SRL of any level as previously defined in the SRL hierarchy, and the three IMs can be applied to each level (i.e., capability, function, and system). Section IV will use a detailed example to show the application of these IMs to each SRL level. A. IM With Respect to TRL/IRL (I P ) This IM determines the impact of change in a component s developmental maturity on SRL. It measures the change of the SRL when the maturity of a specific component (i.e., a TRL or an IRL) changes from its current value to a target value. Take a technology component for an example, let SRL(TRL,IRL) denote the current SRL, and SRL(TRL,IRL TRL i = TRL i ) denote the resultant SRL when TRL i changes to a target maturity level TRL i and all other TRLs and IRLs stay on original maturity values. Then, the IM with respect to TRL (Ii P ) is mathematically denoted as (the same way for integration component) For TRL,Ii P = SRL(TRL,IRL TRL i = TRL i ) SRL(TRL,IRL) For IRL,Iij P = SRL(TRL,IRL IRL ij = IRL ij ). (13) SRL(TRL,IRL) I P implies the effect of change in the readiness level of a given component on SRL, and component, whose variation in readiness level results in the largest variation of the SRL has the highest importance. B. IM With Respect to Cost (I CT ) Zhang et al. [44] specifies the cost ignorance of classical component importance analysis, and states that it is unrealistic to evaluate the importance of components without considering the cost. Hereby, for SRL component importance analysis, we propose to consider the economic factor. This is reasonable by noting that there are always situations, where system developers have to make the investment decisions based on the comparison of the immediate return on the investment of dollars needed to mature components. Presumably, especially with a tight budget, developers allocate budget to the component that can result in the highest system developmental maturity achievement. Therefore, we propose I CT as an IM that takes into account the cost for maturing components to facilitate such comparisons. Since the cost to mature different components varies and the improvement in different components has different effect on SRL, the IM that takes into account the development cost serves as a baseline to compare the investment returns from different components. Let CT i = CT TRLi CT TRLi denote the associated development cost for maturing TRL i from its current readiness level to a target level TRL i, and CT ij = CT IRLij CT IRLij denote the associated development cost from maturing IRL ij from its current readiness level to a target level IRL ij, and then, the formula to calculate the (I CT )isasfollows: For TRL, I CT i = ΔSRL CT i = SRL(TRL,IRL TRL i = TRL i ) SRL(TRL,IRL) CT TRLi CT TRLi For IRL, I CT ij = ΔSRL CT ij = SRL(TRL,IRL IRL ij = IRL ij ) SRL(TRL,IRL) CT IRLij CT IRLij. (14) I CT implies the effect of the cost to mature a given component on SRL, and component, whose readiness improvement from the investment results in the largest gain of SRL has the highest importance. C. IM With Respect to Labor Hour (I LH ) Besides the consideration of cost, there are other situations (e.g., when only certain labor hours are available), where developers care more about the return of the effort needed to improve components. Therefore, we propose another IM (I LH ) that takes into account the associated labor hours to upgrade the component readiness level in order to mature the SRL. Let LH i = LH TRLi LH TRLi denote the associated development labor hours for developing TRL i from its current status to a target level TRL i, and LH ij = LH IRLij LH IRLij denote the associated development labor hours for developing IRL ij

11 TAN et al.: ANALYZING COMPONENT IMPORTANCE IN MFMC SYSTEMS DEVELOPMENTAL MATURITY ASSESSMENT 285 Fig. 3. System diagram. from its current status to a target level IRL ij, and then, the formula for I LH is as follows: For TRL,I LH i = ΔSRL LH i = SRL(TRL,IRL TRL i = TRL i ) SRL(TRL,IRL) LH TRLi LH TRLi For IRL,I LH ij = ΔSRL LH ij = SRL(TRL,IRL IRL ij = IRL ij ) SRL(TRL,IRL). LH IRLij LH IRLij (15) I LH implies the effect of the effort to mature a given component on SRL, and component, whose readiness improvement from the effort investment results in the largest gain of SRL has the highest importance. IV. ILLUSTRATIVE EXAMPLE A. Case Overview 1) System Diagram: The illustrative example, which was also examined in [13] to show the SRL approach, will demonstrate the application of the proposed IMs under the enhanced SRL definition. The architecture (see Fig. 3) represents an endto-end integration of command and control capabilities with a variety of unmanned vehicles and intelligence, surveillance, and reconnaissance sensor packages. These elements are capable of autonomous operations and include both off-the-shelf equipments and cutting-edge development networked seamlessly together to enhance effectiveness and efficiency. Totally, there are 20 technologies and 21 integrations in the whole system, within which there are three functions and nine capabilities (see the following part). The TRL and IRL values shown in Fig. 3 for this system are notional. 2) System Functions and Capabilities: As illustrated in Fig. 4, only the shaded technologies and their corresponding integrations are considered in evaluating the maturity of the corresponding capability of a function. For the notional system in question, there are basically three different functions with three, five, and one capabilities, respectively. The three functions are mine-detection, mine-neutralization, and gun-combat. Within each function, there are several capability alternatives to ensure the success of the corresponding mission. For example, for the mine-detection, one of the capabilities is to pull the sensor by using a submarine, which has longer endurance, but slower speed, and another capability is by using a helicopter, which is faster, but can only stay out for a limited amount of time. Using the enhanced SRL definition, estimates of the actual SRLs at different levels can be obtained before the system is actually deployed. When reviewing the SRL for this system in its current state, the capability and function SRLs are shown in the first row in Fig. 4. The function SRLs, which are the highest capability SRLs of the corresponding functions, are shaded with their capability SRLs. With the given notional numbers for TRLs and IRLs, all of these SRLs fall in a range of (0.58, 0.68), which

12 286 IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 58, NO. 2, MAY 2011 Fig. 4. Functions, capabilities definition, and their SRLs. indicate that the system is undergoing a balanced development and still in the engineering and manufacturing development phase. According to the DoD system development lifecycle (see Table II), during this phase, the main assignments are to develop system capability, reduce integration and manufacturing risk, ensure operational supportability, reduce logistics footprint, implement human systems integration, design for production, ensure affordability, and protection of critical program information and demonstrate system integration, interoperability, safety, and utility [12]. While it would be perfect to have adequate resource to completely mature the system, very often, the situation is that only constrained budget and labor hours can be executed. Seeking proper resource allocation strategy has been more and more imperative in the past two decades, and it has never been so highlighted than in today s tight economy. Therefore, it becomes meaningful to prioritize the limited resource by distinguishing the component importance in terms of the impact of individual maturity on capability, function, and system maturity. The following will show how to evaluate the component importance using the proposed measures in this paper. 3) Resource Consumption: Since we are proposing to take into account the resource consumption (cost and labor hour) in the component importance evaluation, Tables III and IV show them for maturing the components of the system. The cost is in thousand dollars ($1000), and the effort is in labor hours. For example, it requires 202 h of effort and $ to move Technology 1 from level 8 to 9. As stated in [8], it is the obligation of the program manager to obtain these estimates of resource consumption in reality. To maturate the whole system, the estimated cost and effort equal $ and labor hours, respectively. B. Component Importance Identification by Applying Component IMs With the proposed component IMs for I P, I CT, and I LH, we consider two scenarios for each measure to identify the component importance at the capability and function level. We will term them capability and function component importance, respectively. While keeping all the other components remaining on their current maturity levels, the two scenarios are to change the current maturity of the corresponding component to a target value by 1) increasing by one level, which is TRL i = TRL i + 1 or IRL ij = IRL ij +1, and 2) increasing to its highest level, which is TRL i =9or IRL ij =9. 1) Two Scenarios for Calculating I P : The demonstration of the calculation starts from the capability level. As highlighted in Fig. 4, there are seven technologies and seven integrations that enable the capability C 11. By increasing the current status of a component by one level and applying (13) to the capability,

13 TAN et al.: ANALYZING COMPONENT IMPORTANCE IN MFMC SYSTEMS DEVELOPMENTAL MATURITY ASSESSMENT 287 TABLE III RESOURCE CONSUMPTION FOR TRL UPGRADE TABLE IV RESOURCE CONSUMPTION FOR IRL UPGRADE Table V shows the result of the calculation for component importance of capability C 11. As the results show, Technology 2 is the most important component, whose change in developmental maturity has the largest impact on the capability maturity. When Technology 2 is increased by one level, the capability SRL of C 11 is upgraded from its current value of to 0.635, and gives an I P of If the objective is to have the most change on capability SRL with the restriction that only one component can be changed by one level, then Technology 2 is the most important one. The second and third most important components identified are Technology 4 and 6, with an I P of and 1.024, respectively.

14 288 IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 58, NO. 2, MAY 2011 TABLE V CAPABILITY I P FOR THE SCENARIO OF INCREASING BY ONE LEVEL FOR C 11 Following the same way, we calculate every component s importance index of each capability for the scenario of increasing the current maturity of the corresponding component by one level. Since the main objective is to determine the critical components to SRL, we list the top three most important components from the calculation in Table VI. As shown in the table, Technology 2 is always the most important component for all the capabilities under this scenario, while the second and third vary among several components. Statistically, Technologies 2, 4, 8, and 17 are the critical technologies when the consideration is to increase one readiness level. After considering the component importance at the capability level, attention is given to the function level importance. For the same scenario of increasing by one level, Table VII shows the calculation results. Technology 2 is identified to be most critical component for all three functions. Technologies 8, 17, and 18 are the second most important components for functions 1, 2, and 3, respectively. Coincidently, Technology 4 occupies the third position for all the three functions. While it is difficult to tell the component importance from the architecture, the IM enables the ability to do so. Similarly, for the scenario of increasing the corresponding component s developmental maturity level to the highest (i.e., level 9), Table VIII shows the results of top three most important components for capability importance. Table VIII can be regarded as a SRL achievement worth measure, i.e., it is a measure of the change in capability SRL when a component is set to its highest possible maturity, which is level 9. It assumes that the maturity status of a component is perfect, and thus, bumps up the capability SRL. This IM shows how much mature the capability can become as the readiness level of its components are improved, or some of its components are replaced by more mature components. From the calculation results, Technology 2 has the most potential to upgrade the SRL under the condition that only one component is to be developed to its most readiness level for all capabilities. While Technology 2 still dominates the first place for this scenario, Technology 3 is the second most important component for capability C 11, C 12, C 13, C 21, and C 31, and Technology 17 is the second most important component for capabilities C 22, C 23, C 24, and C 25. For the third position, it shifts among Technologies 3, 4, 6, 9, and 17. Table IX shows the results of the function level component importance for the scenario of increasing to the highest level. 2) Two Scenarios for Calculating I CT : The economic factor can also be considered for evaluating the component importance. As previously defined, we performed the calculation for component importance with respect to cost I CT using the same two scenarios of increasing the component maturity level by one level and to the highest. For the scenario of increasing by one level, Table X shows the top three most important components for each capability, while Table XI shows at the function level. In order to understand the tables, take the first data row of Table X for an example, Technology 3 is the most important component of impact to capability C 11 with an importance index I CT of , which means one dollar spent on the development of Technology 3 would upgrade the SRL_C 11 by If there is only limited budget, then Technology 3 should be the first to be considered in order to achieve significant achievement of the capability SRL. Compared to the calculation results of I P, here the top three positions are occupied by components with great shift and no single component dominates the rank. For increasing every component to the highest of level 9, Table XII shows the top three most important components for each capability, and Table XIII shows at the function level. The top three positions are occupied by different components. However, Technology 4 dominates the first place for all the capabilities except capability C 12. 3) Two Scenarios for Calculating I LH : In this part, we consider the component importance with regard to effort (in terms of labor hour). The effort needed to upgrade the components is shown earlier in Tables III and IV. As with component importance with respects to TRL/IRL and cost, hereby we still try the same two scenarios.

15 TAN et al.: ANALYZING COMPONENT IMPORTANCE IN MFMC SYSTEMS DEVELOPMENTAL MATURITY ASSESSMENT 289 TABLE VI TOP THREE MOST IMPORTANT COMPONENTS FOR CAPABILITY I P OF INCREASING BY ONE LEVEL TABLE VII TOP THREE MOST IMPORTANT COMPONENTS FOR FUNCTION I P OF INCREASING BY ONE LEVEL TABLE VIII TOP THREE MOST IMPORTANT COMPONENTS FOR CAPABILITY I P OF INCREASING TO THE HIGHEST LEVEL Table XIV shows the top three most important components for all the capabilities when the scenario is to increase by one level. To understand the table, take the row of C 11 for an example, Technology 6 has been identified to be the most important component with a ratio to labor hours of , which means one labor hour spent on the development of Technology 6 would upgrade the SRL by , which significantly outperforms the performance of other components, as shown in the table. When taking into account the effort to further mature the technology, and thus, upgrade the capability SRL, this result means that the

16 290 IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 58, NO. 2, MAY 2011 TABLE IX TOP THREE MOST IMPORTANT COMPONENTS FOR FUNCTION I P OF INCREASING TO THE HIGHEST LEVEL TABLE X TOP THREE MOST IMPORTANT COMPONENTS FOR CAPABILITY I CT OF INCREASING BY ONE LEVEL TABLE XI TOP THREE MOST IMPORTANT COMPONENTS FOR FUNCTION I CT OF INCREASING BY ONE LEVEL effort put on Technology 6 would result in the most significant achievement of the maturity for capability C 11. If there are only limited labor hours to deliver, then Technology 6 should be the first to be considered in order to achieve most return on the capability SRL. Technologies 3 and 4 follow the importance rank, which sequentially indicates their importance to the capability SRL when comparing their return of investment. Although the positions of the top three components vary for different capabilities, we can see that they remain the same for the capabilities of a specific function. For instance, for C 11, C 12, and C 13 of Function 1, Technologies 6, 3, and 4 always occupy the first, second, and third position, respectively. Table XV gives the component importance results at the function level for the same scenario. The results for the scenario of increasing to the highest readiness level are shown in Tables XVI and XVII at the capability and function level, respectively. In this scenario, Technology 1 is the most important component for all the capabilities and functions, while the second and third positions vary in several other components. C. Discussion of the Application of the Component IMs Component IMs facilitate the comparison among different components with respect to their impact to the capability/function SRL. With an illustrative example, the capability and function level component importance are considered with two scenarios for each measure. As a summary, we extend our

17 TAN et al.: ANALYZING COMPONENT IMPORTANCE IN MFMC SYSTEMS DEVELOPMENTAL MATURITY ASSESSMENT 291 TABLE XII TOP THREE MOST IMPORTANT COMPONENTS FOR CAPABILITY I CT OF INCREASING TO THE HIGHEST LEVEL TABLE XIII TOP THREE MOST IMPORTANT COMPONENTS FOR FUNCTION I CT OF INCREASING TO THE HIGHEST LEVEL TABLE XIV TOP THREE MOST IMPORTANT COMPONENTS FOR CAPABILITY I LH OF INCREASING BY ONE LEVEL analysis to the results of the application and explore the insights as follows. 1) Component Importance Dependency on Interest: As shown from the calculation of the illustrative example, depending on what is of interest to an engineering or project manager (i.e., component importance with respect to TRL/IRL, cost, or effort), these three component IMs give different results of critical components. This makes sense with the reality that component importance should be situation-dependent. I P tells the potential of impact of change of component maturity on the capability or function SRL, I CT measures the impact of cost on the SRL, and I LH measures the impact of effort on the SRL.

18 292 IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 58, NO. 2, MAY 2011 TABLE XV TOP THREE MOST IMPORTANT COMPONENTS FOR FUNCTION I LH OF INCREASING BY ONE LEVEL TABLE XVI TOP THREE MOST IMPORTANT COMPONENTS FOR CAPABILITY I LH OF INCREASING TO THE HIGHEST LEVEL TABLE XVII TOP THREE MOST IMPORTANT COMPONENTS FOR FUNCTION I LH OF INCREASING TO THE HIGHEST LEVEL Further, depending on which capability is of interest, the component IMs give different evaluation of critical components at the capability level. Moreover, depending on which function is of interest, the component IMs give different evaluation of critical components at the function level. 2) Discernibility in Components Importance Determination: As shown in the example, the difference between the two consecutive important components can be very minor. For example, the difference between the top and second most important components in Table V is only 0.5%. Since the values of importance factors (TRL/IRL, developmental cost, and effort) are based on estimates, they are not accurate and will directly influence the determination of components importance. Improving estimation method can decrease this influence. Earlier on, Tan et al. [23], [24] have proposed a probabilistic approach for the objective of getting reliable and confident maturity estimation. Furthermore, for component importance determination, it may be more realistic to determine the importance categories through clustering the calculation results rather than just concluding with the ranking order. Then, the allocation of resources can be prioritized to the development of the components in the top category to meet system maturity objectives. While it is not covered in this paper, the authors plan to investigate it in the future research. 3) Comparatively, Technologies Outperform Integrations in the Component Importance Rank: In the illustrative example, TRLs almost dominate the importance ranks, while integrations merely show up in the top three positions. For I P, integrations never occupy the top three positions for two scenarios; for I CT, only the integration between Technologies 6 and 8 (IRL 6,8 ) ranks third for capability C 12 ;fori LH, IRL 6,7, IRL 8,10, and IRL 18,20

19 TAN et al.: ANALYZING COMPONENT IMPORTANCE IN MFMC SYSTEMS DEVELOPMENTAL MATURITY ASSESSMENT 293 rank second for capabilities C 11, C 12, and C 31, respectively, and IRL 6,9 ranks third for capability C 13. While the technology and integration were assumed to be the two basic elements in a system, we investigate the phenomena by tracing back to the SRL computation, which leads to the conclusion that it could mainly ascribed to the following: 1) each IRL join in two and only two operations (when we define IRL ij = IRL ji ) during the SRL computation; 2) each TRL will join at least two operations, and the more links a technology has, the more operations that TRL will join; and 3) the more operations a component join, the more sensitive the SRL will be when the readiness level of the corresponding component changes. 4) Technologies That Have More Integrations Rank Relatively Higher Than Those Have Less Integrations When all Other Factors are the Same: As previously described, the more links a technology has, the more operations it will involve, and the more weight it assumes to impact the SRL. Take capability C 11,forinstance, Table V shows the importance calculation. Technologies 2 and 4 have the same TRL, but Technology 2 that has more links (i.e., three links) ranks higher in the comparison of component importance than Technology 4 that has only two links. 5) Integrations Whose Associated TRLs are Higher Rank Relatively Higher Than Those Whose Associated TRLs are Lower if all Other Factors are the Same: Still take capability C 11 as example, the importance rank of IRL 6,7 is higher than that of IRL 4,5 because the associated TRLs with the former (i.e., TRL 6 = 6 and TRL 7 = 8) are higher than that with the latter (i.e., TRL 4 = 7 and TRL 5 = 6). V. CONCLUSION AND FUTURE WORK The introduction of the SRL approach enables the maturity assessment at the system level; however, the previous SRL definition was unable to address the assessment of function and capability maturity. This paper proposes an enhanced SRL definition to facilitate the developmental maturity assessment for MFMC systems (e.g., the littoral combat ship, the landing platform dock, etc.) at the capability, function, and system levels. In order to distinguish the impact of components maturity on SRL, i.e., the components maturity on the capability or function maturity, this paper proposes three component IMs with respects to TRL/IRL, cost, and effort, respectively. The example in Section IV demonstrates the application of the IMs and explores the insights from the calculation. The enhanced SRL definition attempts to facilitate the maturity assessment from a function-based and capability-based perspective. Due to the exposure of cost overruns, schedule slips and performance problems with DoD weapon programs from the assessment reports by the U.S. Government Accountability Office [29], [45], the request to better manage the program cost becomes more and more imperative. As a result, the open and flexible platform design, such as the littoral combat ship, that can accommodate multiple functions and capabilities, as well as the ability for adopting future mission packages becomes more and more favorable. With such a trend, decisions to tradeoff among multiple functions and capabilities will be unavoidable. Then, applying the enhanced SRL definition to meet these requirements remains to be researched and explored, which will be of great interest in the future research work of the authors. While the proposal of IMs can enable project and engineering managers to differentiate the roles that are played by a number of components, how to apply these IMs to plan system development and make developmental decisions under constrained budget or limited effort availability will be of further interest. While it is not the coverage of this paper, the researchers plan to devote further research to demonstrate the usage of the proposed IMs to assist decision makers in such situations. ACKNOWLEDGMENT The authors would like to thank Editor-in-Chief R. Sabherwal, Department Editor J. Sarkis, and the reviewers for their insightful comments and direction. REFERENCES [1] Z. W. Birnbaum, On the importance of different components in a multicomponent system, in Multivariate Analysis, vol. 11. New York: Academic, 1969, pp [2] F. K. Hwang, A new index of component importance, Oper. Res. Lett., vol. 28, pp , [3] P. Baraldi, E. Zio, and M. Compare, A method for ranking components importance in presence of epistemic uncertainties, J. Loss Prev. Process Ind., vol. 22, pp , [4] J. E. Ramirez-Marquez and D. W. Coit, Composite importance measures for multi-state systems with multi-state components, IEEE Trans. Rel., vol. 54, no. 3, pp , Sep [5] J. E. Ramirez-Marquez and D. W. Coit, Multi-state component criticality analysis for reliability improvement in multi-state systems, Rel. Eng. Syst. Saf., vol. 92, pp , [6] E. Zio, M. Marella, and L. Podofillini, Importance measures-based prioritization for improving the performance of multi-state systems: Application to the railway industry, Rel. Eng. Syst. Saf.,vol. 92,pp ,2007. [7] R. L. Dillon, M. E. Pate -Cornell, and S. D. Guikema, Optimal use of budget reserves to minimize technical and management failure risks during complex project development, IEEE Trans. Eng. Manage., vol. 52,no. 3, pp , Aug [8] J. E. Ramirez-Marquez and B. J. Sauser, System development planning via system maturity optimization, IEEE Trans. Eng. Manage., vol. 56, no. 3, pp , Aug [9] R. Gove, Development of an integration ontology for systems operational effectiveness, in School of Systems and Enterprises. Hoboken, NJ: Stevens Institute of Technology, [10] B. Sauser, E. Forbes, M. Long, and S. McGrory, Defining an integration readiness level for defense acquisition, presented at the Int. Symp. Int. Counc. Syst. Eng., Singapore, [11] B. Sauser, R. Gove, E. Forbes, and J. Ramirez-Marquez, Integration maturity metrics: Development of an integration readiness level, Inf., Knowl., Syst. Manage., vol. 9, no. 1, pp , [12] B. Sauser, J. Ramirez-Marquez, R. Magnaye, and W. Tan, A systems approach to expanding the technology readiness level within defense acquisition, Int. J. Defense Acquis. Manage., vol. 1, pp , [13] B. Sauser, J. Ramirez-Marquez, R. Magnaye, and W. Tan, System maturity indices for decision support in the defense acquisition process, presented at the 5th Annu. Acquis. Symp., Monterey, CA, [14] B. Sauser, J. Ramirez-Marquez, D. Henry, and D. DiMarzio, A system maturity index for the systems engineering life cycle, Int. J. Ind. Syst. Eng., vol. 3, pp , [15] B. Kober, Technology insertion metrics for spiral development E-2C aircraft, in School of Systems and Enterprises Masters Project in Systems Engineering. Hoboken, NJ: Stevens Institute of Technology, [16] G. Dankulich, System readiness assessment of the Navy s electromagnetic aircraft launch system, in School of Systems and Enterprises Masters Project in Systems Engineering. Hoboken, NJ: Stevens Institute of Technology, [17] D. Lin, Develop a producibility readiness level to complement system readiness level within defense acquisition systems, in School of Systems

20 294 IEEE TRANSACTIONS ON ENGINEERING MANAGEMENT, VOL. 58, NO. 2, MAY 2011 and Enterprises Masters Project in Systems Engineering. Hoboken, NJ: Stevens Institute of Technology, [18] B. Kober and B. Sauser, A case study in implementing a system maturity index, presented at the Amer. Soc. Eng. Manage. Conf., West Point, NY, [19] E. Forbes, R. Volkert, P. Gentile, and K. Michaud, Implementation of a methodology supporting a comprehensive system-of-systems maturity analysis for use by the littoral combat ship mission module program, presented at the 6th Annu. Acquis. Symp., Monterey, CA, [20] D. J. Kim, H. C. Kim, and J. O. Kim, Evaluation of structural importance based on minimal cut set theory, Trans. Korean Inst. Electr. Eng.,vol.58, pp , [21] R. Volkert, Notional assessment methodology for KPP accomplishment in a SoS proposed methodology for measuring performance progress within a system of systems (SoS), PMS 420 white paper, Sep [22] R. Magnaye, B. Sauser, and J. Ramirez-Marquez, System development planning using readiness levels in a cost of development minimization model, Syst. Eng., vol. 13, no. 4, [23] W. Tan, B. Sauser, and J. E. Ramirez-Marquez, Monte-Carlo simulation approach for system readiness level estimation, presented at the Int. Symp. Int. Counc. Syst. Eng., Singapore, [24] W. Tan, J. E. Ramirez-Marquez, and B. Sauser, A probabilistic approach to system maturity assessment, J. Syst. Eng., New York: Wiley, [25] P. Fomin, T. A. Mazzuchi, and S. A. Sarkani, Incorporating maturity assessment into quality functional deployment for improved decision support analysis, risk management, and defense acquistion, presented at the Int. Conf. Syst. Eng. Eng. Manage., San Francisco, CA, [26] R. Cueller and B. Sauser, Dynamic multipoint optimization application to corporate portfolio management, presented at the Acquis. Res. Symp., Monterey, CA, [27] R. Magnaye, B. Sauser, and J. Ramirez-Marquez, Using a system maturity scale to monitor and evaluate the development of systems, presented at the 6th Annu. Acquis. Symp., Monterey, CA, [28] A. Tetlay and P. John, Determining the lines of system maturity, system readiness and capability readiness in the system development lifecycle, presented at the Conf. Syst. Eng. Res., Loughborough, U.K., [29] GAO, Defense acquisitions: Assessments of selected weapon programs, GAO, vol. GAO SP, Washington, DC: U.S. Government Accountability Office, [30] F. C. Meng, Comparing the importance of system components by some structural characteristics, IEEE Trans. Rel., vol. 45, no. 1, pp , Mar [31] A. Gandini, Importance and sensitivity analysis in assessing system reliability, IEEE Trans. Rel., vol. 39, no. 1, pp , Apr [32] J. B. Fussell and W. E. Veseley, New methodology for obtaining cut sets for fault trees, Trans. Amer. Nucl. Soc., vol. 15, pp , [33] R. E. Barlow F. Proschan Importance of system components and fault tree events, Stochastic Processes and their Appl., vol. 3, no. 2, pp , Apr [34] Y. Dutuit and A. Rauzy, Efficient algorithms to assess component and gate importance in fault tree analysis, Rel. Eng. Syst. Saf., vol. 72, pp , [35] F. K. Hwang, A hierarchy of importance indices, IEEE Trans. Rel., vol. 54, no. 1, pp , Mar [36] G. Levitin and A. Lisnianski, Importance and sensitivity analysis of multi-state systems using the universal generating function method, Rel. Eng. Syst. Saf., vol. 65, pp , [37] E. Zio and L. Podofillini, Monte Carlo simulation analysis of the effects of different system performance levels on the importance of multi-state components, Rel. Eng. Syst. Saf., vol. 82, pp , [38] J. D. Andrews and S. Beeson, Birnbaum s measure of component importance for noncoherent systems, IEEE Trans. Rel., vol. 52, no. 2, pp , Jun [39] S. Beeson and J. D. Andrews, Importance measures for noncoherentsystem analysis, IEEE Trans. Rel., vol. 52, no. 3, pp , Sep [40] P. Hilber and L. Bertling, Component reliability importance indices for electrical networks, in Proc. Power Eng. Conf Int., pp [41] Z. Peichao, L. Portillo, and M. Kezunovic, Reliability and component importance analysis of all-digital protection systems, in Proc. IEEE PES Power Syst. Conf. Expo. 2006, pp [42] X. Hengcheng, C. Pu, and Z. Jianguo, The importance analysis of componments in the aviation system using the binary decision diagrams, in IEEE Conf. Proc. Cybern. Intell. Syst. 2008, pp [43] J. Smith An alternative to technology readiness levels for nondevelopmental item (NDI) software, Carnegie Mellon, Pittsburgh, PA, Tech. Rep. CMU/SEI-2004-TR-013, Apr [44] P. Zhang, C. Huo, and M. Kezunovic, A novel measure of component importance considering cost for all-digital protection systems, in Proc. IEEE Power Eng. Soc. Gen. Meet., 2007, pp [45] GAO, Defense acquisitions: Assessments of selected major weapon programs, GAO, vol. GAO , Washington, DC: U.S. Government Accountability Office, Weiping Tan received the B.E. degree with first class honor in automation from Beijing Institute of Technology, Beijing, China, in 2006, and the M.E. degree in engineering management in 2009 and two graduate certificates in project management and infrastructure management from Stevens Institute of Technology, Hoboken, NJ, where he is currently working toward the Ph.D. degree in engineering management from the School of Systems and Enterprises. His research interests include developing methodologies for the developmental maturity assessment for multifunction multicapability systems. Mr. Tan was elected to Epsilon Mu Eta the Engineering Management Honor Society. He was the Vice President for the International Council on Systems Engineering (INCOSE) Stevens Student Chapter. He was the recipient of the Brian Mar Best Student Paper Award at the 2009 INCOSE International Symposium. BrianJ.Sauserreceived the B.S. degree in agricultural development with the specialization in horticulture technology from Texas A&M University, College Station, TX, the M.S. degree in Bioresource Engineering from Rutgers, The State University of New Jersey, New Brunswick, NJ, and the Ph.D. degree in project management from Stevens Institute of Technology, Hoboken, NJ. He is currently an Assistant Professor in the School of Systems and Enterprises, Stevens Institute of Technology, where he is also the Director of the Systems Development and Maturity Laboratory, which seeks to advance the state of knowledge and practice in systems maturity assessment. He was also engaged in government, industry, and academia both as a Researcher/Engineer and Director of programs for more than 12 years. His research interests include the management of complex systems. Dr. Sauser has been nationally recognized and adopted as a decision support tool by organizations within NASA, U.S. Army, U.S. Navy, Northrop Grumman, and Lockheed Martin for his work on system maturity assessment. He is a National Aeronautics and Space Administration Faculty Fellow, Editor-in-Chief of the Systems Research Forum, and an Associate Editor of the IEEE Systems Journal. Jose Emmanuel Ramirez-Marquez received the M.S. and Ph.D. degrees in industrial engineering from Rutgers University, New Brunswick, NJ, and the M.S. (statistics) degree in actuarial science from Universidad Nacional Autonoma de Mexico, Mexico City, Mexico. He is a Fulbright Scholar. He is an Associate Professor of the School of Systems and Enterprises at Stevens Institute of Technology. His research efforts focus on the reliability analysis and optimization of complex systems, the development of mathematical models for networks operational effectiveness, the computational analysis of resilience and, the development of evolutionary optimization algorithms. In these areas, he has conducted funded research for both private industry and government. Also, he has published more than 70 refereed manuscripts related to these areas in technical journals, book chapters, conference proceedings and industry reports. He has presented his research findings both nationally and internationally in conferences such as INFORMS, IERC, ARSym and ESREL. He is an Associate Editor for the International Journal of Performability Engineering. Dr. Ramirez-Marquez Nationally, serves as the current President of the Quality Control and Reliability division board of the Institute of Industrial Engineers. Internationally, he is a member of the Technical Committee on System Reliability for the European Safety and Reliability Association.