An "asymmetric" approach to the assessment of safety-critical software during certification and licensing

Transcription

1 An "asymmetric" approach to the assessment of safety-critical software during certification and licensing Sergiy A. Vilkomir, Vjacheslav S. Kharchenko Abstract The purpose of the present paper is the description of the offered by the authors general approach to the software assessment during certification and licensing. This kind of software assessment has the specific character, taking into account limitation of time, material and human resources available to the experts. The offered asymmetric approach allows to define the most important areas, where the basic efforts of the software assessment should be concentrated. The following tasks have been solved for the formation of this approach: - more precise definition of the purposes of the software safety assessment; - formation of the set of acceptance criteria, which software should satisfy; - development of principles of acceptance criteria application at all stages of the software life cycle; - development of standard contents of stages of the software licensing; - definition of features of software assessment for systems with version redundancy. 1. Introduction The problem of ensuring and assessment of computer systems software has become actual last years in connection with extending application of computer systems in such critical branches, as atomic energetics, astronautics, chemistry, and transportation. Computer systems fulfil more and more crucial functions and accordingly consequences of possible software mistakes increase. Ensuring and assessment of software at various stages of the software life cycle are important parts of the general problem of system dependability. These tasks are considered in plenty of scientific publications, a number of monographs [1-3], national and international standards, for example, in atomic energetics [4, 5]. Several international normative documents in the area of safety-critical software are under approval now [6-8]. Usually the problem is considered from the point of view of two basic participants of a process: a computer system user and a software developer. At the same time in many countries with advanced atomic energetics there is a third participant - state regulatory bodies (authorities), which main task is licensing of activity connected with nuclear and radiation safety. An assessment of software during certification and licensing is usually carried out on behalf of regulatory bodies either by independent experts, or by experts of organisations which provide a scientific and technical support of regulatory bodies. The assessment of software for these purposes has the specific character [9-11] and just this aspect is addressed in the present paper. Irrespective of assessments carried out by the experts, the developer of the system and software has the main responsibility for the safety achievement. For achievement of a required level of software at all stages of life cycle the developer implement verification, i.e. the process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase [12]. During licensing the experts implement the assessment both of software and of the process of its development and verification. Theoretically it is possible the symmetric approach, when 467

2 the experts implement the all-round assessment after each separate phase of the development. In that case volumes of efforts of the experts are in a sense proportional to volumes of efforts of the developer at the appropriate stage. However in practice the symmetric approach can be inexpedient and even unrealisable. The main reason consists in limitation of time, material and human resources available to the experts. It involves a complex task of a choice of optimum volumes of materials for the assessment, allocation of most important software characteristics, development of practicable methods of the software assessment during certification and licensing. The purpose of the present paper is the general description of the offered methodology for the solution of this task, named the methodology of the asymmetric assessment of software. 2. General characteristics of the "asymmetric assessment" of software The offered approach is based on the experience of practical works by the software assessment of computer control systems of nuclear power plants (NPPs) in Ukraine [13]. These works are carried out by Ukrainian State Scientific and Technical Center on Nuclear and Radiation Safety and Nuclear Regulatory Administration of Ministry for Environmental Protection and Nuclear Safety of Ukraine. The following tasks have been solved for the formation of this approach: - more precise definition of the purposes of the software assessment and analysis of an opportunity of these purposes achievement at all stages of the software life cycle (see section 3); - formation of the set of acceptance criteria, which software should satisfy to have a demanded safety level; development of the structure of each criterion internal contents and hierarchy of subcriteria and requirements (see section 4); - development of principles of acceptance criteria application at all stages of the software life cycle depending on the safety system class; consideration of a ratio criterion stages for software of safety systems and software of normal operation systems, important for safety (see section 5); - development of standard contents of stages of the software licensing for safety important digital systems at NPPs (see section 6); - definition of the features of the software assessment for systems with version redundancy (see section 7). The solution of these tasks has allowed to define the most important areas, where the basic efforts of the software assessment should be concentrated. For all specified directions the optimum allocation of resources is irregular (asymmetric). Thus, the offered approach to the software assessment is characterised by the asymmetric distribution of: - the purposes of works on software life cycle stages; - criteria on safety classes of systems; - criteria on software life cycle stages; - works on kinds of used diversity. 3. Purposes of the assessment on various stages of the software life cycle The purpose of the expert s activity on the software assessment within the framework of licensing of important for safety systems is to be convinced that software answers criteria, norms and rules of the safety. The reliance on the high level of software safety, received during the assessment, forms the basis for issuing by regulatory bodies of the sanctions on the software and system use. For achievement of this purpose (we name it as the purpose A) the experts can require of the developer to give the additional information (documents), or to 468

3 specify the information, transferred earlier. However, it is necessary to note that the receipt of the additional information from the developer does not influence in any way on the existing level of the software safety. Therefore, alongside with the purpose A the second purpose (the purpose B) of works of the expert also exists - to exert real influence upon the increase of the safety level. For possibility of achievement of the purpose B the special approach to the software assessment is required. First, it is necessary to carry out the assessment parallel and in rate with the development process at all life cycle stages - software requirements, designing, coding (programming), and testing. Secondly, the operative contact with the developers is necessary, so that all remarks and defects, founded out by the experts were transferred to the developers for elimination. The possibility to achieve both the purpose A and the purpose B is different for each stage of the software life cycle. The optimum distribution of efforts of the experts is schematically shown in Figure 1. purpose A purpose B requiremets design coding testing plan testing report purpose A (using tools) purpose B (using tools) Figure 1: Distribution of the software assessment purposes for each stage of the software life cycle During the assessment after the requirements development, experts in principle can have plenty of the remarks, the part of which can require correcting the documentation. However, as the experience shows, elimination of these remarks basically influences on quality of the documentation and to a lesser degree on the real level of software. The similar situation arises with the assessment on the design stage. Hence, the basic efforts of the experts at these stages are directed on the achievement of the purpose A. The opportunity of the fulfilment of the software assessment after the stage of coding directly depends on the presence of special hardware and software means (tools) allowing carrying out static and dynamic analysis of software. The realisation of such analysis is, as a matter of fact, independent verification that is carried out parallel with works of the developer. In principle, the realisation of independent verification is not a direct task of the regulatory body. Therefore, it is possible not to carry out the assessment after the stage of coding, especially as its realisation is practically impossible without tools. However, if the tools were available, the realisation of selective examination of a part of software would be expedient. Since it gives the possibility to reveal software errors, the basic purpose at this stage is increasing of the software safety level (the purpose B). The assessment of the software testing should be carried out separately for two substages: after issue of the testing plan (but prior to the beginning of the testing) and after issue of the software testing report. For the achievement of the purpose B, all experts remarks should be transferred to the developer for their removing and updating of the plan prior to the beginning of the testing. The experts should pay the special attention to testing completeness and 469

4 volumes of such auxiliary (from the functional point of view, but important from the safety point of view) software functions, as diagnostics of hardware, software self-diagnostics, logic redundancy of signals, reconfiguration and switching to reserve channels and so on. If necessary, the experts should oblige to the developer to carry out the additional testing. If the object of the assessment on this substage has been achieved, i.e. the testing plan has been corrected by the developer and has been approved by the experts, during the assessment of the testing report the experts need only to be convinced, that all tests have been carried out according to the plan and have been finished successfully. So the main purpose of this substage is the purpose A. The adducted above review of the software assessment purposes at the various life cycle stages gives evidence, that the greatest possibility for the experts to affect the safety level increasing occurs at the stage of the software testing plan assessment. The basic efforts of the experts should be concentrated just on this part of the assessment. 4. The set of the criteria of the software assessment For the fulfilment of the software assessment the authors propose the set of the acceptance criteria. These criteria are indicators and rules, according to which the assessment is carried out and the final conclusions about software conformity the safety requirements are done. The proposed set includes five criteria: completeness, documentation, intelligibility, independence and conformity. Software satisfies the criterion of completeness, if: - its specifications completely correspond to the specifications of the system; - all functional requirements to software are reflected in the project; - software corresponds to the general requirements, common to software of all important for safety systems, including the requirements to designing and verification; - the performance of all software functions is checked up with testing. Software satisfies the criterion of documentation, if the composition and structure of the developer s documentation on all stages of the development and verification correspond to the requirements of the standards, norms and rules, and also there is a necessary operational documentation. The criteria of documentation and completeness are interconnected. Thus by the criterion of completeness the substantial aspect, and by criterion of documentation - the formal aspect are estimated. Software satisfies the criterion of intelligibility, if the documentation on the software development and verification is stated in the form, clear and intelligible to the experts who are not directly participating in their realisation. In addition, the traceability of the performance of the requirements to software at the various life cycle stages should be provided. Software satisfies the criterion of independence, if a degree of the independence of software verification corresponds to a safety class of the system. For the most critical systems verification should be carried out by the group of the experts (organisation), administratively and(or) financially independent on the experts (organisation), developing software. For less critical systems the realisation of the development and verification by the different experts is recommended, however the administrative and financial independence is not required. Software satisfies the criterion of conformity, if verification has been successfully completed before putting the system into operation, i.e. if by this moment all found out defects have been analysed and are eliminated (or the reasonable decision on their further elimination has been accepted). The criteria are the important part of the general circuit of the software assessment, illustrated in Figure

5 Norms, Rules, Standards Experts Purposes of software assessment Criteria of software assessment Methods and means Subcriteria Final assessment Result Documenta tion Requir. Software object of assessment Life cycle stages Software developer Figure 2: Software assessment The set of subcriteria, which detail the assessment, corresponds to each criterion. The further detailed elaboration is carried out on the basis of the connection of each subcriterion with the set of the requirements, which are checked during the assessment. So the proposed set of criteria has a two-part structure: hierarchy criterion - subcriterion - requirement on a vertical and five described above criteria on a horizontal. First the assessment by each criterion is formed on the basis of separate subcriteria, and then the final assessment is formulated. Two basic loops (evaluation and correction) are present in the proposed circuit according to two purposes of the software assessment. The loop of evaluation is shown by unbroken arrows. The loop of correction is shown by dotted arrows and corresponds to a situation, 471

6 when on account of the negative assessment the developer should modify software (to correct the errors, to carry out the additional tests and so on). 5. The use of acceptance criteria at the various stages of the software life cycle for systems of the various safety classes The proposed set of the criteria has the specific character of the application at each stage of the software life cycle. Some criteria are applied at all stages, some - only on one stage. The character of the application essentially depends on a safety class of the estimated system. Depending on importance of the system for safety we shall divide software on two groups. Following the Ukrainian normative document [14] for NPPs systems, it is software of safety systems (SS) and software of systems of normal operation, important for safety (SIS). Instead of this, it is also possible to divide according to accepted in the USA approach on classes 1E and not 1E, or into categories of safety agrees IEC Std [15]. For software of SS, it is necessary to carry out the assessment in the greatest possible volumes. For software of SIS, the approach is opposite: should check something only if it is especially necessary or it is the most important for safety. Such approach is explained in smaller influence SIS on the safety in comparison with SS. The realisation of the specified approaches results in the asymmetric distribution of criteria both on stages of the life cycle and on classes of safety, as shown in Table 1. It is necessary to note, that the question is to carry out (+) or to not carry out (-) the assessment according to the criterion, and it is not the question if software should or should not satisfy this criterion. Table 1: Acceptance criteria for each stage of the software life cycle Stages Software Software Software testing Criteria requirements design Planning Reporting Completeness SS SIS Intelligibility SS SIS Documentation SS SIS Independence SS SIS Conformity SS SIS For example, both software of SS and software of SIS should satisfy the criterion of intelligibility at all stages of the life cycle software. The task of the developer is to comply with this requirement. But the approach to the assessment of this requirement by the experts is various for software of SS and software of SIS. The influence on the safety of software of SS is so great, that the conformity to the criterion of intelligibility is necessary for estimating at all stages. For software of SIS, it is possible to carry out this assessment only at the stage of the verification plan, since this stage is the most responsible in the general circuit of the assessment. The application of some concrete criterion at various stages of the life cycle also has the specific character. Let us consider it on an example of the criterion of completeness. At the requirements stage according to the criterion of completeness it is necessary to estimate the 472

7 completeness of conformity of the software requirements to the requirements for the system, and also completeness of reflection of the general requirements in the specifications. These general requirements are common for software of all kinds of systems and include the requirements to diagnostics of hardware, software self-diagnostics, opportunity of periodic testing of functions of the system, software structure and others. At the design stage, by the criterion of completeness it is necessary to estimate the reflection of the software requirements in the design documentation. At the stage of the development of the testing plan it is necessary to estimate the completeness of the stipulated checks of software functions (requirements). Besides, the substantiation of the application of tools for software development and verification should be estimated. At last, at the stage of the testing report on the criterion of completeness it should estimate, as far as the carried out testing corresponds to the testing plan and as far as their results are reflected in the final documentation. When the same criterion is applied at any stage both for SS and for SIS, the assessment by subcriteria for software of SS could be more profound, and the requirements could be stricter. 6. Stages of the software review Standard contents of stages of the software licensing based on the acceptance criteria are considered below for safety important digital systems at NPPs [16]. Before the beginning of digital system licensing, the plan of the review is established. This plan covers as well the software assessment. The schedule of the review is established, including deadlines of representation by the developer of the concrete software documents. The plan of the assessment of software includes: - the brief description of the subject of the assessment; - criteria, on which the assessment is conducted; - the list of methods and actions which are carried out during the assessment. The plan of the software review should contain a detailed list of review stages, which is taking into account specificity of the considered system and its software. The standard content of such list of software review stages at all phases of software life should include: - the evaluation of availability and completeness of software documentation; - the evaluation of the reflection of the detailed requirements to software, composed at the previous stage of review; - the evaluation of conformity to the acceptance criteria; - the transfer of the remarks to the developer and obtaining the corrected and supplemented software documentation. The final experts' report and recommendations to the regulatory bodies about possibility of software use should be issued on last phase of the review. 7. Features of the software assessment for systems with version redundancy Features of the methodology of the asymmetric assessment are considered below for software of computer systems with version redundancy, otherwise named multiversion systems [17, 18]. The various kinds of diversity are using in such systems depending on areas of application. These kinds of diversity are classified [19-21] as object and subject diversity; functional (model), program and hardware diversity; design and operational diversity etc. The multiversion system contains two channels or subsystems (basic and diverse) in simple case and could have up to 4-5 subsystems in general case. 473

8 The following tasks should be in addition solved during the software assessment for multiversion systems: - checking of the presence of the real diversity in the system; - determination of the characteristics (parameters) of the diversity; - the assessment of the achieved level of diversity and its conformity to the requirements. Software of subsystems constructed on the different versions should be estimated in such systems. If in case of a failure of one subsystem (for example, basic) the transition to the second (diverse) subsystem is stipulated, then control and reconfiguration software should also be estimated. Within the framework of the software assessment of multiversion systems, the asymmetry is displayed in two directions: on life cycle stages and on subsystems. The asymmetry on the life cycle stages is displayed depending on the kind of diversity. For systems with the functional (model) diversity in comparison with systems with the program diversity, large efforts should be directed on the assessment of the system requirements and software specifications. It is caused by the fact that the effect from introduction functional diversity can take place only with minimal correlation of the appropriate model versions, and consequently, specifications, on the basis of which the versions of software have been elaborated. Opposite, for systems with program diversity large efforts should be directed on the assessment of design and coding stages. The asymmetry on subsystems is displayed for the software assessment of various channels of multiversion systems. On the one hand, the requirements to the assessment of software of these channels, in comparison with software of channels of nondiverse systems, can be weakened, because the probability of a common failure caused by program errors is lower in multiversion systems. On the other hand, the requirements to the assessment of control and reconfiguration software should be increased. 8. Conclusions and further work The optimum distribution of works of the experts is the basis of the proposed methodology of the asymmetric assessment of software during licensing. The assessment is considered at stages of the software life cycle with using of the set of the acceptance criteria depending on the safety class of the system. The stated results can be widespread on the assessment of hardware and a computer system as a whole. Thus, the asymmetric assessment can be used within the framework of a holistic methodology [22], where the human, hardware and software components are considered by an integrated approach. The described methodology has found practical application with the review of the following safety related computer systems at Ukrainian NPPs: - Rod Group and Individual Control System of Skoda-Controls (Czech Republic) at South- Ukrainian-1 NPP and Khmelnitsky-1 NPP; - Turbine Control System of Shevchenko Plant (Ukraine) for Zaporozhye-1 NPP; - Safety Parameters Display System of Westinghouse (USA) at Zaporozhye-5 NPP, Khmelnitsky-1 NPP and Chernobyl-3 NPP; - Unit Information System of WESTRON (Ukraine-USA) at South-Ukrainian-1 NPP and others. The complex of methodical and tool [23] means is developed for realisation of methodology of the asymmetric assessment. The proposed methods can be adapted for the software assessment for critical application systems in other branches. 474

9 9. References [1] Leveson, N, Safeware: System Safety and Computers, Addison-Wesley, [2] Lyu, M. R. (editor), Handbook of Software Reliability Engineering, McGraw-Hill, [3] Kersken, M. and Saglietti, F., Software Fault Tolerance Achievement Assessment Strategies, Springer-Verlag, Berlin, [4] IEC Std. 880, Software for Computers in the Safety Systems of Nuclear Power Stations, [5] IEEE Std , IEEE Standard Criteria for Digital Computers in Safety Systems of Nuclear Power Generating Stations, [6] IAEA Safety Standard Series, Working ID NS 264, Software for Computer Based Systems Important to Safety in Nuclear Power Plants, Draft Safety Guide, [7] IEC Std , Software for Computers Important to Safety for Nuclear Power Plants as a First Supplement to IEC Publication 880, Draft, [8] IEC Std , Functional Safety of Electrical/ Electronic/ Programmable Electronic Safety - Related Systems, Draft, [9] Vilkomir, S. A. and Zhidok, G. I., Software for Nuclear Power-Generating Unit Protection Systems: Safety and Reliability Problem, Control Systems and Machines, num. 4/5, 1995, pp (In Russian). [10] Karpeta, C., Licensing Aspects of the NPP Temelin I&C Replacement Project, Proceedings of the International Topical Meeting on VVER Instrumentation and Control, April 21-24, 1997, Congress Centre, Prague, Czech Republic, pp [11] Bussac,.J. P., Jover, P. and Conflant, M., The Introduction of Computer Systems into Nuclear Power Plant Instrumentation and Control: the French Safety Approach, International Symposium on Nuclear Power Plant Instrumentation and Control, May 1992, Tokyo, Japan. [12] IEEE Std , Glossary of Software Engineering Terminology, [13] Vilkomir, S. A. and Zhidok, G. I., Experience of Licensing of Software for Digital Safety Related Systems in Ukraine, Project Control for 2000 and Beyond, Proceedings of ESCOM- ENCRESS 98, May 1998, Rome, Italy, pp [14] General provisions on safety assurance at nuclear stations (OPB-88), PNAE G , Moscow, 1989 (In Russian). [15] IEC Std. 1226, Nuclear Power Plants. Instrumentation and Control Systems Important for Safety. Classification, [16] Vilkomir, S. A. and Kharchenko, V. S., Methodology of the Review of Software for Safety Important Systems, Proceedings of ESREL 99 The Tenth European Conference on Safety and Reliability, Munich - Garching, Germany, September 1999, pp [17] Avizienis, A., The N-version Approach to Fault-Tolerant Systems, IEEE Transaction on Software Engineering, vol. 11, no. 12, 1985, pp [18] Kharchenko, V. S., Theory of Defect Tolerant Digital Systems with Version Redundancy, Kharkov Military University, Ukraine, 1996 (In Russian). [19] Saglietti, F., Fault Tolerance by Software Diversity: How and When?, Proceedings of the Safecomp'94, Anaheim, California, USA, October 23-26, 1994, pp [20] Kharchenko, V. S. and Mishchenko, S., Dynamical Model for the Operative Forecasting of the Software Reliability, Transport information and control systems, num. 6, 1996, pp. 3-7 (In Russian). [21] Kharchenko, V. S., Choice of Design Technologies and Basic Architectures for the Defect- Tolerant Digital Control and Computing Real-Time Systems, Space Science and Technology, v. 3, num. 5/6, 1997, pp (In Russian). [22] Pasquini, A., Goerke, W., Kanoun, K. and Rizzo, A., An Holistic Approach to Dependability, Proceedings of the Safecomp'96, Vienna, October [23] Vilkomir, S. A., Kharchenko, V. S., Ponomaryev, A. S. and Gorda, A. L., The System Safety Assessment by the Use of Programming Tools during the Licensing Process, Proceedings of the 17th International System Safety Conference, Orlando, Florida, USA, August 1999, pp