Impact of an compliant quality management system on a University research enterprise AJ Walker Software Engineering Applications Laboratory, Electrical Engineering, University of the Witwatersrand, Johannesburg, South Africa Summary: The Software Engineering Applications Laboratory (SEAL) is a university-based research enterprise with project activity in the areas of software quality management and the development of specialised engineering applications. The SEAL was awarded a third party certification for software development in July 1995. This paper reviews the motivation for developing an compliant quality management system (QMS). It examines some of the special challenges which had to be faced while developing the QMS. Benefits which have accrued from this experience are considered. Keywords:, quality management, documented quality system, education and training, software quality. 1. Introduction The University shares many goals and objectives in common with leading universities the world over, including a commitment to excellence in education, in the conduct of research, and the development of highly skilled human resources for benefit to the individual and the wider community. The key performance indicators of such academic excellence are seen to include: a. the research output of papers published at conferences, and accepted for publication in learned journals. b. the number of under- and postgraduate students graduating in the allotted period of time c. the role played by academics in professional fora d. the commitment to excellence in teaching activity A highly prized feature of academic life is the freedom to pursue self-directed research activity, and to enjoy a high degree of latitude in the conduct of teaching activities. As a consequence universities are staffed by high calibre self-directed individuals. The familiar research model is one in which a research novice (post-graduate student) attaches him/her self to the research team leader (typically a highly respected individual with a good research track record) and thereby learns to do research as a member of the team. After a long apprenticeship, and promotion, there will be an expectation that such an individual will gather promising students into the research discipline. In this way the process of developing skilled researches repeats itself. From a traditional viewpoint, the system is effective. If the situation is viewed from a quality perspective, the model is flawed, and rarely achieves maximum potential. 1.1 The SEAL, Goals and Objectives The Software Engineering Applications Laboratory (SEAL) was established in August 1988 as a research facility with a view to promoting the development of a culture of quality in software product development in South Africa. The SEAL is the smallest of the research groups in the Department of Electrical Engineering within the Faculty of Engineering, which include: Power systems engineering Control engineering Communications engineering Software engineering The Mission of the Laboratory is to be: Partners with Industry in the pursuit of excellence in Software Engineering. Specifically, the SEAL goal is to become a world-class centre of expertise, education and training, in Software Engineering. This goal requires that the SEAL will: a. Play the leading role in education and training in software engineering in Southern Africa, as evidenced by feedback from the following stakeholders: i. Course participants ii. Employer of participant iii. Industry at large b. Demonstrate a significant research capability and promote, implement and extend the boundaries of international best practices in our chosen areas of expertise, as evidenced by the following indicators: i. Output of Technical papers ii. Output of graduate and post-graduate students QS16030.010 Revision 0.10 08 January 1997 Page 1
iii. Receipt of Financial Support and income from contracts and other industry-oriented activities. iv. Level and quality of contracts or consulting v. Membership of, and contribution to professional fora 1.2 Full-time staff and postgraduate students actual duration was closer to two years, largely due the time taken to crystallise the actual requirements of the research project, and the time taken to produce the research dissertation. 2. Quality Problems and improvement opportunities At the present time there at two full-time staff members, and 15 post-graduate students. The latter is highly variable as new intake swells the number and graduations reduce the cohort. The SEAL Post-graduate Laboratory can accommodate 10 full-time students comfortably, and any time there might be 5 part-time research students. The nominal duration of a full-time Masters research programme is 12 calendar months. Until recently, the Quality Problem While the official Faculty period of executing a full-time M Sc. is 12 months most students take from anywhere between 15 and 36 months. Most software developed by higher degree students is unusable by current and future project stakeholders. Few research projects actually result in technical publications at conferences or in learned journals The following issues are responsible for nonachievement of research potential: ineffective project management, Table 1 Quality Problems and improvement opportunities Improvement Opportunity Cutting down the time spent on the M Sc. to 12 months can double productivity. Cutting down rework and scrap represents large opportunities for productivity improvement. A large number of research projects are underway, a review of which indicate that the results are publishable Reasons for problem research progress is hampered by poorly defined project requirements resulting in research delays of up to 9 months research scope is excessive for the type of degree programme producing the dissertation thesis takes from 4-8 months The low level of reusability is largely to due to absence of document control non-existent software configuration management. In the conventional M Sc. model the major documentation effort is devoted to the dissertation write-up. Once this is completed the student is usually not available to write up the material as publications. poor or non-existent project support processes, including document and configuration control Process Change Required more realistic scoping of research projects invest more effort in developing project requirements and specifications at the start of the project adopt the technical paper model for M Sc. project documentation procedures and practices for document and data control, and software configuration management need to be installed facilities need to be acquired to support this process Change the documentation model to emphasise: write-up of project results and work products as the project progresses Change the M Sc. documentation model from multi-chapter to technical paper model. Impact of SEAL QMS strong emphasis on project planning and specification and traceability of requirements has helped enormously to invest quality thought and effort early in the project. the technical paper model has proved to be a win win model - benefiting students and staff. project definition for all SEAL activity defined fully electronic definition for QMS SEAL File server facility acquired and installed The QMS project model ensures that management and technical artefacts are products as the project progresses. All research projects now deliver technical papers, most of which are in a form suitable for publication. poor records, lack of visibility of project processes poor track record in documenting research results QS16030.010 Revision 0.10 08 January 1997 Page 2
As a consequence of the above, research output suffers on account of: documentation while the project is in progress is minimal or non-existent - resulting in a major documentation effort at the close of the research project i.e. writing up the thesis a process which familiarly occupies from 3 months to two years. unless research results are written up in the form of a technical paper prior to submission of the examinable output i.e. the dissertation\thesis, there is a strong likelihood that the research results will not be written up at all - since the research supervisor will be too busy with other commitments. the emphasis of the research documentation activity is on the wrong product - i.e. the dissertation or thesis. This product will be examined typically by 2 or 3 persons the internal and external examiners. Once the examination process is completed the product is physically archived and loses visibility The manner in which some of these quality issues have been addressed in the SEAL are listed in Table 1. 3. SEAL approach to achieving compliance to The life-cycle defined for SEAL software development activity is a spiral development model in which in the early spirals the focus is upon elaborating product requirements. Once these have begun to consolidate and stabilise the focus in the model changed to requirements consolidation, product test specification development, and to product verification, validation and testing. A brief overview of the application of the process clauses of the [1] Standard to the SEAL software development process is given in Table 2. QMS and requirements 4.1 Management Responsibility 4.2 Quality System 4.3 Contract Review One of the problems of the small staff complement of the SEAL is that key responsibilities are all carried by the Director - including ongoing responsibility of the maintenance of the QMS and its infrastructure. All aspects of SEAL activity development are managed in compliance with the document quality system The few areas where this can be formally applied is in the relationship of the student to QMS and requirements 4.4 Design Control 4.5 Document and data control the supervisor, and when contract work for outside entities is undertaken. The issues of design input, design output, design changes, design verification and validation are applied to all technical products emanating from the project. All document and data control requirements are applied to all management and technical products and quality records emanating from the project. All products are considered to have their primary representation in electronic format. Document and data distribution is managed by ftp and e-mail from the SEAL File server. 4.6 Purchasing Not seen as a serious quality problem area in the SEAL. All standard university purchasing procedures are followed. 4.7 Control of customer supplied product 4.8 Product Identification and traceability 4.9 Process Control 4.10-4.12 Inspection and Test, Control of inspection, measuring and test equipment, Inspection and Test Status 4.13 Control of non- Not viewed as a problem area since there is little in the way of customer supplied product Mainly interpreted in terms of security and safety issues in the workplace. Comprehensive configuration control is applied to all management, technical and quality records of the project. The concept is also applied to maintaining traceability of software project supplied to clients. (See 4.15) The processes applied to the project are those applied commonly to all projects. The SEAL QMS defines how a project will be managed and what templates will be applied. The requirements of these clauses are applied by means of the following technical products: Product test Plan, Product Test Specification, Product Test Report, Procedure: Verification and Validation; Register: Product Requirements Verification and Validation. These requirements, taken together with the actions taken QS16030.010 Revision 0.10 08 January 1997 Page 3
QMS and requirements conforming product 4.14 Corrective and preventive action 4.15 Handling, storage, packaging, preservation and delivery 4.16 Control of Quality Records 4.17 Internal Quality Audits on 4.8, are applied to prevent the inadvertent release of product with known defects. These requirements are supported at the system level (Project Issue Reports) and at the product level (Product Exception Reports). All such reports are logged and supported in the project QMS as quality records and are reviewed at project level. The policy of the SEAL is to supply product by electronic means to the greatest possible extent. Physical product is only supplied where the situation is best served using this means i.e. sample product supply at conferences for promotional purposes. All records emanating from the project i.e. correspondence, minutes, audit reports, document issue notices, project issue reports, product exception reports, product test records etc. are maintained under configuration control in the project and maintained in the Master Document List. The project is subject to the project level and system level audits on an ongoing basis to verify compliance to the planned project arrangements. 4.18 Training The SEAL is a research enterprise but has a vital role in education and training. Training records are addressed and maintained as part of the Human resource management issues in the Project Management Plan for each project. 4.19 Servicing Not considered a serious issue for the SEAL since there is little requirement for after sales service following the conduct of a research project. 4.20 Statistical techniques At the process level, the ISO\IEC 15504 [2] model for software process assessment is been applied to the SEAL QMS and project definition. With a number of minor exceptions, QMS and requirements 4. Project Practices the SEAL QMS shows compliance in the process categories of CUS, ENG, SUP and MAN to Capability Level 3 requirements. This level of capability is considered adequate for the present. The implications of Level 4 compliance in selected process is being explored. At product level, statistics are captured including effort commitment, meeting of scheduled project activity. Product defects are all recorded in the project quality records. Key SEAL project practices are briefly reviewed. 4.1 Project communication Project communication depends heavily upon the use of the Internet for communication and document distribution. The SEAL File Server is the repository of the project documents and records. All project members have username and password access to the management products, technical products and quality records supported on the SEAL File Server. Selected SEAL products are publicly available and are accessible using anonymous ftp access. E-mail lists supported using the list-server facility are maintained to support exchange of information and ideas between project group members, inside and outside of the University. 4.2 Document management and control The 4.5 document control requirements are comprehensively applied to all documents recorded and emanating from the project. Briefly, the impact of such compliance includes: a. Revision control is applied to all documents. b. All documents and records are numbered and recorded in the project Master Document List (MDL). Documents are issued by placing them on the SEAL File Server and then issuing an e-mail Document Issue Notice which advises members of the Document Name, Revision, Number and file path. Each individual is QS16030.010 Revision 0.10 08 January 1997 Page 4
responsible for downloading the updated documents using ftp. 4.3 Software configuration management All project artefacts are supported by a Configuration Management Plan, which describes the: a. Document naming conventions b. Record naming conventions c. Description of project directory structure d. Archiving arrangements These conventions are defined at the start of the project and have ensured a common understanding of the project artefacts as the project evolves. 4.4 Project tracking and control The Project Management Plan supports details of the following: a. Project dependencies b. Human resources required c. Human resources available d. Team roles and responsibilities e. Hardware and software required f. Any training needs g. The Work breakdown Structure (WBS), obligations and schedule, and log of completed tasks Items a) - f) are assembled at the start of the project and are largely static. The WBS is actively managed as the project evolves. 4.5 Project reviews These are undertaken in the context of formal meetings of the core group members. The purpose of the meeting is to review the status of the project and to plan in detail the next phase of product development. 5. Discussion An assessment of our documented quality system was undertaken by the South African Bureau of Standards (SABS) on 16 February 1995. Certain nonconformances were identified and were subsequently cleared during April. Having met the requirements of the International Standard, the award of the certification took place on 19 July 1995. The award of the certification was highly significant for the SEAL, as it is believed that the SEAL is the first academic enterprise internationally to have met the requirements of for the development of software. 5.1 Significant achievements Some of the consequences of this commitment to formal quality management in the SEAL include: a major reduction of wasted effort in the management and conduct of SEAL teaching and project activity. provision of visible evidence to local industry of the SEAL commitment to apply current good international practices to its own project and course activities. provision of a disciplined framework for research, education and training into software quality systems management. provision of an open environment for industry to see at first hand the workings of a software quality management system, noting that most company quality management systems are highly confidential and not open to detailed inspection by interested outsiders. 5.2 Education and training The SEAL QMS has had a major impact on the quality of student research project activity, in the following ways: there is a consistent framework for managing projects, which once the issues are understood, has been accepted by the students as a positive and valuable contribution to getting work successfully done and providing experience which is perceived to be valuable in the wider commercial environment. project practices such as keeping minutes of meetings with project supervisors and being subjected to project audits took a while to get used to, but such activities are now seen as positive practices with added value there has been a marked appreciation for the extensive template for the cradle to grave project experience, recognising that templates encapsulate a wealth of experience and once understood, make the job easier. the SEAL QMS has made it possible to reconcile two opposing needs in developing software project management skills: on the one hand to develop software as a team experience, and on the other hand is the need to assessment individual performance. The highly instrumented SEAL QMS project process has made it possible to achieve this end consistently and meet both requirements, to the benefit of the student. 5.3 SEAL interaction with local industry The certification of the SEAL to requirements has not had an immediate impact on the local community - except amongst those companies already certified to QS16030.010 Revision 0.10 08 January 1997 Page 5
. Amongst such enterprises it has been possible to develop close working relationships out of a common concern for software process improvement in the local community and to try and improve the competitiveness of the South African software industry. The SEAL has pioneered electronic support for quality management in the local community, and the SEAL File server, taken together with ease of access to the Internet has made collaborative project effort possible, with national and international dimensions. This would not have been possible without the discipline of the ISO 9001 QMS. 5.4 Impact of SEAL QMS on university stakeholders Perhaps not surprisingly, the certification of the SEAL to has had limited impact on the Department of Electrical Engineering, and almost nil impact beyond that. There are a number of reasons perceived for this: there is little awareness of quality issues in terms of reduction of waste and improvement of productivity. While there are ongoing calls and initiatives to reduce wastage in the University, these are piecemeal efforts and are uncoordinated, having no long-term impact. Universities are home to individuals with high level skills in narrow domains - which too easily become barriers that restrict communication and the flow of experience across disciplines. sadly the university environment can easily lead to intellectual arrogance and disdain of that which has to do real-world engagement. While on one hand it is accepted that to teach quality management is part of education, but giving effect to the same discipline is viewed as training - and hence not a serious subject in a traditional university. a quality practitioner has to tread carefully since mistakes are all too easily seized upon as evidence that quality management does not work. The only approach in this situation is to keep at it and rather let the results, when they become evident, speak for themselves, and in so doing create an environment receptive to positive change. which do not contribute to improved productivity and elimination of wasted effort. 6. References [1] (1994) Quality Systems - Model for Quality Assurance in Design\Development, Production, Installation and Servicing [2] ISO\IEC 15504-2. Software Process Assessment Part 2: A Reference Model For Processes And Process Capability Date: PDTR, October 1996 7. Author Contact Details Alastair Walker is presently an associate professor in the Department of Electrical Engineering, University of the Witwatersrand. He was responsible for establishing the Software Engineering Applications Laboratory in 1988. Since that time, the SEAL has become known as the leading provider of advanced professional courses on topics in Software Engineering, particularly in the areas of application of object-oriented design methodology in industrial applications and in software quality systems management. He is a Certified Quality Analyst, and a Certified Software Quality Systems Auditor. He is the leader of the SABS work group involved in the development of the new international standard for software process assessment He is the South African representative on the ISO\IEC\JTC1\SC7 Advisory Board. Contact details: Professor AJ Walker, Software Engineering Applications Laboratory, Department of Electrical Engineering, University of the Witwatersrand, Private Bag 3,P O Wits, 2050 Johannesburg. Office Phone: +27-11-716-5469, Mobile: +27-82-453-0933; Fax: +27-11-403-1929, Internet E-mail: walker@odie.ee.wits.ac.za ❶ 5.5 Ongoing improvement of the quality system It would be misleading to give the impression that the SEAL has eliminated all significant quality problems. A major concern is the fact that the support of the SEAL falls heavily on the shoulders of the full-time staff members, and the Director in particular. With most universities in South Africa subject to severe financial stringency there is little opportunity to take on additional full-time members to share the burden of the QMS support. From a different perspective though, this has proved to be a valuable ongoing discipline in that the QMS is subject to continuous scrutiny to eliminate activities QS16030.010 Revision 0.10 08 January 1997 Page 6