Software Process Improvement via IS0 9000 ... - Semantic Scholar

Proceedings of the 29th Annual Hawaii International Conferenceon SystemSciences- 1996

Software Process Improvement via IS0 9000? Results of two surveys among European software houses Dirk Stelzer, Werner Mellis, Georg Herzwurm University of Cologne, Chair of Business Computing, 50923 Cologne, Germany

Abstract

1 Introduction

Much has been said andpublished about the advantages and disadvantages of the IS0 9000 family for the sofMare industry. However, there is very little empirical evidence to what extent software process improvementshave really been achieved via IS0 9000. This paper presents results of two surveys among European software houses. One of the targets of the studies was to find out whether software companies have actually achieved software process improvementsvia IS0 9000, The first survey was conducted among twenty German software houses that have received an IS0 9001 certificate. The study focuses on five elements of an IS0 9000 quality system: code reviews and inspections, software testing, product and process measurements,measurementof quality costs, and demonstration of quality improvements. Many software houses included in our first survey have not carried out any modfications of the Jive elements. Thus, it seemed that IS0 9000 had not led to significant improvements.Nevertheless,nearly 100% of the companies would decide in favor of implementing an IS0 9000 quality systemonce again. We decided to conduct a second study to gain a better understanding of the software process improvements achieved via IS0 9000, We analyzed experiencereports and conducted interviews with quality managersffom a total of 36 European software houses. It turned out that software houses have indeed achieved process improvements. Unfortunately, most sofnvare housesare not able to quantt$ the bene$ts of implementing an IS0 9000 quality system. We identtfted ten key success factors that the respondentsof our studies considered to be the most helpful when implementing an IS0 9000 quality system. Astonishingly, only two of the ten factors are explicit requirements of IS0 9001. This shows that it is necessaryto implement a more comprehensive approach to achieve substantial software process improvements.

In Europe, the IS0 9000 family is one of the most important standards in the field of quality management. Many software suppliers try to accomplish software process improvements by implementing a quality system based on the IS0 9000 standards.Most of these software housestry to display their achievements by obtaining an IS0 9001 certificate. Much has been said and published about the advantagesand disadvantagesof the IS0 9000 family for the software industry. However, there is very little empirical evidence to what extent sottware process improvements have really been achieved via IS0 9000. This paper presents results of two surveys among European so&ware houses. One of the targets of the studies was to find out whether software companieshave actually achieved software process improvementsvia IS0 9000.

2 Software process improvement and IS0 9000 IS0 9000-l defines a process as “a set of interrelated resourcesand activities which transform inputs into outputs. ... Resources may include personnel, finance, facilities, equipment, techniquesand methods”[ 11.Correspondingly, a software process can be defined as “a set of activities, methods, practices, and transformations that people use to develop and maintain software and the associatedproducts (e.g., project plans, design documents, code, test cases,and user manuals)”(Paulk, Curtis, Chrissis, and Weber [2]). The term ‘software process improvement’ denotes the “changes implemented to a software process that bring about improvements” (Olson, Humphrey, and Kitson [3]). The intent of software process improvement is (Paulk, Weber, Garcia, Chrissis, and Bush [4]): l improving software product quality, 9 increasing productivity, l and decreasingthe cycle time for product development.

703 1060-3425/96$5.00 0 1996 IEEE

Proceedings of the 1996 Hawaii International Conference on System Sciences (HICSS-29) 1060-3425/96 $10.00 © 1996 IEEE

Proceedings of the 29th Annual Hawaii International Conferenceon SystemSciences- 1996 In other words, software process improvement aims at achieving competitive advantage for software houses. Worldwide, there are various approaches to implementing software process improvement. North American companies seem to prefer the Capability Maturity Model (Pat& Curtis, Chrissis, and Weber [2]). Top level Japanese companies mainly apply Total Quality Management (Matsubara [5]). European firms tend to use the IS0 9000 family to enhance their capabilities. Whereas the Capability Maturity Model is designed exclusively for software processes,Total Quality Management and the IS0 9000 standardsare not limited to software production. The IS0 9000 family is a set of internationally recognized standardson quality management.The standards are intended “to provide a generic core of quality system standards applicable to a broad range of industry and economic sectors” [l]. A major purpose of quality management is “to improve the systems and processes so that continual improvement of quality can be achieved”. “The standards describe what elements quality systems should encompass but not how a specific organization should implement these elements”[l]. The IS0 9000 family is based on the assumptionthat all work is accomplishedby a process. Accordingly, quality management means managing all processesin an organization. Figure 1 gives an overview of the IS0 9000 standards.

requirements is called certification or registration. IS0 9000-l to 9000-4 [9 and lo] are guidelines for the selection and application of IS0 9001, 9002, and 9003. IS0 9000-3 Ill], for example, is a guideline for the application of IS0 9001 to the development, supply and maintenance of software. Apart from the standards shown in figure 1, the IS0 9000 family also comprises the standards IS0 8402, and IS0 10001 to 10020. The IS0 9000 family is mainly used in two different situations: l It provides guidance for quality management, i.e. for implementing a quality system,and l it is a framework for third-party certification or registration. The authors of the IS0 9000 family strongly recommend taking the following route: A company should first use IS0 9004-l and other applicable parts of IS0 9004 to design and to implement a quality system. Once the quality system has been installed, the company may use the quality assurance models of IS0 9001, IS0 9002, or IS0 9003 to demonstrate the adequacy of the quality system. Obviously, a certification on the basis of IS0 9001, 9002, or 9003 is the most effective way to conduct this demonstration. Software suppliers usually choose the IS0 9001 as a basis for certification.

3 Scope and objectives of the first survey

Production,

Application

of IS0

Installation,

Final Inspection

II

1

Figure I. Overview of the IS0 9000 standard. The IS0 9004 series comprises standardsfor internal use by an organization. They provide guidance for the design and implementation of a quality system. A quality system is a set of “organizational structures,procedures,processesand resources needed to implement quality management”[l]. IS0 9001, 9002 and 9003 [6, 7, and 81 are models for external quality assurance. They specify a set of requirements. The demonstration of conformance to these 704


From June to September 1994, we conducted a survey among the first twenty German software houses that have received an IS0 9001 certificate. We identified quality managers of the companies on the telephone and asked whether they were willing to participate in the survey. All companies agreed and received a questionnaire that consisted of 36 questions. Eighteen questionnaires were returned. We analyzed the answers, prepared a report, and sent it back to the respondents.We asked them to comment on our results in order to avoid misinterpretations. However, not a single statementof the report was criticized. The main businessobjective of all companies included in the survey is software development. Some companies produce standard software packages, others mainly provide individual applications for clients, Most of the companies also provide support, maintenance, training, consultancy services,etc. Five companieshad received the certificate less than six months before the survey was conducted. Thirteen software houses had been certified for more than half a year. The average period between certification and our survey was fourteen months. Ten companieshave received an IS0 900 1 certificate for the quality system of the entire company. In eight companies the certificate is limited to the software development division. In these companies other business areas, for

Proceedings of the 29th Annual Hawaii International Conferenceon SystemSciences- 1996 example consultancy services or the training division, are not covered by the certificate. The number of employees in the certified divisions reaches from 15 to 900 persons, with an average of 221 persons. Figure 2 shows the size of the certified divisions included in our study.

l

z 50 staff

50-25ostaff

> 250 stai? I

by clause 6 of IS0 9004-l. (“It is important that the effectiveness of a quality system be measured in financial terms” [9]) Quality costs are divided into prevention, appraisal and failure costs. IS0 9004-I describesthree approachesto gather, analyze and present quality-related costs. However, cost of quality measurementis neither required by IS0 9001, nor by IS0 9000-3. Demonstration of quality improvements Demonstration and documentation of quality improvements are recommended by clauses 4 and 5 of IS0 9000-I (,, ... the management of an organization should ensurethat the (quality) system will facilitate and promote continuous quality improvement”. “Appropriate documentation is essential for ... quality improvement (and for) maintaining the improvements”[I]) IS0 90044 consists of guidelines for quality improvements. However, demonstration of quality improvements is not an explicit requirement of IS0 900 1, nor of IS0 9000-3.

I

Figure 2. Size of the certl$ed divisions. One of the targets of the survey was to find out whether software houses have actually achieved improvements. The survey concentrated on five elements of IS0 9000 quality systems: l Code reviews and inspections Code reviews and inspections are suggestedby clause 12 of IS0 9004-I. (“Verification, typically by inspectionsor tests, should be considered at appropriate points in the process to verify conformity” [9]) They are also recommended by clause 4.10 of IS0 9001 and 5.6 of IS0 9000-3. l Software testing Software testing is a central element of an IS0 9000 quality system. It is suggestedby clause 12 of IS0 90041 (product verification), by clause 4.10 of IS0 9001 (“The supplier shall establish and maintain documented procedures for inspections and testing activities in order to verify that the specified requirements for the product are met” [6]) and by clauses 5.7 (testing and validation) and 5.8 (acceptancetest) of IS0 9000-3. l Product and processmeasurements Product and process measurements are suggested by clause 20 of IS0 9004-l (use of statistical methods), by clause 4.20 of IS0 9001 and 6.4 of IS0 9000-3. (“Metrics should be reported and used to manage the development and delivery process and should be relevant to the particular software product” [ 1 I]) l Measurementof quality costs Measurement of quality costs is suggestedby clause 4.3 of IS0 9004-l (“The calculation and evaluation of costs associated with all quality elements and objectives should always be an important consideration ...‘I [9]) and 705


Thus, we have included three elementsthat are required or suggestedby IS0 9001 and two other elements that are suggested by other pm of IS0 9000, but that are not explicitly mentioned in IS0 9001. We wanted to find out whether companies make use of the entire IS0 9000 family or whether they focus on IS0 900 1. The results of the survey among the twenty German software suppliers are described in chapter 5 of this paper. Analyzing the results, we realized that some interesting questions concerning software process improvement were still unanswered.We decided to conduct a second study to gain a better understanding of the software process improvements achieved via IS0 9000. The results of the second survey are discussedin chapter 6 of this paper.

4 Reasons for implementing an IS0 9000 quality system The authors of the IS0 9000 family mention two different motivations for implementing an IS0 9000 quality system: l The stakeholder-motivatedapproach The company implements a quality system in responseto demandsby customersor other stakeholders. l The management-motivatedapproach The company’s own management initiates the effort to install a quality system. The management-motivated approach corresponds to software process improvement initiatives. It usually aims at significant increasesin productivity and product quality as well as reductions in costs and time to market. The authors of IS0 9000-I state that the “management-motivated approachwill normally be more comprehensiveand fiuittil than the model used for demonstrating the adequacy of the

Proceedings of the 29th Annual Hawaii International Conferenceon SystemSciences- I996 quality system” [l] (i.e., the stakeholder-motivated approach). Most companies included in our study gave more than one reason for the implementation and certification of their quality system. Some software houses are urged to obtain an IS0 9000 certificate by their clients. Others suppose that a certificate would be a strong marketing argument. At the same time, most companies realize that they should improve their processes.In other words, most so&are houses follow a combination of the ‘stakeholder-’ and the ‘managementmotivated approach’. Figure 3 shows which approaches were used in how many companies.

expected by the twelve companies that followed the management-motivatedapproach.

Figure 4. A closer look at expectedprocess improvements (including multiple answers).

Figure 3. Reasonsfor implementing a qualify system (including multiple answers). More than 90% of the software houses started to implement a quality system becausethey assumethat an IS0 9000 quality system will be a barrier to entry into the software and services market in Europe in the near future. They regard an IS0 9001 certificate as a prerequisite for increasing sales figures. We have subsumed these reasons under the heading ‘stakeholder-motivated’ approach. Only one company in our survey did not explicitly mention this reason. Two thirds of the companies included in our study follow the management-motivated approach. They implement the quality system to improve their internal organization. IS0 9000 is regarded as a route to software process improvement and to achieving competitive advantage. Five companies mentioned other reasons that could not be classified as ‘stakeholder-’ or ‘management-motivated’. For example, one company plans to offer consulting services in quality management. Another respondent prefers an independent assessmentof the quality system. We conducted a more detailed analysis of twelve companies that followed the management-motivated approach. Fifty percent of the twelve companies expected clearly defined, transparent processes. Fifty percent have implemented a quality system in order to improve product quality. Approximately a third of the companies tried to reduce costs or to increase productivity. Only one company expected a significant reduction of the time needed to develop software. Figure 4 shows the process improvements

706


The implementation of an IS0 9000 quality system does not necessarily require a certification. However, most German sofhvare houses are seeking certification when they have installed a quality system. Respondents of our study mention two reasons: 1. Most quality managers prefer an assessment by an independent organization, once a quality system is installed. This assessment helps to demonstrate the success of the quality system to the company’s management. 2. Many customers ask for a certificate or for any other proof of the supplier’s quality achievements. An IS0 9001 certificate is the most recognized way to meet this requirement.

5 Software process improvements achieved via IS0 9000 We asked whether the quality systems had come up to the respondents’expectations.Figure 5 shows the answers.

J Figure 5. Overall assessmentof quality systems. Forty-seven percent of the respondents said that their expectations were met. Fifty-three percent of the respondents answered that the quality system had only

Proceedings of the 29th Annual Hawaii International Conferenceon SystemSciences- 1996 partially come up to their expectations. Not a single respondent said that the implementation of the quality systemhad ended in failure. We tried to find out why the companies thought the installation of the IS0 9000 quality systems was more or less successful. Our first assumption was that the software houses would have achieved significant improvements in conducting code reviews and inspections, in software testing, in applying software metrics, in accounting quality costs or in demonstrating quality improvements. We tried to verify this assumption. We asked whether and how the software houses apply the five elements.It is not possible to describe all answersin detail, Figure 6 shows the percentageof software housesthat have implemented the five quality systemelements.

the results of a more detailed analysis of the modifications.

Figure 7. Modification of theBve quality systemelements.

Figure 6. Implementation of thefive quality system elements. All companies regularly conduct code reviews and inspections as well as software tests. Eleven percent do neither measureproduct quality, nor processquality. A more detailed analysis of this figure shows that 17% of the companies do not conduct process measurementsregularly. Twenty-eight percent do not collect and analyze product metrics. Twenty-two percent have not established any measures to prove quality improvements. Thirty-three percent of the companies do not have an accounting system to control quality costs. It is interesting that the two elements that are not explicitly required by IS0 9001 (i.e., demonstration of quality improvements and measurementof quality costs) are less prevalent than the other three elements. This finding indicates that some companies focus on IS0 9001 only, and do not make use of the entire IS0 9000 family. Presumably this is due to the fact that the respective software houses have focused on becoming certified to IS0 900 1, and not on improving their processesin the first place. This paper describessoftware processimprovementsthat have been achieved with the help of the IS0 9000 family. Any improvement does necessarily require a modification. Forty percent of the software houses did not change any of the five quality system elements considered in our study. In other words, these companies have not achieved any improvements in any of the five elements. Figure 7 shows

Only 22% of the companies have modified their testing techniques during the implementation of the IS0 9000 quality system. Twenty-eight percent have adopted new measuresto analyze quality costs. Thirty-three percent of the software houses have modified their approach to conduct code reviews and inspections. Thirty-nine percent have modified the measures taken to analyze quality improvements, product metrics and process metrics. Many software houses, however, have not carried out any modifications of the elementswe considered in our study. Thus, it seemedthat IS0 9000 had not led to substantial improvements in many sotiare houses. This result is in sharp contrast to our findings discussedat the beginning of chapter 5. Forty-seven percent of the respondents said that their expectationswere met, and 53% said their expectations were met partially. Consequently, there must have been some kind of improvement. Obviously, the results of our study did not reflect the value of IS0 9000 for software houses correctly. Therefore, we decided to take a closer look at software process improvementsachieved via IS0 9000.

6 A closer look at software process improvements achieved via IS0 9000 Besides our first survey, we conducted various interviews with representatives of companies who had recently sought certification. Furthermore, we examined progress reports and case histories of software houses that had implemented an IS0 9000 quality system. This second study was carried out between October 1994 and May 1995. The results are based on interviews and reports from a total of 36 European software houses. The study also draws on interviews with IS0 9000 auditors and consultants working in the software industry, and on the authors’ experiences gathered by consulting software houses that are currently implementing a quality system and preparing for IS0 9001 certification. The objective of this second study was to answer the following questions: l Are IS0 9000 quality systemssuccessful? l What are the key successfactors?

707


Proceedings of the 29th Annual Hawaii International Conference on System Sciences- 1996 l

Can quality managers quantify the success of the IS0 9000 quality systems?

The objective of the second study was to gain deeper insights into the essenceof IS0 9000 quality systems, and to understand whether and how companies have achieved software process improvements via IS0 9000. Therefore, it is a qualitative study and does not present statistical material. 6.1 Are IS0 9000 quality systems successful? We asked quality managers - and also top executives of software houses - to assessthe value of the quality system. Our interview partners even passed a more favorable judgment on IS0 9000 quality systems than the respondents of our first survey. Nearly 100% of the company representatives would decide in favor of implementing an IS0 9000 quality system once again. They are convinced that the benefits of installing a quality system exceed the costs. 6.2 What are the key success factors? We asked how exactly the companies have achieved improvements during the implementation of the IS0 9000 quality system. Our aim was to identify the most helpful elements of IS0 9000 for software process improvement. We identified ten key success factors. The term “success factor” describes the aspects that the respondents of our study considered to be the most helpI when implementing an IS0 9000 quality system.

a predictable and stable software process when they started to implement a quality system. Managers usually had a general idea of the software process, but they did not understand any details. Even in small companies employees usually did not know how colleagues fulfilled their tasks. The software houses suffered from a lack of organizational structures and clearly defined procedures. The consequence of these deficits was a waste of time, money and effort. In many software houses the design of the IS0 9000 quality system was the first opportunity to discuss details of software development in the context of business processes. Usually, it turned out that a company had numerous different procedures, e.g. for software testing. Often, every single software developer had invented his own procedures, methods and tools. In most cases the difference in the way software was designed, developed, tested, and maintained could not be justified by technical or business reasons. It seemed that most so&are developers had regarded their approach to software development as a private affair. Thus, the description of the status quo was a difficult task. Interestingly enough, only very few staff members had realized the vast dimensions of the weaknesses before the IS0 9000 project was started. Once the status quo was defined and documented, everyone recognized that the way the company had operated before was extremely inefficient. Weaknesses became obvious to everyone involved in the process. Since the deficiencies were made explicit, it was now possible to suggest improvement initiatives and to discuss necessary actions. Employees gained a better understanding of their contribution to the company’s business success. Hence, they were able to submit helptil proposals how to improve the software process. 2 Identification of best practices. When software developers described how they perform their daily work it often became obvious that some employees were wellversed in specific tasks (e.g., software tests, configuration management, code inspections, etc.). These outstanding procedures and methods were identified as best practices. Other colleagues imitated them at once. Some IS0 9000 teams documented best practices and made them part of the internal training scheme. 3 Identification of business processes. Most software houses do not only develop software; they also provide

Figure 8. Key successfactors of sofiware process improvement via IS0 9000.

consultancy

services,

hotline

SUPPOG

training

and

maintenance. However, only very few software companies have clearly defined these business processes. Some soflware suppliers, for example, have not clarified the difference between training and hotline support. Consequently, areas of responsibility are not accurately determined and most processesare organized inefficiently. Some companies used the IS0 9000 project as a chance for the identification of business processes. First, they described and analyzed their key business processes.

The sequence of the ten factors in figure 8 does not indicate their importance. We will now discuss the ten factors in detail. 1 Definition and documentation of the status quo. The implementation of the quality system has helped to define and to document procedures. Many companies did not have

708


Proceedings of the 29th Annual Hawaii International Conferenceon SystemSciences- 1996 Secondly, they analyzed strengths and weaknessesof the current processes.Thirdly, they elaboratedbetter procedures to perform specific tasks, improved business process interfaces, and clarified areas of responsibility. These improvements were documented and defined as standards. Thus, the IS0 9000 initiative helped to make the business processesmore efficient.

really interested in improvements they made valuable proposals and committed themselves to the IS0 9000 project. Many quality managerssaid that they were surprised by the positive response the IS0 9000 initiative created among employees. Once the quality system was installed, the certification process became a driving force. Most quality managers realized that the IS0 9001 certificate is a strong incentive. The certification process is comparable to a sports competition. Everyone knows the enormous efforts that have been made. All staff members have noticed that top managers are eager to succeed in the certification process. Accordingly, everyone is highly motivated.

4 Simplification of routine procedures. The implementation of the quality system has helped to standardize and to simplify routine procedures. Before the IS0 9000 project was started, some companies had large losses because of missing guidelines for preparing an offer or submitting a tender. Other software houses did not have defined procedures, check lists and tools for software testing. Some small software houses did not even have standardized templates for writing letters, reports, invoices or software documentation. Thus, even routine procedures required intensive efforts. Nevertheless, the results were often incorrect. During the implementation of the IS0 9000 quality systems the employees prepared guidelines, check lists, templates and tools to facilitate routine procedures. Often this was not a very difficult task. Some staff members had already used their individual instruments before the quality system was designed.The IS0 9000 initiative helped to collect all the material, and to analyze, improve, document and distribute it to all employees. 5 Internal audits. A software supplier “shall establish and maintain ... internal quality audits to verify whether quality activities and related results comply with planned arrangements and to determine the effectiveness of the quality system” [l]. Our interview partners considered the internal quality audits to be one of the most effective elements of IS0 9001. The audits are often used as an internal consultancy service. Accordingly, most employees do not consider the audits an instrument of internal control or supervision. On the contrary, they are happy to discuss details of their daily work with the auditors. During the audits they have the chance to reflect on their contributions from a general business perspective. The audits provide an opportunity to discuss strengths and weaknesses of the quality system, to submit proposals for improving the system, or to complain about disadvantagesand drawbacks. Internal quality audits are a helpfid activity to support a continuous improvement initiative. 6 Impetus and incentive. In most companiesemployees understood the deficiencies in the current processes.They noticed that many practices were inadequateand inefficient. Some software developers really suffered from these deficiencies. However, they never had a chance to express their dissatisfaction. The design and implementation of an IS0 9000 quality system was an enormous impetus for most staff members. Once they realized that management was

7 Team spirit. The implementation of an IS0 9000 quality system usually affects the entire company. The IS0 9000 project cannot be accomplished by a few persons. It requires a joint effort of management and all employees. Inevitably, all divisions and departments have to define common objectives, agree upon documentation standards, and discuss the status quo of sohare development and service provision. It requires interdisciplinary workshops and teamwork. The collective effort to design and implement the quality system and the will to succeedin the certification process usually creates a team spirit among all employees. This helps to improve cooperation, teamwork, and the working climate in most software houses. This effect is still advantageousafter the certificate has been granted. 8 Workshops and regular meetings. Many software houses have introduced regular meetings and workshops during the implementation of the quality system. The meetings are held to discuss problems of the software process.Quality managersand software developers consider these meetings to be a very efficient way to present new ideas for process improvements, to discuss options, to analyze costs and benefits and to communicate the results to all relevant persons. Of course, meetings had also been held before the IS0 9000 initiative was started, but these meetings were often arranged to discuss pressing problems in projects. Regular meetings to analyze procedures and practices were very infrequent. 9 Definition of a common language. Before the IS0 9000 initiative was started most companies suffered from a lack of teamwork and of inter-departmental cooperation. Consequently, different teams spoke different languages. Expressions and terms that were well understood in one team were confused by other teams. When two or more teams had to cooperate they usually had to resolve misunderstandingsbefore they were able to work efficiently. The design of an IS0 9000 quality system forces the teams to define a common language. This common language facilitates communication in all projects.

709


Proceedings of the 29th Annual Hawaii International Conferenceon SystemSciences- 1996 10 Customer perception surveys. Most companieshave introduced formal customer perception surveys. The objective of the surveys is to analyze how customers perceive the products and services provided by the company. The survey results attract close attention among staff members and management. Most software developers feel that for the first time they now receive an unbiased and unprejudiced response to their work. Therefore, customer perception surveys are not only an essential tool for market researchbut also for improving job satisfaction.

l

l

Summary of key success factors. Astonishingly, only two of the ten key successfactors identified in our study are explicit requirements of IS0 9001: definition and documentation of the status quo, and internal audits. Eight other factors are not mentioned in IS0 9001. A quality manager of a software house who would only apply the elements of IS0 9001 would probably ignore other important successfactors. This shows that it is necessaryto implement a more comprehensive approach to achieve substantial software process improvements. It seems, however, that most companies included in our study have realized this weakness and have elaborated more comprehensivequality systems. An interesting question is whether the eight factors that are not explicitly mentioned in IS0 9001 would have been implemented anyway. Without the IS0 9000 initiative the eight factors would probably not have been established at this time. The respondents of our study mentioned the factors when they were asked for the most helpful aspects when implementing an IS0 9000 quality system. Therefore, we suppose that the implementation of the IS0 9000 quality system was the impetus for implementing these factors. 6.3 Can quality managers quantify the successof the IS0 9000 quality systems? Nearly all quality managers assume that the implementation of the IS0 9000 quality system was successful. Some companies report fewer customer complaints, earlier detection of software faults and interest savings by quicker presentation of invoices. However, only very few managers are able to quantifjr the benefits of the quality system. They have not collected data, and they have not produced documented records of success. Most respondents are not sure to what extent the implementation of the IS0 9000 quality system has led to a reduction of cycle time, working effort or product failures. As mentioned before, 22% of the companies included in our first survey do not establish any records in proof of quality improvements, 28% do not conduct product measurements, and 33% do not measure quality costs, A more detailed analysis into the answers of the software houses shows that 0 only a third of all companies can support their opinion 710


about the success of the quality system with facts and figures, a third of all companies say that it is too early to come to a balanced conclusion because the quality system has only been operating for a short time, and a third of the companies will probably never be able to quantify the success of the quality system because they do not conduct comprehensive measurementsof product quality, cycle time or costs.

Nevertheless, managers are convinced that the implementation of the IS0 9000 quality systems has been successfid. As mentioned before, the results of our first study seemed to indicate that IS0 9000 had not led to any improvements. However, this impression was not correct. When we took a closer look at organizational changes in software houses brought about by IS0 9000, it turned out that they had indeed achieved substantial improvements. As the changes did only marginally affect the five elements considered in our fust study we did not perceive them directly. Unfortunately, most software houses are not able to quantify the benefits of implementing an IS0 9000 quality system.

7 Cost of implementing an IS0 9000 quality system The average time needed to implement an IS0 9000 quality system is 1.5 years. Software houses that have already installed a quality system before the IS0 9000 initiative is started need approximately one year to adapt it to the requirements of IS0 9000. Companies that have to design a quality system ‘from scratch’ usually need two years or more. German software houses calculate DM 300,000 to 500,000 for the implementation and certification of a quality system (DM 1 = approx. US$ 0.70). This sum includes expenses for external consulting and certification audits (DM 20,000 to 50,000). The quality managers need 50% to 100% of their working hours to design the quality system. All managers and most software developers must write reports, review documents and attend workshops. The objective of the workshops is to discuss organizational changes and options for software process improvements. The meetings cause the largest proportion of the expenses for the IS0 9000 quality system. Furthermore, all employees have to be trained. Although one might h&e expected that a quality system in a big company is more expensive than in a small one, the costs of implementing and certifying a quality system do not correlate with the size of the company. It seems that the organizational structures of larger companies are usually better documented and more clearly defined. This makes it much easier to install an IS0 9000 quality system.

Proceedings of the 29th Annual Hawaii International Conference on System Sciences -

8 Significance of IS0 9001, IS0 90003, IS0 9004-1, and IS0 9004-4 As already mentioned before, the authors of IS0 9000 recommend making use of applicable parts of IS0 9004 to design and to implement a quality system. The quality systemmay later be certified on the basis of IS0 900 1. IS0 9000-3 provides guidelines for the application of IS0 9001 to the development, supply and maintenanceof software. However, most software houseshave not made intensive use of IS0 9004-l and IS0 9004-4. They gather information directly from IS0 9001. Consequently, software houses do not apply the suggestionsof IS0 9004. For example,they do not follow the advice to control cost of quality, to conduct continuous quality improvement and to implement a metrics program to analyze the correlation of process and product quality. Surprisingly, also IS0 9000-3 is not of great help for software houses. Quality managers only read it when they begin to study the IS0 9000 family. When they realize that IS0 9000-3 is as difficult to read as all other parts of IS0 9000, they ignore it and use IS0 900 1 instead.

9 Conclusions Interviews with managers of software companies to assess the value of IS0 9000 for software process improvement draw a very favorable judgment on IS0 9000 quality systems. Nearly 100% of the company representativeswould decide in favor of implementing an IS0 9000 quality system once again. They are convinced that the benefits of installing a quality system exceed the costs. Nevertheless, the favorable judgment must be modified in severalrespects: Software companies that had already established stable processesbefore the implementation of an IS0 9000 quality system have not achieved software process improvements via IS0 9000. They only documented the status quo and sought certification. The majority of the software houses, however, have accomplished substantial improvements during the implementation of the IS0 9000 quality system. They established stable processes, explicit organizational structures, and software engineering standards and proceduresfor the first time during the IS0 9000 initiative. IS0 9001 identifies only minimal requirements for a quality system. When a software company aims at achieving software process improvements, it is necessary to address more aspects. We have identified ten characteristics that successtil IS0 9000 projects have in common. IS0 9004-l and IS0 9004-4 provide guidelines for continuous process improvements.Unfortunately, only very few companieshave applied IS0 9004. Most software suppliers focus on IS0 900 I, instead. Consequently,they cannot take full advantage of the IS0 9000 family. The results of our surveys indicate that the effects and

1996

benefits of IS0 9000 quality systemsare different from what one might have expected.The most important effects of IS0 9000 for software process improvement are only marginally reflected by modifications of IS0 9000 quality elements, e.g. code reviews and inspections, software testing, product and processmeasurements,measurementof quality costs, or demonstrationof quality improvements. Other factors are responsiblefor the managers’favorable judgment about IS0 9000. The implementation of IS0 9000 quality systems helped to overcome the lack of organizational structures and clearly defined procedures. It helped to identify weaknessesand instable processes.During the IS0 9000 projects employeessubmitted various options to standardizeand to improve procedures and practices. Top management’scommitment to success in the certification process had a positive effect on staff motivation. Most companies introduced formal customer perception surveys. The survey results gave developers an unbiased and unprejudiced responseto their work for the first time. Job satisfaction was also improved by internal quality audits. Usually audits are not used to supervise staff members, but to provide coaching and to discussstrengths and weaknesses of the quality system. Regular meetings specifically devoted to analyze proceduresand practices also had positive effects. Finally, the IS0 9000 projects helped to prepare guidelines, check lists, templates and tools to facilitate routine procedures.Thus, IS0 9000 helped to save time, money and effort. Unfortunately, most software houses are not able to quantify the benefits of implementing an IS0 9000 quality system. It seemsthat it is not the technical contents of the IS0 9000 family that makes it specifically appropriate for software process improvement. The culture created by a company-wide improvement program seems to be more important. Implementing an IS0 9000 quality system is a helpful impetus for a cultural change. In this paper we describedthe ten key successfactors, which the respondents of our studies considered to be the most helpful for achieving software processimprovement via IS0 9000.

References Ill

ISO, “Quality management and quality assurancestandards, Part 1: Guidelinesfor selectionand use”,IS0 9000-l: 1994, Geneva,1994

PI M. Paulk,B. Curtis, M. Chrissis,and C. Weber,“Capability Maturity Model for Software,Version 1.1‘I, TechnicalReport CMU/SEI-93-TR-024, Pittsburgh,1993 r31 T. Olson, W. Humphrey,and D. Kitson, “ConductingSEIAssistedSoftwareProcessAssessments”, Technical Report CMU/SEI-89-TR-7, Pittsburgh,1989

711


Proceedings of the 29th Annual Hawaii International Conferenceon SystemSciences- 1996 [8] ISO, “Quality systems. Model for quality assurance in final inspection and test”, IS0 9003: 1994, Geneva, 1994

[4] M. Paulk, C. Weber, S. Garcia, M. Chrissis, and M. Bush, “Key Practices of the Capability Maturity Model, Version 1. 1I’, Technical Report CMU/SEI-93-TR-025, Pittsburgh, 1993

[9] ISO, “Quality management and quality system elements. Part 1: Guidelines”, IS0 9004-l : 1994, Geneva, 1994

[5] T. Matsubara, “Does IS0 9000 really help improve Software Quality?“, American Programmer, pp. 38-45, No. 2, 1994 [6] ISO, “Quality systems. Model for quality assurancein design, development, production, installation and servicing”, IS0 9001: 1994, Geneva, 1994 [7] ISO, “Quality systems. Model for quality assurance in production, installation and servicing”, IS0 9002: 1994, Geneva, 1994

712


[lo] ISO, “Quality management and quality system elements. Part 4: Guidelines for quality improvement”, IS0 9004-4: 1993, Geneva, 1993 [l l] ISO, “Quality management and quality assurance standards. Part 3: Guidelines for the application of IS0 9001 to the development, supply and maintenance of software”, IS0 9000-3: 1992, Geneva, 1992