Two Case Studies on Small Enter pr ise Motivation and Readiness for CMMI Mark Staples1,2, Mahmood Niazi3 1
NICTA, Australian Technology Park Eveleigh NSW 1430, Australia 2 School of Computer Science and Engineering, University of New South Wales, NSW 2052, Australia
[email protected] 3
School of Computing and Mathematics, Keele University Staffs ST5 5BG, United Kingdom
[email protected]
Abstr act. The Capability Maturity Model Integration (CMMI) is a structured representation of process areas that can support an organization’s software development competency. CMMI can be used as a framework for Software Process Improvement (SPI). Some large organizations have received productivity and product quality benefits from achieving high “levels” of CMMI. However, CMMI is sometimes thought to be difficult to apply to Small and Medium-sized Enterprises (SMEs), and its relevance to SMEs is not yet clear. This paper discusses early results from two case studies of small enterprises that have chosen to use CMMI for SPI. In particular, this paper examines these enterprises’ motivation for adopting CMMI for SPI, and also looks at their organizational readiness for SPI. We suggest, in contrast to common wisdom about the adoption of SPI, that readiness for SPI is not associated with the specific motivations for adopting SPI. Instead, we suggest that having clear goals or objectives for SPI is critical, even if those goals are not stereotypical for SPI.
1 Intr oduction Software Process Improvement (SPI) aims to improve the effectiveness of the software development process in an organization. SPI models such as the CMMI [26] focus on defining characteristics of processes that are intended to produce quality software. Most research on CMMI-based SPI has focused on large organizations [2, 9, 13], and large enterprises using processes based on CMMI have reported higher quality software, reduced development cost and time, and increased development productivity [28]. There is world-wide academic and industrial interest in similar research on Small and Medium-sized Enterprises (SMEs), but fewer results have been reported. CMMI is sometimes seen as difficult to apply to SMEs, due to the cost, time and resource needs associated with its application [7, 23]. We have conducted longitudinal case studies with two small enterprises that have started to adopt CMMI-based SPI. We report findings about why these companies had chosen to adopt CMMI-based SPI, how ready they were for SPI, how their pre-existing processes compared to the requirements for CMMI level 2 process areas, and how their adoption of CMMI progressed. This data lets us discuss four of the research questions in our project: RQ1 Why do SMEs embark on SPI initiatives? RQ2 What motivates SME employees for SPI? RQ3 What de-motivates SME employees for SPI? RQ4 How ready are SMEs for SPI? The two small companies in our case study are similar in many ways, but have contrasting reasons for adopting SPI. This helps us to think about the differences in their readiness and adoption of CMMI. The data and analysis in this paper contributes to the body of knowledge about the adoption of SPI, in particular for SMEs. The research reported in this paper is part of a wider project to investigate the impacts of CMMI-based SPI on SMEs. The results of this project are intended to provide insights to SMEs about the nature, size and timing of CMMI-based process improvement benefits, in order to help SMEs commit to an effective SPI program. This paper is organised as follows. Section 2 describes the background and context for our research. Section 3 describes our case study methodology. The main case study findings are discussed in Section 4, and are illustrated with some example responses. Section 5 provides the discussions and conclusion.
2 Background Our research concerns the use of CMMI-based SPI by SMEs. Here we give some relevant background about CMMI and SMEs in Australia. 2.1 CMMI CMMI (and its precursor CMM) is a framework describing software engineering process areas. These process areas are intended to cover many important best practices within the software engineering industry, and CMMI can be used for SPI. The most well-known representation of CMMI is the “staged” representation, with five “capability maturity levels” ranging from “Initial” (Level 1) to “Optimizing” (Level 5). For an organization to reach a maturity level, they must satisfy the goals of the process areas for that level and all lower levels. An organization can judge its process strengths, weaknesses, and maturity level by conducting an “appraisal” against the requirements of a specific level. There are three classes of appraisal method, which each have different standards of evidence. The most well-known is a “Class A” appraisal, which is an intensive process audit, conducted by a team led by an SEI-authorized CMMI Lead Appraiser. Less intensive “Class B” and “Class C” appraisals can be run by an unauthorized process expert, require less evidence, and are usually much less expensive and time-consuming. Only Class A appraisals result in an official “rating” of the organization at a specific maturity level, but all appraisals result in a list of agreed process strengths and weaknesses. A number of studies have been conducted to observe the impact of CMMI and CMM on organizational software development capability [2, 3, 6, 9, 11, 18, 24, 31]. Most of these studies have collected data from large organizations, almost invariably having undergone Class A appraisals. However, little research has been conducted to observe the impact of CMMI-based SPI on SMEs, especially where these organizations use Class B or Class C appraisals. 2.2 Small Enter pr ises There are more than a million SMEs in Australia, and these make a significant contribution to the Australian economy [22]. ICT producing SMEs make up 16% of the total number of Australian SMEs [29]. The definition of SMEs varies between organizations and countries. The World Bank uses three criteria to define SMEs: number of employees, total assets and total sale [17]. The APEC Centre for Technology Exchange and Training for Small and Medium Enterprises (ACTETSME) uses only one criterion (number of employees) to define Manufacturing and Services SMEs in Australia [1], as shown in Table 1. Table 1 Definition of SMEs Manufacturing Enterprises Services Enterprises
Small Medium Small Medium
< 100 employees 100-199 employees < 20 employees 20-199 employees
There is no single definition of SMEs with the software community. Since this study will be conducted on Australian SMEs, the ACTETSME’s definition of Australian Services SMEs seems to be more appropriate for this research project.
3 Resear ch Design This research uses an empirical approach [5] and collects and analyses empirical data from two longitudinal case studies. 3.1 Tar get and Sample Population The target population for our research is software-developing SMEs. The sample population consists of two SMEs that had decided to undergo an initial CMMI Level 2 Class B appraisal and to implement a subsequent improvement plan. The two
SMEs agreed to participate in our study under conditions of anonymity. Both of these organizations are small services enterprises according to ACTETSME’s criteria. [1] 3.2 Data Collection Methods The organizations in this study are dynamic entities acting in the world within a complex business context. In order to extract as much understanding about the participant’s experiences as possible, this research uses a series of semi-structured interviews for data collection. Each of the two longitudinal case studies is based largely on three interviews with a participating SME: 1. Prior to the initial appraisal. 2. Immediately after the initial appraisal. 3. Significantly after the presentation of the appraisal results and improvement plan to the organization. (Six months, or earlier if indicated by the organization.) The interview questions were adapted from a list of questions that had been used previously [21, 25]. The chosen questions were suited to the nature of our research questions, and related to CMMI impact on SMEs. The list of questions was designed to limit the amount of time required of the study’s participants. The first interview questions (immediately before the initial appraisal) are divided into five sections: • Respondent background • Demographics • Process quality data • Process improvement data • Process improvement readiness The first interview forms were handed out in person, and questions are answered in turn. The interview questioning is both open and close-ended with frequent probing to elaborate and clarify meaning. The interview duration is two hours, however, the researcher and interviewee would determine the pace of the interview. The second interview questions (immediately after the initial appraisal) are divided into four sections: • Respondent background • Process improvement • Appraisal experiences • Future plans For the second interview we used a questionnaire form, supported by telephone or email. The questions were fairly simple and participants could respond effectively without face to face interaction. The third interview questions (significantly after the presentation of the appraisal results and improvement plan) 3.3 Data Analysis Methods In order to analyse the interviews a content analysis technique was used. Content analysis is a research method that uses a set of procedures to make valid inferences from text [20]. Identification of themes is one of the important stages in analysing qualitative data [8]. In order to identify common themes of CMMI impact on SMEs, the following process has been used in this research. The interview forms were read to identify the major themes. These were noted down and compared to the notes made during the interviews in order to provide confirmation that the forms being analysed are indeed a true reflection of the discussion in the interviews. Sometimes different themes are grouped together. For example, SPI motivators ‘sales and marketing’ and ‘tendering initiatives’ were grouped together in a category ‘sales and marketing’. Similarly, SPI de-motivators ‘no immediate benefits’ and ‘long term approach’ were grouped together under a SPI de-motivator category ‘no immediate benefits’.
4 Two Case Studies The two companies in our case study are called “Company A” and “Company B”. After describing these companies below, we discuss the findings from our case studies in terms of each of the research questions listed in the introduction.
Company A is a small Australian-based company that provides hardware and software services to private sector. The company employs fewer than 20 professionals. Their annual sales total is less than three million Australian dollars. Company A has been in existence for more than 4 years. The company develops software solutions for workflow, inventory control, and time tracking for manufacturing and distribution companies in Australia. It develops these applications mainly using a Microsoft platform and data integration with accounting systems and other databases. The company is also productizing two internally-developed R&D projects: a time tracking product (also used internally to track software development work); and a content management system (also used in-house for internet and intranet websites). Company B is a small Australian company that provides consultancy and information technology services on e-business and online projects. The company employs fewer than 20 professionals. The annual sales total is less than three million Australian dollars. Company B has been in existence for more than 12 years and has completed more than 200 successful projects. The company provides two main streams of services – a full range of website and e-business services, and a specialization in enterprise portal and content management systems. 4.1 Why do SMEs embar k on SPI initiatives? The reason why an organization adopts a SPI initiative can influence the implementation and impact of the initiative. The SEI has discouraged organizations from pursing ratings as an end in itself, (e.g. purely for marketing purposes) [30]. Hall et al. found the most common reasons for introducing SPI were: improving software quality, reducing costs and reducing timescales. [14] Hall et al. found relatively few companies introduced SPI for marketing purposes. Nonetheless, it is a reality that business value can accrue from ratings themselves, not just from improved organizational software development capability. We collected the reasons that SMEs gave for embarking on their CMMI-based SPI initiative. These are shown in Table 2. Company A cited only one reason for embarking on the initiative: for marketing purposes. Company B cited eight reasons, which included external drivers such as marketing purposes, customer demand and customer satisfaction, but also included internal software development capability drivers. Company B’s reasons are in line with common responses from Hall et al.’s study [14], but Company A’s reason is not. Table 2 Why do SMEs embark on SPI initiatives? ‘X’ denotes ‘yes’ and ‘-‘ denotes ‘no’ Reason Public relations/ marketing To reduce software development cost To shorten software development cycle times To increase productivity To improve the quality of the software developed To improve the relevant development documentation To meet customers demand To improve customers satisfaction To improve management visibility in software development To reduce time-to-market
Company A X -
Company B X X X X X X X X -
Company A was aware of the potential longer-term benefits for improved software development capability from CMMIbased SPI. However, they saw this potential benefit as an incidental one – it was not the driver for their adoption. Company A believed that their main opportunity for improved business value was from independent recognition of their software development capability. Although none of Company A’s customer had an explicit demand for CMMI-based ratings, Company A believed that their customers would respect and reward independent evidence of process quality. Company B reported many reasons for undertaking CMMI-based SPI. It may be that these were all drivers for adoption, but it is also possible that the company did not have a focussed goal for SPI or was unsure about specific benefits from SPI. 4.2 What motivates SME employees for SPI? Technical quality initiatives such as CASE tools and organizational initiatives such as CMMI target improved software processes. We suggest that whether a quality initiative is technical or organizational, insufficient attention to the human
aspects of implementing SPI can significantly affect its success. McDermid and Bennet [19] have argued that the human factors in SPI have been largely ignored and this has damaged the effectiveness of SPI. Hall and Wilson [15, 16] have also suggested that the experiences, opinions and perceptions of software practitioners impact indirectly on the quality of software produced. One such human aspect is practitioners’ motivations for SPI initiatives. The objective for our research question RQ2 “What motivates SME employees for SPI?” is to understand the nature of issues that motivate SME practitioners for SPI initiatives, so that SME practitioners can better manage these issues. So far no other study has been conducted on this topic with SMEs. Table 3 shows the list of motivators cited by practitioners of both Company A and Company B. The employee motivators cited at both companies are distinct from each other but are well aligned with the reasons given in Table 2 about why the company is embarking on CMMI-based SPI. Table 3 What motivates SME employees for SPI? ‘X’ denotes ‘yes’ and ‘-‘ denotes ‘no’ Motivator Certification Enhance customers satisfaction Improve business Improve job satisfaction Process consistency Reduction in cost of poor quality Sales and marketing
Company A X X X
Company B X X X X -
These motivators are also not similar to those reported in the literature about studies conducted with larger companies [3, 10, 13, 24, 31], except for job satisfaction. Further study is required to investigate this potential difference in motivators identified by SMEs compared to larger companies. These differences may be due to the distinguishing characteristics of SMEs, or may only have an affect within a particular environment. One plausible hypothesis is that in smaller companies employee motivations are more closely aligned with company goals than in larger companies. If these differences are valid, we suggest that it will be essential that SPI practitioners be aware of them so that they can design SPI initiatives appropriate for SMEs. 4.3 What de-motivates SME employees for SPI? The previous section discussed employee motivators for SPI. However, de-motivators are not necessarily just the opposite of the motivators, and so we have chosen to separately investigate the research question RQ3 “What de-motivates SME employees for SPI?” Table 4 provides a summary of SPI de-motivators cited by the two companies in our study. Again, different demotivators were reported between the two companies. Table 4 What de-motivates SME employees for SPI? ‘X’ denotes ‘yes’ and ‘-‘ denotes ‘no’ Reason Resource hungry No immediate benefits Risk of adding complexity Concerns about moving to a ‘one-size-fits-all’ methodology Uncertain about relevance of CMMI for SMEs
Company A X X -
Company B X X X X
At both companies, ‘resource hungry’ was identified as a de-motivator. This supports previous results from Baddoo and Hall [4]. The belief of practitioners and researchers that SPI initiatives require a lot of resources may well be true not only for large companies but also for SMEs. There are three major elements involved in SPI initiatives: SPI appraisal, process definition and process deployment [12]. All elements require significant resources, including: • money to hire lead appraisers,
• time away from work for staff to be interviewed, • time for process definition Company A also cited ‘no immediate benefits’, which has been previously noted by Baddoo and Hall [4]. We think that this de-motivator will be important for many SMEs. SMEs have limited resources and budget often need immediate benefits to support their survival. A recent report of Software Engineering Institute reports the average number of months taken to successfully move from one maturity level of CMM® to the next [27]: • Maturity level 1 to 2 is 22 months • Maturity level 2 to 3 is 19 months • Maturity level 3 to 4 is 25 months • Maturity level 4 to 5 is 13 months In order to fully implement a SPI initiative, organizations need to invest resources for a long time. The calculation of expected benefits must also consider the risk that the SPI initiatives may not achieve the desired results. An interesting de-motivator cited by Company B is ‘Uncertain about relevance of CMMI for SMEs’. This reported uncertainty among employees supports the possibility discussed in section 4.1 that the company reported many reasons for adopting CMMI-based SPI because they were not sure about which specific benefits were likely. There is little evidence available that CMMI is as effective for SMEs as it is for large companies. This is one factor that may be addressed by our research project. We believe that this problem can be reduced by publishing more support guidance from quality bodies and by promoting benefits of SPI [15, 16, 21], and a good understanding of the costs and risks of SPI. Research has shown that many practitioners complain about the lack of useful guidance about quality and how to achieve higher quality [15]. SPI awareness is important to ensure the support of management and practitioners to continue SPI initiatives [21]. 4.4 How r eady ar e SMEs for SPI? We collected information from both companies which we used to determine how prepared they were for SPI implementation. SPI readiness was assessed using an Implementation Maturity Model (IMM) [21]. The IMM is structurally similar to the CMMI [26] and has four maturity or readiness levels (as shown in Figure 1): Initial, Aware, Defined and Optimising. These maturity levels contain different critical success factors and critical barriers. For each factor, practices have been identified that influence the factor. According to the IMM, the SMEs in the ‘Initial’ level are not ready at all for SPI programs, while SMEs in the ‘Optimising’ level are fully ready for SPI programs. The SPI readiness results of the two companies in our case studies are summarised in Table 5. Company A achieved Level-2 ‘Aware’ of the IMM, and only one factor of Level-3 ‘Defined’ was not fully implemented: ‘Creating process action teams’. In order to achieve a maturity level it is important that all the factors that belong to that level are satisfied. Company B was only at IMM Level-1 ‘Initial’, because one factor of Level-2 ‘Aware’ was not fully implemented: ‘Training and mentoring’. Company B also satisfied fewer of the Level 3 areas. Table 5 Summary of SPI readiness assessment results, SPI? ‘X’ denotes ‘Satisfied’ and ‘-‘ denotes ‘Not satisfied’ IMM Maturity Levels IMM Level-2 ‘Aware’
IMM Level-3 ‘Defined’
IMM Level-4 ‘Optimising’
Assessment Implementation Factors Senior management commitment Training and mentoring Staff involvement Awareness of SPI Lack of support Creating process action teams Experienced staff Staff time and resources Defined SPI implementation methodology Time pressure Organizational politics Reviews of SPI implementation methodology
A X X X X X X X X X X -
B X X X X X -
5 Discussion and Conclusions Although the size and nature of the two companies we have been studying are similar, the reasons why they are adopting CMMI-based SPI are different, and their employees’ motivators and de-motivators are likewise mostly different. The difference in their reasons for adopting CMMI-based SPI help us to consider other similarities and differences in their employee’s motivators and de-motivators, and also in their organizational readiness for SPI. As discussed in section 4.1, Company A’s reason for adopting CMMI-based SPI was for marketing purposes, which is an organizational motivation that is usually maligned within the SPI community. Company B listed many organizational motivations for adopting CMMI-based SPI that correspond well to the received views of SPI. However, information about employee de-motivators as discussed in section 4.3 suggested that Company B listed many reasons partly because they were unclear about the expected relevance of CMMI for SMEs and its specific benefits. Both companies were motivated by the marketing advantage to be gained by achieving certification. This motivation is considered suspect by many in the SPI researchers and practitioners. However it is no doubt a common motivation that can deliver real business value. Both companies also have a valid perception about resources required for CMMI-base SPI. This concern is even more critical for SMEs than for large enterprises. This issue is hard to address, but leaner SPI models, better integrated tool support, and improved software engineering education may help. For SPI readiness SMEs have some advantages – organizational politics is not as big an issue in SMEs, due to the smaller number of staff members and clearer central control. It would be comparatively easy for SMEs to successfully deploy any new processes than large companies. However, SMEs have also some disadvantages for SPI readiness – CMMI-based SPI in particular features many roles for staff members, which requires SMEs employees to don many hats. It is hard for SMEs to have dedicated and independent process action team. The ultimate impact of these SPI initiatives is yet to be seen, but our initial SPI readiness assessment indicates that Company A is in satisfactory position for SPI, despite its unorthodox organizational motivation. An organization’s clarity of purpose may be just as important as the nature of their purpose in preparing them for SPI. Our study has findings with both similarities and differences to other studies conducted with large companies [4]. By focusing on studying the reasons for these, researchers and practioners may better understand SPI initiatives for SMEs. This information may help quality bodies with guidance for SMEs that are planning to embark on SPI initiatives. Our case study has also revealed some hypotheses that could be investigated in future research. In particular, in smaller companies employee motivations may be more closely aligned with company goals than in larger companies.
Acknowledgments NICTA is funded by the Australian Government as represented by the Department of Broadband, Communications and the Digital Economy and the Australian Research Council through the ICT Centre of Excellence program.
Refer ences 1. 2. 3. 4. 5. 6. 7. 8. 9.
ACTETSME.: SME profile in Australia , APEC Centre for Technology Exchange and Training for Small and Medium Enterprises (ACTETSME), http://www.actetsme.org/aust/aus98.html, Site visited 8-09-2004. (2004) Ashrafi, N.: The impact of software process improvement on quality: in theory and practice, Information and Management 40 (7). 677-690. (2003) Baddoo, N., Hall, T.: Motivators of software process improvement: An analysis of practitioner's views, J. of Systems and Software (62). 85-96. (2000) Baddoo, N. and Hall, T.: De-Motivators of software process improvement: An analysis of practitioner's views, J. of Systems and Software 66 (1). 23-33. (2003) Black, T.: Doing qualitative research in the social sciences: An integrated approach to research design, measurement and statistics. Sage. (1999) Butler, K.: Process lessons learned while reaching Level 4, CrossTalk (May). 1-4. (1997) Cater-Steel, A.: Process Improvement in Four Small Software Companies. In 13th Australian Software Engineering Conference (ASWEC'01), IEEE Computer Society. (2001) Creswell, J.: Research Design: Qualitative and quantitative approaches. Sage. (1994) Diaz, M., Sligo, J.: How Software Process Improvement helped Motorola, IEEE Software 14 (5). 75-81. (1997)
10. El-Emam, K., Fusaro, P., Smith, B.: Success factors and barriers for software process improvement. Better software practice for business benefit: Principles and experience, IEEE Computer Society (1999) 11. Fitzgerald, B., O'Kane, T.: A longitudinal study of software process improvement, IEEE Software (May/June). 37-45. (1999) 12. Garcia, S.: Preliminary Insights Working with CMMI in Small Organizations. CMMI technology conference, Carnegie Mellon University. (2003) 13. Goldenson, D. R., Herbsleb, J. D : After the appraisal: A systematic survey of Process Improvement, Its benefits, And Factors That Influence Success. SEI, CMU/SEI-95-TR-009 (1995) 14. Hall, T., Rainer, A., Baddoo, N.: Implementing software process improvement: an empirical study, Software Process Improvement and Practice 7 (1). 3-15. (2002) 15. Hall, T., Wilson, D.: Views of software quality: a field report, IEEE Proceedings on Software Engineering 144 (2). (1997) 16. Hall, T., Wilson, D.: Perceptions of software quality: a pilot study, Software Quality J. (7). 67-75. (1998) 17. IFG. SME Definition, International Finance Group - The World Bank Group, http://www.ifc.org/sme/html/sme_definitions.html, Site visited 14-08-2004. 18. Jiang, J., Klein, G., Hwang, H.-G., Huang, J., Hung, S.Y.: An exploration of the relationship between software development process maturity and project performance, Information and Management (41). 279-288. (2004) 19. McDermid, J., Bennet, K.: Software Engineering research: A critical appraisal, IEEE Proceedings on Software Engineering 146 (4). 179-186. (1999) 20. Michael, S., Lewis, B.: Research practice: International handbooks of quantitative application in social sciences. Sage. (1994) 21. Niazi, M., Wilson, D., Zowghi, D.: A Maturity Model for the Implementation of Software Process Improvement: An empirical study, J. of Systems and Software 74 (2). 155-172. (2005) 22. NOIE: Taking the plunge: Sink or Swim - small business attitudes to electronic commerce, http://www.archive.dcita.gov.au/__data/assets/pdf_file/0004/21829/SinkorSwim2000.pdf: visited 14th January 2010, (2000) 23. Paulk, M.: Using the Software CMM in small organizations. The Joint 1998 Proceedings of the Pacific Northwest Software Quality Conference and the Eighth International Conference on Software Quality. Portland. 350-361. (1998) 24. Rainer, A., Hall, T.: Key success factors for implementing software process improvement: a maturity-based analysis, J. of Systems and Software (62). 71-84. (2002) 25. Rainer, A., Hall, T.: A quantitative and qualitative analysis of factors affecting software processes, J. of Systems and Software 66 (1). 7-21. (2003) 26. SEI: Capability Maturity Model® Integration (CMMISM), Version 1.1. SEI, CMU/SEI-2002-TR-029 (2002) 27. SEI: Process Maturity Profile. Software Engineering Institute Carnegie Mellon University. (2004) 28. SEI: 2005 Performance Results of CMMI, http://seir.sei.cmu.edu/cmmiresearch/results/2005results.asp, last accessed 14 Jan 2010. (2005) 29. Sensis: ICT production in Australian SMEs. Business index special report for the Department of Communications, Information Technology and the Arts. (2004) 30. Shrum, S.: Roundtable Interview on CMMI. In SEI Interactive, Software Engineering Institute Carnegie Mellon University, Dec (1999) 31. Stelzer, D., Werner, M.: Success factors of organizational change in software process improvement, Software Process Improvement and Practice 4 (4). (1999)
