Forecasting enterprise resource planning software effort using ...

This article appeared in a journal published by Elsevier. The attached copy is furnished to the author for internal non-commercial research and education use, including for instruction at the authors institution and sharing with colleagues. Other uses, including reproduction and distribution, or selling or licensing copies, or posting to personal, institutional or third party websites are prohibited. In most cases authors are permitted to post their version of the article (e.g. in Word or Tex form) to their personal website or institutional repository. Authors requiring further information regarding Elsevier’s archiving and manuscript policies are encouraged to visit: http://www.elsevier.com/copyright

Author's personal copy

Available online at www.sciencedirect.com

International Journal of Project Management 30 (2012) 967 – 977 www.elsevier.com/locate/ijproman

Forecasting enterprise resource planning software effort using evolutionary support vector machine inference model Jui-Sheng Chou a,⁎, Min-Yuan Cheng a , Yu-Wei Wu a , Cheng-Chieh Wu a

b

National Taiwan University of Science and Technology, Taipei, Taiwan b Data Systems Consulting Co., Ltd., Taichung, Taiwan

Received 20 June 2011; received in revised form 24 January 2012; accepted 16 February 2012

Abstract Despite significant advances in procedures that facilitate project management, the continued reliance of software managers on guesswork and subjective judgment causes frequent project time overruns. This study uses an Evolutionary Support Vector Machine Inference Model (ESIM) for efficiently and accurately estimating the person-hour of ERP system development projects. The proposed ESIM is a hybrid intelligence model integrating a support vector machine (SVM) with a fast messy genetic algorithm (fmGA). The SVM mainly provides learning and curve fitting while the fmGA minimizes errors. The analytical results in this study confirm that, compared to artificial neural networks and SVM, the proposed ESIM provides preliminary prediction at early phase of ERP software development effort for the manufacturing firms with superior accuracy, shorter training time and less overfitting. Future research can develop user-friendly expert systems with window or browser interfaces that can be used by planning personnel to flexibly input related variables and to estimate development effort and corresponding project time/cost. © 2012 Elsevier Ltd. APM and IPMA. All rights reserved. Keywords: Enterprise resource planning; Software effort prediction; Project management; Hybrid intelligence

1. Introduction Enterprise resource planning (ERP) enables developers to enhance the global competitiveness and sustainability of their client enterprises by ensuring efficient resource allocation. In practice, ERP software functions and specifications are highly unpredictable at early stages of R&D; thus, initial cost estimation relies mostly on the subjective judgments of experienced software engineers. Although knowledgeable sales managers or estimators may generate accurate cost assessments via a cooperative approach, professionals in small and medium-sized software enterprises are often difficult to train and highly mobile. Thus, the difficulty of retaining experienced personnel with project knowledge results in such problems as loss of project know-how. Despite the significant advances in the procedures that facilitate project management (PMI, 2008), product managers in the ⁎ Corresponding author at: 43 Sec. 4, Keelung Rd., Taipei 106, Taiwan. Tel.: +886 2 2737 6321; fax: +886 2 2737 6606. E-mail address: [email protected] (J.-S. Chou). 0263-7863/$36.00 © 2012 Elsevier Ltd. APM and IPMA. All rights reserved. doi:10.1016/j.ijproman.2012.02.003

software industry still encounter problems requiring guesswork and subjective judgment, which often result in inaccurate estimates. Effort estimation is not functionally related to the basic drivers of ERP system development. Companies can thus lose market share and orders when attempting to attract customers during the early marketing phase. Although human experts can achieve satisfactory outcomes, shortfalls typically result from inefficient information management. Shortcomings in current subjective assessments or analogous methods indicate the urgent need and opportunity for improvement. Unlike traditional manufacturing, most software product development costs are incurred by investment in human resources. As software is a virtual intelligence and customer service–oriented product, software developers must estimate project completion time at early stages. Many studies (Ahmed and Muzaffar, 2009; de Barcelos Tronto et al., 2008; Elish, 2009; Finnie et al., 1997; Huang and Chiu, 2006; Huang et al., 2008; Jørgensen, 2010; Kazemifard et al., 2011; Lopez-Martin, 2011; Mair et al., 2000; Oliveira et al., 2010; van Koten and Gray, 2006) have proposed cost or effort forecasting methods for software development


968

J.-S. Chou et al. / International Journal of Project Management 30 (2012) 967–977

projects. Nevertheless, the effectiveness and efficiency of approximate inference methods for estimating ERP system development efforts via hybrid intelligence (i.e., combination of multiple artificial intelligence techniques) are rarely addressed. Developing deterministic mathematical models for solving project prediction problems is both difficult and expensive. Approximate inference, a fast and cost effective approach, is a viable alternative to deterministic mathematical modeling. Inference is the process of deriving new knowledge from known information. As the known information changes, the inference process adapts accordingly. Prediction problems are complex and often involve substantial uncertainty, vagueness, and incomplete or inexact data. Therefore, the inference process must fit environmental conditions (Mareels and Polderman, 1996). Humans can process and solve complex problems, even those involving uncertainty, imprecision, and incomplete information. Therefore, imitating the process of human inference is an effective approach to solving project prediction problems. The primary objective of this research was predicting development time for ERP software project by using an Evolutionary Support Vector Machine Inference Model (ESIM). After reviewing the relevant literature and collecting data and ERP software specifications from past projects in the manufacturing industry, an ESIM was proposed for simultaneously searching for the fittest support vector machine (SVM) parameters within a globally optimized model. An early accurate time estimate during the negotiation stage with potential clients can eliminate unnecessary bargaining and cost changes in subsequent processes. The rest of this paper is organized as follows. Section 2 reviews pertinent literature in software project estimation. Section 3 then discusses the research methodology and model adaptation process. Next, Section 4 illustrates the applicability of the ESIM technique in a case study of a leading ERP software system provider. Conclusions are finally drawn in Section 5 along with managerial implications and suggestions for future research. 2. Literature review The ERP software systems typically manage internal enterprise resources and departmental business operations. Functions include business management, production management, financial planning, quality management, and human resource management. The typical phases of a software development project (Yeo, 2001) are, in chronological order, system analysis, system design, programming, testing, deployment, and online operations. Estimation accuracy heavily depends on the amount and quality of information available at estimation time. Thus, providing accurate preliminary project estimates is extremely challenging, particular during the software development phases (LopezMartin, 2011). A major limitation of current practices is that, when contacting clients for make-to-order quotes, project managers must estimate time-to-complete based solely on known product attributes whereas virtually all software costs at early stages of software development are implicitly determined by investment in human resources. High-quality ERP software development projects require efficient project control mechanisms to schedule and manage

software development progress. Project estimation is thus a prerequisite for all subsequent planning activities. Although none of the studies have quantified the effort or duration requirements for ERP software development, project estimation schemes proposed for other software project and industries include casebased reasoning (Belecheanu et al., 2003; Chou, 2009b; Finnie et al., 1997; Mendes et al., 2002, 2003; Yang and Wang, 2008), multiple regression analysis (Adriano L.I, 2006; Bashir and Thomson, 2001, 2004; M. Camargo et al., 2003; Caputo and Pelagagge, 2008; Chou, 2009a; Chou et al., 2006; Ferens, 1998; Finnie et al., 1997; Huang et al., 2008; Marban et al., 2008; Mittas and Angelis, 2010; Sentas and Angelis, 2006), activitybased costing (Baykasoglu and Kaplanoglu, 2008; Ben-Arieh and Qian, 2003; Tornberg et al., 2002), artificial neural networks (Attarzadeh and Ow, 2010; Berlin et al., 2009; Caputo and Pelagagge, 2008; de Barcelos Tronto et al., 2008; Delen et al., 2005; Finnie et al., 1997; Huawang and Wanqing, 2008), support vector machine (Adriano L.I, 2006; An et al., 2007; Lu and Tsai, 2008; Pal and Deswal, 2008), genetic algorithms (Huang and Chiu, 2006; Oliveira et al., 2010) and function point analysis (Albrecht and Gaffney, 1983; Finnie et al., 1997; IFPUG, 2009; Longstreet, 2002; Mendes et al., 2003; Myrtveit et al., 2005; Yinhuan et al., 2009). However, the above works considered single techniques rather than hybrid intelligence schemes. Artificial intelligence (AI)-based approaches are related to computer system designs that attempt to resolve problems intelligently by emulating human brain processes. As AI technology enhances the ability of computer programs to handle tasks for which humans are still superior (Haykin, 1999), AI models are typically used to solve project estimation problems. Various scientific and engineering fields have recently combined different AI paradigms to enhance efficacy. Numerous studies confirm that hybrid AI schemes outperform single techniques in project estimation (Chen, 2007; Kim and Shin, 2007; Lee, 2009; Li et al., 2005; Min et al., 2006; Nandi et al., 2004; Wu, 2010; Wu et al., 2009). Fast messy genetic algorithm (fmGA) (Goldberg et al., 1993) and the SVM (Vapnik, 1995) are two such tools that have proven effective for solving various project management problems. Specifically, the SVM in the ESIM is used mainly for learning and curve fitting while the fmGA optimizes prediction error. Given the characteristics and merits of fmGA and SVM, the ESIM proposed in this study combines them for predicting development effort in ERP software projects. Fig. 1 shows the software project quotation and engineer-to-order processes integrated with the proposed ESIM. This ESIM was designed to achieve the fittest C and gamma parameters with minimal prediction error. The proposed approach considers implicit knowledge from historical cases in the software industry so that project managers or decisionmakers can overcome challenges during early cost quotation. 3. Evolutionary support vector machine inference model 3.1. Support vector machine integrated with fast messy genetic algorithm Support vector machine was first introduced by Vapnik (1995) and colleagues at AT&T Bell Laboratories (Vapnik,



Marketing department

Routine visits Observe the dynamic global market Acquire conceptual product attributes

Seel & identify potential customers & market places

Lose order

Visit customers & understand their needs

Price quotation and negotiation process

After-sale service

969

ESIM (estimated model)

Offer accepted Procurement/legal affairs/design/production departments

Historical Cases

Make to order by conducting detailed design

Product delivery

Deliver software product with quality, as scheduled, and within budgetary estimates

Project close-out

Fig. 1. The engineering-to-order processes with the proposed ESIM prediction model.

1995, 1998). The numerous applications of SVMs in various other disciplines include customer churn prediction (Coussement and Van den Poel, 2008), reliability forecasting in engine systems (Chen, 2007), document categorization (Hao et al., 2007), performance rating (Lee, 2007; Ravi et al., 2008), content-based image retrieval (Seo, 2007), Fault detection and diagnosis (Konar and Chattopadhyay, 2011; Widodo and Yang, 2007), feature selection and pattern recognition (Acir, 2006; Huang and Dun, 2008; Kim and Shin, 2007; Li et al., 2005), prediction of air entrainment rate and aeration efficiency of weirs (Baylar et al., 2009), and bankruptcy prediction (Chaudhuri and De, 2011; Min et al., 2006; Shin et al., 2005). The SVM classifies data with different class labels by determining a set of support vectors that are members of the set of training inputs that outline a hyper plane in a feature space. A kernel function of a generic mechanism is used to fit the hyper plane surface to training data. The input-output model in this study was constructed by Epsilon Support Vector Regression (ε-SVR) (Smola and Schölkopf, 2004), a variation of generic SVM for function estimation. In SVM regression, the input x is first mapped onto an m-dimensional feature space by using fixed (nonlinear) mapping. A linear model is then constructed in this feature space

(Fig. 1). The linear model (in the feature space) f(x,ω) can be expressed mathematically as f ðx;ωÞ ¼

m X

ω j g j ð xÞ þ b

ð1Þ

j−1

where gj(x), j= 1, …, m denotes a set of nonlinear transformations, and b is the “bias” term. Because data are typically assumed to have a mean of zero (and are therefore obtainable at the preprocessing stage), the bias term is dropped. Estimation quality is measured by the loss function L(y,ƒ(x,ω)). The SVM regression uses the following ε-insensitive loss function, which was proposed by Vapnik (1998): 0 if jy−f ðx; ωÞj≤ε Lε ðy; f ðx; ωÞÞ ¼ ð2Þ otherwise jy−f ðx; ωÞj−ε Linear regression performed in SVM regression in the highdimension feature space uses ε-insensitive loss and reduces model complexity by minimizing ‖ω‖ 2. This procedure is implemented when (non-negative) slack variables, ξi, ξi*,i = 1, … n, are introduced to identify training samples that deviate from the


970


ε-insensitive zone. Thus, SVM regression is formulated as a minimization of the following function: n X 1 min kωk2 þ C ξi þ ξ i 2 i¼1 8 < yi−f ðxi ; ωÞ ≤ ε þ ξi subject to f ðxi ; ωÞ−yi ≤ ε þ ξi : ξi ; ξi ≥ 0; i ¼ 1; …n

ð3Þ

This optimization problem can be transformed into the dual problem, which is solved by f ðxÞ ¼

nSV X α i −α i K ðxi ; xÞ

subject to 0 ≤ α i ≤ C; 0 ≤ α i ≤ C

i¼1

ð4Þ where nSV is the number of support vectors, and the kernel function is K ðx; xi Þ ¼

m X

g j ðxÞg j ðxi Þ

ð5Þ

j¼1

SVM generalization performance (estimation accuracy) requires correct settings for meta-parameters C and γ, and for ε kernel parameters (that are related to the kernel type, e.g., γ parameter in RBF kernel (K(x, y) = exp(− γ||x − y|| 2)); default values of kernel parameters clearly depend on the kernel parameter type and the implementing software. Existing software

implementations of SVM regression usually treat metaparameters as user-defined inputs (Smola and Schölkopf, 2004; Vapnik, 1995). Selecting specific kernel types and kernel function parameters usually requires application-domain knowledge and should reflect the distribution of training data inputs. A user may select the SVM kernel function (e.g., linear, radial basis, polynomial, or sigmoid function) during the training process to identify support vectors along the function surface. Since the RBF kernel maps samples nonlinearly into a higher dimensional space and presents fewer numerical difficulties, RBF is a reasonable first choice. However, when using an SVM, users must determine the optimal kernel parameters. Therefore, SVM parameters must be obtained when optimizing parameter settings. For enhanced SVM prediction accuracy, parameter optimization should include penalty parameter C, and the gamma of the radial basis function (RBF) kernel. Fig. 2 shows the generic SVM procedure. Fast messy genetic algorithm (fmGA), which was first developed by Goldberg et al. (1993), is an alternative means of identifying the best C and gamma parameters when using the RBF kernel function. In contrast with the conventionally adopted simple GA (sGA), which uses fixed length strings to represent possible solutions, an fmGA applies messy chromosomes to form strings of various lengths. Efficient solution optimization for large-scale permutation problems enables fmGA to generate SVM parameters C and gamma simultaneously. The proposed ESIM uses SVM for learning and curve fitting while the fmGA optimizes prediction error. The purpose of the

Fig. 2. The SVM procedure.



Inference engine (SVM)

Output

971

Input patterns

C, γ Actual output

Optimal prediction model

Yes

Optimization (fmGA)

Termination criteria

No

Desired output Fig. 3. Flowchart of ESIM.

model was to obtain the fittest C and gamma parameters with minimized prediction error. The ESIM adaptation process can be demonstrated with a pseudo code algorithm (Section 3.2) developed by the authors for efficient project prediction. Fig. 3 shows the ESIM flowchart. The steps are as follows. (1) Training the SVM. In this step, the SVM uses default parameters and a training dataset to train a prediction model. (2) Fitness evaluation. For each chromosome representing C and γ, a training dataset is used to train the SVM and to calculate prediction accuracy. When accuracy is sufficient, each chromosome is evaluated using a fitness function. (3) Termination criteria. The process stops once termination criteria are satisfied. If any criteria are unsatisfied, the model proceeds to the next generation. (4) The search for fmGA parameters. In this stage, the model searches for superior solutions via genetic operations.

Evaluate(Pop(Era),template);

//Evaluate

fitness

value } //Juxtapositional Phase j = 0; While (not juxtapositional termination condition) { Thresholding_Selection(Pop(Era)); Cut_and_Splice(Pop(Era)); Mutation(Pop(Era)); Evaluate(Pop(Era), template); j = j + 1; } template = Optimal_string(Pop(Era), Epoch); Era = Era + 1; } Epoch = Epoch + 1;

3.2. ESIM adaptation process } The ESIM adaptation process can be programmed by the following pseudo code algorithm. Begin Epoch = 1; Generate the competitive template = random string; While (not termination condition) //Outer Loop { Repeat //Inner Loop { //Initialization Phase Era = 0; Probabilistic_Initialize(Pop(Era), Epoch); Evaluate(Pop(Era), template); //Evaluate fitness value //Primordial Phase While (not primordial termination condition) { Episode = 0; While (Episode b Episode_max(Era)) { Thresholding_Selection(Pop(Era)); Episode = Episode + 1; } Building-Blocks_Filtering(Pop(Era));

End 3.2.1. Probabilistic_Initialize This step simulates a natural chromosome by randomly assigning 0s and 1s to gene values to produce random variables, including C and γ. 3.2.2. Evaluate The objective function of the model (f ob) combines model accuracy and model complexity. The fitness function is the reciprocal of the objective function. 3.2.3. Thresholding_Selection To restrict competition between highly dissimilar building blocks, a generic thresholding mechanism was used to limit tournament selection to pairs of strings that shared a greater than expected number of common genes. In random strings of two different hlengths, λ1, λ2, ithe expected number of common genes is θ ¼

λ1 λ2 l

þ c0 ðα 0 Þσ , where σ is the standard devia-

tion in the number of genes that two randomly chosen strings of possibly differing lengths have in common, and parameter c´(α´) is simply the ordinate of a one-sided normal distribution with tail probability α.


972


3.2.4. Building-Blocks_Filtering A building block repetition factor γ acquires the desired 3 2 λi−1 6 7 λi 7 building block, where γ ¼ 6 4 λi−1 −k 5. λi −k

(c) (d)

3.2.5. Cut_and_Splice In this step, a cut probability recombines different strings into new strings.

(e)

3.2.6. Mutation The mutation produces spontaneous random changes in various chromosomes to protect against premature loss of important notations. For the ESIM, the purpose of mutation is to adjust the value of C and γ for better performance. It alters one or more genes with a probability (p ge), which is smaller than or equal to the mutation rate (p mu). Mutation operation incrementally compares the p ge of the gene with p mu. If p ge ≤ p mu, then the gene value is altered.

(g)

(f)

(h) (i)

(j)

4. Project description and analytical results The ERP software development projects are funded mainly by direct investment in human resources. Therefore, if the number of person-hours can be estimated accurately, then the software development costs can be determined rationally by multiplying the known corresponding hourly rate of engineers required during the preliminary phase. This study collected and analyzed empirical data for 182 ERP software projects developed by a leading Taiwan software provider over the last five years for the manufacturing units.

(k)

(l) (m)

(n)

4.1. Variable selection and definition 4.1.1. Output variables In practice, ERP software development projects normally estimate costs based on the devoted person-hour of effort. Project development costs are obtained by multiplying the development time by the hourly rate. The output variable in this investigation referred to software development effort (SDE) in order to accurately represent in-house project costs and to prevent distortions due to inconsistent project quotes or discounts from external sources. 4.1.2. Input variables Based on interviews with experienced in-house project managers, this work identified factors that affect ERP SDE. The attributes were then designated as input variables and defined as follows: (a) Program Type_Entry (PTE) allows business users to compile basic information for future business transactions and applications. (b) Program Type_Report (PTR) extracts data from the ERP system database and generates charts with specific

(o)

(p)

(q)

formats to verify transactions or to analyze management practices. Program Type_Batch (PTB) processes the information stored in the ERP system database. Program Type_Query (PTQ) extracts information stored in the ERP system database and compiles them appropriately for presentation on the system interface. Program Type_Transaction (PTT) manages the creation of various internal transaction forms in an enterprise. Number of Programs (NP) indicates the number of scheduled programs in the development project. Number of Zoom (NZ) is the source data in the operating interface column. Number of Columns in Form (NCF) refers to the number of columns in the operating interface. Number of Actions (NA) refers to specific keys typically arranged to correspond to functions specified by the program. Number of Multi-angle Trade Tasks (NMTT) refers to the trade model that a business uses to accept orders from overseas clients, to purchase supplies from a third country, and to either deliver the goods to the overseas clients directly from third country suppliers or to transport the goods to the client through Taiwan. Number of Multi-unit Tasks (NMT) requires the use of multiple measurement units, including inventory, purchasing, sales, costs, and products. Number of Reference Calls (NRC) calls for an external system for independent tasks. Number of Confirmed Tasks (NCT) refers to data generated through file and invoice creation programs which are officially adopted through confirmation procedures. Number of Post Tasks (NPT) calculates the inventory quantity and control functions for incoming and outgoing goods and materials for daily enterprise operations. Number of Batch Serial Numbers (NBSN) is used for customer complaints about product quality, and batch serial numbers are used to monitor these problems so effective solutions can be found. Number of Signature Tasks (NST) refers to the use of a signature function in workflow software programs so that important tasks can be verified by administrators. Number of Industry Type Tasks (NITT) indicates the number of different companies from the same enterprise, which may use various business models and may require specialized system designs.

4.2. Validation and descriptive statistics Model validation usually involves checking the model against observed or independent data (Neter et al., 1996). The predictive ability of the ESIM model was checked through the hold-out data extracted from the original ERP projects. The purpose of test data is to examine whether the model developed from the earlier data is still applicable for other projects. Since it's a common approach to randomly split data into 90–10 or 80–20% (Witten and Frank,


J.-S. Chou et al. / International Journal of Project Management 30 (2012) 967–977 Table 1 Variables and descriptive statistics for ERP software project effort prediction. Variable

Min. Max. Mean

Standard Data type deviation

SDE (person-hour) PTE PTR PTB PTQ PTT NP NZ NCF NA NST NBSN NMTT NMT NRC NCT NPT NITT

4 0 0 0 0 0 1 0 3 0 0 0 0 0 0 0 0 0

394.690

2694 1 1 1 1 0 88 2028 3216 1645 15 11 22 21 528 21 12 21

258.55 Dummy variable Dummy variable Dummy variable Dummy variable Referential category 16.73 100.22 397.75 288.44 .39 .31 .55 1.10 13.96 1.50 .23 .80

19.122 255.395 548.057 339.614 1.772 1.497 2.662 3.409 49.923 3.990 1.330 2.967

Numerical Boolean Boolean Boolean Boolean – Numerical Numerical Numerical Numerical Numerical Numerical Numerical Numerical Numerical Numerical Numerical Numerical

2005), historical cases were randomly separated into training sets (163 projects, 90%) and test sets (19 projects, 10%). Table 1 summarizes the descriptive statistical data obtained by randomly filtering 90% of collected data for 163 historical projects. The minimum SDE was 4 (hours), the maximum value was 2,694 (hours), and the standard deviation was 394.690 (hours). The statistical results indicated that the size and complexity of each development project depend on the features and functions of program they incorporate. The minimum value for the number of programs is 1, the maximum value is 88, and the standard deviation is 19.122. The empirical data indicate that the ERP SDE is proportional to the NP in a project, and NZ, NCF, and NA vary according to the functions of each program. Meanwhile, NST, NBSN, NMTT, NMT, NCT, NPT, and NITT are related to specific function areas. Restated, not all programs in each project are equipped with this functional requirement. Therefore, the average values in Table 1 are relatively low, ranging from 0.23 to 1.5. However, as the number of specified features increases, ERP SDE significantly increases. 4.3. Model performance and prediction accuracy The following performance measures were used to evaluate the proposed estimating techniques.

973

4.3.1. Root mean squared error Root mean squared error (RMSE) is the square root of the mean square error. The RMSE is thus the average distance of a data point from the fitted line measured along a vertical line. The RMSE is given by the following equation:

RMSE ¼

vffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi uP un u ðy^ −yÞ2 ti¼1

ð6Þ

n

where y is actual value, y^ is the predicted value, and n is the number of data samples. 4.3.2. Mean magnitude of relative error Mean magnitude of relative error (MMRE) is a statistical measure of predictive accuracy. It usually expresses accuracy as a percentage. The MMRE is commonly used in quantitative forecasting methods because it indicates relative overall fit (goodness-of-fit). The MMRE is given by the following equation: MMRE ¼

n ^ 1X y−y n i¼1 y

ð7Þ

4.4. Cross-fold validation When comparing the predictive accuracy of two or more methods, researchers often use k-fold cross-validation to minimize bias associated with the random sampling of the training and holdout data samples. Specifically, the cross-fold technique can be employed to examine the internal model validity while the comparison with other techniques can assess the external validity. Thus, this work further includes conventional artificial neural networks (ANNs) and support vector regression (SVR) as baseline models. Since cross-validation requires random assignment of individual cases into distinct folds, a common practice is stratifying the folds themselves. In stratified k-fold cross-validation, the proportions of predictor labels (responses) in the folds are intended to approximate those in the original dataset. Empirical studies show that, compared to regular kfold cross-validation, stratified cross-validation tends to reduce bias in the comparison results. Kohavi (1995) showed that ten folds were optimal (i.e., performed validation testing in the shortest time and with

Table 2 Cross-fold validation. Measure Fold Model 1 MMRE

RMSE

ESIM SVR ANNs ESIM SVR ANNs

242.86% 1295.88% 248.06% 22.92174 49.07410 10.24634

2

3

4

5

6

7

8

9

10

Average

New case prediction

42.92% 380.98% 47.35% 33.47718 82.70228 36.49448

61.18% 711.85% 113.40% 15.74532 56.85330 14.84068

46.09% 425.50% 56.73% 23.77867 39.67703 18.42085

43.53% 283.47% 45.47% 31.70817 43.81111 23.76794

44.33% 360.01% 52.79% 13.65386 43.63377 8.680917

27.05% 73.70% 20.59% 21.52452 44.51864 27.02532

34.80% 478.62% 68.29% 19.38907 43.28778 8.542289

31.66% 141.06% 26.14% 25.32623 60.43048 25.56236

27.72% 205.85% 36.94% 13.41490 36.01950 20.38687

60.21% 435.69% 71.58% 22.09397 50.00080 19.39681

26.16% 177.63% 29.09% 37.32115 135.39524 44.44957


974


cross-validation accuracy CVA, where k is the number of folds used and Ai is the accuracy measure of each fold. CVA ¼ ∑k1

Fig. 4. The ESIM predicted versus actual person-hour with training data.

acceptable bias and variance) (Kohavi, 1995). Thus, to assess estimation performance across artificial intelligence techniques, a stratified 10-fold cross-validation approach was used. The entire dataset was divided into ten mutually exclusive subsets (or folds) with class distributions approximating those of the original dataset (stratified). The subsets were extracted in five steps: 1. Randomize the dataset 2. Extract one tenth of the original dataset size from the randomized dataset (single fold). 3. Remove the extracted data from the original dataset. 4. Repeat steps 1-3 eight times. 5. Assign the remaining portion of the dataset to the last fold (10th fold). This procedure was first used to obtain ten distinct folds. Each fold was used once for performance tests of the prediction models, and the remaining nine folds were used for training, which obtained ten independent performance estimates. The cross-validation estimate of overall accuracy was calculated by simply averaging the k individual accuracy measures for

Ai k

ð8Þ

Based on the cross-fold technique, the experiment compared prediction performance in the three regression models, i.e., ESIM, SVR, and ANNs. Table 2 shows MMREs and RMSEs for the three models. The comparison results show that ESIM generally outperformed SVR and ANNs in terms of average MMRE. Interestingly, ESIM and ANNs have similar RMSE rates which are both better than SVR. Notably, to test the prediction power, this study collects five more newly completed project cases. The prediction performance of ESIM is also better than ANNs and SVR in terms of MMRE and RMSE (Table 2). 4.5. Model performance The previous internal and external validity assessments have proven the superiority of ESIM and ANNs in ERP project effort prediction. This study further randomly split the whole case dataset into training (90%) and testing (10%) groups for evaluating these two models’ performance. Fig. 4 compares the ERP project training results for the person-hour predicted by the ESIM with those for the actual effort. Although the range of these individual prediction errors (PEs) was .00–177.00%, roughly 60% of the training cases had PEs lower than 20%, and about 40% had PEs lower than 10%. The overall MMRE of the ESIM training dataset was 26.8%, and the RMSE was 234.0157 (hours). Model validity can be verified by the data in Fig. 5, which shows the difference between the predicted and actual development effort for the test dataset; 58% of the test cases had PEs lower than 25.4%. The overall MMRE of the ESIM testing set was 27.3% with a smaller dispersion of RMSE (97.2667 hours). Based on the randomly split training and test datasets, the analytical results confirm the efficacy of ESIM for preliminary prediction. Table 3 lists the specified C constant and gamma value required as input patterns for the ESIM case study. This investigation adopted the same set of historical project data used in ESIM to construct a general back-propagation ANNs model for comparison purposes. This work used the PASW modeler 13.0 data mining tool (formerly Clementine) for the analyses. The network-related parameters included Table 3 ESIM Results.

Fig. 5. Models comparison via testing cases.

C, γ

110, 0.9995

RMSE (Training) MMRE (Training) RMSE (Testing) MMRE (Testing) No. of training cases No. of testing cases

234.0157 hr 26.8% 97.2667 hr 27.3% 163 19



975

Table 4 Model performance. Case No.

1

Actual Output (hr) ANNs (hr) PE 1 (%) ESIM (hr) PE (%)

688 58

351 58

364 658 53

571 17 600 12.7

196 44 119 66.0

164 55 214 41.1

1

2

90 56 51 11.6

3

4

53 9 43 25.4

5

6

419 36 399 39.4

7

20 61 66 24.4

8

9

16 17

18

19

143 12 862 317 410 68

383 168 65 44

12

156 –

241 69 82 42.4

582 52 482 25.8

7 42 18 51.2

259 66 222 42.5

17 45 13 9.4

10

746 13 861 0.1

11

186 41 257 18.8

12

491 20 332 19.0

13

58 15 57 16.8

14

15

198 18 178 6.0

54 17 68 5.2

89 102 71 61.0

MMRE PEmin PEmax RMSE (%) (%) (%) (hr)

– 36.5 – 27.3

–

–

–

– 0.2 – 0.1

– 133.9 – 66.0

– 84.17 – 97.27

Prediction error.

input ranges, number of neurons in the input layer, number of neurons in the hidden layers, number of input neural networks, learning rate-related initial value, decay rate, maximum value, minimum value, and inertia factor (alpha). Training was curtailed when either of the following conditions was met: (a) accuracy higher than the designated value; (b) training cycle; (c) training time. The ANNs simulation was set to 2–9 to avoid model overfitting. Since empirical experience indicated that low learning rates obtain highly precise solutions, the initial value was set to 0.05. The error value may be nonlinear, possibly resulting in the classification of a solution obtained by steepest descent method as the local optimal solution. To avoid local optimal solutions during training, a higher inertia factor of 0.95 is suggested (SPSS, 2009). Table 4 shows the project person-hour predicted by the model for individual test cases along with their actual output. Although the ANNs clearly generated less prediction dispersion, the ESIM was more accurate and had a narrower range of prediction error for the ERP software project development. 5. Conclusions and future directions This study demonstrated an ESIM approach for predicting development time for ERP software projects. The ESIM is a hybrid intelligence technique integrating an SVM with an fmGA. The SVM provides learning and curve fitting by mapping nonlinear input and output, and the fmGA primarily deals with global optimization concurrently while minimizing prediction error. According to the empirical case study, the proposed systematic approach efficiently predicts the development effort of ERP software project at early stages as prediction error is within a satisfactory limit. Through the cross-validation and prediction power testing, ESIM and ANNs show better performance than conventional SVR in ERP project effort prediction. Further, analytical results via random sampling process indicate that the overall MMRE of ESIM is 26.8% for training datasets. For the randomly separated data for testing, the MMRE achieved by ESIM was 27.3%, which is lower than the 36.5% by ANNs (Table 3) with a 25.2% improvement in prediction accuracy. Particularly, the current in-house analogous practice is 35.0% in average indicating an improvement of 22.0% via ESIM over the MMRE of the software provider. Notably, the effort required to complete software development projects also depends on collaboration among multiple engineers, their experience level, and

location. The historical project data considered by the ESIM did not include such information and could not be covered in this research. Therefore, to further improve the ESIM performance, future studies are needed to gather comprehensive project data. Software companies typically have high employee turnover. Therefore, they may often have difficulty retaining experiencebased and implicit corporate knowledge. Conversely, only incomplete design specifications are available during the early quotation and negotiation stages. Rapidly responding to customer enquiries using little available information is critical to securing orders. Fairly accurate tools that can be used to predict the manpower requirements for developing an ERP system are needed so that engineered-to-order suppliers can maintain competitive advantage in global markets. With experience, good judgment, and timely adjustments, systematic knowledge discovery from sales records offers project managers in the software industry an effective and efficient way of estimating project effort and quoting prices in the early stages of software development for customers. Future research can develop user-friendly expert systems with window or browser interfaces that can be used by planning personnel to flexibly input related variables and to estimate development effort. Such a system can be designed to forecast completion date based on project size and participating project resources.

References Acir, N., 2006. A support vector machine classifier algorithm based on a perturbation method and its application to ECG beat recognition systems. Expert Systems with Applications 31, 150–158. Adriano, L.I.O., 2006. Estimation of software project effort with support vector regression. Neurocomputing 69, 1749–1753. Ahmed, M.A., Muzaffar, Z., 2009. Handling imprecision and uncertainty in software development effort prediction: A type-2 fuzzy logic based framework. Information and Software Technology 51, 640–654. Albrecht, A.J., Gaffney Jr., J.E., 1983. Software Function, Source Lines of Code, and Development Effort Prediction: A Software Science Validation. IEEE Transactions on Software Engineering SE-9, 639–648. An, S.-H., Park, U.Y., Kang, K.-I., Cho, M.-Y., Cho, H.-H., 2007. Application of support vector machines in assessing conceptual cost estimates. Journal of Computing in Civil Engineering 21, 259–264. Attarzadeh, I., Ow, S.H., 2010. A novel soft computing model to increase the accuracy of software development cost estimation. 2nd International Conference on Computer and Automation Engineering, ICCAE 2010, February 26, 2010 - February 28, 2010. IEEE Computer Society, Singapore, Singapore, pp. 603–607.


976


Bashir, H.A., Thomson, V., 2001. Models for estimating design effort and time. Design Studies 22, 141–155. Bashir, H.A., Thomson, V., 2004. Estimating design effort for GE hydro projects. Computers and Industrial Engineering 46, 195–204. Baykasoglu, A., Kaplanoglu, V., 2008. Application of activity-based costing to a land transportation company: A case study. International Journal of Production Economics 116, 308–324. Baylar, A., Hanbay, D., Batan, M., 2009. Application of least square support vector machines in the prediction of aeration performance of plunging overfall jets from weirs. Expert Systems with Applications 36, 8368–8374. Belecheanu, R., Pawar, K.S., Barson, R.J., Bredehorst, B., Weber, F., 2003. The application of case based reasoning to decision support in new product development. Integrated Manufacturing Systems 14, 36–45. Ben-Arieh, D., Qian, L., 2003. Activity-based cost management for design and development stage. International Journal of Production Economics 83, 169–183. Berlin, S., Raz, T., Glezer, C., Zviran, M., 2009. Comparison of estimation methods of cost and duration in IT projects. Information and Software Technology 51, 738–748. Camargo, M., Rabenasolo, B., Jolly-Desodt, A.-M., Castelain, J.-M., 2003. Application of The Parametric Cost Estimation in The Textile Supply Chain. Journal of Textile and Apparel, Technology and Management 3, 1–12. Caputo, A.C., Pelagagge, P.M., 2008. Parametric and neural methods for cost estimation of process vessels. International Journal of Production Economics 112, 934–954. Chaudhuri, A., De, K., 2011. Fuzzy Support Vector Machine for bankruptcy prediction. Applied Soft Computing 11, 2472–2486. Chen, K.-Y., 2007. Forecasting systems reliability based on support vector regression with genetic algorithms. Reliability Engineering and System Safety 92, 423–432. Chou, J.-S., 2009a. Generalized linear model-based expert system for estimating the cost of transportation projects. Expert Systems with Applications 36, 4253–4267. Chou, J.-S., 2009b. Web-based CBR system applied to early cost budgeting for pavement maintenance project. Expert Systems with Applications 36, 2947–2960. Chou, J.-S., Peng, M., Persad, K.R., O'Connor, J.T., 2006. Quantity-based approach to preliminary cost estimates for highway projects. Transportation Research Record 22–30. Coussement, K., Van den Poel, D., 2008. Churn prediction in subscription services: An application of support vector machines while comparing two parameter-selection techniques. Expert Systems with Applications 34, 313–327. de Barcelos Tronto, I.F., da Silva, J.D.S., Sant'Anna, N., 2008. An investigation of artificial neural networks based prediction systems in software project management. Journal of Systems and Software 81, 356–367. Delen, D., Walker, G., Kadam, A., 2005. Predicting breast cancer survivability: A comparison of three data mining methods. Artificial Intelligence in Medicine 34, 113–127. Elish, M.O., 2009. Improved estimation of software project effort using multiple additive regression trees. Expert Systems with Applications 36, 10774–10778. Ferens, D.V., 1998. Software parametric cost estimation: wave of the future. Engineering Costs and Production Economics 14, 157–164. Finnie, G.R., Wittig, G.E., Desharnais, J.M., 1997. Comparison of software effort estimation techniques: Using function points with neural networks, cased-based reasoning and regression models. Journal of Systems and Software 39, 281–289. Goldberg, D.E., Deb, K., Kargupta, H., Harik, G., 1993. Rapid, accurate optimization of difficult problems using fast messy genetic algorithms. Proceedings of the Fifth International Conference on Genetic Algorithms. Morgan Kaufmann, pp. 56–64. Hao, P.-Y., Chiang, J.-H., Tu, Y.-K., 2007. Hierarchically SVM classification based on support vector clustering method and its application to document categorization. Expert Systems with Applications 33, 627–635. Haykin, S., 1999. Neural networks: a comprehensive foundation. Prentice-Hall, New York.

Huang, S.-J., Chiu, N.-H., 2006. Optimization of analogy weights by genetic algorithm for software effort estimation. Information and Software Technology 48, 1034–1045. Huang, C.-L., Dun, J.-F., 2008. A distributed PSO-SVM hybrid system with feature selection and parameter optimization. Applied Soft Computing 8, 1381–1391. Huang, S.-J., Chiu, N.-H., Liu, Y.-J., 2008. A comparative evaluation on the accuracies of software effort estimates from clustered data. Information and Software Technology 50, 879–888. Huawang, S., Wanqing, L., 2008. The integrated methodology of rough set theory and artificial neural-network for construction project cost prediction. 2nd International Symposium on Intelligent Information Technology Application, IITA 2008, December 21, 2008 - December 22, 2008. Inst. of Elec. and Elec. Eng. Computer Society, Shanghai, China, pp. 60–64. IFPUG, 2009. Function Point Counting Practices Manual Release 4.3. International Function Point Users Group. Jørgensen, M., 2010. Identification of more risks can lead to increased overoptimism of and over-confidence in software development effort estimates. Information and Software Technology 52, 506–516. Kazemifard, M., Zaeri, A., Ghasem-Aghaee, N., Nematbakhsh, M.A., Mardukhi, F., 2011. Fuzzy Emotional COCOMO II Software Cost Estimation (FECSCE) using Multi-Agent Systems. Applied Soft Computing 11, 2260–2270. Kim, H.-j., Shin, K.-s., 2007. A hybrid approach based on neural networks and genetic algorithms for detecting temporal patterns in stock markets. Applied Soft Computing 7, 569–576. Kohavi, R., 1995. A study of cross-validation and bootstrap for accuracy estimation and model selection. The 14th International Joint Conference on Artificial Intelligence (IJCAI). Morgan Kaufman, San Francisco, CA, pp. 1137–1145. Konar, P., Chattopadhyay, P., 2011. Bearing fault detection of induction motor using wavelet and Support Vector Machines (SVMs). Applied Soft Computing 11, 4203–4211. Lee, Y.-C., 2007. Application of support vector machines to corporate credit rating prediction. Expert Systems with Applications 33, 67–74. Lee, M.-C., 2009. Using support vector machine with a hybrid feature selection method to the stock trend prediction. Expert Systems with Applications 36, 10896–10904. Li, L., Jiang, W., Li, X., Moser, K.L., Guo, Z., Du, L., Wang, Q., Topol, E.J., Wang, Q., Rao, S., 2005. A robust hybrid between genetic algorithm and support vector machine for extracting an optimal feature gene subset. Genomics 85, 16–23. Longstreet, D., 2002. Function Points Analysis Training Course. Lopez-Martin, C., 2011. A fuzzy logic model for predicting the development effort of short scale programs based upon two independent variables. Applied Soft Computing 11, 724–732. Lu, C.J., Tsai, D.M., 2008. Independent component analysis-based defect detection in patterned liquid crystal display surfaces. Image and Vision Computing 26, 955–970. Mair, C., Kadoda, G., Lefley, M., Phalp, K., Schofield, C., Shepperd, M., Webster, S., 2000. An investigation of machine learning based prediction systems. Journal of Systems and Software 53, 23–29. Marban, O., Menasalvas, E., Fernandez-Baizan, C., 2008. A cost model to estimate the effort of data mining projects (DMCoMo). Information Systems 33, 133–150. Mareels, I., Polderman, J.W., 1996. Adaptive systems: an introduction. Birkhauser, Boston, Massachusetts. Mendes, E., Mosley, N., Counsell, S., 2002. The application of case-based reasoning to early web project cost estimation. Proceedings - IEEE Computer Society's International Computer Software and Applications Conference, pp. 393–398. Mendes, E., Watson, I., Triggs, C., Mosley, N., Counsell, S., 2003. A comparative study of cost estimation models for web hypermedia applications. Empirical Software Engineering 8, 163–196. Min, S.-H., Lee, J., Han, I., 2006. Hybrid genetic algorithms and support vector machines for bankruptcy prediction. Expert Systems with Applications 31, 652–660.


J.-S. Chou et al. / International Journal of Project Management 30 (2012) 967–977 Mittas, N., Angelis, L., 2010. Visual comparison of software cost estimation models by regression error characteristic analysis. Journal of Systems and Software 83, 621–637. Myrtveit, I., Stensrud, E., Shepperd, M., 2005. Reliability and validity in comparative studies software prediction models. IEEE Transactions on Software Engineering 31, 380–391. Nandi, S., Badhe, Y., Lonari, J., Sridevi, U., Rao, B.S., Tambe, S.S., Kulkarni, B.D., 2004. Hybrid process modeling and optimization strategies integrating neural networks/support vector regression and genetic algorithms: study of benzene isopropylation on Hbeta catalyst. Chemical Engineering Journal 97, 115–129. Neter, J., Kutner, M.H., Nachtsheim, C.J., Wasserman, W., 1996. Applied Linear Statistical Models, fourth ed. IRWIN. Oliveira, A.L.I., Braga, P.L., Lima, R.M.F., Cornélio, M.L., 2010. GA-based method for feature selection and parameters optimization for machine learning regression applied to software effort estimation. Information and Software Technology 52, 1155–1166. Pal, M., Deswal, S., 2008. Modeling pile capacity using support vector machines and generalized regression neural network. Journal of Geotechnical and Geoenvironmental Engineering 134, 1021–1024. PMI, 2008. A Guide to the Project Management Body of Knowledge, 4th ed. Project Management Institute. Ravi, V., Kurniawan, H., Thai, P.N.K., Kumar, P.R., 2008. Soft computing system for bank performance prediction. Applied Soft Computing 8, 305–315. Sentas, P., Angelis, L., 2006. Categorical missing data imputation for software cost estimation by multinomial logistic regression. Journal of Systems and Software 79, 404–414. Seo, K.-K., 2007. An application of one-class support vector machines in contentbased image retrieval. Expert Systems with Applications 33, 491–498. Shin, K.-S., Lee, T.S., Kim, H.-j., 2005. An application of support vector machines in bankruptcy prediction model. Expert Systems with Applications 28, 127–135. Smola, A.J., Schölkopf, B., 2004. A Tutorial on Support Vector Regression. Statistics and Computing 14, 199–222.

977

SPSS, 2009. PASW Modeler 13, Chicago, Illinois. Tornberg, K., Jämsen, M., Paranko, J., 2002. Activity-based costing and process modeling for cost-conscious product design: A case study in a manufacturing company. International Journal of Production Economics 79, 75–82. van Koten, C., Gray, A.R., 2006. Bayesian statistical effort prediction models for data-centred 4GL software development. Information and Software Technology 48, 1056–1067. Vapnik, V.N., 1995. The Nature of Statistical Learning Theory. Springer-Verlag, New York. Vapnik, V.N., 1998. Statistical Learning Theory. John Wiley and Sons, New York. Widodo, A., Yang, B.-S., 2007. Application of nonlinear feature extraction and support vector machines for fault diagnosis of induction motors. Expert Systems with Applications 33, 241–250. Witten, I.H., Frank, E., 2005. Data Mining: Practical Machine Learning Tools and Techniques Second Edition ed. Morgan Kaufmann. Wu, Q., 2010. The hybrid forecasting model based on chaotic mapping, genetic algorithm and support vector machine. Expert Systems with Applications 37, 1776–1783. Wu, C.-H., Tzeng, G.-H., Lin, R.-H., 2009. A Novel hybrid genetic algorithm for kernel function and parameter optimization in support vector regression. Expert Systems with Applications 36, 4725–4735. Yang, H.-L., Wang, C.-S., 2008. Recommendation system for IT software project planning: A hybrid mining approach for the revised CBR algorithm. Institute of Electrical and Electronics Engineers Computer Society, Piscataway, NJ 08855-1331, United States, Melbourne, Australia, p. 4598501. Yeo, A.W., 2001. Global-software development lifecycle: An exploratory study. Proceedings of the SIGCHI conference on Human factors in computing systems ACM, NY, USA, pp. 104–111. Yinhuan, Z., Beizhan, W., Yilong, Z., Liang, S., 2009. Estimation of software projects effort based on function point. Proceedings of 2009 4th International Conference on Computer Science & Education. IEEE, pp. 941–943.