Identification Method of Improvements in User ... - IEEE Xplore

0 downloads 0 Views 1MB Size Report
This article has been accepted for publication in a future issue of this journal, but has not been fully ...... [3] A Guide To The Project Management Body Of Knowledge (PMBOK ... ment skill-up simulator,” in Proceedings of the 6th International.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2017.2673019, IEEE Access JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

1

Identification Method of Improvements in User Operations on Project Manager Skill-Up Simulator Manami Otsuki, Masanori Akiyoshi, Member, IEEE, and Masaki Samejima, Member, IEEE Abstract—Project managers need to determine the appropriate operations for solving problems in project management. In order to obtain the necessary skills for determining these operations, project managers learn and practice the theory of project management, which requires great deal of time and money. To provide more efficient training for project managers, we developed a skill-up simulator to allow the project manager to practice such operations on a computer. However, the simulator did not show the project manager which operations need to be improved. Our proposed method identifies the operations that need to be improved in comparison to other operations. Because it is time-consuming to collect operations from other project managers, agent programs automatically generate the operations in the proposed method. By inputting the generated operations to the simulator, the proposed method obtains project results for the operations. Operations to be improved are identified by a decision tree trained with other operations and evaluations. The experimental results confirm that the proposed method can correctly identify operations that need to be improved. Index Terms—e-Learning, Project Management, Operation Analysis, Decision Tree.

F

1

I NTRODUCTION

D

UE to the complication and diversification of software systems, only 30% of software development projects are completed successfully [1]. Project manager training methods have received attention for increasing the number of project managers who have superior skills for handling projects. In these training methods, project managers are trained through an iterative process of learning and practicing the theory of project management [2]. For learning theory, project managers study the contents of textbooks based on the Project Management Body of Knowledge (PMBOK) [3]. For practicing theory, project managers are trained through On-the-Job Training (OJT), case studies, and role playing [4]. OJT involves project managers joining real projects, requires a lot of training time and may lead to failures by the learning managers [5]. Case studies and role playing require instructors who explain the cases and evaluate the learners; these are difficult to iterate due to the high costs involved. To provide more efficient training to project managers, research has been conducted on skill-up simulators that allow project managers to have a simulated experience [6], [7]. Advantages of training with a simulator include reduced costs and avoided troubles in real projects. We have also developed a simulator for learning how to manage a project



M. Otsuki was with Graduate School of Information Science and Technology, Osaka University, 1-5, Yamadaoka, Suita, Osaka, 5650871, Japan.



M. Akiyoshi is with Faculty of Engineering, Kanagawa University, 3-271, Rokkakubashi, Kanagawa-ku, Yokohama, Kanagawa, 2218686, Japan. e-mail: [email protected]



M. Samejima is with Graduate School of Information Science and Technology, Osaka University, 1-5, Yamadaoka, Suita, Osaka, 5650871, Japan. e-mail: [email protected]

This work was supported by JSPS KAKENHI Grant Number 23650537.

during the implementation and testing phases [8]. Our developing simulator receives input data of an operation from learners to improve a project’s status, which is evaluated by quality, cost and delivery date of the software to be developed [9]. Based on the input operation, the simulator estimates the project status after performing the operation and prompts the learners to input additional operations. Once the development of the software is completed in the simulator through user operations, the simulator shows the final result of the status. The learner then determines the operations in the simulation that should be improved based on the final result. However, the project status considers only quality, cost and delivery date, and does not include information on which operations have a negative effect on the project; for example, an early operation of overworking. Therefore, learners are unable to determine all operations should be improved based on the final status result. In this paper, we aim to identify operations to be improved among all the operations input by the learner during the simulation.

2

P ROJECT M ANAGER S KILL - UP S IMULATOR

2.1 Outline of the simulator Our developing simulator aims to show learners various project situations, rather than reproducing real projects in detail. The simulator generates situations based on a project model and a project management difficulty level provided by a trainer. Figure 1 shows the outline of the project manager skill-up simulator [8]. The trainer provides a project model including event rules based on the trainer’s policy. Next, the simulator outputs a Gantt chart showing the progress of the project simulated with the project model and the event rules. If the learner inputs an operation to improve the progress, the

2169-3536 (c) 2016 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2017.2673019, IEEE Access JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

2

simulator generates the next status of the project based on the operation. Once the project is completed, the learner can confirm the simulation results shown by the simulator as an evaluation of the operations. The input data of a project model and a set of event rules, as well as the simulation process, are described in the following sections. •



Project model Table 1 shows a project model. A project model contains information on tasks to be completed and human resources in a project, as follows: –





Difficulty level of a task We evaluate the difficulty of a task according to three levels: H is the highest, followed in order by M and L. Table 1 indicates that modules 3 and 4 are the most difficult of the five. The simulator quantifies the difficulty: H , M , and L are converted to the uniform random numbers [0.7, 1.0], [0.3, 0.7], and [0.0, 0.3], respectively. These values are reflected to indicate the status of the project by the simulator, as described further in the paper. Because the same level of difficulty may differ slightly as a result of the randomness, a learner must determine operations not only based on the level of the difficulty, but also based on the status of the project. Starting time and duration of a task The time of starting a task and its duration are preliminarily planned and provided to the simulator, but these will change based on the status of the project. Submodule Developing a module sometimes requires other modules that have already been developed. We call the requiring module a “sub-



Set of event rules Event rules are aimed at causing bugs unexpectedly in a simulated project in order to have learners understand how to deal with various bug frequencies. An event rule is a pair made up of an event and a firing condition. –



Trainer Set

Project Model (modules, members, etc.)

module” of the required module. The relationship means that it is impossible to start developing submodules before developing the modules required by the submodule. Therefore, in Table 1, module 5 can only be started after developing module 4. Member’s skill level We evaluate a member’s skill based on three levels: the highest is A, followed in order by B and C . The simulator quantifies the skill level: A, B , and C are converted to the uniform random numbers [0.7, 1.0], [0.3, 0.7], and [0.0, 0.3], respectively. In the example of Table 1, a member with a skill level of C is assigned to a task of with a difficulty level of B . Because the member is lacking the required skill level in this case, delays and bugs occur during development of the module.

Refer

Event We assume that three negative events take place in a project: bug occurrence, increased tasks due to requirement changes, and decreased member performance. The impact of each events is represented by three levels: large, middle, and small. A trainer provides the type and impact of each event. Firing condition A firing condition is the condition that causes an event based on the status of the project. A trainer sets the firing condition with a threshold for the status, and the event occurs stochastically when the status is over or below the threshold.

Simulator Estimate

Event rules



Status of project (progress, bugs) Learner

Input operations

0

Module 1 Show Module 2 Gantt chart Module 3

10 Supervising Overworking

20 Overworking

Simulator The simulator repeatedly determines the progress of each task and the number of bugs that have occurred during the task based on a project model and event rules, until a project is completed. –

Supervising Overworking

Fig. 1. Outline of project manager skill-up simulator

Progress of a task When a member is assigned to a task with a difficulty level that is not less than the member’s skill level, the member needs more than the standard time to complete the task. In addition, it becomes challenging to keep the

TABLE 1 An example of a project model

Module 1 Module 2 Module 3 Module 4 Module 5

Difficulty M L H H M

Starting date (duration) 1(10) 11(10) 21(10) 1(10) 11(10)

Submodule Module 2 Module 3 – Module 5 –

Developer’s skill B B B B C

2169-3536 (c) 2016 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2017.2673019, IEEE Access JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

3

member motivation of the member, which further delays the task. Therefore the simulator determines the progress Pi of each task i based on the difficulty Di and the skill level Si of the member assigned to the task [10]:

Pi = 1 − Mi (Di − Si )

(1)

where Mi is the motivation of the member assigned to the task i, and is determined by the following formula with a given parameter α(< 1): { 1 (Di < Si ) Mi = (2) α (Di ≥ Si ) –

Number of occurred bugs Bugs first occur when a member writes source code during the coding phase. Thereafter, the member finds and fixes bugs during the debugging phase, which sometimes causes other bugs. The simulator determines the number of bugs Bic and Bid in the coding and debugging phases, respectively. The number of bugs tends to increase when a member’s skill level is not sufficient for the difficulty of the task to be developed. Assuming that the number of bugs is proportional to the amount of a task completed in a day, and considering the gap between the difficulty of the task and the skill level of the assigned member, we implement the following formulas to calculate Bic and Bid in the simulator [11], [12]:

0

(3)

Bid = max(rPid (Di − Si ), 0)

(4)

2.2 Research Purpose During training, a learner confirms the status of a project and inputs operations. We implement real operations in projects: overworking members for completing a larger than usual amount of a task than usual, and supervising, which allows members to complete the task with reduced bugs even if the member’s skill is not sufficient. These operations increase costs. The implemented operations are not always positive, but are necessary for learners to know to what extent operations affect the project. After training, the simulator outputs an evaluation of the training using quality, cost, and delivery date (QCD) as standard metrics: Quality is evaluated according to the number of bugs.

10

20

Module 1

Evaluation

Operation on the 8th day

Module 2

Bad: Low quality 23rd

Module 3 Operations leading to the best result 0

26th

Small differences often do not affect evaluation but are wrongly identified as improvements 10

20

Module 1 Module 2 Module 3

Operation on the 7th day

Evaluation

22nd

25th

Good

Difference that affects the evaluation should be identified as improvements

Fig. 2. An example of the na¨ıve method

• •

Cost is accumulated according to a learner’s input operations. Delivery date is the number of days until a project is completed.

Based on the quality, cost, and delivery ultimately shown to learners, the learners evaluate the operations that they inputted during the simulation. We call the operations inputted by the learner user operations. The goal of evaluating user operations is to determine which operations need to be improved among the user operations. However, it is difficult to improve user operations based on quality, cost, and delivery date, because of the following reasons: •

Bic = rPic (1.0 + Di − Si )

where r is the ratio of bugs caused for the amounts Pic and Pid of the coding and debugging tasks. As shown in formula (3), bugs occur during the coding phase even if the skill level is much higher than the difficulty level. However, as shown in formula (4), no bugs occur during the debugging phase if the skill level is higher than the difficulty level.



User operations



Quality, cost, and delivery date are the results of user operations and do not indicate which operations have a positive or negative influence on the project. Learners input multiple operations during training, which makes it difficult to determine the relationships between each operation and the project.

In this study, we develop a method for identifying operations to be improved as improvement in order to support learners’ training with the simulator. 2.3

Research issue in identifying improvements

A na¨ıve method for identifying improvements is comparing user operations to operations that lead to the best evaluation of the project, and regarding different operations as improvements. We define the best evaluation as the minimum sum of quality, cost, and delivery date. The best operations leading to the best evaluation can be found using a brute force method with the simulator. Figure 2 shows an example of comparing leaner’s operations to the best operations. The downward arrow in Figure 2 indicates an operation that is being performed at the time. In the example in Figure 2, the first of the user operations is performed on the 8th day, and the first of the best operations is performed on the 7th day. We can find small differences between both operations, but the differences have a minor influence on the project. Because the na¨ıve method identifies all the differences as improvements, a small difference that is not an improvement is also shown to

2169-3536 (c) 2016 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2017.2673019, IEEE Access JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

the leaner. Therefore, it is necessary to show only improvements that have a significant effect on the project.

3

I DENTIFICATION METHOD OF IMPROVEMENTS IN

USER OPERATIONS

3.1 Outline of the proposed method In order to identify improvements in user operations, the proposed method records user operations, fired events, and the evaluation of the simulated project. Figure 3 shows an outline of the identification method of improvements in the recorded user operations. The proposed method splits the time frame from the starting date to the delivery date into multiple time slots with a length of T in order to aggregate different small operations in the same time slot. If the frequencies of operations in each time slot differ between user operations and best operations, the proposed method can identify the differences (with the exception of small ones) as improvements. However, the identified improvements include user operations, which have too minor an influence on the project to be considered improvements. The proposed method determines whether each of the user operation has a large influence on the project in comparison to all the possible operations. Because it is possible to evaluate the project for any operations carried out by the simulator, the operations that have a positive effect on the evaluation can be found among all the possible operations. If the user operations do not include operations that have a positive effect on the evaluation, the operations can be regarded as improvements. As shown in Figure 3, in order to determine the operations that affect the evaluation, we use a decision tree [13], [14], which finds the features contributing to the classes. In this research, the features are the frequencies of operations and the classes are the evaluations. Because the nodes on the decision tree represent the frequencies of operations that affect the evaluation, it is possible to determine improvements by comparing user operations to the operations on the nodes.

Input

4

3.2 Training decision tree for identifying improvements Training the decision tree needs supervised data: frequencies of operations as features and evaluations as classes. The proposed method automatically generates operations, counts the frequencies of operations in each time slot, and evaluates a project by inputting the operations to the simulator. For generating operations, we implement five agent programs to simulate user operations, as shown in Table 2. The proposed method randomly assigns the agent programs to modules in a project, and the assigned agent programs determine the operations based on their behavior policies and the simulated status of the project. By repeating this process N times, the proposed method obtains N sets of operations and evaluations. Frequencies of operations generated by the agent programs are calculated within each time slot and used as supervised data. However, the evaluation based on quality, cost, and delivery date is a continuous value, so operations that are classified into the continuous value are too detailed, which makes the decision tree overfit a certain continuous value. In order to avoid overfitting, we convert a continuous value to a symbolic value with five levels. The conversion process is as follows: 1)

Find the maximum values Qmax , Cmax , and Dmax and minimum values Qmin , Cmin , and Dmin among quality Qn , cost Cn , and delivery date Dn (1 ≤ n ≤ N)

TABLE 2 Agent programs to generate operations automatically Agent name Normative Random More overworking More supervising Late operation

Behavior policy Input overworking when the project is delayed and supervising when the member’s skill level is not higher than the difficulty level Input overworking and supervising randomly Input overworking frequently Input supervising frequently Input operations later than Normative inputs

Output

Identify improvements

User operations

Improvements: operations that made evaluation worse

Operations lead to bad evaluation on Decision Tree T Improvements

All the possible operations with evaluation

Train Decision Tree

good

good

(Automatically generated)

Nodes branching to different evaluations

Decision Tree is trained so as to classify operations via nodes.

It is possible to regard nodes as operations that affect evaluations.

bad good

good

bad

Fig. 3. Outline of the identification method of improvements

2169-3536 (c) 2016 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2017.2673019, IEEE Access JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

5

TABLE 3 Conversion from a continuous evaluation value to a symbolic value Evaluation Very good Good Fair Bad Very bad

′ Range of En ′ < 0.2 0.0 ≤ En ′ < 0.4 0.2 ≤ En ′ < 0.6 0.4 ≤ En ′ < 0.8 0.6 ≤ En ′ ≤ 1.0 0.8 ≤ En

User operations Overworking for Module 1 between 10-12th days Supervising for Module 3 between 10-12th days

Sets of frequencies of operations and evaluation (Supervised data)

>1

Overworking for Module 3 between 10-12th days

(Automatically generated) N sets

Overworking for Module 1 between 13-15th days

good

1st set

0

Bad

2nd set

1

Good

=0

>1 Supervising for Module 2 between 13-15th days

Frequencies in each time slot Evaluation good

>1

Improvement 1

>1 bad

All the possible operations with evaluation

=0

Operations leading to the best evaluation

good

Supervising for Module 2 between 10-12th days

=0 Very good

>1 Very good

=0 bad

Improvement 2

Fig. 5. An example of identifying improvements with the decision tree . Decision Tree

Training

Overworking for Module 1 between 7-9th days Supervising for Module 2 between 7-9th days >1 =0 good

The frequency that affects evaluation is included as a node.

classify the result according to whether the frequency is 0 or 1.

Overworking for Module 3 between 10-12th days >1 =0

bad

good

3.3

bad

Fig. 4. Training the decision tree

2)

Normalize Qn , Cn , Dn to Q′n , Cn′ , Dn′ :

Qn − Qmin Qmax − Qmin Cn − Cmin Cn′ = Cmax − Cmin Dn − Dmin ′ Dn = Dmax − Dmin Q′n =

3)

(6) (7)

Calculate evaluation En as a sum of Q′n , Cn′ , Dn′ and normalize it to En′

En = Q′n + Cn′ + Dn′ En − Emin En′ = Emax − Emin

4)

(5)

(8) (9)

where Emin and Emax are the minimum and maximum values of En . Determine the symbolic value of the evaluation according to the conversion table shown in Table 3.

The proposed method trains the decision tree with N sets of operation frequencies in each time slot and evaluation of the project. Figure 4 shows the process of training the decision tree. From N sets of operations and a continuous value of evaluation, the proposed method generates the supervised data of N sets of operation frequencies in each time slot and a symbolic value of the evaluation. Decision Tree is trained so as to classify the frequencies of operations into the symbolic values of evaluations. In the example in Figure 4, the result becomes bad when the frequency of overworking is 0 for module 3 in the time slot between the 10th and 12th days while the result becomes good when the frequency is 1 for the same module and time slot. This supervised data is included as a node in the decision tree to

Identification of improvements

In the proposed method, improvements are identified as user operations to be improved to obtain an evaluation of very good. Figure 5 shows an example of identifying improvements. The operation frequencies necessary to obtain a very good evaluation appear on the edges from nodes that lead to leaves, indicating very good on the decision tree. Therefore, the proposed method compares the edges of user operations to the edges of the operations leading to the leaves indicating very good, and identifies the different edges between both operations as improvements. In the example shown in Figure 5, an overworking time of one for module 1 between the 10th and 12th days, shown as a branch from the root node (improvement 1), and no supervising operation for module 2 between the 10th and 12th days (improvement 2) are different branches of operations that both lead to very good, which are identified as improvements in user operations. If the trained decision tree contains multiple leaves indicating very good, the proposed method selects one of the leaves for identifying improvements. When training decision trees, leaves indicating very good do not always include supervised data corresponding to very good. Therefore, the proposed method selects the leaf into which the supervised data corresponding to very good is classified with the highest probability, known as fitness. If multiple leaves have the highest fitness, the proposed method selects the leaf that has more supervised data corresponding to very good.

4 4.1

E VALUATION EXPERIMENT Experimental setup

We obtained the user operations that eight learners, L1 , · · · , L8 , input to the simulator, then applied the proposed method to identify improvements on the operations. Two project models were used for the training: project model 1 is shown in Table 1 in section 2.1 and project model 2 is shown in Table 4. Five learners, L1 , · · · , L5 used project model 1 and three learners, L6 , · · · , L8 , used project model 2.

2169-3536 (c) 2016 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2017.2673019, IEEE Access JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

6

TABLE 4 Project model 2

Module 1 Module 2 Module 3 Module 4 Module 5 Module 6 Module 7

Difficulty M L H H M H M

Starting date (duration) 1(10) 11(10) 21(10) 1(10) 11(10) 1(15) 16(10)

We used r = {0.5, 0.3, 0.1} for difficulty levels {H, M, L}, and α = 0.7 for determining the motivation Mi as for parameter values of project models. The agent programs in the proposed method generate 1000 sets of operations and evaluations as supervised data (N = 1000). In order to train the decision tree, we used C4.5 algorithm implemented in a WEKA data mining tool [15]. For evaluating how effectively the proposed method can identify improvements, we used the recall rate and precision rate, as follows:

Ic Ic + In Ic Precision rate = Ic + Iw Recall rate =

(10) (11)

where Ic , In , and Iw are the numbers of improvements that the proposed method correctly identifies, does not identify, and wrongly identifies false improvements as, respectively. Project management experts identify true improvements, and determine Ic , In , and Iw . From a training point of view, the precision rate is more important than the recall rate. In addition to the proposed method, we applied the na¨ıve method to compare the user operations to the best operations and to identify different operations as improvements, as described in section 2.3.

Submodule Module 2 Module 3 – Module 5 – Module 7 –

Developer’s skill B B B B C B A

method. Because of the importance of the precision rate in training learners, the proposed method is more appropriate for training than the na¨ıve method. Next, we discuss the influence of the interval T . When T is too small, the proposed method can not identify large differences in operations as improvements, which decreases the recall rates. When T is too large, the proposed method can not identify small differences in operations as improvements, which also decreases the recall rates. In these project models, T = 3 is the most appropriate parameter value for the proposed method. According to project management experts, once a project manager performs an operation, they monitors the effect of the operation for approximately three days or half a week, without improving or changing the operation immediately. Therefore, T = 3, which is equivalent to half a week, is a valid parameter value. Recall rate 100%

90%

80%

Naïve Method Proposed Method 90% 90% 86% 86% 76%

60%

86%

48%

43% 40% 24%

4.2

Experimental results

Because the result of identifying improvements depends on the interval T of each time slot, we applied the proposed method and the na¨ıve method with value of T = {1, 2, 3, 4, 5}. Fig. 6 and Fig. 7 show the recall rates and precision rates in the case of project model 1, while Fig. 8 and Fig. 9 show the recall rates and precision rates in the case of project model 2. It ca be seen in Figure 6 and Figure 8 that the recall rates of the na¨ıve method are often better than those of the proposed method. If the interval T is appropriately set (in this case, T = {3, 4}), it is possible to obtain high recall rates using the proposed method as well as the na¨ıve method. The reason the recall rates of the na¨ıve method tend to be high is that this method identifies all the different operations as improvements, so it includes both true and false improvements. From Figure 7 and Figure 9, we can see that it is difficult to obtain high precision rates using the na¨ıve method, the output of which include true and false improvements. However, the proposed method selects only the user operations that have a major influence on the evaluation of the project, which results in better precision rates than the na¨ıve

20% 0% 1

2

3

4

5

Interval T Fig. 6. Recall rates in the case of project model 1

Naïve Method 100%

Precision rate 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0%

Proposed Method 95%

100%

86% 63% 43%

1

59%

59% 50% 40%

2

3

4

5

Interval T Fig. 7. Precision rates in the case of project model 1

2169-3536 (c) 2016 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2017.2673019, IEEE Access JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

7

TABLE 5 Breakdown of the results in the case of project model 1

Learner True improvements Correctly identified Ic Not identified In Wrongly identified Iw

Na¨ıve method L3 L4 L5

Proposed method L3 L4 L5

Total

L1

L2

3

21

5

4

4

5

3

21

4

3

19

5

3

3

5

2

18

1

1

0

2

0

1

1

0

1

3

2

3

1

13

0

0

1

0

0

1

L1

L2

5

4

4

5

5

4

3

0

0

3

4

Total

TABLE 6 Breakdown of the results in the case of project model 2

Learner True improvements Correctly identified Ic Not identified In Wrongly identified Iw 0

L6

Na¨ıve method L7 L8 Total 3

3

9

3

3

3

9

3

3

3

9

3

3

3

9

0

0

0

0

0

0

0

0

4

12

4

20

0

0

0

0

10

20

30 Overworking Supervising

Module 2 Module 3

Fig. 8. Recall rates in the case of project model 2

Module 4 Module 5

Precision rate Naïve Method 100% 100%

Proposed Method 100% 100%

100%

Fig. 10. Operations by L3 0

80%

10

20

30 Necessary overworking

Module 1

60%

Module 2

40%

31% 24%

Proposed method L7 L8 Total

3

Module 1

100%

L6

31%

31%

Unnecessary overworking

Module 3

Unnecessary supervising

Module 4

22%

20%

Module 5

0% 1

2

3

4

5

Fig. 11. Improvements in L3 ’s operations identified by the proposed method

Interval T 0

Fig. 9. Precision rates in the case of project model 2

Module 1 Module 2

Table 5 and Table 6 show a breakdown of the results for T = 3 in the case of project model 1 and project model 2, respectively. According to the tables, the na¨ıve method sometimes shows the learners a large number of wrongly identified improvements. The number of improvements that are incorrectly identified by the na¨ıve method increases as the scale of the project for training increases. Therefore, it is difficult to apply the na¨ıve method when training with large scale project models. However, the proposed method shows no more than one operation wrongly identified as an improvement, even if the project model scale is large. Finally, we discuss the worst result of the proposed method, namely the result of L3 ’s operations in project

Module 3 Module 4 Module 5

10

20

30 Necessary overworking Unnecessary overworking Necessary supervising

Fig. 12. True improvements in L3 ’s operations identified by an expert

model 1 in order to determine which improvements are difficult to identify. Figure 10, Figure 11 and Figure 12 show L3 ’s operations, improvements identified by the proposed method, and true improvements identified by an expert, respectively. As shown in Figure 10, L3 inputted an over-

2169-3536 (c) 2016 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2017.2673019, IEEE Access JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

working operation to module 4 on the 13th day because the learner thought the development would be delayed. The proposed method suggests that the learner inputted the operation on the 7th day instead of the 13th. In addition, the proposed method determines that supervising operation on the 10th day is unnecessary because the early operation on the 7th day would be sufficient. However, as shown in Figure 12, an expert determines that supervising operation on the 10th day is necessary. The development of module 4 has a difficulty level of A and is an early task. Once the development is delayed, this delay is propagated to the development of later modules. According to the expert, additional supervising operations are necessary in comparison to the improvements shown by the proposed method in order to minimize the delay in the development of module 4. Therefore, improvements that can be observed not only by the status of the project but also from information in the project model are difficult to determine using the proposed method. However, because the proposed method can identify the other improvements, the expert confirms that the proposed method is sufficient for use in real simulationbased training.

5

8

[8]

[9] [10] [11] [12]

[13] [14] [15]

K. Iwai, M. Akiyoshi, M. Samejima, and H. Morihisa, “A situationdependent scenario generation framework for project management skill-up simulator,” in Proceedings of the 6th International Conference on Software and Data Technologies, vol. 2, 2011, pp. 408– 412. A. Babu and N. Suresh, “Project management with time, cost, and quality considerations,” European Journal of Operational Research, vol. 88, no. 2, pp. 320 – 327, 1996. T. Abdel-Hamid and S. E. Madnick, Software Project Dynamics: An Integrated Approach. Upper Saddle River, NJ, USA: Prentice-Hall, Inc., 1991. R. J. Madachy, Software Process Dynamics. Wiley-IEEE Press, 2008. B. Ramesh and T. Abdel-Hamid, “Integrating genetic algorithms with system dynamics to optimize quality assurance effort allocation,” in Software Engineering with Computational intelligence, T. M. Khoshgoftaar, Ed. Kluwer Academic Publishers, 2003, pp. 44–68. P.-N. Tan, M. Steinbach, and V. Kumar, Introduction to Data Mining. Addison Wesley, 2005. T. M. Mitchell, Machine Learning, 1st ed. New York, NY, USA: McGraw-Hill, Inc., 1997. M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and I. H. Witten, “The weka data mining software: An update,” ACM SIGKDD Explorations Newsletter, vol. 11, no. 1, pp. 10–18, 2009.

C ONCLUSION

In this paper, we proposed an identification method of improvements in user operations in the project manager skill-up simulator. While the na¨ıve method of comparing the best operations to user operations incorrectly identifies improvements based on small differences between both operations, the proposed method identifies improvements based on operation frequencies during time slots in order to summarize the slightly different operations. Using the frequencies as features of user operations, the proposed method trains a decision tree and identifies improvements that affect the evaluation of the project significantly by using the trained Decision Tree. The experimental results show that the proposed method identifies improvements with the same recall rate and a better precision rate than the na¨ıve method.

R EFERENCES [1] [2] [3] [4] [5]

[6] [7]

A. Ahmed, Software Project Management: A Process-Driven Approach. Auerbach Publications, 2011. G. Lister, Mastering Project, Program, and Portfolio Management: Models for Structuring and Executing the Project Hierarchy. Pearson FT Press, 2014. A Guide To The Project Management Body Of Knowledge (PMBOK Guides). Project Management Institute, 2004. T. R. Henry and J. LaFrance, “Integrating role-play into software engineering courses,” J. Comput. Sci. Coll., vol. 22, no. 2, pp. 32–38, Dec. 2006. P. C. Blumenfeld, E. Soloway, R. W. Marx, J. S. Krajcik, M. Guzdial, and A. Palincsar, “Motivating project-based learning: Sustaining the doing, supporting the learning,” Educational Psychologist, vol. 26, no. 3-4, pp. 369–398, 1991. A. Drappa and J. Ludewig, “Simulation in software engineering training,” in Proceedings of the 22nd International Conference on Software Engineering, ser. ICSE ’00, 2000, pp. 199–208. E. O. Navarro and A. van der Hoek, “Simse: An educational simulation game for teaching the software engineering process,” in Proceedings of the 9th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education, 2004, p. 233.

2169-3536 (c) 2016 IEEE. Translations and content mining are permitted for academic research only. Personal use is also permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.