A Particle Swarm Optimization Approach to Composing ... - CiteSeerX

8 downloads 43581 Views 668KB Size Report
In this paper, a particle swarm optimization-based approach is proposed to ..... As such the initial swarm contains solutions that have good objective .... The entire test sheet is presented in one Web page with a scroll bar for .... For further application, collaborative plans with some local e-learning companies are proceeding, ...
Yin, P.-Y., Chang, K.-C., Hwang, G.-J., Hwang, G.-H., & Chan, Y. (2006). A Particle Swarm Optimization Approach to Composing Serial Test Sheets for Multiple Assessment Criteria. Educational Technology & Society, 9 (3), 3-15.

A Particle Swarm Optimization Approach to Composing Serial Test Sheets for Multiple Assessment Criteria Peng-Yeng Yin and Kuang-Cheng Chang Department of Information Management, National Chi Nan University, Pu-Li, Nan-Tou, Taiwan 545, R.O.C.

Gwo-Jen Hwang Department of Information and Learning Technology, National University of Tainan, 33, Sec. 2 Shulin St.,Tainan city 70005, Taiwan, R.O.C. [email protected] Tel: 886-915396558 Fax: 886-6-3017001

Gwo-Haur Hwang Information Management Department, Ling Tung University, Taichung, Taiwan 40852, R.O.C.

Ying Chan Graduate Institute of Educational Policy and Leadership, Tamkang University Tamsui, Taipei County, Taiwan 251, R.O.C.

ABSTRACT To accurately analyze the problems of students in learning, the composed test sheets must meet multiple assessment criteria, such as the ratio of relevant concepts to be evaluated, the average discrimination degree, difficulty degree and estimated testing time. Furthermore, to precisely evaluate the improvement of student’s learning performance during a period of time, a series of relevant test sheets need to be composed. In this paper, a particle swarm optimization-based approach is proposed to improve the efficiency of composing near optimal serial test sheets from very large item banks to meet multiple assessment criteria. From the experimental results, we conclude that our novel approach is desirable in composing near optimal serial test sheets from large item banks and hence can support the need of evaluating student learning status.

Keywords Computer-assisted testing, serial test-sheet composing, particle swarm optimization, computer-assisted assessment

1. Introduction As the efficiency and efficacy of the deployment of computer-based tests have been confirmed by many early studies, many researchers in both technical and educational fields have engaged in the development of computerized testing systems (Fan et al., 1996; Olsen et al., 1986). Some researchers have even proposed computerized adaptive testing, which uses prediction methodologies to shorten the length of the test sheets without sacrificing their precision (Wainer, 1990). A well-scrutinized test is helpful for teachers wanting to verify whether students well digest relevant knowledge and skills and for recognition of students’ learning bottlenecks (Hwang et al., 2003a). In a computerized learning environment, which provides students with greater flexibility during the learning process, information concerning the student learning status is even more important (Hwang, 2003a). The key to a good test depends not only on the subjective appropriateness of test items, but also on the way the test sheet is constructed. To continuously evaluate the learning performance of a student, it is usually more desirable to compose a series of relevant test sheets to meet a predefined set of assessment criteria such that those test sheets in the same series will not contain identical test items (or contain only an acceptable percentage of overlapped test items). Because the number of test items in an item bank is usually large and the number of feasible combinations to form test sheets thus grows exponentially, an optimal test sheet takes enormous time to build up (Garey & Johnson, 1979). Previous investigation has even shown that a near-optimal solution is difficult to find when the number of candidate test items is larger than five thousand (Hwang et al., 2003b), not to mention the composition of a series

ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at [email protected].

3

of relevant test sheets from larger item banks for evaluating the improvement of student’s learning performance during a period of time. To cope with the problem in composing optimal serial test sheets from large item banks, a particle swarm optimization (PSO)-based algorithm (Kennedy and Eberhart, 1995) is proposed to optimize the selection of test items to compose serial test sheets. By employing this novel approach, the allocation of test items in each of the serial test sheets will meet the needs of multiple criteria, including the expected testing time, the degree of difficulty, the expected ratio of unit concepts, and the acceptable percentage of overlapped test items among test sheets to approximate the optimal allocation. Based on this approach, an Intelligent Tutoring, Testing and Diagnostic (ITED III) system has been developed. Experimental results indicated that the proposed approach is efficient and effective in generating near-optimal compositions of serial test sheets that satisfy the specified requirements.

2. Background and Relevant Researches In recent years, researchers have developed various computer-assisted testing systems to more precisely evaluate student’s learning status. For example, Feldman and Jones (1997) attempted to perform semi-automatic testing of student software using Unix systems; Rasmussen, et al. (1997) proposed a system to evaluate student learning status on computer networks while taking Feldman and Jones’ progress into consideration. Additionally, Chou (2000) proposed the CATES system, which is an interactive testing system developed in a collective and collaborative project with theoretical and practical research on complex technology-dependent learning environments. Unfortunately, although many computer-assisted testing systems have been proposed, few of them have addressed the problem of finding a systematic approach for composing test sheets that satisfy multiple assessment requirements. Most of the existing systems construct a test sheet by manually or randomly selecting test items from their item banks. Such manual or random test item selection strategies are inefficient and are unable to meet multiple assessment requirements simultaneously. Some previous investigations showed that a well-constructed test sheet not only helps in the evaluation of student’s learning status, but also facilitates the diagnosis of the problems embedded in the learning process (Hwang, 2003a; Hwang et al. 2003a; Hwang 2005). Selecting proper test items is very critical to constitute a test sheet that meets multiple assessment criteria, including the expected time needed for answering the test sheet, the number of test items, the specified distribution of course concepts to be learned, and, most importantly, the maximization of the average degree of discrimination (Hwang et al, 2005). Since satisfying multiple requirements (or constraints) when selecting test items is difficult, most computerized testing systems generate test sheets in a random fashion. Hwang et al. (2003b) proposed a multiple-criteria where test sheet-composing problem is formulated as a dynamic programming model (Hillier and Lieberman, 2001) to minimize the distance between the parameters (e.g., discrimination, difficulty, etc.) of the generated test sheets and the objective values subject to the distribution of concept weights. A critical issue arising from the use of a dynamic programming approach is the exceedingly long execution time required for producing optimal solutions. As the time-complexity of the dynamic programming algorithm is exponential in terms of input data, the execution time will become unacceptably long if the number of candidate test items is large. Consequently, Hwang et al. (2005) attempted to solve the test sheet-composing problem by optimizing the discrimination degree of the generated test sheets with a specified range of assessment time and some other multiple constraints. Nevertheless, in developing an e-learning system, it is necessary to conduct a long-term assessment of each student; that is, only optimizing a test sheet is not enough for such long-term observation for the student. Therefore, a series of relevant test sheets with multiple assessment criteria need to be composed for such a continuously learning performance evaluation. As the problem is much more difficult than that of composing a single test sheet, a more efficient and effective approach is needed. In this paper, a particle swarm optimizationbased algorithm is proposed to find quality approximate solutions in an acceptable time. A series of experiments will be also presented to show the performances of the novel approach.

3. Problem Description In this section, a mixed integer programming model (Linderoth and Savelsbergh, 1999) is presented to formulate the underlying problem. In order to conduct a long-term observation on the student’s learning status, a series of K relevant test sheets will be composed. The model aims at minimizing the differences between the average 4

difficulty of each test sheet and the specified difficulty target, with a specified range of assessment time and some other multiple constraints. Assume K serial test sheets with a specific difficulty degree will be composed out of a test bank consisting of N items, Q1, Q2, …, QN. To compose test sheet k, 1 ≤ k ≤ K, a subset of nk candidate test items will be selected. Assume that in total M concepts will be involved in the K tests. With the specified course concepts to be learned, say Cj, 1 ≤ j ≤ M , each test item is relevant to one or more of them. For example, to test the multimedia knowledge of students, Cj might be “MPEG”, “Video Streaming” or “Videoon-Demand”. We shall call this problem the STSC (Serial Test Sheets Composition) problem. In the STSC problem, we need confine the similarities between each pair of tests. Such a constraint imposed upon each pair of tests k and l, 1 ≤ k, l ≤ K, is specified by parameter f, which indicates that any two tests can have at most f items in common. The variables used in this model are given as follows: ¾ Decision variables xik, 1 ≤ i ≤ N and 1 ≤ k ≤ K: xik is 1 if test item Qi is included in test sheet k; 0, otherwise. ¾ Coefficient di, 1 ≤ i ≤ N : degree of difficulty of Qi. ¾ Coefficient D, target difficulty level for each of the serial test sheets generated. ¾ Coefficient rij, 1 ≤ i ≤ N , 1 ≤ j ≤ M : degree of association between Qi and concept Cj. ¾ ¾ ¾ ¾ ¾

Coefficient ti, 1 ≤ i ≤ N : expected time needed for answering Qi. Right hand side hj, 1 ≤ j ≤ M: lower bound on the expected relevance of Cj for each of the K test sheets. Right hand side l: lower bound on the expected time needed for answering each of the K test sheets. Right hand side u: upper bound on the expected time needed for answering each of the K test sheets. Right hand side f: the maximum number of identical test items between two composed test sheets;

Formal definition of the STSC Model:

⎛ ⎜ ⎜ Minimize Zk = ⎜ ∑ ⎜ 1≤ k ≤ K ⎜ ⎝ subject to: N

∑r x ij

ik

i =1 N

N



p

d i x ik

i =1 N



− D x ik

i =1

⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠

1

p

≥ h j , 1 ≤ j ≤ M , 1 ≤ k ≤ K;

(1)

∑t x

ik

≥ l , 1 ≤ k ≤ K;

(2)

∑t x

ik

≤ u, 1 ≤ k ≤ K ;

(3)

i

i =1 N

i

i =1 N

∑x x

ij ik

≤ f , 1 ≤ j ≠ k ≤ K;

(4)

i =1

In the above formula, constraint set (1) indicates the selected test items in each generated test sheet must have a total relevance no less than the expected relevance to each concept assumed to be covered. Constraint sets (2) and (3) indicate that total expected test time of each generated test sheet must be in its specified range. Constraint set (4) indicates that no pair of test sheets can contain more than f identical test items.

⎛ ⎜ ⎜ In the objective function, Zk = ⎜ ∑ ⎜ 1≤ k ≤ K ⎜ ⎝

N



p

d i x ik

i =1 N



i =1

− D x ik

⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠

1

p

is the p-norm between the average

difficulty degree of each test sheet from the target difficulty degree specified by the teacher. In particular, the objective function indicates the absolute distance when p = 1, and it calculates the root squared distance when p = 2. Therefore, the objective of this model seeks to select a number of test items such that the average difficulty 5

of each generated test sheet is closest to the target difficulty value D. Without loss of generality, we let p = 2 for simulation. The computation complexity for obtaining the optimal solution to the STSC problem is analyzed as follows. The number of possible combinations of test items for composing a single test sheet is ⎛⎜ N ⎞⎟ where Ω is the range

∑⎜ i∈Ω

⎟ ⎝i⎠

for number of test items that could be answered within the specified time frame [l, u], while the parameters hj and f will affect the number of feasible solutions in those combinations. For composing K serial test sheets, it K requires a computation complexity of ⎛⎜ ⎛⎜ ⎛ N ⎞ ⎞⎟ ⎞⎟ which is extremely high. Hence, seeking the optimal solution O⎜ ∑ ⎜ ⎟ ⎟ ⎜ ⎜⎝ i∈Ω ⎜⎝ i ⎟⎠ ⎟⎠ ⎟ ⎠ ⎝

to the STSC problem is computationally prohibitive.

4. PSO-based Algorithm for Serial Test Sheet Composition Linderoth and Savelsbergh (1999) conducted a comprehensive computational study manifesting that the mixed integer programming problems are NP-hard, which implies that composing optimal serial test sheets from a large item bank is computationally prohibitive. To cope with this difficulty, a particle swarm optimization (PSO)based algorithm is proposed to find quality approximate solutions with reasonable time. A. STSCPSO (Serial Test Sheets Composition with PSO) Algorithm The PSO algorithm was developed by Kennedy and Eberhart (1995). It is a biologically inspired algorithm which models the social dynamics of bird flocking and fish schooling. Ethologists find that a swarm of birds/fishes flock synchronously, change direction suddenly, scatter and regroup iteratively, and finally stop on a common target. The collective intelligence from each individual not only increases the success rate for food foraging but also expedites the process. The PSO algorithm facilitates simple rules simulating bird flocking and fish schooling and can serve as an optimizer for nonlinear functions. Kennedy and Eberhart (1997) further presented a discrete binary version of PSO for combinatorial optimization where the particles are represented by binary vectors of length d and the velocity represents the probability that a decision variable will take the value 1. PSO has delivered many successful applications (Eberhart and Shi, 1998; Yoshida et al., 1999; Shigenori et al., 2003). The convergence and parameterization aspects of the PSO have also been discussed thoroughly (Clerc and Kennedy, 2002; Trelea, 2003). In the followings, a PSO-based algorithm, STSCPSO (Serial Test Sheets Composition with PSO approach), is proposed to find quality approximate solutions for the STSC problem. Input: N test items Q1, Q2, …, Qn, M concepts C1, C2, …, Cm, the target difficulty level D, and the number of required test sheets, K. Step 1. Generate initial swarm Since all decision variables of the STSC problem take binary values (either 0 or 1), a particle in the STSCPSO algorithm can be represented by x = x11 x 21 ⋅ ⋅ ⋅ x N 1 x12 x 22 ⋅ ⋅ ⋅ x N 2 ⋅ ⋅ ⋅ x1K x2 K ⋅ ⋅ ⋅ x NK , which is a vector of NK

[

]

binary bits where xik is equivalent to 1 if test item Qi is included in test sheet k and 0 otherwise. Due to the constrains (2) and (3) with test time, the number of selected items in any test sheet is bounded in [l max{t i }, u min{t i }] . Hence, we should enforce the integrity rule: i =1~ N

i =1~ N N i =1 ik

l max{t i } ≤ ∑ x ≤ u min{t i }, ∀k = 1,2,..., K , during every step of our algorithm. To generate the i =1~ N

i =1~ N

initial swarm, we randomly determine the number of items for each test sheet according to the integrity rule. The selection probability of each item is based on the selection rule which gives higher selection probability to the items that have closer difficulty level to the target. In particular, the selection probability of item Qi is defined as

(S − d

i

− D ) S where S is a constant. As such the initial swarm contains solutions that have good objective

6

values but may violate constraint sets. Then the particle swarm evolves to quality solutions that not only optimize the objective function but also meet all of the constraint sets. Step 2. Fitness evaluation of particles The original objective function of the STSC problem measures the quality of a candidate solution which meets all the constraints (1)-(4). However, the particles generated by the PSO-based algorithm may violate one or more of these constraints. To cope with this problem, the merit of a particle is evaluated by incorporating penalty terms into the objective function if any constraint is violated. The penalty terms corresponding to separate constraints are described as follows. ¾ α penalty for violating concept relevance bound constraint K

M



N





i =1



α = ∑∑ ⎜ h j − ∑ rij xik ⎟ . k =1 j =1

¾

This term sums up relevance deficit of selected test items to the specified relevance lower bound of each concept over all test sheets. β penalty for violating test time bound constraint K





N





N

⎞⎞

k =1





i =1





i =1

⎠⎠

β = ∑ ⎜⎜ max⎜ l − ∑ t i xik ,0 ⎟ + max⎜ 0, ∑ t i xik − u ⎟ ⎟⎟ .

¾

This term penalizes the case where the expected test times are beyond the specified lower bound or upper bound. γ penalty for violating common item constraint



N



γ = ∑ ⎜ ∑ xij xik − f ⎟ . j ≠k

⎝ i =1



This term penalizes the case where the number of common items between two different tests exceeds the threshold f. ¾

Function J(⋅) for evaluating the fitness of a particle x Minimize

J ( x ) = Z k + w1α + w2 β + w3γ .

w1, w2, and w3 denote relative weights for the three penalty terms. As such the fitness of a particle x accounts for both of quality (objective value) and feasibility (penalty terms). The smaller the fitness value, the better the particle. Step 3. Determination of pbesti and gbest using the bounding criterion In the original PSO, the fitness evaluation of particles which is a necessity for determination of pbesti and gbest is the most time-consuming part. Here we propose a bounding criterion to speed up the process. We observe that the fitness value of a particle is only used for determination of pbesti and gbest, but not directly used for velocity update. Since Zk and J(⋅) are both monotonically increasing functions, we can use the fitness of the incumbent pbesti as a fitness bound and terminate the fitness evaluation of the ith particle when the intermediate fitness value has exceeded the bound. Also, only those pbesti that have been updated at the current iteration need to be compared with gbest for its possible updating. The use of bounding criterion can save the computational time significantly. Step 4. Update of velocities and particles The updating of velocities and particle positions follow the discrete version of PSO, i.e., the velocity is scaled into [0.0, 1.0] by a transformation function S(⋅) and is used as the probability with which the particle bit takes the value 1. In this paper, we adopt the linear interpolation function, S (vij ) =

vij

2 vmax

+ 0.5 , to transform velocities

into probabilities.

7

C. An Illustrative Example Herein, an illustrative example for the STSCPSO algorithm is provided. Assume that two test sheets with target difficulty level D = 5 are required to be generated from 10 test items. The 10 test items are relevant to 3 concepts, and the relevance association (rij) between each test item and each concept is shown in Table 1. The estimated answering time (ti) and difficulty degree (di) for the 10 test items are tabulated in Table 2. Let h1 = 2, h2 = 2, h3 = 1, l = 10, u = 16, f = 3, w1 = w2 = w3 = 0.01. The algorithm proceeds as follows. Table 1. Relevance association between each test item and each concept C1 C2 C3 Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10

1 0 0 0 0 0 1 1 0 0

0 1 0 1 0 1 0 1 0 1

0 0 1 0 1 0 0 1 0 0

Table 2. Estimated answering time and difficulty degree for each test item ti di Q1 4 0.5 Q2 3 0.9 Q3 3 0.1 Q4 5 0.7 Q5 3 0.4 Q6 2 0.5 Q7 4 0.2 Q8 3 0.6 Q9 5 0.3 Q10 4 0.5 Initial swarm generation Let the algorithm proceed with a swarm of two particles. To generate the initial swarm, the range for the feasible number of selected items in a test sheet is first determined using the integrity rule: N l max{t i } ≤ ∑ xik ≤ u min{t i } . Hence, each test sheet can select 2 to 8 items from the test item bank. i =1~ N

i =1

i =1~ N

According to our particle representation scheme, each particle is represented as a binary vector with 20 bits and is generated based on the selection rule S − d i − D S which gives higher selection probability to the items

(

)

that have closer difficulty level to the target D. It is observed from Table 2 that test items Q1, Q6, and Q10 will have the highest selection probability. With the integrity and selection rules, the initial swarm can be generated as shown in the first generation in Figure 1. Particle 1 selects items Q1, Q6, and Q8 for the first test sheet, and chooses Q1, Q4, Q5, Q8, and Q10 for the second test sheet. As for particle 2, the first test sheet consists of Q2, Q4, Q8, and Q10 and the second test sheet is composed of Q1, Q3, Q7, and Q9.

Generation 1:

Generation 2:

test sheet 1

test sheet 2

Zk

α

β

γ

J

particle 1 1000010100

1001100101

0.05

0

3

0

0.08

particle 2 0101000101

1010001010

0.285

3

0

0

0.315

particle 1 0100010100

1000100101

>0.08

-

-

-

-

particle 2 1000000100 1000101101 0.078 1 5 Figure. 1. Swarm evolution and the corresponding fitness evaluation

0

0.138 8

Particle fitness evaluation The particle fitness evaluation function

J ( x ) = Z k + w1α + w2 β + w3γ consists of objective value and

penalty terms which can be easily computed. In particular, particle 1 attains an objective value of 0.05 and incurs β penalty of 3 because the expected test time exceeds the upper limit, resulting a fitness value of 0.08. While particle 2 has an objective value of 0.285 and incurs α penalty of 3 due to the deficit of concept relevance. The fitness of particle 2 is thus 0.315. For the initial swarm, the personal best experience (pbesti) is the current particle itself. Since the fitness value of particle 1 is smaller, it is considered as gbest. The fitness values of pbesti and gbest will be used as bounds to expedite the process of next generation. Update of velocities and particles As for particle 1, the incumbent particle is equivalent to pbesti and gbest, resulting in the same vij values as previous ones and thus the same probabilities. Only a very small number of bits will be changed. Assume that particle 1 replaces Q1 by Q2 for the first test sheet and removes Q4 for the second test sheet (see generation 2 in Figure 1). For the case of particle 2, pbesti is the particle itself, but gbest is equivalent to particle 1, vij will be changed at the bits where pbesti and gbest are different. In essence, particle 2 will be dragged, to some extent, toward particle 1 in the discrete space. Assume that particle 2 becomes [10000001001000101101]. Use of bounding criterion for determining pbesti and gbest Now we proceed with the fitness evaluation for the two particles. For the case of particle 1, we find that the intermediate objective value during computation has already exceeded the current bound (0.08); hence, the computation is terminated according to the bounding criterion. There is also no need to derive penalty terms. As such the computational time is significantly reduced. As for particle 2, it attains an objective value of 0.078 and incurs α and β penalties of 1 and 5, resulting in a fitness of 0.138. Compared to incumbent pbest2 (fitness = 0.315), the fitness is improved, so pbest2 is updated to current particle 2; while gbest is not changed (fitness = 0.08). The STSCPSO algorithm iterates this process until a given maximum number of iterations has passed and gbest is considered as the best solution found by the algorithm.

5. Experiment and Discussion The STSCPSO approach has been applied to the development of an Intelligent Tutoring, Evaluation and Diagnosis (ITED III) system, which contains a large item bank for many science courses. The interface of the developed system and the experiments for evaluating the performance of the novel approach are given in the following subsections. A. System Development The teacher interface of ITED III provides a step-by-step instruction to guide teachers in defining the goal and parameters of a test. In the first step, the teacher is asked to define the type and date/time of the test. The test type might be “certification” (for performing a formal test), personal practice (for performing a test based on the to-be-enhanced concepts of each student), or group practice (for performing a test to evaluate the computer knowledge of a group of students). Each student is asked to take the test, using an assigned computer in a monitored computer room, and is allowed to answer the test sheet only within the specified test date/time range. In the following steps, the teachers are asked to define the parameters for composing the test sheets, such as the lower bound and upper bound of the expected test times (i.e., l and u), and the lower bound on the expected relevance of each concept or skill to be evaluated (i.e., hj). Figure 2 demonstrates the third step for defining a test with lower bound and upper bound of the expected test times being 60 minutes and 80 minutes, respectively. Moreover, in this step, the teacher is asked to define the lower bounds on the expected relevance of the concepts for the test, which are all set to 0.9 in the given example.

9

Figure. 2. Teacher interface for defining the goal and parameters of a test The entire test sheet is presented in one Web page with a scroll bar for moving the page up and down. After submitting the answers for the test items, each student will receive a scored test result and a personalized learning guide, which indicates the learning status of the student for each computer skill evaluated. Figure 3 is an illustrative example of a personalized learning guide for evaluating the skills of web-based programming. Such a learning guidance has been shown to be helpful to the students in improving their learning performance if remedial teaching or practice can be conducted accordingly (Hwang, 2003a; Hwang et al. 2003a; Hwang 2005).

Figure. 3. Illustrative example of a personalized learning guide 10

B. Experimental Design To evaluate the performance of the proposed STSCPSO algorithm, a series of experiments have been conducted to compare the execution times and the solution quality of three competing approaches: STSCPSO algorithm, Random Selection with Feasible Solution (RSFS), and exhaustive search. The RSFS program generates the test sheet by selecting test items randomly to meet all of the constraints, while the exhaustive search program examines every feasible combination of the test items to find the optimal solution. The platform of the experiments is a personal computer with a Pentium IV 1.6 GHz CPU, 1 GB RAM and 80G hard disk with 5400RPM access speed. The programs were coded with C# Language. To analyze the comparative performances of the competing approaches, twelve item banks with number of candidate items ranging from 15 to 10,000 were constructed by randomly selecting test items from a computer skill certification test bank. Table 3 shows the features of each item bank. Item bank 1 2 3 4 5 6 7 8 9 10 11 12

Table 3. Description of the experimental item banks Number of test Average Average expected answer time items difficulty of each test item (minutes) 15 0.761525 3.00000 20 0.765460 3.25000 25 0.770409 3.20000 30 0.758647 2.93333 40 0.720506 2.95000 250 0.741738 2.94800 500 0.746789 2.97600 1000 0.751302 2.97200 2000 0.746708 3.00450 4000 0.747959 3.00550 5000 0.7473007 2.99260 10000 0.7503020 3.00590

The experiment is conducted by applying each approach twenty times on each item bank with the objective values (Zk) and the average execution time recorded. The lower bounds and upper bounds of testing times are 60 and 120 minutes, respectively, and the maximal number of common test items between each pair of test sheets is 5. To make the solutions hard to obtain, we set the target difficulty level D = 0.5 which sufficiently deviates from the average difficulty of item banks. The STSCPSO algorithm is executed with 10 particles for 100 generations. The execution time of RSFS is set the same as that of the STSCPSO algorithm, while the maximal execution time of the exhaustive search is set to 7 days to obtain the optimal solution.

N 15 20 25 30 40 250 500 1000 2000 4000 5000 10000

Table 4 Experimental results STSCPSO RSFS Average Zk Average Zk Time (sec) Time (sec) 0.44 50 0.62 50 0.44 63 0.70 63 0.44 80 0.66 80 0.44 102 0.68 102 0.40 134 0.76 134 0.36 815 0.54 815 0.28 1805 0.68 1805 0.22 3120 0.46 3120 0.18 6403 0.60 6403 0.16 12770 0.62 12770 0.16 15330 0.54 15330 0.14 21210 0.56 21210

Optimum Solution Zk Average Time (day) 0.42 2 days > 7 days -

11

Table 4 shows the experimental results of objective values (Zk) and execution times using the three methods. We observe that the exhaustive search method can only obtain the optimal solution for the smallest test item bank with N = 15 since the computation complexity of the STSC problem is extremely high as described in Section III. As for the STSCPSO algorithm and the RSFS, approximate solutions to all of the test item banks can be obtained with reasonable times ranging from 50 seconds to 5.89 hours, but the solution quality delivered by the STSCPSO algorithm is significantly better. In particular, for N = 15, the objective value obtained by the STSCPSO algorithm is 0.44 which is very close to the optimal value (0.42), while the objective value obtained by the RSFS is 0.62. For the other cases with larger item banks, the superiority of the STSCPSO algorithm over the RSFS becomes more prominent as the size of the item banks increases. Figure 4 shows the variations of the objective value obtained using the STSCPSO algorithm and the RSFS. The objective value derived by the RSFS fluctuates, while the objective value derived by the STSCPSO algorithm constantly decreases since more candidate test items can be selected to construct better solutions as the test bank is larger.

Figure. 4. Variations of the objective value as the size of test item banks increases Figure 5 shows the fitness value obtained by gbest of the STSCPSO algorithm as the number of generations increases for the test item bank with N = 1000. We observe that the global swarm intelligence improves with a decreasing fitness value as the evolution proceeds. This validates the feasibility of the proposed particle representation and the fitness function fits the STSC problem scenario.

Figure. 5. Variations of the objective value as the size of test item banks increases

12

To analyze the convergence behavior of the particles, we testify whether the swarm evolves to the same optimization goal. We propose the information entropy for measuring the similarity convergence among the particles as follows. Let pij be the binary value of the jth bit for the ith particle, i = 1, 2, …, R, and j = 1, 2, …, NK, where R is the swarm size. We can calculate probj as the conditional probability that value one happens at the jth bit given the total number of bits that take value one in the entire swarm as follows.

∑ p = ∑ ∑ R

prob j

i =1

R

i =1

ij

NK

.

p h =1 ih

The particle entropy can be then defined as

Entropy = −∑ j =1 prob j log 2 ( prob j ) . NK

The particle entropy is smaller if the probability distributions are denser. As such, the variations of particle entropy during the swarm evolution measure the convergence about the similarity among all particles. If the particles are highly similar to one another, the values of the non-zero probj would be high, resulting in denser probability distributions and less entropy value. This also means the swarm particles reach the consensus about which test items should be selected for composing the test sheets. Figure 6 shows the variations of particle entropy as the number of generations increases. It is observed that the entropy value drops drastically during the first 18 generations since the particles exchange information by referring to the swarm’s best solution. After this period, the entropy value is relatively fixed due to the good quality solutions found and the high similarity among the particles, meaning the particles are resorting to the same high quality solution as the swarm converges.

Figure 6. The particle entropy as the number of generations increases

6. Conclusions and Future work In this paper, a particle swarm optimization-based approach is proposed to cope with the serial test sheet composition problems. The algorithm has been embedded in an intelligent tutoring, evaluation and diagnosis system with large-scale test banks that are accessible to students and instructors through the World-Wide Web. To evaluate the performance of the proposed algorithm, a series of experiments have been conducted to compare the execution time and the solution quality of three solution-seeking strategies on twelve item banks. Experimental results show that serial test sheets with near-optimal average difficulty to a specified target value can be obtained with reasonable time by employing the novel approach. For further application, collaborative plans with some local e-learning companies are proceeding, in which the present approach is used in the testing and assessment of students in elementary school and junior high schools.

13

Acknowledgement This study is supported in part by the National Science Council of the Republic of China under contract numbers NSC-94-2524-S-024-001 and NSC 94-2524-S-024 -003.

References Chou. C. (2000). Constructing a computer-assisted testing and evaluation system on the World Wide Web-the CATES experience. IEEE Transactions on Education, 43 (3), 266-272. Clerc, M., & Kennedy, J. (2002). The particle swarm explosion, stability, and convergence in a multidimensional complex space. IEEE Transaction on Evolutionary Computation, 6, 58-73. Eberhart, R. C., & Shi, Y. (1998). Evolving artificial neural networks. In Proceedings of the International Conference on Neural Networks and Brain, PL5-PL13. Fan, J. P., Tina, K. M., & Shue, L. Y. (1996). Development of knowledge-based computer-assisted instruction system. The 1996 International Conference Software Engineering: Education and Practice, Dunedin, New Zealand. Feldman, J. M., & Jones, J. Jr. (1997). Semiautomatic testing of student software under Unix(R). IEEE Transactions on Education, 40 (2), 158-161. Garey, M. R., & Johnson, D. S. (1979). Computers and Intractability: A Guide to the Theory of NPCompleteness, San Francisco, CA: Freedman. Hillier, F. S., & Lieberman, G. J. (2001). Introduction to Operations Research, 7th Ed., New York: McGrawHill. Hwang, G. J. (2003a). A concept map model for developing intelligent tutoring systems. Computers & Education, 40 (3), 217-235. Hwang, G. J. (2003b). A Test Sheet Generating Algorithm for Multiple assessment criteria. IEEE Transactions on Education, 46 (3), 329-337. Hwang, G. J., Hsiao, J. L., & Tseng, J. C. R. (2003a). A Computer-Assisted Approach for Diagnosing Student Learning Problems in Science Courses. Journal of Information Science and Engineering, 19 (2), 229-248. Hwang, G. J., Lin, B. M. T., & Lin, T. L. (2003b). An effective approach to the composition of test sheets from large item banks. 5th International Congress on Industrial and Applied Mathematics, Sydney, Australia, 7-11 July, 2003. Hwang, G. J. (2005). A Data Mining Algorithm for Diagnosing Student Learning Problems in Science Courses. International Journal of Distance Education Technologies, 3 (4), 35-50. Hwang, G. J., Yin, P. Y., Hwang, G. H., & Chan, Y. (2005). A Novel Approach for Composing Test Sheets from Large Item Banks to Meet Multiple Assessment Criteria. The 5th IEEE International Conference on Advanced Learning Technologies, Kaohsiung, Taiwan, July 5-8, 2005. Kennedy, J., & Eberhart, R. C. (1995). Particle swarm optimization. In Proceedings of the IEEE International Conference on Neural Networks, IV, 1942-1948. Kennedy, J., & Eberhart, R. C. (1997). A discrete binary version of the particle swarm algorithm. In Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics, 4104-4108. Linderothm J. T., & Savelsbergh, M. W. P. (1999). A computational study of search strategies for mixed integer programming. INFORMS Journal on Computing, 11 (2), 173-187.

14

Olsen, J. B., Maynes, D. D., Slawson, D., & Ho, K. (1986). Comparison and equating of paper-administered, computer-administered and computerized adaptive tests of achievement. The Annual Meeting of American Educational Research Association, California, April 16-20, 1986. Rasmussen, K., Northrup, P., & Lee, R. (1997). Implementing Web-based instruction. In Web-Based Instruction, Khan, B. H. (Ed.), Englewood Cliffs, NJ: Educational Technology, 341–346 Shigenori, N., Takamu, G., Toshiku, Y., & Yoshikazu, F. (2003). A hybrid particle swarm optimization for distribution state estimation. IEEE Transaction on Power Systems, 18, 60-68. Trelea, I. C. (2003). The particle swarm optimization algorithm: convergence analysis and parameter selection. Information Processing Letters, 85, 317-325. Wainer, H. (1990). Computerized Adaptive Testing: A Primer, Lawrence Erlbaum Associates, Hillsdale, NJ. Yoshida, H., Kawata, K., Fukuyama, Y., & Nakanishi, Y. (1999). A particle swarm optimization for reactive power and voltage control considering voltage stability. In Proceedings of the International Conference on Intelligent System Application to Power Systems, 117-121.

15