Technology for Testing Nondeterministic Client/Server Database ...

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING,

VOL. 30,

NO. 1,

JANUARY 2004

1

Technology for Testing Nondeterministic Client/Server Database Applications Gwan-Hwan Hwang, Sheng-Jen Chang, and Huey-Der Chu Abstract—The execution of a client/server application involving database access requires a sequence of database transaction events (or, T-events), called a transaction sequence (or, T-sequence). A client/server database application may have nondeterministic behavior in that multiple executions thereof with the same input may produce different T-sequences. In this paper, we present a framework for testing all possible T-sequences of a client/server database application. We first show how to define a T-sequence in order to provide sufficient information to detect race conditions between T-events. Second, we design algorithms to change the outcomes of race conditions in order to derive race variants, which are prefixes of other T-sequences. Third, we develop a prefix-based replay technique for race variants derived from T-sequences. We prove that our framework can derive all the possible T-sequences in cases where every execution of the application terminates. A formal proof and an analysis of the proposed framework are given. We describe a prototype implementation of the framework and present experimental results obtained from it. Index Terms—Concurrent programming, reachability testing, client/server, database management system.

æ 1

INTRODUCTION

G

speaking, a client/server program is a type of distributed program (or concurrent program). Let P be a distributed program. Multiple executions of P with the same input may produce different results—this is called nondeterministic behavior [1], [2]. Because of this, when testing P with input X which is a sequence of inputs for processes in P, a single execution is insufficient to determine the correctness of P with X. Even if P with input X has been executed successfully many times, it is possible that a future execution of P with X will produce incorrect results. An execution of a distributed program exercises a sequence of synchronization events called a synchronization sequence (or SYN-sequence). Examples of process synchronization include P and V primitives applied to a shared semaphore, monitor-entry procedures, send/receive message primitives, and general sharing of memory [3], [4]. The distributed program (or concurrent program) exhibits nondeterministic behavior because different executions of P with the same input X may produce distinct SYN-sequences. The meanings of some terms are not standard. In this paper, a test of P with input X is to execute P with input X once to obtain a SYN-sequence and check the result of execution. Duplicating test means that different tests of P with input X produce the same SYN-sequence. Exhaustive testing of P ENERALLY

. G.-H. Hwang is with the Department of Information and Computer Education, National Taiwan Normal University, 162, Hoping E RD. Sec. 1, Taipei, Taiwan 106. E-mail: [email protected]. . S.-J. Chang is with Telecommunications Laboratories, Chunghwa Telecom Co., Ltd., No. 12, Lane 551, Min-Tsu Road Sec. 5 Yang-Mei, Taoyuan, Taiwan 326. E-mail: [email protected]. . H.-D. Chu is with the Department of Management Information Systems, Takming College, #56, Huan Shan Rd., Sec. 1, Ney Hwu, Taipei, Taiwan 114. E-mail: [email protected]. Manuscript received 1 Oct. 2002; revised 6 Nov. 2003; accepted 14 Nov. 2003. Recommended for acceptance by J. Offutt. For information on obtaining reprints of this article, please send e-mail to: [email protected], and reference IEEECS Log Number 117488. 0098-5589/04/$20.00 ß 2004 IEEE

with input X is to conduct a lot of tests which exercise all feasible SYN-sequences of P with input X. In this paper, we focus on the problem of testing client/ server SQL database applications that exhibit nondeterministic behavior. The client interacts with the servers using the SQL [5], which is a powerful set-oriented language consisting of a few commands; it was created as a language for databases that adhere to the relational model [6]. Consider a client/server SQL application P with input X. The X contains input for each client as well as the initial state of each SQL database. During the execution of P, clients send SQL transactions to database servers concurrently. Whenever there are two or more SQL transactions, sharing the same portion of data in the same database server, a race condition may occur. Executing these SQL transactions in different orders, i.e., with different race outcomes, may produce different results. Thus, our target programs are distributed programs in which multiple clients race for data in relational databases. We present a framework using the reachability testing scheme to test SQL client/server applications. Our scheme provides a way of overcoming the nondeterministic behavior. If every execution of P with input X terminates, reachability testing of P with input X can accomplish exhaustive testing of P with input X and, thus, can determine the correctness of P with X. However, existing methods for reachability testing can only be applied to the testing of shared-memory [7] and asynchronous-messagepassing [8], [9], [10] distributed programs. First, we design the data structure of the SYN-sequence for race analysis of SQL transactions. It is the transaction sequence (T-sequence) that consists of many transaction events (T-events). The T-sequence can be used to perform a prefix-based replay in the reachability testing. The prefixbased replay is to start the execution at a state other than the initial state. We believe that the T-sequence is the most general form of the defined SYN-sequences [7], [8], [11]. Published by the IEEE Computer Society

2


Since the SQL transaction actually has the most complicated data-access models, the T-sequence can also handle other types of synchronization model, including shared-memory, semaphore, monitoring, and message-passing models. Second, we design new algorithms to derive race variants from a T-sequence. Because the data-access models of shared memory and message passing are far simpler than the SQL database, the algorithms for deriving race variants presented in [7], [8], [9], [10] can no longer be applied. We present two algorithms in this paper. The first one is a modified version of the old algorithm presented in [7]. It generates a race-variant diagram (RV diagram) to derive race variants. The second is a novel algorithm that analyzes the order of T-events in a T-sequence to generate race variants. Formal proofs and complexity analyses are given for both algorithms. The experimental results show that the second algorithm is much more efficient than the first one, and we consider it a significant breakthrough in reachability-testing technology. Third, we develop a scheme for performing prefix-based replay for an SQL client/server database program. We present two approaches: the first one is based on the intrinsic synchronization mechanism of an SQL database server, and the other is based on thread synchronization. We have implemented the entire architecture and have performed many experiments on it. The experimental results show that the scheme we propose in this paper is a practical method for testing client/server database applications. In addition, the codes we implemented for the experiments can be obtained via the Internet (see Section 7). This paper is organized as follows: Section 2 surveys previous work on database testing and several approaches to the testing of distributed programs. Section 3 discusses how to performa race analysis for SQL transactions. In Section 4, we present two approaches for deriving race variants from a T-sequence. Section 5 presents two approaches for prefix-based replay. Section 6 presents the results of experiments and Section 7 concludes this paper.

2

PREVIOUS WORK AND TESTING FOR DISTRIBUTED PROGRAMS

APPROACHES

No techniques targeted specifically toward the nondeterministic behavior of database applications have been described in the software-testing research literature. However, some studies have investigated techniques for generating automated and semiautomated tests for database systems and applications. The work of Davies et al. [12] on dataconstraint testing investigated the feasibility of automatically producing test data to a database for load testing and establishing the integrity of the stored data. They proposed a scheme to check if the database has the ability to store, retrieve, update, and delete records correctly. The work in [13] proposed a testing technique to help assure that database programs meet a user’s specification. Some other work has assessed the performance of database management systems rather than tested applications for correctness [14], [15], [16], [17].

VOL. 30,

NO. 1,

JANUARY 2004

Fig. 1. Reachability testing.

The only related work is that by Chu and Dobson [18], which involved a client/server database program that exhibited nondeterministic behavior. But, they did not propose any solution. However, there has been a lot of work on testing technologies for ordinary nondeterministic concurrent programs that do not involve database access. The remainder of this section provides a brief overview of this work. Nondeterministic testing (also called multipleexecution testing) of a concurrent program P involves the following steps: 1) Select a set of inputs of P, and 2) for each selected input X, execute P with X many times and examine the result of each execution. Nondeterministic testing of P with input X has two major problems; one is that some feasible SYN-sequences of P with input X may be executed many times, and the other is that some feasible sequencesmay never be executed [7], [19]. Deterministic testing of a concurrent program P involves the following steps: 1) Select a set of tests, each of the form (X,S), where X and S are an input and a SYN-sequence of P, respectively, and 2) for each selected test (X,S), perform a deterministic execution of P with input X according to S. The forced execution determines whether S is feasible for P with input X. Deterministic testing of P allows the use of SYN-sequences selected according to the implementation and specification of P. However, deterministic testing has additional problems to solve. One major problem is deciding which pairs of inputs and SYN-sequences to select for a concurrent program. A number of methods for solving this problem have been proposed [20], [21], [22]. Fig. 1 illustrates the concept of reachability testing. Assume that S is the SYN-sequence of an execution of P with input X. Reachability testing of P with input X and SYN-sequence S involves the following steps: 1.

2.

Use S to derive a set of prefixes of other feasible SYN-sequences of P with input X. Such prefixes (race variants of S) are derived by changing the outcome of race conditions in S. An execution which follows a race variant of S will always exercise a SYN-sequence which is different from S. For each new race variant derived in Step 1, perform a prefix-based replay of P with input X and the race variant to execute and collect an additional SYN-sequence for P with input X. The prefix-based replay of P with a race variant R includes two phases:

HWANG ET AL.: TECHNOLOGY FOR TESTING NONDETERMINISTIC CLIENT/SERVER DATABASE APPLICATIONS

Replay phase: replay all the synchronization events in R. b. Monitor phase: subsequent synchronization events are recorded after replaying R. 3. For each new SYN-sequence collected in Step 2, repeated Steps 1 and 2. Note that R is not a complete SYN-sequence of an execution of P. Thus, after the replaying of R, we have to record the subsequent synchronization events to obtain a SYN-sequence of an execution of P. a.

3

RACE ANALYSIS TRANSACTIONS

FOR

SQL DATABASE

The concurrent queries of several clients to the same database may access the same portion of data. The majority of modern SQL servers are transaction servers; the client invokes remote procedures that reside on the server with an SQL database engine. The remote procedures on the server execute a group of SQL statements called a transaction, and the SQL statements contained within this group either all succeed or fail as a unit. Also, a transaction will lock the accessed data automatically. The special case of a transaction is one with only a single SQL statement. All process synchronizations in a parallel program were modeled in [11] as operations on shared data (or shared memory). It showed that this characterization of process interactions is not restrictive, since many communication and synchronization primitives can be reduced to operations on shared data. In particular, message passing can be modeled as communication through a shared port or mailbox. The work in [7] showed how to perform reachability testing of concurrent programs using read and write operations to shared memory. Let P be a concurrent program using read and write operations. Each shared variable in P is assigned a version number, which is initialized to zero and increased by one after each write operation on it. An execution of P involves two types of synchronization events: read and write. A read event is denoted as R(U,V), which refers to a read operation on variable U with the version number being V, and a write event is denoted as W(U,V), which refers to a write operation on variable U with the resulting version number being V. Thus, if W(A,1) and R(A,1) are issued by different processes in the concurrent program, we can determine that W(A,1) and R(A,1) have a race condition and that W(A,1) happened before R(A,1). In [9], Bechini and Tai presented algorithms for vector timestamps developed to determine the “happen before” relations between events of an execution of a message-passing program. The algorithms can be applied to perform race analysis. In [10], Lei and Tai improved the work in [8]. It showed how to derive all race variants for asynchronous message-passing programs for efficient searching and insertion without using prefix-based testing. However, compared to read/write access to shared memory, the data-sharing model of SQL is far more complicated. First, the data manipulation language of SQL includes the instructions INSERT, DELETE, UPDATE, SELECT, JOIN, STORED PROCEDURE, TRIGGER, RULE, and CURSOR. Second, a single SQL statement may refer to

3

TABLE 1 Car

multiple tables which contain multiple rows (as opposed to the shared-memory model, in which each read or write operation only accesses a single shared variable). Third, the shared-memory model has a fixed number of shared variables which are declared and initialized before program execution. However, with the INSERT and DELETE operations in SQL, the number of rows in any table always varies depending on the SQL transaction operations of clients. We can treat read/write access to shared memory as a special case in the data access model of SQL transaction in which each row represents a shared variable. Consider the essential data structure of an SQL database. Data are stored as rows in tables. A primary key is a field whose value must be unique for each row. It seems that the most precise way to identify the accessed rows in an SQL transaction is to store the table names as well as the primary keys of the accessed rows. However, we will show that there are some situations in which storing the primary key of each access row of an SQL transaction does not help to detect the race condition. We provide three examples to illustrate this point. The first example considers the case where two concurrent database transactions are the SQL statements INSERT and SELECT, respectively. Assume the table name is car and that column id is the primary key of this table (see Table 1). The two SQL transactions issued by two concurrent clients are T1 and T2 , where T1 is BEGIN_TRANSACTION INSERT INTO car VALUES (5, LS430, Lexus, 3,500,000) END_TRANSACTION and T2 is BEGIN_TRANSACTION SELECT * FROM car WHERE car.car_price > 600,000 END_TRANSACTION Transaction T1 inserts a row (5, LS430, Lexus, 3,500,000) into table car, and transaction T2 selects rows from car for which car_price > 600,000. Assume that these transactions are issued by two concurrent clients. If T1 is executed prior to T2 , then the primary keys of accessed rows of the SQL transaction issued by T1 and T2 are 5 and 1, 2, and 5, respectively. By observing the primary-key information in the two T-events, it is obvious that the two clients race because they access the same row, i.e., the row with primary key 5. However, if T2 is executed prior to T1 , then the primary keys of accessed rows of T1 and T2 are 5 and 1 and 2, respectively. Since f5g [ f1; 2g ¼ , the primary-key information in the T-events is not sufficient for identifying the race condition. However, since it is possible for the two SQL transactions to access the same row in the database, we should still consider that the two SQL transactions exhibit a race condition.

4


The second example is where the two concurrent database transactions are SELECT and UPDATE. The two database transactions issued by two concurrent clients are T3 and T4 , where T3 is BEGIN_TRANSACTION SELECT * FROM car WHERE car.car_price > 600,000 END_TRANSACTION and T4 is BEGIN_TRANSACTION UPDATE car SET car_price = 550,000 WHERE car_name = "Altis" END_TRANSACTION If T3 is executed prior to T4 , then the primary keys of accessed rows of the SQL transaction issued by T3 and T4 are 1 and 2, and 2, respectively. By observing the primarykey information in the two T-events, it is obvious that the two clients race because they access the same row. However, if T4 is executed prior to T3 , then the primary keys of accessed rows of the SQL transaction issued by T4 and T3 are 2 and 1, respectively. In this case, the primarykey information in the T-events does not exhibit the race condition. The above two examples show that it is inappropriate to use the primary keys of accessed rows in database transactions to decide whether two SQL transactions exhibit a race condition. This is because the accessed rows are determined by the Boolean predicates which are not related to the primary key in the WHERE clause of the SQL statement. Consider the following example where the Boolean predicates are bound to the primary key. The two database transactions issued by two concurrent clients are T5 and T6 , where T5 is BEGIN_TRANSACTION SELECT * FROM car WHERE car.id in (1,3) END_TRANSACTION and T6 is BEGIN_TRANSACTION UPDATE car SET car_price = 71,532 WHERE car.id=2 END_TRANSACTION Irrespective of whether T5 executes prior to T6 or T6 executes prior to T5 , the primary keys of the accessed rows of the SQL transaction issued by T5 and T6 are 1 and 3, and 2, respectively. Since f1; 3g [ f2g ¼ , there is no data race between the two transactions. Using the above three examples, we can design the data structures of the T-event and T-sequence so that they contain enough information to perform a race analysis. Let the client/server application P contain clients CL1 ; CL2 ; . . . ; CLn , and the SQL database comprise DB1 ; DB2 ; . . . ; DBm among all the database transaction servers, where n > 1 and m > 0 (note that there may be more than one database in a database server). An execution of P exercises a sequence of SQL transactions (the T-sequence). More specifically, the T-sequence of P is denoted as (T1 ; . . . ; Ti ; . . . ; Tx ), where Ti ; 1 i x, is a T-event. We define the T-event as the following data structure:

VOL. 30,

NO. 1,

JANUARY 2004

Ta = {Database_name, Database_transaction_order, Client_name, Client_transaction_order, [Table_name1 , Operation1 , Primary_Key_List1 ], ... [Table_namei , Operationi , Primary_Key_Listi ], ... [Table_names , Operations , Primary_Key_Lists ] } Ta is a T-event. The header of a T-event records the name of the database and the name of the requested client, which here are Database_name and Client_name, respectively. Each database is associated with a transaction number, which is initialized to one and increased by one after each transaction is performed on this database. Database_ transaction_order records the order of this transaction relative to all transactions executed on the database by different clients. The Client_transaction_order indicates the order of this transaction relative to all transactions executed by the client. After the header, there is a series of accessed table records. We have mentioned that a single SQL statement may access multiple rows in multiple tables. For the race analysis, we translate each SQL statement into multiple accessed table records. Each accessed table record is represented by Table namei , Operationi , and Primary Key Listi , 1 i s. Table namei is the table name. We summarize the row operations as having INSERT, READ, UPDATE, and DELETE instructions. Note that these four instructions are operations to rows rather than SQL statements. The combination of these four operations can represent any SQL data-manipulation statements, including static and dynamic SQL [5]. Primary Key Listi has one of following two forms: The {*} represents that the primary keys of the accessed rows depend on the status of the database and the WHERE clause in the SQL statement (e.g., SQL transactions T2 ,1 T3 , and T4 shown above). It means that an operation is possible to act on any row in a table. 2. If the primary keys of the access rows do not depend on the status of the database and the WHERE clause in the SQL statement, Primary Key Listi is a set of primary keys of the form fk1; k2; . . .g which records the primary keys of accessed rows (e.g., SQL transactions T1 , T5 , and T6 shown above). Example 1 (Fig. 2) shows an SQL transaction with five SQL statements and its corresponding T-event. In the following, we define a race relation of two T-events (see Fig. 3). Two T-events are said to exhibit a transaction race if they can access the same portion of data in the same database, and if the results of their execution may differ with their order of execution. 1.

1. Note that transactions T1 and T6 are different from the transactions in Example 1.


5

Fig. 2. Example 1.

Referring to Definition 1, conditions 1-3 mean that Tu and Tv are two SQL transactions which act on the same table in a database, but are issued by different clients. Condition 4 means that the operations are not both READ, and condition 5 determines if the two operations issued by two clients may act on the same row. Note that {*} means that an operation is able to act on any row in a table.

Fig. 3. Definition 1.

In the above discussion, we have provided examples which illustrate that the race may occur between INSERT and READ or between READ and UPDATE; two INSERT operations may also race. Consider the case where two clients C1 and C2 are trying to insert a row with the same primary key into the same table in a database. If Client C1 executes prior to Client C2, then the SQL transaction of C2

6


VOL. 30,

NO. 1,

JANUARY 2004

Fig. 4. Algorithm 1.

will fail; conversely, C1 will fail is C2 executes first. The execution order of the two INSERT operations changes the execution results. Another case is when the two operations are DELETE. Consider the case where two clients are attempting to delete the same row from the same table. The first client will delete the row successfully, after which the second client will receive a reply stating that it cannot successfully delete the row from the database server. The client may perform another transaction if it cannot successful delete the row from the table. Thus, it is obvious that the order of execution of transactions issued by the two clients affects the results. This is also a type of race between two transactions.

4

DERIVE RACE VARIANTS SEQUENCE

FROM A

TRANSACTION

Section 3 defined the format of a T-sequence and showed how to detect a transaction race between two T-events in a T-sequence. This section presents algorithms for deriving race variants from a T-sequence. Let the client/server application P contain clients CL1 , CL2 ; . . . ; CLn , and the SQL database comprise DB1 , DB2 ; . . . ; DBm among all the database transaction servers, where n > 1 and m > 0. Let TS be a feasible T-sequence of P with input X. X contains input for each client and the initial state of each database. For convenience, we define the following notation: TSðCLi Þ is the ordered set of the T-events of Client CLi , and TSðCLi ; jÞ is the jth T-event of Client CLi .

4.1 The Race Graph of a T-Sequence Before we describe how to derive race variants for a T-sequence, we show how to construct a directed partialordered graph of a T-sequence (see Fig. 4). We call this the

race graph of a T-sequence. The race graph is used to determine if there exists a race condition whose outcome can be changed. The vertices in a race graph represent T-events. The added edge between two T-events e1 and e2 represents that e1 must happen before e2 . We use the following example to illustrate Algorithm 1 (Fig. 4). Assume there are two databases, DB1 and DB2: TA1 and TA2 are tables of DB1; TA3 is a table of DB2; there are three clients C1, C2, and C3; and TS includes the following T-events: TS(C1,1) = {DB1,1,C1,1,[TA1,read,*], [TA2, read,*]} TS(C1,2) = {DB2,1,C1,2,[TA3,insert,4]}, TS(C2,1) = {DB1,2,C2,1,[TA1,insert,5]}, TS(C3,1) = {DB2,2,C3,1,[TA4,insert,5]}, TS(C2,2) = {DB2,3,C2,2, [TA3,read,*], [TA4,read,*]}, TS(C2,3) = {DB1,3,C2,3,[TA1,delete,*], [TA2,read,*]}, TS(C1,3) = {DB1,4,C1,3,[TA1,read,*], [TA2,read,*]}, We summarize the race relation between the T-events in TS in Table 2. After Steps 1 and 2 of Algorithm 1, we have the graph shown in Fig. 5a. After Step 3, we obtain the final race graph, as shown in Fig. 5b. Given a T-sequence TS, we denote the race graph of TS as Race-graph(TS). In the following, we give definitions which are related to the race graph, which will be referred to in the algorithm for deriving race variants (see Figs. 7 and 8). Note that e1 happened before e2 means not only that e1 occurred before e2 , but also that there is a causality relation between e1 and e2 [23].

TABLE 2 Race Relationships between T-Events in T-Sequence TS

Note that “yes” means there exists a transaction race between two T-events.


Fig. 5. An example of the race graph.

If f is a feasible prefix of a T-sequence TS, then f corresponds to a feasible intermediate execution state which can reach the execution state of TS. We use Fig. 6 to illustrate Definition 3. P1 is a feasible prefix of TS, but P2 is not a feasible prefix of T because TS(C2,1) happened before TS(C1,3). Since TS(C1,3) is in P2, TS(C2,1) should also be included in P2 to make P2 a feasible prefix of TS.

4.2

Generate an RV Diagram to Derive Race Variants In this paper, we propose two schemes for deriving race variants from a T-sequence TS in Sections 4.2 and 4.3, respectively. In the first scheme, we construct the RV diagram for TS, which is a tree with the path from the root to the node representing a feasible prefix or race variant of TS. The nodes in the RV diagram for TS are generated by considering all the possible interleavings of T-events. It is an n-ary tree if the number of concurrent clients is n. For a node in the RV diagram for TS, the path from the root node of the diagram to this node is a totally ordered sequence of T-events. The RV diagram scheme was first proposed in [7], which uses the version number of variable access to determine if there exists a different race outcome in the nodes of the RV diagram. However, the transaction order is insufficient for race analysis in a T-sequence. As shown in the example of Section 4.1 (Fig. 5b), two T-events TS(C1,2) = {DB2, 1, C1, 2, [TA3, insert, 4]} and TS(C3,1) = {DB2, 2, C3, 1, [TA4, insert, 5]} access the same database with different transaction orders, but do not race. The reordering of these two T-events cannot change the execution result. Thus, the



7

Fig. 6. An example illustrating the definition of feasible prefix.

algorithm for generating race variants from the read-write sequence presented in [7] cannot be applied to the T-sequence. Since the version number is insufficient for determining whether race conditions are changed, the RV diagram defined in [7] is insufficient. Algorithm 2 (see Fig. 9) is our new algorithm for generating an RV diagram based on the race graph. Each node N in the RV diagram for TS contains a client-transaction-order vector and a race graph: Client-transaction-order vector: (I1 ; I2 ; . . . ; In ), where Ij ; 1 j n, is the order number of the last T-event in the jth client that is executed for the generated node N. Race graph: This represents a race variant if N is a racevariant node; otherwise, it represents a feasible prefix of the T-sequence if N is a prefix node. The following algorithm (Fig. 9) shows how to drive race variants from a T-sequence. Algorithm 2 generates an RV diagram to simulate the execution of the issuing of T-events by clients. Each generating of a child node represents an execution of a T-event by a client. A prefix node corresponds to a feasible prefix of TS which does not have different race outcomes. A race-variant node contains a sequence of T-events which have different race outcomes. In case a node is marked, it is either a race-variant node or a prefix node which does not need to generate child nodes anymore. We use the T-sequence TS in the running example of Section 4.1 (Fig. 5b) to illustrate Algorithm 2. Fig. 10 is part of the RV diagram generated from TS. Note that TS(Ci,j) is abbreviated to Ti,j in the figure. There are two race-variant

8


VOL. 30,

NO. 1,

JANUARY 2004


nodes. Consider the prefix node with client transaction order (1,1,0). In Step 3 of Algorithm 2, observe the following: j=1: This generates a prefix node whose client transaction order is (2,1,0). Since there is already a prefix node with client transaction order (2,1,0), this prefix node is labeled “marked.”

j=2: This generates a node whose client transaction order is (1,2,0), which adds TS(C2,2) to the race graph of its own. In the target T-sequence TS, TS(C3,1) happened before TS(C2,2), but TS(C3,1) is not in its race graph. Thus, it is a race-variant node. j=3: This generates a prefix node whose client transaction order is (1,1,1). Fig. 11 shows all the race variants derived by Algorithm 2 with the T-sequence TS in the running example of Section 4.1 (Fig. 5b) as input. There is a total of 10 race variants derived: RV2-1 to RV2-10. We can easily find out that some race variants are feasible prefixes of others. For example, RV2-3 is a feasible prefix of both RV2-6 and RV2-10; RV2-10 is a feasible prefix of RV2-6. It will make the prefix-based replay of the three race variants RV2-3, RV2-6, and RV2-10 able to generate the same T-sequence. This means that during the testing process it will be possible to duplicate tests. We discuss this issue in more detail in Section 4.5.

4.3

Fig. 10. An example of the RV diagram.

Analyzing Edges in a Race Graph to Derive Race Variants In this section, we propose another algorithm for deriving race variants. Instead of generating an RV diagram, it only has to analyze the edges in the race graph of a T-sequence. The algorithm tries to reverse the direction of the edges so as to derive race variants of a T-sequence. The analysis of the edges is based on the in-edges of each T-event in the race


9

Fig. 12. In-edges and out-edges of a T-event in a race graph.

analysis and investigating the duplication of the same tests by the two algorithms.

Fig. 11. Race graphs of the race variants derived by Algorithms 2 and 3.

graph (see Definition 4 (Fig. 13) for the definitions of the inedge and out-edge). We use Fig. 12 to illustrate Definition 4. The in-edges of T-event TS(2,2) are (TS(1,1), TS(2,2)) and (TS(1,2), TS(2,2)). The out-edges of T-event TS(2,2) are (TS(2,2), TS(3,1)) and (TS(2,2), TS(3,3)). With Definition 4, we now present Algorithm 3 (see Fig. 14). For each T-event, if it is with k in-edges, the algorithm generates 2k 1 subsets of in-edges and tries to reverse them so as to generate race variants. Also, after reversing some edges, a cycle means that the sequence is infeasible, i.e., e1 happened before e2 and e2 happened before e1 is not possible. We use Fig. 15 to illustrate Algorithm 3. In Step 4 of Algorithm 3, assume E is the second event of Client 2. There are two in-edges of E. We show the three cases for i ¼ 1, i ¼ 2, and i ¼ 3. Note that i is the index variable in Step 4. Fig. 11 shows all the race variants derived by Algorithm 3 with the T-sequence TS in Section 4.1 (Fig. 5b) as input. Note that TS(Ci,j) is abbreviated to Ti,j. There is a total of six race variants derived: RV3-1 to RV3-6. However, Algorithm 2 derives 10 race variants from the same T-sequence. This motivates us to study the correctness of the two algorithms and the differences between them. In Section 4.4, we prove that reachability testing with the two algorithms to derive race variants can execute all the possible T-sequences if the client/server program always terminates. In Section 4.5, we compare the two algorithms. We focus on the running time


4.4 The Correctness of Algorithms 2 and 3 In this section, we investigate whether the two algorithms can perform the exhaustive testing for a client/server program. We first need two additional definitions (see Figs. 17 and 18). In Step 3 of Algorithm 4 (see Fig. 19), it first adds a T-event e and its edges to GMCF P if e is a not-happenedbefore node in both RG1 and RG2 . Then, it removes e from RG1 and RG2 . This process continues until there does not exist any T-event which is a not-happened-before node in both RG1 and RG2 . The algorithm shows one method of deriving the MCFP from two T-sequences, from which we can easily know that the MCFP of two T-sequences is unique. Fig. 16 shows an example for the definition of the MCFP. In the example, the MCFP has three T-events. In the extreme case, the MCFP of two T-sequences has no T-event. Now, we show why all the feasible T-sequences can be derived if we use Algorithms 2 and 3 to derive race variants in reachability testing (see Theorem 1 and Theorem 2). Theorem 1. Assume that every execution of a client server program P with input X terminates. According to Algorithm 2, reachability testing of P with input X derives and executes all feasible T-sequences of P with input X. Proof. Refer to Fig. 1. Assume we have a T-sequence S of P with input X. Since every execution of P with input X terminates, we assume that the number of T-events in S is n, where n is a positive integer, which we denote in the following as jSj ¼ n. Assume a feasible T-sequence S0 of P with input X. We prove that conducting the process shown in Fig. 1 by applying Algorithm 2 to derive race variants can always perform a testing of P with X and S0 . Let TM be the MCFP of S and S0 , and let jTM j ¼ m, m is a positive integer. It is trivial that m jSj and m jS0 j.

10


VOL. 30,

NO. 1,

JANUARY 2004


Reachability testing will first invoke Algorithm 2 with S as its input (see Fig. 1). As the nodes in the RV diagram are generated by considering all the possible interleavings of T-events of S, there must exist at least one prefix node whose race graph is the same as the race graph of TM in the generated RV diagram. This is because TM is also a feasible prefix of S. We denote this node NM . In the following, we prove that there must be at least one child node of NM which is a feasible prefix of S0 . Let TSM be the set of T-events in S but not in TM , and let TS0 M be the set of T-events in S0 but not in TM . Without loss of generality, let e be a not-happened-before event in Race-graph (TS0 M ). First, we have that TM þ e is

Fig. 15. An example to illustrate Algorithm 3.

a feasible prefix of S0 . It is trivial that e must not be a nothappened-before event in Race-graph (TSM ). If e is a not-happened-before event in both Race-graph (TSM ) and Race-graph (TS0 M ), then it must be that e 2 TM . According to Step 3 of Algorithm 2, this will generate a child node thathas Race-graph (TM þ e) as its race graph. Since there must exist an event thathas happened before e in Race-graph(S), this child node is a race-variant node of the generated RV diagram. Then, the reachability testing will conduct a prefixbased replay of TMþe . Assume that this produces a T-sequence S1 . It is trivial that TMþe is a feasible prefix of S1 and S0 . Let TMþ1 be the MCFP of S1 and S0 . We easily see that jTMþ1 j jTM j þ 1 ¼ m þ 1. Again, applying Algorithm 2 to S1 will derive a race variant of size jTMþ1 j þ 1. Repeating the above process, we will eventually derive a u T-sequence that is equal to S0 in the prefix-based replay.t

Fig. 16. Two examples illustrating the definition of MCFP.


11




Theorem 2. Assume that every execution of a client/server program P with input X terminates. According to Algorithm 3, reachability testing of P with input X derives and executes all feasible T-sequences of P with input X. Proof. Refer to Fig. 1. Assume that we have a T-sequence S of P with input X. Since every execution of P with input X terminates, we assume that jSj is n, n is a positive integer. Assume a feasible T-sequence S0 of P with input X. We prove that conducting the process in Fig. 1 by applying Algorithm 3 to derive race variants can always perform a testing of S0 . Let TM be the MCFP of S and S0 , and let jTM j ¼ m, where m is a positive integer. It is trivial that m jSj and m jS0 j. The reachability testing will first invoke Algorithm 3 with S as its input (see Fig. 1). The same way as the proof of Theorem 1, let TSM be the set of T-events in S but not in M, and TS0 M be the set of T-events in S0 but not in M. Without loss of generality, let e be a not-happened-before event in Race-graph (TS0 M ). It is trivial that e must not be a not-happenedbefore event in Race-graph (TSM ). Then, there must be some in-edges to e from the events in Race-graph (TSM ) (see Fig. 20). By changing the directions of the in-edges of e from the nodes in Race-graph (TSM ), we can obtain a feasible prefix TM þ e of S0 . Because it has no knowledge of TM in the process of deriving race

variants, Step 4 in Algorithm 3 will change the direction of the in-edges of an event of S in any combination, i.e., 2jnumber of in-edgesj 1. The changing of in-edge directions of e in S to derive TM þ e must fall into one of the 2jnumber of in-edgesj 1 cases (see Fig. 20). Then, the reachability testing will conduct a prefixbased replay of TM þ e. Assume that this produces a T-sequence S1 . It is trivial that TM þ e is a feasible prefix of S1 and S0 . Let TMþ1 be the MCFP of S1 and S0 . We easily see that jTMþ1 j jTM j þ 1 ¼ m þ 1. Again, applying Algorithm 3 to S1 will derive a race variant of size jTMþ1 j þ 1. Repeating the above process, we will eventually derive a T-sequence that is equal to S0 in the prefixbased replay. u t

Fig. 20. Changing the direction of some of the in-edges of e to obtain a feasible prefix S0 .

12


4.5 Analysis of Algorithms 2 and 3 The analyses of the algorithms are divided into two parts. One is the complexity analysis of Algorithms 1, 2, and 3 (Complexity Analyses 1-3). The other is the analysis of duplicating tests during reachability testing (Theorems 3-9). Complexity Analysis 1: The running time of the algorithm for deciding if Tu and Tv exhibit a transaction race in Definition 1 is Oðm n MP2 KL Þ in which m, n, and MP KL are the number of accessed table records in Tu and Tv , and maximal number of elements in the Primary_ Key_List, respectively. Proof. For each pair of the ½Table nameui ; Operationui ; Primary Key Listui in Tu and ½Table namevj ; Operationvj ; Primary Key Listvj in Tv , 1 i m, 1 j n, it needs the running time OðM2PKL Þ to check if Primary Key Listui [ Primary Key Listvj is the empty set. Since there are m n pairs of accessed table records to be checked, the running time is Oðm n M2PKL Þ. u t Complexity Analysis 2: Assume that the T-sequence TS is from the execution of P with n clients. Let |TS|, K, MAT R , and MP KL be the number of T-events in TS, the number of edges between different clients in the race graph of TS, the maximal number of accessed table records in T-events of TS, and the maximal number of elements in Primary_Key_List, respectively. The running time for Algorithm 1 to generate 2 2 2 the race graph of TS is OðMAT R MP KL jT Sj þ KÞ. The running time for Algorithm 2 to derive race variants from TS 2 2 2 is OðjT Sj2 K þ njT Sj þ MAT R MP KL jT Sj Þ. 2 2 Proof. For Algorithm 1, it first needs OðMAT R MP KL 2 2 2 jT Sj ðjT Sj 1Þ=2Þ ¼ OðMAT R MP KL jT Sj Þ to compute all the transaction race relationships between jT Sj T-sequences. Steps 1, 2, and 3 of Algorithm 1 are bound on OðjT SjÞ, OðjT SjÞ, and OðKÞ, respectively. Thus, the 2 2 2 running time of Algorithm 1 is OðMAT R MP KL jT Sj þ 2 2 2 jT Sj þ jT Sj þ KÞ ¼ OðMAT R MP KL jT Sj þ KÞ. For Algorithm 2, the running time for identifying the happened-before relations between jT Sj T-events is OðjT Sj2 KÞ. The RV diagram for T S is an n-ary tree with a maximal depth of jT Sj. It is because each expanding of a child node from a prefix node simulates an execution of a T-event and the total number of T-events is jT Sj. The number of nodes in an n-ary tree with depth jT Sj is ðnjT Sjþ1 1Þ=ðn 1Þ. We have OððnjT Sjþ1 1Þ=ðn 1ÞÞ ¼ OðnjT Sj Þ. Algorithm 2 is bound on the running time to generate the race graph and the number of nodes in the generated RV diagram. We have the running time of Algorithm 2 2 2 2 2 is OðjT Sj2KþnjT Sj þMAT RMP KL jT Sj þKÞ ¼ OðjT Sj 2 jT Sj 2 2 Kþn þ MAT R MP KL jT Sj Þ. u t

VOL. 30,

NO. 1,

JANUARY 2004

Complexity Analysis 3: The running time for Algorithm 3 to derive race variants from TS is OðjT Sj2 2 2 2 2K þ MAT R MP KL jT Sj þ KÞÞ. Note that jT Sj, K, MAT R , and MP KL are defined in Complexity Analysis 2. Proof. Assume that the jT Sj T-events are V1 ; V2 ; . . . , and VjT Sj . Let ki be the number of in-edges of Vi . Then, we have K ¼ k1 þ k2 þ . . . þ kjT Sj . In addition to generating the race graph for TS, the running time of Algorithm 3 is bound on the time for executing Step 4 for each T-event, i.e., OðjT Sj 2ki1 Þ for Vi . Note that OðjT SjÞ is to check if there is a loop in G0 . Thus, the running time of Step 4 is jT Sj 2k11 þ jT Sj 2k21 þ . . . þ jT Sj 2kjT Sj1 < jT Sj 2K þ jT Sj 2K þ . . . þ jT Sj 2K ¼ jT Sj2 2K . The running 2 2 2 u t time is OðjT Sj2 2K þ MAT R MP KL jT Sj þ KÞÞ. According to Fig. 11, we can easily determine that the number of race variants generated by Algorithms 2 and 3 may be different. That is, given the same T-sequence, Algorithms 2 and 3 may derive different sets of race variants. We can divide the race variants into six groups. For example, in the race variants derived by Algorithm 2, RV2-3 is a feasible prefix of both RV2-6 and RV2-10 and RV2-10 is a feasible prefix of RV2-6 in group 5. A similar situation also holds in groups 1 and 6. Theorem 3. Assume that RV1 and RV2 are two race variants derived by Algorithm 2 from a T-sequence. It is possible that RV1 is a feasible prefix of RV2. Proof. This comes directly from the example shown in Fig. 11, where RV2-3 is a feasible prefix of both RV2-6 and RV2-10. u t Theorem 4. Assume two racevariants RV1 and RV2. If RV1 is a feasible prefix of RV2, then the prefix-based replay of RV1 and RV2 with the same input may produce the same T-sequence. Proof. Assume that the prefix-based replays of RV1 and RV2 produce two feasible T-sequences T1 and T2, respectively. T1 is prefixed with RV1. The T-events of T1 after RV1 are obtained in the monitor phase of the prefix-based replay. Since the execution of T-events in the monitor phase is not controlled, it is possible to produce any feasible sequences of T-events. Since T2 is also prefixed with RV1 (because T2 is prefixed with RV2 and RV2 is prefixed with RV1), it is possible that T1 is equal to T2. u t Theorem 3 and Theorem 4 show that it is possible to duplicate the same test if we employ Algorithm 2 to derive the race variant. However, it is impossible for Algorithm 3 to duplicate the same test in this situation (see Theorem 5). Theorem 5. Assume a T-sequence TS, and that RV1 and RV2 are two race variants derived by applying Algorithm 3. It is impossible that RV1 is a feasible prefix of RV2. Proof. According to the operation of Theorem 3, we consider the following two cases: 1) We assume that RV1 and RV2 are generated by altering the in-edges of the same node in step 4 of Theorem 3. If the node is e, then e must be in a T-event in RV1 and RV2. However, according to Step 4 of Theorem 3, the in-edges of e of RV1 and e of RV2 must be different. Thus, it is


13

Fig. 21. A race graph G and two of its race variants RV1 and RV2.

impossible for RV1 to be equivalent to RV2 in this case. 2) We assume that RV1 and RV2 are generated by altering the in-edges of the different nodes in Step 4 of Theorem 3. Assume that RV1 is generated by altering the in-edges of node e. If RV1 is a feasible prefix of RV2, e must also be a T-event of RV2. However, since we apply Theorem 3 to change the in-edges of e to derive RV1 and RV2 are derived by altering the in-edges of another node, the inedges of e must be different in RV1 and RV2, so it is impossible for RV1 to be a feasible prefix of RV2. It is clear that each case leads to a contradiction. u t We have shown in Theorem 5 that it is impossible for Algorithm 3 to derive two racevariants from the same T-sequence, for which one is the other’s feasible prefix. Also, in the example shown in Fig. 11, none of the race variants derived by Algorithm 3 is another’s feasible prefix. However, this does not mean that performing reachability testing with Algorithm 3 to derive race variants will not duplicate the same test. Consider the race graph G of a T-sequence shown in Fig. 21. By changing the in-edges of two T-events E1 and E2 in G, we can derive at least two race variants RV1 and RV2. Fig. 21 also shows that it is possible for two T-sequences obtained by applying prefix-based replay of RV1 and RV2 to be identical. Since each prefixbased replay of a racevariant conducts a test, it is obvious that Algorithm 3 cannot also avoid duplication of the same test. The example shown in Fig. 21 motivates Theorem 6. In addition to Theorem 4, Theorem 6 shows another situation in which the prefix-based replay of two race variants may produce the same T-sequence. Theorem 6. Assume two race variants RV1 and RV2. Let M be the MCFP of RV1 and RV2. If (set of nodes in Race-graph (RV1-M) \ set of nodes in Race-graph (RV2-M))2 is the empty set, then the prefix-based replays of RV1 and RV2 may produce the same T-sequence. Proof. The prefix-based replaying of a racevariant includes two steps: 1) controlling the execution of the concurrent program by following the execution order specified in the racevariant, and then 2) executing the concurrent 2. Note that the “-” in (RV1-M) denotes a set difference.

program without any control and recording the order of the executed synchronization events. Consider the prefix-based replaying of RV1. Because we do not control the execution of the concurrent program in Step 2, it is possible to produce a T-sequence thathas RV2 as its feasible prefix. u t Following Theorem 6, Theorem 7 discusses another situation in which it is impossible for the prefix-based replays of two race variants to produce the same T-sequence. Theorem 7. Assume two race variants RV1 and RV2. Let M be the MCFP of RV1 and RV2. If (set of nodes in Race-graph (RV1-M) \ set of nodes in Race-graph (RV2-M)) is not the empty set, then the prefix-based replays of RV1 and RV2 produce different T-sequences. Proof. We first assume e 2 (set of nodes in Race-graph (RV1-M) \ set of nodes in Race-graph (RV2-M)). Note that e is a T-event. Let N1 and N2 be the sets of nothappened-before nodes in Race-graph (RV1-M) and Race-graph (RV2-M), respectively. According to the definition of MCFP, we have N1 \ N2 ¼ . We consider the following two cases: 1) Assume that node e is in one of N1 and N2. Without loss of generality, we assume e is in N1. Since N1 \ N2 ¼ , we have that e is in Racegraph (RV2-M-N2). It is obvious that the replay of RV1 and RV2 will not produce the same T-sequence because one of the nodes in N2 must occur before e in RV2. 2) Assume that e is not in N1 and N2 (i.e., e is in Racegraph (RV1-M-N1) and Race-graph (RV2-M-N2)). It is impossible for the replaying of RV1 and RV2 to produce the same T-sequence because there must exist a node x 2 N1 that has occurred before e in RV1 plus a node y 2 N2 that has occurred before e in RV2. Since N1 \ N2 ¼ , we have x 6¼ y. u t Theorem 8. Assume a T-sequence TS, and that RV1 and RV2 are two race variants derived by applying Algorithm 2. It is possible that (set of nodes in Race-graph (RV1-M) \ set of nodes in Race-graph (RV2-M)) is the empty set. Proof. This comes directly from the example shown in Fig. 11. Assume that M is the MCFP of RV2-3 and RV2-10. Actually, M is RV2-3. Then, (set of nodes in

14


VOL. 30,

NO. 1,

JANUARY 2004

Race-graph (RV2-3-M) \ set of nodes in Racegraph(RV2-10-M)) is the empty set. u t Theorem 9. Assume a T-sequence TS, and that RV1 and RV2 are two race variants derived by applying Algorithm 3. It is possible that (set of nodes in Race-graph (RV1-M) \ set of nodes in Race-graph (RV2-M)) is the empty set. Proof. This comes directly from the example shown in Fig. 21. Set of nodes in Race-graph (RV1-M) is E1 and set of nodes in Race-graph (RV2-M) is E2. E1 \ E2 is the empty set. u t With Theorems 6 and 9, we can conclude that it is also possible to duplicate the same test if we employ Algorithm 3 to derive the race variants during reachability test. From the above discussion, we conclude that both Algorithms 2 and 3 may conduct the same test. However, the probability of Algorithm 3 duplicating the same test seems lower. Fig. 11 is an illustrative example. The experimental results in Section 6 also demonstrate this.

5

PREFIX-BASED REPLAY TRANSACTIONS

FOR

DATABASE

In addition to the implementation of Algorithms 2 and 3, we have also implemented a prefix-based replay technique for race variants based on T-sequences. The prefix-based replay of a racevariant includes two steps: 1) controlling the execution of the program by following the execution order specified in the racevariant, and then 2) executing the program without any control and recording the order of the executed T-events. Note that the two steps are the replay and monitor phases we mentioned in Section 2. After the two steps, the program has been executed once. This procedure conducts a test and produces a T-sequence. For example, see Fig. 21. RV1 is the race variant to be replayed in the first step and we obtain six T-events in the second step. It produces a T-sequence with nine T-events. A scheme for performing monitoring and replaying of concurrent shared-memory programs has been presented previously [11]. The scheme cannot perform prefix-based replay of a racevariant because it cannot switch between the monitoring and replay phases during execution. It must first replay the execution order according to the target race variant and then monitor the execution of the concurrent program to conduct the two steps of prefix-based replay. A method has been proposed [8] for performing prefix-based replay of a racevariant, but it can only be applied to concurrent software which uses shared memory or semaphores to perform process synchronization. In this paper, we propose a scheme that performs prefix-based replay for an SQL client/server program. Fig. 22 shows how to modify the original client/server program to perform prefix-based replay of a race variant. First of all, there must be some variables added to the modified program, which are used to control the replaying and monitoring of the program. They can be divided into two classes: 1) those stored in the tables of databases and 2) local variables in each client process. These variables should be initialized before performing the prefix-based replay. For each T-event E in the original program, an entry

Fig. 22. How to modify the source program to perform prefix-based replay.

and exit protocol must be inserted before and after E, respectively. In this paper, we present two methods for performing prefix-based replay based on the data structure of the T-sequence defined in this paper. For the replay phase in Step 1 and monitor phase in Step 2 of the prefix-based replay, we need a mechanism to synchronize the execution of concurrent clients. In this paper, we present the entry and exit protocols based on two types of synchronization mechanism: one is based on the built-in synchronization functions in the SQL database system, and the other uses Java thread synchronization. We first present the entry and exit protocols based on the built-in SQL database synchronization mechanism. Note that Client1 to Clientn are processes executed in different machines. For clients executed in the same machines, we can consider it as special case of it. There are some variables added to the databases and client processes. The variables added to the databases are as follows: We add a table Transaction_order to each database. There is only one row in this table. The primary key is DTO, which represents the transaction order of the database in replaying and monitoring. The initial value of DTO is 1. . We add a new database called Un_replay_client, which has only one table, Un_finish_replay_client. There is only one row in this table, and its primary key is UFRC. This variable represents the number of clients that have not finished the replay phase. The initial value of UFRC is the total number of clients in the application. The local variables added to each client process are as follows: .

.

.

Mode: This variable represents whether the client is in Monitor or Replay mode. Its initial value is Replay. Number of replay events: This constant variable represents the number of T-events which the client should replay. Assume that RV is the race variant


15

Fig. 23. The entry protocol based on built-in database synchronization.

that is going to be prefix-based replayed. It is initialized to the number of the T-events of the client in RV. . Client transaction order: This variable records the order of T-events which the client is going to replay or monitor. Its initial value is 1. The following local procedures should be invoked in the client process:

Lock_DB(Un_replay_Client); Num_UFRC = Select UFRC From Un_replay_client. Un_finish_replay_ client; Update Un_replay_ client. Un_finish_replay_ client Set UFRC = Num_UFRC - 1; UnLock_DB(Un_replay_Client);

LockDB(Name of Database): This is a function to lock the use of the database Name of Database. While a database is locked by a client, all the database transactions issued by other clients are detained until the client calls UnLockDB(Name of Database). Almost all implementations of database servers support a way to lock the database, such as the setTransactionIsolation member function of the Connection object in Java Database Connectivity [24]. . UnLockDB(Name of Database): This is a function to unlock the database Name of Database. . My_next_DB_transaction_order(i): This function returns the data transaction order according to the client transaction order i. Figs. 22 and 23 show the pseudocodes of entry and exit protocols, respectively, for the prefix-based replay of a race variant. In the second case, we have Client1 to Clientn as threads in the same machine, which means that they can use their shared memory to synchronize themselves. This is based on the thread synchronization procedure of the Java programming language [25], for which the entry and exit protocols are very similar to the protocols shown in Figs. 22 and 23. However, the following code fragment uses the built-in database synchronization to protect the access of variable Un_finish_replay_client:

synchronized private void UFRC() { Un_finish_replay_client = Un_finish_replay_client - 1; }.

.

which is replaced with the following Java subroutine:

Also, the Lock_DB(DB) and UnLock_DB(DB) functions are replaced with wait(DB.sem) and signal(DB.sem), which are Java subroutines that implement the binary semaphore operations defined in [4]. The DB.sem subroutine implements a semaphore with an integer value of either 0 or 1. Figs. 24 and 25 are the entry and exit protocols based on Java thread synchronization.

6

EXPERIMENTS

This section presents the results from the experimental testing of three programs: Program 1 is a client/server program with three clients, where each client issues three SQL transactions. Program 2 is a client/server program with three clients, where each client issues five SQL transactions. Program 3 is a client/server application extracted from [18]. There are three clients which issues three, five, and four SQL transactions, respectively. We test each example program using the following schemes: ME is the multipleexecution testing with random delays during execution. RT_1 is the reachability testing using Algorithm 2 to

16


VOL. 30,

NO. 1,

JANUARY 2004

Fig. 24. The exit protocol based on built-in database synchronization.

generate the race variants, with Figs. 22 and 23 as the entry and exit protocols for prefix-based replay of a race variant. RT_2 is the reachability testing using Algorithm 2 to generate the race variants, with Figs. 24 and 25 as the entry and exit protocols for prefix-based replay of a race variant. RT_3 is the reachability testing using Algorithm 3 to generate the race variants, with Figs. 22 and 23 as the entry and exit protocols for prefix-based replay of a race variant. RT_4 is the reachability testing using Algorithm 3 to generate the race variants, with Figs. 24 and 25 as the entry and exit protocols for prefix-based replay of a race variant. All the client processes and database servers are executed on PCs with 800-MHz Intel Pentium III processors and the MS Windows 2000 operating system. Client

programs and database servers are located in the same network segment (with a bandwidth of 100Mbps). The database system is Microsoft SQL Server 7.0 [26]. The client application programs and code for implementing of reachability testing are compiled and executed in Java Runtime Environment, Standard Edition (build 1.3.0_01) [27]. Fig. 27a shows the experimental results froma multipleexecution test of Program 1. It is obvious that this program duplicates a lot of tests since, from 1,000 tests, an average of only 5.2 different T-sequences are derived. We repeat the testing 10 times. Fig. 27b shows the experimental results from using schemes RT_1, RT_2, RT_3, and RT_4 to test Program 1. Each test is repeated three times. We discover

Fig. 25. The entry protocol based on Java thread synchronization. For ease of understanding, we still show the pseudocode rather than real Java souce code.


17

Fig. 26. The exit protocol based on Java thread synchronization.

that using Algorithm 2 to derive race variants tends to duplicate more tests than using Algorithm 3. Fig. 27c shows the time required to perform tests ME, RT_1, RT_2, RT_3, and RT_4 on Program 1. This figure shows that using Java thread synchronization (Figs. 24 and 25) to perform prefix-based replay is much more efficient than synchronizing clients with the database synchronization mechanism (Figs. 22 and 23). Consider the programs which use the SQL database synchronization mechanism, i.e, ME, RT_1, and RT_3. Although the multiple-execution testing scheme required the least time to perform a test (i.e., 8.8 seconds), it could only derive an average of six different T-sequences from 1,000 tests. Note that the time to perform a test includes initializing the tables in the database and the execution of all clients. Fig. 27d focuses on two points. The first one is the required time to generate race variants from a T-sequence, i.e., the execution times of Algorithms 2 and

3. The second one is the number of duplicate T-sequences during reachability testing. The figure shows that Algorithm 2 is slightly faster than Algorithm 3. However, considering the number of duplicated tests, Algorithm 2 is much worse than Algorithm 3. In this example, Algorithm 3 (RT_3 and RT_4) duplicates less than five tests among 24 T-sequences, whereas Algorithm 2 (RT_1 and RT_2) duplicates more than 80 tests. The experimental results of Programs 2 and 3 are shown in Figs. 28 and 29, respectively. From these experimental results, we draw the following conclusions. First, the multiple-execution testing scheme is not a practical way of testing SQL client/server programs since a tremendous number of the tests are duplicated. Also, most T-sequences are never exercised. Second, although Algorithm 2 can always exhaustively execute all possible tests; this is associated with the duplication of

Fig. 27. Experimental result of Program 1. (a) Using ME to test Program 1. (b) Reachability testing of Program 1. (c) Time required to perform reachability testing of Program 1. (d) Time required to generate the race variants for Program 1.

18


VOL. 30,

NO. 1,

JANUARY 2004

Fig. 28. Experimental result of Program 2. (a) Using ME to test Program 2. (b) Reachability testing of Program 2. (c) Time required to perfrom reachability testing of Program 2. (d) Time required to generate the race variants for Program 2.

Fig. 29. Experimental result of Program 3. (a) Using ME to test Program 3. (b) Reachability testing of Program 3. (c) Time required to perfrom reachability testing of Program 3. (d) Time required to generate the race variants for Program 3.

many tests. Algorithm 3 duplicates fewer tests than Algorithm 2, with the result that, although Algorithm 3 is slightly slower than Algorithm 2, reachability testing using Algorithm 3 to generate race variants is more efficient than using Algorithm 2. Third, using the intrinsic synchronization mechanism in an SQL server to perform prefix-based replay is much slower than using Java thread synchronization. Finally, RT_4 provided the best performance in all of our experiments.

7

general type of model and, hence, the algorithm can be applied to any data-access model for reachability testing. The Java byte code of it can be obtained at http:// bashful.ice.ntnu.edu.tw/~ghhwang/RV_gen.zip. We consider the algorithm to be a significant breakthrough in reachability-testing technology. To test a concurrent program having a huge or infinite number of feasible T-sequences, we can combine reachability testing with some strategies for selecting T-sequences. There is room for further investigation.

CONCLUSION

In this paper, we have provided a framework for exhaustive testing of a client/server database application that exhibits nondeterministic behavior. We prove theoretically that the proposed scheme can derive all the possible T-sequences of a database application if each execution of application terminates, and our experimental results support this. In the key technology of reachability testing (i.e., for deriving race variants from a SYN-sequence), instead of generating an RV diagram (Algorithm 2) we have developed a new algorithm (Algorithm 3) that is based on the edge analysis of the race graph of the target SYN-sequence. Our theoretical analysis suggests that Algorithm 3 duplicates fewer tests, which is confirmed by the results of our experiments. This algorithm can be applied to the dataaccess model of database applications, which is the most

ACKNOWLEDGMENTS The authors would like to thank the anonymous referees for a number of useful suggestions to improve the paper. G.-H. Hwang and S.-J. Chang’s work was supported in part by the Republic of China National Science Council under grant 89-2218-E-260-016 and ROC MOE/NSC program for promoting academic excellence of universities under grant 89-E-FA04-1-4.

REFERENCES [1] [2]

C.E. Mcdowell and D.P. Helmold, “Debugging Concurrent Programs,” ACM Computing Surveys, vol. 21, no. 4, Dec. 1989. K.C. Tai and R.H. Carver, “Testing of Distributed Programs,” Parallel and Distributed Computing Handbook, A.Y. Zomaya, ed., chapter 33, McGraw-Hill, 1996.


[3] [4] [5]

[6] [7] [8] [9] [10] [11] [12] [13] [14] [15]

[16] [17] [18]

[19] [20] [21] [22] [23] [24] [25] [26] [27]

A. Dinning, “A Survey of Synchronization Methods for Parallel Computers,” Computer, July 1989. A. Silberschatz, P. Baer Galvin, and G. Gagne, Operating System Concepts, sixth ed. John Wiley & Sons, June 2001. Int’l Organization for Standardization, Information Technology— Database Languages-SQL-Part 1: Framework (SQL/Framework), ISO/IEC 9075-1: 1999 and Information Technology—Database Languages-SQL-Part 2: Foundation (SQL/Foundation), ISO/IEC 9075-2: 1999, http://www.iso.org, 1999. R.A. Elmasri and S.B. Navathe, Fundamentals of Database Systems, third ed. Addison-Wesley, Jan. 2000. G.-H. Hwang, K.C. Tai, and T.L. Huang, “Reachability Testing: An Approach To Testing Concurrent Software,” Int’l J. Software Eng. and Knowledge Eng., vol. 5, no. 4, pp. 493-510, Dec. 1995. K.-C. Tai, “Reachability Testing of Asynchronous MessagePassing Programs,” Proc. Second Int’l Workshop Software Eng. for Parallel and Distributed Systems, 1997. A. Bechini and K.-C. Tai, “Timestamps for Programs Using Messages and Shared Variables,” Proc. Int’l Conf. Distributed Computing Systems, pp. 266-273, May 1998. Y. Lei and K.-C. Tai, “Efficient Reachability Testing of Asynchronous Message-Passing Programs,” Proc. Eighth IEEE Int’l Conf. Eng. for Complex Computer Systems, pp. 35-44, Dec. 2002. T.J. LeBlanc and J.M. Mellor-Crummey, “Debugging Parallel Programs with Instant Replay,” IEEE Trans. Computers, vol. 36, no. 4, pp. 471-482, Apr. 1987. R.A. Davies, R.J.A. Beynon, and B.F. Jones, “Automating the Testing of Databases,” Proc. First Int’l Workshop Automated Program Analysis, Testing and Verification, June 2000. D. Chays, S. Dan, P.G. Frankl, F.I. Vokolos, and E.J. Weyuker, “A Framework for Testing Database Applications,” Proc. ACM Int’l Symp. Software Testing and Analysis, 2000. M.J. Carey, D.J. DeWitt, and J.F. Naughton, “The 007 Benchmark,” Proc. 1993 ACM SIGMOD Int’l Conf. Management of Data, pp. 12-21, May 1993. J. Gray, P. Sundaresan, S. Englert, K. Baclawski, and P.J. Weinberger, “Quickly Generating Billion-Record Synthetic Databases,” SIGMOD Record (ACM Special Interest Group on Management of Data), vol. 23, no. 2, pp. 243-252, June 1994. D. Slutz, “Massive Stochastic Testing of SQL,” Proc. Conf. Very Large Databases, pp. 618-622, Aug. 1998. Trans. Processing Performance Council, TPC-Benchmark C, 1998. H. Chu and J. Dobson, “Towards Quality Programming in the Automated Testing of Client/Server Application,” Proc. Pacific Northwest Software Quality Conf. ’98 and Proc. Int’l Conf. Software Quality ’98, Oct. 1998. D. Helmbold and D. Luckham, “Debugging ADA Tasking Programs,” IEEE Software, vol. 2, no. 2, pp. 66-74, 1985. R.N. Taylor, “A General-Purpose Algorithm for Analyzing Concurrent Programs,” Comm. ACM, vol. 21, no. 7, July 1978. M. Young and R.N. Taylor, “Combining Static Concurrency Analysis with Symbolic Execution,” IEEE Trans. Software Eng., vol. 14, no. 10, Oct. 1988. C.E. McDowell, “A Practical Algorithm for Static Analysis of Parallel Programs,” J. Parallel and Distributed Computing, vol. 6, pp. 515-536, 1989. L. Lamport, “Time, Clocks, and the Ordering of Events in a Distributed System,” Comm. ACM, vol. 21, no. 7, pp. 558-565, July 1978. M. Gruber, “Mastering SQL,” Sybex, Jan. 2000. G. Steele, J. Gosling, and G. Bracha, Java(TM) Language Specification, second ed. B. Joy, ed., Addison-Wesley, June 2000. S. Bjeletich, G. Mable, and D.W. Solomon, “Microsoft SQL Server 7.0 Unleashed,” Sams, first ed., May 1999. Sun Microsystem, The Source for Java(TM) Technology, http:// java.sun.com, 2002.

19

Gwan-Hwan Hwang received the BS and MS degrees while in the Department of Computer Science and Information Engineering at National Chiao-Tung University, in 1991 and 1993, respectively, and the PhD degree while in the Department of Computer Science at National Tsing-Hua University, HsinChu, Taiwan, in 1998. He has been an assistant professor in the Department of Information and Computer Education at National Taiwan Normal University, Taiwan, since 2001. His research interests include concurrent program testing, parallelizing compilers, Internet security, thin-client technologies, and groupware. Sheng-Jen Chang received the BBA degree while in the Department of Management Information Systems at National Cheng-Chi University in 2000 and the MBA degree while in the Department of Information Management at National Chi-Nan University in 2002, respectively. He has been an associate researcher in the Telecommunication Laboratories of Chunghwa Telecom Co., Ltd., Taiwan, since 2002. His research interests include concurrent program testing and database system testing. Huey-Der Chu received the PhD degree (1998) from the Centre for Software Reliability at the University of Newcastle upon Tyne, England, funded by the National Science Council in Taiwan. He is an associate professor at Takming College in Taiwan, as well as the executive director of the Chinese Software Quality Association. His current research interests are in software process improvement, knowledge management, and quality management.

. For more information on this or any other computing topic, please visit our Digital Library at http://computer.org/publications/dlib.