Performance Driven Database Design for Scalable ... - Springer Link

Performance Driven Database Design for Scalable Web Applications Jozsef Patvarczki, Murali Mani, and Neil Heffernan Worcester Polytechnic Institute, Department of Computer Science 100 Institute Road, Worcester, Massachusetts, 01609, US {patvarcz,mmani,nth}@cs.wpi.edu

Abstract. Scaling up web applications requires distribution of load across multiple application servers and across multiple database servers. Distributing load across multiple application servers is fairly straightforward; however distributing load (select and UDI queries) across multiple database servers is more complex because of the synchronization requirements for multiple copies of the data. Different techniques have been investigated for data placement across multiple database servers, such as replication, partitioning and de-normalization. In this paper, we describe our architecture that utilizes these data placement techniques for determining the best possible layout of data. Our solution is general, and other data placement techniques can be integrated within our system. Once the data is laid out on the different database servers, our efficient query router routes the queries to the appropriate database server/(s). Our query router maintains multiple connections for a database server so that many queries are executed simultaneously on a database server, thus increasing the utilization of each database server. Our query router also implements a locking mechanism to ensure that the queries on a database server are executed in order. We have implemented our solutions in our system, that we call SIPD (System for Intelligent Placement of Data). Preliminary experimental results illustrate the significant performance benefits achievable by our system. Keywords: Scalability, Web application, database design.

1

Introduction

There are thousands of web applications, and these systems need to figure out how to scale up their performance. Web applications typically have a 3-tier architectures consisting of clients, application, and database server that are working together. Significant work has been done in load balancers to solve the possible scalability issues and to distribute requests equally among multiple application servers. However, issues related to the increased database server usage and to distribute requests among multiple database servers have not been adequately addressed. The increasing load of the database layer can lead to slow response time, application error, and in the worst case, to different type of system crashes. J. Grundspenkis, T. Morzy, and G. Vossen (Eds.): ADBIS 2009, LNCS 5739, pp. 43–58, 2009. c Springer-Verlag Berlin Heidelberg 2009

44

J. Patvarczki, M. Mani, and N. Heffernan

Our work is motivated by the ASSISTment Intelligent Tutoring system [6]. In the ASSISTment system the increasing number of sessions can be easily balanced among application servers but the continuous database read (select) queries, and update, delete and insert (UDI) queries decrease the system response time significantly. Currently, the ASSISTment system supports 3000 users that include 50 teachers from 15 public schools over Massachusetts. It consists of multiple application servers, a load balancer, and a database server. A characteristic of web applications such as our ASSISTment system, is that we know all the incoming query templates beforehand as the users typically interact with the system through a web interface such as web forms [5]. Traditional solutions for distributing load across multiple database servers, on the other hand, do not have this property [10]. This allows us to propose additional solutions for balancing load across multiple servers in the scenario of web applications, above and beyond what is supported for traditional applications. 1.1

Current Techniques for Distributing Load across Multiple Database Servers

Several techniques are known for distributing load across multiple database servers; one of them is replication [10]. In replication, a table is placed on more than one database server. In such a case, a select query on that table can be executed by any one of the database servers that have a replica of that table. An UDI query on that table however needs to be executed on all the database servers that have a replica of that table. If we do not know all the queries that the application may need to process beforehand, then one of the database servers must hold the entire data (all the tables) of that application. Such a layout of the data is needed to answer a query that needs to access all the tables. A drawback of this technique is that every UDI query needs to be executed against the node/(s) that hold the entire data and thus these nodes become the bottleneck for performance. Such an architecture is supported by Oracle, and is referred to as a master-slave architecture. In this case, the master node holds the entire data; every UDI query is executed against the master node and propagated to slave nodes as necessary using log files. In the case of web applications, we no longer need a node that holds the entire data (assuming that none of the queries access all the data). We can therefore do a more intelligent placement of the data such that there is no node that must execute all UDI queries; thus we can remove the bottleneck node for UDI queries that is inherent in non-web applications. This improves performance of read queries while not significantly impacting the performance of UDI queries. We discussed a simple master slave architecture above, where there is a single master node. However other master-slave architectures are possible where there is more than one master node. If there is more than one master node, there is no single-point of failure, but there is a higher synchronization (or update propagation) cost. In case of full-replication (all nodes are effectively master nodes), any node can act as a master when the original master fails, and the

Performance Driven Database Design for Scalable Web Applications

45

routing of queries to the nodes is straightforward as any node can answer any query, but the updates have to propagated to all the nodes. Another technique for distributing load across multiple database servers in web applications is partitioning of data, which includes both horizontal and vertical partitioning. Horizontal partitioning splits the table up into multiple smaller tables containing the same number of columns, but fewer rows. This can speed up query performance if data needs to be accessed from only one of the partitions. However, horizontal partitioning cannot be done in all circumstances, if we want a query to be answered by one of the nodes (a good assumption for such systems). For instance, if there are two queries in the workload that access the same table, one which selects based on a column say C1, and another which selects based on a column C2, then if we do horizontal partitioning based on the values in C1, then this partitioning cannot be used to answer queries based on C2. Vertical partitioning splits the table into smaller ones with the same number of rows but fewer columns. It is a reasonable approach when the system does not want to combine the records between the partitions. However, just like horizontal partitioning, vertical partitioning cannot be done in all scenarios also. For instance suppose the query workload consists of a select query on a table, and there is also an insert statement which inserts values into all columns of the same table. In this case, performing inserts after vertical partitioning is cumbersome. Another big disadvantage for both the partitioning schemes is that the system needs to maintain the partitions and balance the amount of data with a built in application logic. De-normalization [13] can optimize the performance of database systems as well. In de-normalization, one moves from higher to lower normal forms in the database modeling and add redundant data. The performance improvement is achieved because some joins are already pre-computed. However there are disadvantages, for instance handling UDI queries are cumbersome when performed against de-normalized data, as we need to synchronize between duplicates. 1.2

Proposed Solution

In this paper, we propose a generic architecture for balancing load across multiple database servers for web applications. There are two core parts of our system: (a) the data placement algorithm produces a layout structure, describing how the data needs to be laid out across multiple database servers for best possible performance and performs the actual layout; this algorithm is independent of the placement techniques considered, and (b) the query router that utilizes the layout structure produced by the data placement algorithm for routing queries and ensuring that the database servers are being utilized effectively. For determining the best possible layout structure, the data placement algorithm uses the given query workload (the percentage of queries for each template), the time it takes to execute a select/UDI query type (this time is measured as will be described in Section 4), and the time it takes to execute a select/UDI query if the table/(s) are partitioned, replicated or de-normalized (this time can

46


be either measured or estimated as will be described in Section 4). After determining the best possible layout structure, the data is laid out across the different database servers, and the system is ready to start processing the incoming requests from the applications. For determining on which node/(s) a query must be executed, we have developed a query router. Our query router is efficient and manages multiple connections per database server so that any database server is executing multiple queries simultaneously; this is to ensure that each database server is utilized as efficiently as possible. The query router also performs a simple locking mechanism to handle conflicting requests. We have integrated our placement algorithm and query router into our prototype system that we call SIPD (System for Intelligent Placement of Data). Our system is quite general: it can be used by any web application, and new placement techniques can be integrated as needed. 1.3

Contributions

Our contributions in this paper include the following: – We propose a data placement algorithm that is general. Our placement algorithm considers the given query workload (consisting of select and UDI queries) and the time for each query and determines the best possible placement of data across multiple database server nodes. Our placement algorithm is general as other techniques for placement can be integrated into our algorithm. – We propose an efficient distributed query router architecture that routes the queries to the different database servers, while ensuring that all the database servers are utilized efficiently. For ensuring that each database server is utilized efficiently, our query router maintains multiple connections to each database server; thus any database server is executing multiple queries simultaneously. – We have integrated our data placement algorithm and our query router into a prototype system that we call SIPD (System for Intelligent Placement of Data). Our system is general in that it can be used by any web application. – We have performed initial performance evaluation of our system. As an illustration, we describe the performance benefits observed by one of the placement techniques: horizontal partitioning. We also illustrate the overall performance benefit for a web application, the ASSISTment system. Outline of the Paper: In Section 2 we define the data placement problem. Our solutions for data placement, and for routing the queries is described in Section 3. Our prototype system implementation (SIPD) is discussed in Section 4. Experimental results are discussed in Section 5, and in Section 6 we discuss other aspects for improving performance. Section 7 describes related work; Section 8 concludes the work and discusses future directions.


2

47

The Data Placement Problem

Our general architecture for a web-based application is shown in Figure 1. First, the data is placed on different database servers. Different clients connect and issue requests which are distributed across different application servers by the load balancer. Balancing the load across different application servers can be done effectively by scheduling the requests using simple schemes such as round-robin, or scheduling the next request on the current least loaded server; these are not discussed further in this paper. A request may need to access data in the database server, in which case a query is issued to the query router. The query router has the logic to route the queries to the appropriate database server/(s). In short, the query router maintains the information about how the data is placed across different database servers. Let us motivate the data placement problem using a very, thinned down schema of the ASSISTment system. The portion of the schema that we consider includes users (students), schools, user roles (that maintains the school that a user attends), problems and logged action (that maintains all the actions of every user, including logins of a user, problems that a user has attempted). We collected 16 query templates for illustration as shown in Table 1. Note that for illustration purposes, we used only simple queries that do not perform a join. This data was collected over the duration of one week from our real application, and we counted the number of queries for each template. The total number of queries for these 16 templates over the week was about 360,000. We also have shown the number of rows of each table, at the end of the week over which the data was collected. Before we describe our data placement algorithm, let us examine Table 1 closely, and study what issues the placement algorithm may have to tackle.

AS1

Global Computer Network AS2

Load Balancer

.. ASn

{ Query Router

DB1

.. .

DB2

DBm Database Servers

Application Servers

Fig. 1. General Architecture for a web application. The requests are distributed among the different application servers by the load balancer. Requests that need to access the data are sent to the query router, that routes the query to the appropriate database server/(s).

48


As there are many updates against the logged action table, if logged action is replicated, the costs of performing these updates will be very high. Instead it might be better to perform a horizontal partitioning of the logged action table and place the different partitions on the different database server nodes. We notice that there are lot of updates against the problems table as well (ratio of UDI queries to select queries is roughly 1:14). However, Q8, Q9 and Q10 all access the problems table, but perform selects on different columns (Q11 and Q12 use the same column as Q9). In this case, we may want to consider maintaining only one copy of the problems table (rather than replicating the table or horizontally partitioning the table). Once a table is placed on only some of the database server nodes, the load on the different database servers may now be skewed. For instance, suppose problems table is placed on node 1, there is additional load on node 1 as compared to the other nodes. This may impact the horizontal partitioning, for instance when logged action is partitioned across nodes 1 and 2, a smaller partition may now be kept on node 1 as opposed to node 2. Let us now define the data placement problem as: we are given a query workload, that describes all the query templates for an application, and the percentage of queries of each template that the application typically processes. Determine the best possible placement of the tables on the different database server nodes. One can optimize based on different criteria: for instance, we can minimize response time, maximize total throughput, minimize latency, minize the maximum load on the database servers etc.

3

Balancing the Load across Multiple Database Servers

As described before, our solution to balance the load across multiple database servers consists of two core parts: (a) the data placement algorithm that produces the best possible layout structure and distributes the tables across the multiple database servers according to this layout structure, and (b) the query router that utilizes this layout structure for routing queries while ensuring that the database servers are utilized efficiently. 3.1

Data Placement Solution

Given the query workload, we want to determine a possible placement, such as the one shown in Figure 3. Figure 3 shows that the users, schools and user roles tables are fully replicated across all nodes. The problems table is placed on node 1, and the logged action is horizontally partitioned uniformly across nodes 2-5. In this section, we describe our algorithm that uses a cost-based approach that given any query workload determines the best possible placement of the tables. Our data placement algorithm is shown in Figure 2. Let us examine this data placement algorithm in detail. The dataLayout is the data structure that returns the best possible placement as determined by our algorithm. First, a pair (described in Step 3) consists


49

Step0. Determine the cost for each query template by running each template on a lightly loaded node. Step1. Initialize an array dataLayout that maintains the current data placed on each database server. The initial data on each database server is set to empty. Step2. Initialize an array, currLoad that maintains the current load on each database server. The initial load for each database server is set to 0. Step3. For each pair, initialize setOfOptions to all possible options. // for instance setOfOptions = {replication, horizontal partition, vertical partition, de-normalization} Step4. For every query in template, remove invalid options from the setOfOptions. // vertical partition and denormalization are invalid if there is an update on the table Step5. Sort the query templates according to the cost, from the most expensive to the least expensive. Step6. Iterate through the sorted list of query templates in a greedy fashion, and for each query template, Step 6.1. “Search” for the best possible placement for every table in the query. Step 6.2. Update the dataLayout array to indicate the data on each database server after this placement. Step 6.3. Update the currLoad array to indicate the load on each database server after this placement. // The currLoad array will reflect the cost for updates on these tables as well. Step7. Layout the tables across the different database servers according to the dataLayout array.

Fig. 2. Data Placement Algorithm. The dataLayout array returns the best possible layout of the tables across the different database servers.

of the table that is accessed by the template. For instance, for Q1 in Table 1, we consider , whereas for Q4, we consider . For a join query, say Qi that joins tables T1, T2, we consider and . Also, the set of options described in Steps 3 and 4 can be modified based on what options are suitable for a specific application. One could perform an exhaustive search for determining the best possible placement of the tables, but such an exhaustive search would be exponential in both the number of query templates as well as the number of nodes, which is not reasonable. Therefore our solution uses a greedy algorithm, considering the most expensive query first. This ensures that the algorithm is polynomial in the number of query templates. Step 6 is the crux of the algorithm. Step 6.1 searches for the best possible placement of the tables for a specific query. Here again, the options considered significantly impacts the performance of the algorithm. For instance, what different ratios of placement do we consider for horizontal partitioning; on which

50


Table 1. Example illustrating Query Templates and Workload. # of rows denotes the number of rows in the tables accessed by the query. Query Template Table name % of queries # of rows 1 SELECT * FROM schools schools < 1% 321 WHERE school.id=? 2 SELECT * FROM schools schools < 1% WHERE schools.name=? 3 SELECT * FROM schools schools < 1% 4 SELECT * FROM users users 19% 30826 WHERE users.id=? 5 SELECT * FROM users users < 1% WHERE users.login=? 6 UPDATE users WHERE users < 1% users.id=? 7 INSERT INTO users users < 1% 8 SELECT * FROM problems problems 13% 20566 WHERE problem.assignment id=? 9 SELECT * FROM problems problems 15% WHERE problems.id=? 10 SELECT * FROM problems problems < 1% WHERE problems.scaffold id=? 11 UPDATE problems WHERE problems 1% problems.id=? 12 DELETE problems WHERE problems 1% problems.id=? 13 SELECT * FROM user roles user roles 19% 42248 WHERE user roles.id=? 14 INSERT INTO user roles user roles < 1% 15 UPDATE logged action WHERE logged action 16% 7274174 logged action.user id=? 16 INSERT INTO logged action logged action 16%

database servers do we replicate a table? If we consider k options for placement of a table per database server, the number of options to be considered is k n (exponential in the number of database servers). In our implementation (discussed in Section 4), we decrease the number of options considered by several means. For instance, for horizontal partitioning of a table, we consider only one option: partition the table based on the currLoad on the different database servers. This ensures that our algorithm is polynomial in the number of nodes as well. Once the layout of the tables for a query template has been determined, Step 6.3 of our placement algorithm updates the load on the different database servers. For determining the load on the different database servers, there are multiple options: we can actually perform the layout and empirically measure the cost, or we can estimate using other means. Step 7 performs the actual best possible layout of the data across the database servers. For the example in Table 1, our placement algorithm determined that the final best possible placement is as shown in Figure 3.

Performance Driven Database Design for Scalable Web Applications Table Node

schools

users

user_roles

problems

replication

replication

replication

placement

replication

replication

replication

horizontal partition Ratio: 25%

replication

replication

replication


replication

replication

replication


replication

replication

replication


51

logged_action

node1

node2

node3

node4

node5

Fig. 3. Optimum Placement as Determined by our Intelligent Placement Algorithm for the Query Workload in Table 1

3.2

Routing the Queries

After the data is laid out across the different database servers, the system is ready to start processing the queries. The query router will route the queries to the appropriate database server/(s): a select query is sent to the appropriate database server, and an UDI query is sent to all the appropriate database servers. For performing the routing, the query router utilizes the dataLayout that is returned by the data placement algorithm. In addition to routing the queries correctly, the query router must also ensure that the database servers are utilized effectively. For this, we need to be executing multiple queries on any database server at any instant, while also maintaining the correct in-order semantics specified by the application. Our soultion includes an efficient query router that maintains multiple connections for each database server, thus enabling multiple concurrent queries on a database server. When multiple queries are executed on a single database server concurrently, we need to implement a locking mechanism to ensure the correct in-order semantics. Relying on the locking mechanism available at a database server is not sufficient. The locking mechanism provided by the query router must ensure that when there are two UDI queries against the same table, the two updates are performed in the order in which the requests arrived at the query router (similarly for an UDI query and a select query). Our implementation includes a simple locking mechanism for handling conflicting queries as will be described in detail in Section 4.

4

System Implementation

In this section, we describe our SIPD (System for Intelligent Placement of Data) implementation. We describe the choices made for our data placement algorithm, and the details of our query router implementation. Our implementation is based on the Python (http://www.python.org) language and uses the PostgreSQL (http://www.postgresql.org) database server.

52

4.1


Implementation of Data Placement

One of the first things that we need is to determine the cost for executing a query on a node. For this, we have multiple options: we can estimate the cost (using EXPLAIN or EXPLAIN ANALYZE), or we can empirically measure the cost. The technique that we use for determining the cost is orthogonal to the rest of our solution. For our implementation, we follow the approach described in [5], where the authors observed that the costs are more accurately determined by executing the queries on a lightly loaded node and measuring the cost. We have implemented a simplified version of the data placement algorithm mentioned in Figure 2, where a table is either horizontally partitioned, fully replicated, or placed on exactly one node. Once the best possible placement is determined, the tables are actually laid out onto the different database servers. For performing the placement, suppose a table is determined to be partitioned across k nodes (say node 1 through node k) based on column c, and pi percentage of data must be placed on node i. We partition the data using the values of the c column using a hash function that results in 100 buckets. On each database server node, we place the appropriate range of buckets. Note that this may result in some skewness in the data placement, and the placement may not exactly obey the percentages determined as optimum; however, if we choose a good hash function, the skewness can be minimized. After the placement is done, we are now ready to process the incoming requests, as we will describe in the following section. 4.2

Query Router Implementation

To process a request, we need a query router that routes the query to the appropriate database server. Our detailed architecture for routing queries is shown in Figure 4.

T h r e a d # 1 Q u e u e # 1 k connections AS1

DB1

.

Main Thread

..

Request Queue

Thread Connector

T h r e a d # 2 Q u e u e # 2 k connections

..

ASn

..

DB2

Q u e u e # m k connections

..

DBm Thread#m

Application Servers

Database Servers

Fig. 4. Architecture of the Query Router. The query router routes the queries from the different application servers to the appropriate database server/(s). The thread that handles the requests for a database server maintains a queue of requests that the server needs to process, multiple connections to the server for executing multiple queries concurrently, and a lock table for ensuring in-order semantics among requests to the same server.


53

The queries from all the application servers are sent to the query router, where the requests are queued. The query router also maintains how the tables are placed on the different database server nodes (using the dataLayout structure returned by the placement algorithm); this information is used to route a query to the appropriate database server node/(s). In our system, how to route a query is determined statically and does not vary based on the current load on the database servers. A select query is routed by the query router to one database server, whereas an update query is routed to all the appropriate database servers. For example, a query of type Q1 may be routed to node 1; a query of type Q2 may be routed to node 5; a query of type Q6 has to be routed to all the five nodes. For replicated tables when a query can be answered by more than one node, our system routes the queries in a simple round-robin fashion. This ensures that the database servers are equally loaded. Note that we have made several assumptions: all database servers are homogeneous and take the same time to execute a query; the number of hops and the bandwidth from any application sever to any database server are equal, thus guaranteeing the same network latency; if multiple database server nodes have a replica of a table, then the load across these server nodes for this table is distributed uniformly. Each database server is managed by a thread that maintains two data structures: a queue of requests it has received, and a lock table to handle conflicting select and UDI queries. In order to increase the performance of each database server, the thread for the database server maintains multiple connections to that server; thus multiple queries can be executed simultaneously on a single server (See Figure 4). If multiple queries can be scheduled simultaneously on a database server, we need to implement a simple locking mechanism. Let us illustrate how the locking mechanism is implemented in our system using a lock table. Consider queries of type Q4 and Q7 that are conflicting: Q4 reads from the users table while Q7 inserts into the users table. If there is a query of type Q4 and a query of type Q7 both waiting to be serviced in that order, they cannot be scheduled simultaneously. Rather, we have to wait for Q4 to be finished before Q7 is scheduled. We cannot let the database server handle the conflict management, because it will not guarantee the serial order of Q4 and Q7. Such conflicts are handled using the lock table as follows: first the thread for the database server examines the current query and sees if it can obtain the appropriate locks (read/exclusive lock). If the locks are available, then the query is scheduled on one of the available connections; otherwise, it waits till the lock is available and then the query is scheduled on one of the connections. When the query is finished, the locks are updated accordingly. While a query is waiting for a lock to be available, the following queries in the thread queue are not scheduled (even though locks may be available for those queries); this is done for simplifying our architecture.

5

Experimental Results

Our first set of experiments are aimed at illustrating the performance benefits of some of the data placement techniques. In Figure 5, we show the performance

54


Fig. 5. Illustrating the benefits of horizontal partition on select and update queries. As the size of the partition decreases, the time to execute a select or update query decreases.

benefits achieved by horizontally partitioning the data for select and UDI queries. We started with a single partition having 100% of the data, then two partitions each having 50% of the data till 10 paritions each having 10% of the data. The time to execute a select query decreased as the size of the partition decreased; so does the time to execute an update query. We obtained similar numbers for other placement techniques as well (vertical paritioning and denormalization). After this initial set of experiments, we evaluated the results obtained from our intelligent placement algorithm. We compared the throughput of this best possible layout with full replication. For our tests, we ran 3262 queries in the same ratio as in the query workload described in Table 1. Our five database server nodes had the following configurations: node 1 is an Intel Pentium 4, 3 GHz machine with 2 GB RAM, running 32 bit Windows XP; nodes 2 - 5 are Intel Xeon 4 Core CPU, with 8 GB RAM running FreeBSD 7.1 i386. The database software used on all the five nodes is Postgres version 8.2.9. Our simulated application server which issued the 3262 queries was an Intel Pentium 4, 3 GHz machine with 4 GB RAM running Ubuntu 4.1.2 OS. The code for this application is written in Python version 2.5. The bandwidth between the application server and the different database server nodes is 100 Mbps, and the number of hops from the application server to the database servers are equal. For our layout, the problems table was placed on database server node 5, and the logged action table was horizontally partitioned based on user id column equally across nodes 1 through 4. Figure 6 illustrates the total time that it took for each database server node to finish executing all the queries routed to that server by the query router. Also for full replication, it took the five nodes a total of around 180 seconds to finish executing all the 3262 queries. For the optimum placement, the five nodes finished executing all the 3262 queries in a total of around 81 seconds. Note that the database server node 5 is more heavily loaded under optimum placement because it has to execute all the queries on the problems table (Q8 - Q12). It is


55

Fig. 6. Illustrating our Data Placement Algorithm Results. For each node, it shows the time it took to execute the query set for full replication, and for optimum placement, along with the standard deviation.

possible to schedule fewer of the queries on the fully replicated tables (Q1 - Q5) on node 5 to make all the nodes more equally loaded.

6

Discussion

The techniques described in this paper assume that the data is distributed on different database servers in such a manner that any select query is answered by one database server. However there are significant work that have studied scenarios without this constraint. Distributed databases and distributed query processing [10] have long studied how to process queries over data distributed across multiple nodes. However the constraint that any select query is answered by one database server is applicable to several applications, especially web applications where all the query templates are known beforehand. This constraint also greatly simplifies query processing and optimization, as no data needs to be exchanged between nodes. Therefore such a system has to only determine which database server needs to execute a query, and then the optimization and execution of the query proceeds on that server as if it was a non-distributed database. Also, as we examine web applications, we see that the load across application servers can be easily balanced and the application server layer scales up easily. This is because the application server logic can be easily replicated across any number of nodes. One potential opportunity for database scalability is to pull some of the database functionality that can be easily replicated out of the database server. For instance, a selection operation that scans a set of rows and selects rows based on a filter condition can be pulled outside the database server. The selection operation can be easily replicated across multiple servers. However this comes at a cost: the database server may be able to perform the selection more efficiently, for instance, by building an index, whereas these options may not be available in the selection operation outside the database server. Note

56


that this is different from full-fledged distributed query processing where different nodes can perform different operations. We believe that this is a promising direction, that we plan to investigate in the future. In real systems, we encounter system crashes quite often, and these crashes also need to be handled. In this paper, we did not consider fault tolerance. Incorporating fault tolerance into the problem definition could potentially lead to interesting results. For instance, one way of formulating the problem definition with fault tolerance is to impose a constraint that every data item is present in at least two nodes. This is also a promising research direction, worth investigating in future. Another aspect of fault tolerance is how to handle if an UDI query fails on some nodes, and succeeds in other nodes. How do we detect this scenario, and also how do we remedy such an inconsistency. One can think of a distributed transaction protocol, but such distributed transactions are very heavy weight, and drastically bring down the performance of a system. We therefore need to investigate different semantics as may be applicable for these scenarios, and which can be implemented without drastically impacting the performance of the overall system.

7

Related Work

There has been considerable amount of work for distributing load across multiple database servers. In [3,8,9,12], the authors study full replication, where all the nodes have an exact replica of the data, and where data consistency is achieved using distributed transactions that are heavy-weight. Commercial systems such as Oracle also support full replication, but UDI queries are not performed using distributed transactions; rather, the updates are performed on a master and then the update logs are propagated to the slave nodes. In [4], the application programmers can choose the data replication and distribution strategies, but choosing such strategies efficiently is not easy for an application programmer. Partial replication is studied in [14,5]; in [14], the replication is at a recordlevel granularity requiring a node to hold the entire database and thus being the bottleneck; in [5], the replication is at a table-level granularity, and no node needs to have the entire database. For improving the performance of database systems, de-normalization has been studied in several projects [13,7,16]. One of the main purposes of denormalization is to decrease the number of tables that must be accessed to answer a query; this is because some joins are already pre-computed during the de-normalization process. Another technique that is critical for improved performance of applications is caching [15,1]. If the results of a query are cached, it is possible for the application server to answer a query directly from the cache without accessing the database server. This can be critical if the network bandwidth between the application and database server is low. Efficiently maintaining the consistency of cache is studied in [15].


8

57

Conclusions and Future Work

In this paper, we studied the problem of scalability in web applications, in specific we considered distributing load across multiple database servers. We proposed a data placement algorithm that can consider multiple data placement techniques and determine the best possible layout of tables across multiple database servers for a given query workload. For routing the queries, we have developed an efficient query router; the query router routes the queries to the appropriate database server/(s). The query router maintains multiple connections for each database server to ensure that the database servers are utilized efficiently; also a simple locking mechanism is supported to handle conflicting queries. Our solutions are integrated into the SIPD (System for Intelligent Placement of Data) that we have developed. Experimental results indicate the significant performance benefits achieved by our system. There are several issues and approaches that need to be investigated for scalability of database servers. Some of the potential future directions for research include pulling some functionality out of the database server to enable easy replication of this logic, distributed query processing in general, considering fault tolerance as an application constraint, and handling inconsistencies that may result if an operation fails on some nodes and succeeds on other nodes. Also with respect to our approach, other effective locking mechanisms that operate at a finer granularity and that can achieve better performance need to be investigated. To increase the systmem performance, we have to investigate different techniques to decentralize our query router and to avoid inappropriate locking mechanism with proper caching [11] or query planning [2] solutions. Evaluating our solution against many different web applications to illustrate the benefits of our approach will also be useful.

References 1. Amiri, K., Park, S., Tewari, R., Padmanabhan, S.: DBProxy: A Dynamic Data Cache for Web Applications. In: IEEE Int’l Conference on Data Engineering (ICDE), Bangalore, India (March 2003) 2. B¨ ohm, K., Mlivoncic, M., Weber, R.: Quality-aware and load-sensitive planning of image similarity queries. In: Proceedings of the 17th International Conference on Data Engineering, Washington, DC, USA, pp. 401–410 (2001) 3. Cecchet, E.: C-JDBC: A Middleware Framework for Database Clustering. IEEE Data Engineering Bulletin 27(2), 19–26 (2004) 4. Gao, L., Dahlin, M., Nayate, A., Zheng, J., Iyengar, A.: Application Specific Data Replication for Edge Services. In: Int’l World Wide Web Conf. (WWW), Budapest, Hungary (May 2003) 5. Groothuyse, T., Sivasubramanian, S., Pierre, G.: GlobeTP: Template-Based Database Replication for Scalable Web Applications. In: Int’l World Wide Web Conf. (WWW), Alberta, Canada (May 2007) 6. Heffernan, N.T., Turner, T.E., Lourenco, A.L.N., Macasek, M.A., Nuzzo-Jones, G., Koedinger, K.R.: The ASSISTment Builder: Towards an Analysis of Cost Effectiveness of ITS creation. In: FLAIRS, Florida, USA (2006)

58


7. Inmon, W.H.: Information Engineering for the Practitioner: Putting Theory Into Practice. Prentice Hall, Englewood Cliffs (1988) 8. Kemme, B., Alonso, G.: Don’t be Lazy, be Consistent: Postgres-R, a New Way to Implement Database Replication. In: Int’l Conference on Very Large Data Bases (VLDB), Cairo, Egypt (September 2000) 9. Plattner, C., Alonso, G.: Ganymed: Scalable Replication for Transactional Web Applications. In: Jacobsen, H.-A. (ed.) Middleware 2004. LNCS, vol. 3231, pp. 155–174. Springer, Heidelberg (2004) 10. Ramakrishnan, R., Gehrke, J.: Database Management Systems, 3rd edn. McGrawHill, New York (2003) 11. R¨ ohm, U., B¨ ohm, K., Schek, H.-J.: Cache-aware query routing in a cluster of databases. In: Proceedings of the 17th International Conference on Data Engineering, Washington, DC, USA, pp. 641–650 (2001) 12. Ronstrom, M., Thalmann, L.: MySQL Cluster Architecture Overview. MySQL Teachnical White Paper (April 2004) 13. Schkolnick, M., Sorenson, P.: Denormalization: A Performance Oriented Database Design Technique. In: AICA Congress, Bologna, Italy (1980) 14. Sivasubramanian, S., Pierre, G., van Steen, M.: GlobeDB: Autonomic Data Replication for Web Applications. In: Int’l World Wide Web Conf. (WWW), Chiba, Japan (May 2005) 15. Tolia, N., Satyanarayanan, M.: Consistency-Preserving Caching of Dynamic Database Content. In: Int’l World Wide Web Conf. (WWW), Alberta, Canada (May 2007) 16. Westland, J.C.: Economic Incentives for Database Normalization. Information Processing and Management 28(5), 647–662 (1992)