Multidatabase Query Optimization - CiteSeerX

Distributed and Parallel Databases, 5, 1{39 (1997)

1997 Kluwer Academic Publishers, Boston. Manufactured in The Netherlands. c

Multidatabase Query Optimization * CEM EVRENDILEK ASUMAN DOGAC SENA NURAL AND FATMA OZCAN

Software Research and Development Center Middle East Technical University (METU) 06531 Ankara Turkiye

[email protected]

Editor: Tamer O zsu Abstract. A multidatabase system (MDBS) allows the users to simultaneously access heterogeneous, and autonomous databases using an integrated schema and a single global query language. The query optimization problem in MDBSs is quite dierent from the query optimization problem in distributed homogeneous databases due to schema heterogeneity and autonomy of local database systems. In this work, we consider the optimization of query distribution in case of data replication and the optimization of intersite joins, that is, the join of the results returned by the local sites in response to the global subqueries. The algorithms presented for the optimization of intersite joins try to maximize the parallelism in execution and take the federated nature of the problem into account. It has also been shown through a comparative performance study that the proposed intersite join optimization algorithms are ecient. The approach presented can easily be generalized to any operation required for intersite query processing. The query optimization scheme presented in this paper is being implemented within the scope of a multidatabase system which is based on OMG's object management architecture. Keywords: Multidatabases, query optimization, Heterogeneity

1. Introduction Thousands of heterogeneous information resources on various platforms could be immediately made available for many users through the existing computer networks. However, database autonomy and heterogeneity still form a severe bottleneck for the development of eective interoperable information systems. Thus, there is a growing need for tools to maximize the portability, reusability and interoperability of arbitrary computing services while maintaining the autonomy of the pre-existing databases in a federated approach. Database providers, participating in such a federation, do not lose control over their data, and ideally do not have to re-engineer their database management systems and applications to allow interoperability. One way of achieving interoperability among heterogeneous, and autonomous database management systems (DBMS) is through a multidatabase system that supports a single common data model and a single global query language on top of * This work is partially being supported by the Turkish State Planning Organization, Project Number: AFP-03-12DPT.95K120500, by the Scienti c and Technical Research Council of Turkey, Project Number: EEEAG-Yazilim5, by Motorola (USA) and by Sevgi Holding (Turkey)

2

C. EVRENDILEK AND A. DOGAC

dierent types of existing database systems. The global schema of a multidatabase system results from the integration of the schemas exported from the underlying local databases which may be relational and/or object oriented DBMSs. A global query language is available to the users of a multidatabase system to specify queries against the global schema. Three steps are necessary to process global queries which are issued against the global schema: First, a global query is decomposed into global subqueries such that the data needed by each subquery are available from one local database. Next, each global subquery is translated to a query or queries of the corresponding local database system and sent there for execution. Finally, the results returned by the global subqueries are combined into the nal answer. Query decomposition, itself, is usually accomplished in two steps. In the rst step, the global query is modi ed such that it references the names in the export schemas rather than the names in the global schema. Since the modi ed queries may still reference more than one database, they are decomposed. The decomposed queries are in the global query language so the global subqueries must also be translated into the respective local languages. Provided that a platform which allows semantic interoperation and easy integration of participating information systems exists, giving a logically uniform and a single view of a DBMS, the importance of exible and ecient access and retrieval of information increases. At this stage, distribution and optimization of queries become important and these problems have not yet been fully explored. Distinctive features of multidatabase systems give rise to a number of unique issues which do not arise in query optimization for homogeneous distributed database management systems. The important issues are the following [17]: Any component DBMS may terminate its services without any advance notice (Site Autonomy). Statistical information needed for eective global query optimization are not readily available and may not remain accurate as component systems evolve over time (Site Autonomy). Multidatabase systems can only interact with component DBMSs via their application program interfaces. This means that there exists no opportunity for low level cooperation accross systems and hence query processing techniques (e.g., semijoin) proposed for distributed DBMSs may no longer be applicable (Execution Autonomy). Communication protocols used, availability and processing costs of component sites are dierent which have a signi cant impact on the generation of query processing plans (System Heterogeneity). In a multidatabase system, same data may be represented dierently in dierent component DBMSs. So the identi cation of the same real world objects and context mediations required imply the need for new query processing methods

MULTIDATABASE QUERY OPTIMIZATION

3

and has an important impact on query processing and optimization (Semantic Heterogeneity). In this paper, we describe an approach to multidatabase query processing and optimization. Speci cally, we consider two problems related to multidatabase systems: (1) The optimization of distribution of global subqueries after query decomposition in case of data replication, and (2) the optimization of intersite joins, that is, the join of the results returned by the local sites in response to the global subqueries [11]. The approach we propose could also apply to distributed DBMSs. In such a case, however, the algorithms would become much simpler since parameters coming from heterogeneity and autonomy disappear. For example, there would be no need to estimate appearance times, and the process of scheduling an intersite join would reduce to the decision on whether the involved semijoin is bene cial or not. We formulate the query distribution in multidatabases in case of data replication as the following optimization problem: Given an initial assignment of relations, class extensions and fragments to the sites, assign the global subqueries to the sites after they are modi ed and decomposed such that the maximum of the parallel execution times is optimized. For this NP-Complete assignment problem we propose a heuristic solution. It should be noted that the non-trivial task of query decomposition and modi cation which has been extensively studied in the literature [2, 24], is not within the scope of this paper. In optimizing intersite joins, a two-step heuristic algorithm is proposed which incorporates parameters stemming from the nature of MDBSs such as the time taken by global subqueries at the local sites, transmission overhead incurred while communicating intermediate results during intersite join processing and the cost of converting global subquery results into the canonical data model. The rst step produces a linear order from the global query graph where the most pro table joins are juxtaposed in a linear sequence. In the second step, this order is exploited to obtain a schedule which increases the parallelism in execution of intersite joins. In order to increase the parallelism further, the paper proposes another algorithm which works on a dierent order obtained through a greedy traversal of the global query graph. The results of a comparative performance analysis indicate that the suggested technique used in the intersite join optimization algorithms has a superior performance. The paper is organized as follows: Section 2 summarizes the related work and the issues in multidatabase query processing and optimization. The architecture of a multidatabase management system adopted in our implementation and processing of an intersite join in such an environment are outlined in Section 3. Section 4 addresses global subquery assignment in the presence of data replication. Optimization of intersite joins is discussed in Section 5 and two optimization algorithms are presented. In Section 5.1, we present our approach for nding a linear order for a given global query that places the most pro table join pairs together in a linear order. The second step of the algorithm not only exploits this property of the linear order in nding the bushy inners, but also takes the appearance times, the communication and the conversion costs of subqueries into account so that asynchronous

4


execution among global query processors and overlapping of communication and computation are considered. We refer to the process of nding bushy inners as parenthesization. This step is described in Section 5.2. In Section 5.3, another algorithm to increase the utilization of available parallelism in multidatabases is presented. By means of this algorithm, we investigate whether it is possible to increase the performance of query processing by nding ways of increasing the parallelism. Although the technique presented increases the parallelism, its complexity hinders its performance. Section 6 presents the performance evaluation of the intersite join optimization algorithms proposed in this paper in comparison to the work presented in [9]. Finally, the conclusions and future work are given in Section 7.

2. Related Work Some previous work in global query optimization in multidatabases have used the techniques developed for homogeneous distributed database systems. However, such direct application of these query-processing algorithms to the multidatabase environments is not appropriate because of the heterogeneity and the autonomy of local DBMSs. Local autonomy makes it impossible to intervene with local query processing. Furthermore, preparing data for transmission in a multidatabase system may be much more expensive than that in a homogeneous distributed database system and this may aect the optimization algorithm. The distributed query optimization algorithm described in [31] is organized as a sequence of steps such that at each step all local DBMSs work in parallel to evaluate the cost of execution plans for partial queries of increasing sizes, and send their cost estimates to other local DBMSs that need them for the next step. No duplicate computation is performed and the execution plans that may not lead to an optimal solution are discarded as soon as possible. The algorithm works as follows: Each site evaluates the cost estimates for its own local subquery and sends the corresponding partial query execution plan (PQEP) and its estimated cost to all of its neighbors. The neighbors of a node are determined from the query graph as the set of nodes that have an edge to this node. When a node receives PQEPs, it selects the one with the least estimated cost from each set of comparable PQEPs and discards the others. For each surviving PQEP, new PQEPs are generated by including this site's partial result. The process is repeated until the complete query execution plan (QEP) is obtained, at which point all the sites involved in the query know the optimal QEP and are ready to cooperate in the query execution plan. The number of steps involved in the algorithm is linear with the number of sites, n. The number of PQEPs evaluated by the algorithm depends on the topology of the query graph and for a fully connected graph, it is O(n2n). The number of messages exchanged, on the other hand, is O(n2 2n ) in the case of a fully connected graph [31]. Although the query optimization is performed in parallel, this algorithm produces only serial query execution plans. Furthermore, the algorithm does not consider


5

the plans with bushy inners. Bushy tree plans are binary trees where both left and right subtrees may contain join nodes. In a left deep tree, however, join nodes are not allowed in the right subtree, at each node a single table is joined with the results of the previous join in a pipelined fashion [14]. [34] states that left deep trees do not necessarily include the optimal plan for a parallel environment. It is further shown in [35] through experiments that bushy tree plans perform better. An intuitive example to clarify the point is as follows: Assume the left deep join tree (((AB )C )D) is given. Although the least costly join to select rst is (BC ), the best plan could be ((AB )(CD)). A technique is proposed in [9] to reduce the query response time based on the following observations: Execution of multidatabase queries diers from that of traditional queries in that sort merge and hash joins are more often favored, as nested loop join requires repeated accesses to external data sources. Yet, traditional optimizers (e.g., System-R style [33]) produce left deep join trees which are not suitable for sort merge and hash joins with respect to response time, due to the long delay for a sort merge (or hash) join node to produce its last result after the subordinate join node did. Therefore, the proposed algorithm rst produces an optimal left deep join tree, and then reduces the response time using simple tree transformations. Three algorithms, namely top-down, bottom-up and hybrid are proposed to balance the left deep join trees. The complexity of these algorithms are O(n2 ), where n is the number of nodes in the query graph. The algorithms do not guarantee that the resulting tree is optimal with respect to total response time [9]. The tree transformation algorithms suggested in [9] try to reduce the response time by nding cost reducing bushy inners. However, since they start with a left deep join tree and thus do not consider the bushy inners, they may miss less costly bushy trees. Multidatabase query processing, has also been addressed in [1, 3, 7, 19, 17, 23, 32]. But, [9] and [31] are the only ones that focus on intersite join optimization and only [9] has considered available parallelism to be exploited for query execution in multidatabases. The problem of improving query response time that is addressed in [9] and in our work is orthogonal to the problems studied in the related work. Our technique or the technique presented in [9], can be used in conjunction with any of the query processing strategies outlined in the related work. In [7], through synthetic database calibration, logical cost model parameters of a local DBMS in a MDBS are estimated. In [17], it is claimed that global query optimization problem in multidatabases is fundamentally dierent from global query optimization in homogeneous distributed database systems, and major issues in the design of a multidatabase query optimizer are highlighted. In [23], a survey of the previous research on multidatabase query optimization is presented. In [3], it is stated that conventional distributed DBMSs use a variety of tactics for reducing the cost of processing queries. One commonly used tactic is to perform selections and projections at individual sites before executing any intersite joins. A second common tactic is to distribute selections, projections and joins over

6


unions. Another distributed query processing strategy involves using semijoins for reducing the sizes of join operands. These tactics are of limited applicability in a multidatabase system where queries can include outerjoins [28] and aggregates. However, it is still desirable whenever possible to perform local selections before moving data to the global site. The semiouterjoin technique is proposed in [3] as an approach to make local selections possible. Semiouterjoin operation produces two results, a semijoin and its complement. As pointed out in the paper this technique works well in all cases, no matter what the aggregation and selection conditions are. In [1] and [32], role-based query processing in multidatabase systems are considered and methods are proposed. The approach suggested in [32] is based on the idea of presenting the answer to a query as a set of sets representing the distinct intersections between the relations corresponding to the various roles. This set is called a role set. Role-based query processing of [32] aims at minimizing the amount of data to be transmitted between the local sites and the global site as well as reducing the processing costs at the global site for dealing with intermediate data and producing the nal joins as in [3]. The intermediate data to be generated at the global site requires the generation of private and overlap parts of a query. For the join operation, after selections are performed, the global site sends the identi ers to each local site and each local site sends join attributes corresponding to each identi er [32]. If the join condition involves an aggregation, aggregate values are computed. Then, the role-sets are found by scanning each relation once comparing the join attribute values and the quali ed tuples are stored in the buer. Using role-sets, a new method for data integration that maintains the materialization autonomy of local database systems is developed. In addition, this method reduces data shipments between sites by allowing local selections to be performed for all queries with/without aggregation. The method suggested in [3], on the other hand, requires expensive semiouterjoins in order to nd the private and overlap parts which is done more eciently via a merge-sort in [32]. In [19], query optimization and processing in Federated Database Management Systems (FDBMS) are discussed and methods for query processing and optimization are proposed. Query processing in [19] is treated at two levels: federation level and local level. Activities are assigned at respective levels to Federation Query Manager (FQM) and Federated Query Agent (FQA). In [19], gateways between FQAs and component DBMSs make the local DBMSs appear to be supporting the common data model. Gateways handle query statements in the common data manipulation language, and translate them into local query commands. The results returned by the local DBMS are transformed to the common data model. Each gateway has to make its supported set of relational operations known to the federation so that the gateway will not be asked to handle the unsupported operations during query execution. It is stated in [19] that in a FDBMS, the distinct feature of autonomy implies that a detailed cost-based query optimization is very dicult, if not impossible. For example, if local autonomy is strictly observed, and usage pattern of other


7

local applications is not available, it is unlikely that extensive eorts to optimize federated queries based on some hypothetical local cost models (including disk and CPU costs) will produce a global optimal plan. However, since the cost dierence between a good plan and bad plan can be enormous, it is more appropriate to consider avoiding expensive plans in the optimization as primary objective rather than achieving a global optimal solution. Our work is in uenced by the existing work in the literature in the following respects: The query processing architecture suggested in [19] aected the design of the system described in [16], [5], [6], [21] and [22]. Yet, our architecture is based on OMG's distributed object management architecture [30], as explained in Section 3. In this paper, the work presented in [7] is used in estimating the time taken by global subqueries. In [7], it is assumed that participating DBMSs are autonomous and may not be able, even if they are willing, to provide the cost model parameters. The strategy in [7] is to deduce the necessary information by calibrating a given DBMS. They propose a calibrating database which is synthetically created so as to make the process of deducing the cost model coecients reasonably devoid of the unpredictability problems. We use deduced cost model coecients to estimate the time taken by global subqueries at the local DBMSs. Yet, the intersite join optimization algorithms proposed in this paper can still be used eectively when these estimations are not available. Also, in case it is not possible to access statistical information, the global subqueries can be modi ed to obtain some parameters, such as the cardinalities, from the local databases to be used in our cost calculations.

3. An Environment for Multidatabase Query Processing The query optimization approach presented in this paper is being implemented in the MIND (METU INteroperable DBMS) project described in [5, 6, 8, 16, 21, 22]. MIND architecture is based on OMG's1 distributed object management architecture [30]. The infrastructure of the system is built on a CORBA implementation, namely DEC's ObjectBroker2 [16]. We have de ned an interface of a generic Database Object through CORBA IDL and developed multiple implementations of this interface for Sybase3 , Oracle74, Adabas D5 and MOOD (METU ObjectOriented DBMS) [4]. The problem of dealing with the growth of the number of component systems while using a global schema as the reference model is discussed in [24]. What clients see are homogeneous DBMS objects accessible through a common interface. The basic components of the system are a Global Database Agent (GDA) class (Object Factory in CORBA terminology) and a Local Database Agent (LDA) class. When a client wants to interact with MIND, a GDA object (GDAO) is created by sending a create message to the Object Factory. ORB provides the location and implementation transparency for GDAOs. The location of GDAOs


8 CORBA Interface Client Part of the Client Interface

CORBA Interface Global Data Repository

Server Side of the Client Interface

Interface

OQL Parser

Client Program

Cursor Implementation

OQL to Local SQL Type Converters Translator Global Query Processor

Client

Interface to Local DBMS (e.g. CLI)

Local Database Agent Client

Client

GDR

GDAO

LDAO LD

GDAO LDAO LDAO

LD

LD

CORBA Interface

Client side of Interface to CORBA

Client Interface

Server side of Client Interface Cursor Impl.

Server Side of GDR Interface

OQL Parser Decomposer Optimizer

G D R Int.

Global Transaction Manager

GDR Manager

Global Database Agent

Interface to OODBMS

OODBMS

Global Data Repository GDR : Global Data Repository GDA : Global Database Agent LDA: Local Database Agent LD: Local Database

Client Interface Local DBMS Interface Global Data Repository Interface

Global View of MIND

Figure 1. Global Query Processing Architecture.


9

are dynamically determined by the ORB using the information provided by the ORB administrator. The default site for the GDAO is usually the local host to prevent communication costs. A possible con guration dynamically attained upon requests by MIND users, is as shown in Figure 1. A GDAO contains objects of two classes, one responsible for global transaction management, namely Global Transaction Manager (GTM) class, and the other mainly dealing with global query decomposition and optimization, namely Global Query Manager (GQM) class. The results of global subqueries are called partial results. When a global query is submitted to a GDAO, an object of GQM class is created which parses, decomposes and optimizes the query and submits it to a GTM object (GTMO). GQM object (GQMO) obtains the global schema information necessary for the decomposition of the query from a Global Schema Information Manager Object. A GTMO, a GQMO and a LDA Object (LDAO) for each local database connected are created for each session of each client. The GTMO in coordination with the GQMO controls the execution of the global query at the related global query processor components of LDAOs. GTMO sends the global subqueries and the query execution plan to the LDAOs of the related sites. A LDAO after translating the global subquery to the respective local query language, submits it to the local DBMS. Partial results produced by local DBMSs are then converted into the canonical form and are passed to the global query processors of the respective LDAOs. Global query processors are responsible for intersite joins. Due to autonomy, sites have the right to reject cooperation. In this case, global queries are aborted. The global query processors execute the intersite joins in accordance with the execution plan provided by the GQMO. Notice that the query execution plan may require global query processors to pass intermediate results to one another. In other words, global query processors may communicate partial results or results of previously computed intersite joins to each other according to the query execution plan. According to this framework, the parameters involved in global query optimization are as follows:

The appearance times of global subqueries. These are the estimates of global

subquery execution times, i.e., the time it takes from submitting a global subquery to a LDAO until when the partial result appears at the local DBMS. Note that even when the local loads of the local DBMSs are not available it is possible to rationalize the global load of local DBMSs by taking previously scheduled global queries into account

The conversion cost parameter which includes entity identi cation [1] and generalized attribute derivation [19]. For the sake of simplicity, the conversion cost parameter is incorporated into the appearance times of global subqueries. This assumption is realistic since conversion is performed only once in the LDAOs, the rest of global query processing being in the canonical data model. The conversion cost parameter may be a dominating factor, since it takes more time to prepare data for transmission in a multidatabase system [19]. This is another

10


justi cation of our framework that the appearance times should be taken into account in multidatabase query optimization.

The cost of transferring intermediate results among global query processors of

involved LDAOs. The communication cost, is assumed to be proportional to the size of the intermediate result.

The cost of processing a join operation, the operands of which may either be

partial results and/or results of previously computed joins pending at the respective global query processors of LDAOs. Notice that since a global query processor immediately proceeds with the processing of the join operation, the communication and computation may overlap at the global query processors.

All these parameters have an impact on determining the cost of an intersite join operation. The last parameter which is associated with the join operation itself is a special case of an intersite join operation when both operands are available at the same site ready to be joined. It should, therefore, be noted that the cost of a join operation generally accounts for only a fraction of the cost of an intersite join operation. The strategies described in [3] and [32] for processing the relational algebra operators such as selection, projection and join involving aggregation for multidatabases can readily be used within the optimization framework described in this paper. Different join processing strategies like the one given in [32], may easily be incorporated into our query processing and optimization model conforming to the infrastructure of a multidatabase system.

4. Optimizing global subquery assignment in case of data replication Query decomposition has been extensively studied in [3] and [2]. The details of query decomposition in multidatabase systems have also been described in [24] for MIND. In this section, we consider only the problem of distributing the global subqueries to the related sites in case there is more than one candidate site that can process the subquery. Note that this phase starts after the query decomposition process has been completed. When there is data replication in a multidatabase system, there may be more than one site where each global subquery can be executed. The parts of a global query issued against the global integrated schema after decomposition, are termed as global subqueries. We use the terms subqueries and global subqueries interchangeably in the context when the meanings are clear. We state the problem formally as follows: Given the weights w(qi , sj ) which denote the cost of executing global subquery qi at local site sj , and the local load lj of site sj , we need to nd a mapping f : fqi g ! fsj g giving the execution site of each subquery where qi 2 f1; :::; ng and sj 2 f1; :::; mg, such that the cost of mapping, expressed as


max fl + s 2f1;:::;mg j j

X qi :[f (qi )=sj ]

11

w(qi ; sj )g

is minimized. Since nding such a mapping is an NP-Complete problem6 , we suggest a greedy heuristic algorithm that tries to achieve the least increment possible at each step as presented in Algorithm 4.1. In this algorithm, after initializing the set of global subqueries Q and the set of sites S , the cost of executing each global subquery at each possible site are computed. The cost values are copied to a temporary matrix w0 and the local loads of the local sites are added to these temporary cost values if they are available. The while loop continues until all subqueries are assigned to sites. At each iteration of the while loop, a global subquery having the maximum cost in w0 for a site is chosen. If there are more than one global subquery having this maximum value, the subquery with the smallest cost for a site in w0 is chosen. The minimum cost site in w0 is, then, chosen for this global subquery. Finally, the selected global subquery is deleted from Q and the corresponding cost values in w0 of the remaining global subqueries are incremented by the cost w of the deleted global subquery for the selected site.

Algorithm 4.1 (Optimizing Query Decomposition) begin Q=f1,...,ng denotes the set of subqueries ; S=f1,...,mg denotes the set of sites ; Calculate all w(qi , sj ), qi 2 Q and sj 2 S ; w0 (qi ; sj ) = w(qi ; sj ); qi 2 Q and sj 2 S ; w0 (qi ; sj ) = w0 (qi ; sj ) + lj ; qi 2 Q and sj 2 S ; while (Q = 6 ) do f Set subquery=qi 2 Q such that 8qr 2 Q ? qi ; maxsj 2S fw0 (qi ; sj )g maxsk 2S fw0 (qr ; sk )g and 8qr 2 fqj maxsj 2S fw0 (qi ; sj )g = maxsk 2S fw0 (q; sk )g; q 2 Q ? qi g; minsj 2S fw0 (qi ; sj )g minsk 2S fw0 (qr ; sk )g; set site=sj such that w0 (subquery; sj ) = minsk 2S fw0 (subquery; sk )g; Q=Q-subquery; /* subquery assigned to site */ w0 (qi ; site) = w0 (qi ; site) + w(subquery; site); qi 2 Q; g

end. 2 The algorithm assumes that the time taken by the global subqueries can be estimated. The local loads of the local sites are used only if they are available, otherwise the local loads of the sites are accepted as zero. Note that if the local loads of the sites are available, they are collected by the Global Query Manager Object of GDAO shown in the Global Query Processing Architecture given in Figure 1. Note that the cost of processing a subquery at a site can be calculated by obtaining the logical cost parameters through synthetic database calibration [7] and by obtaining the related database statistics from the data dictionaries; If a data


12

Table 1. Initial w0 Values.

Subquery 1 2 3 4

Site 1 w0 (1; 1)=5 w0 (2,1)=6 w0 (3,1)=6 w0 (4,1)=15

Site 2 w0 (1,2)=7 w0 (2,2)=8 w0 (3,2)=7 w0 (4,2)=13

Site 3 w0 (1,3)=9 w0 (2,3)=8 w0 (3,3)=6 w0 (4,3)=14

dictionary is not available then statistics such as cardinality can be obtained from the underlying databases through querying. We have a global schema where schema con icts are resolved by identifying the type of con icts as described in [24]. This global schema contains all the information to decide on query decomposition. If there are n sites and k global subqueries, this algorithm requires the construction of k heaps of n elements each and maintenance of the remaining heaps adjusted at each iteration. A signi cant amount of time will be spent in searching the right element in a heap to update the value, which requires O(n) time. If a separate array that directly points to each element in the heap is kept, we can access the element in O(1) time. A max-heap can be constructed in O(n) time without sorting. Then the time complexity of Algorithm 4.1 is dominated by O(nk) for building k heaps of n elements each and by O(k2 logn) for performing k iterations to nd the maximum value root among the remaining heaps, selecting the minimum value from the heap and for updating the corresponding values in other heaps. As a result, time complexity of Algorithm 4.1 is given by O(nk+k2logn) with the locate array and by O(k2n) without the locate array [11]. We will explain how Algorithm 4.1 works brie y through Example 4.1.

Example 4.1 Assume there are three sites with no local loads and the costs of

executing the subqueries at those sites are given in Table 1 (w values correspond to the initial w0 values). In the rst iteration, the largest value in the table is w0 (4; 1)=15, therefore subquery is set to 4. Since minsj 2f1;2;3gfw0 (subquery; sj )g=w0 (4; 2)= 13, site is chosen as 2, that is, subquery 4 will be executed at site 2 with cost 13. Then subquery 4 is deleted and all w0 (i; 2) where i 2 f1; 2; 3g are increased by w(4,2)=13. For the second iteration, we start with the con guration shown in Table 2. In this iteration, w0 (2; 2) = 21 is maxqi ;sj 2f1;2;3g fw0 (qi ; sj )g resulting in the assignment subquery=2. For minsj 2f1;2;3g fw0 (subquery; sj )g= w0 (2; 1)= 6, subquery 2 will be executed at site 1 with cost 6. Following the deletion of subquery 2, all w0 (i; 1) i 2 f1; 3g are incremented by the value w(2,1)=6. For the third iteration, we start with the con guration given in Table 3.


13

Table 2. w0 Values for the Second Iteration.

Subquery 1 2 3

Site 1 w0 (1,1)=5 w0 (2,1)=6 w0 (3,1)=6

Site 2 w0 (1,2)=20 w0 (2,2)=21 w0 (3,2)=20

Site 3 w0 (1,3)=9 w0 (2,3)=8 w0 (3,3)=6

Table 3. w0 Values for the Third Iteration.

Subquery 1 3

Site 1 w0(1,1)=11 w0(3,1)=12

Site 2 w0(1,2)=20 w0(3,2)=20

Site 3 w0 (1,3)=9 w0 (3,3)=6

In the third iteration, both w0 (1; 2) and w0 (3; 2) have value 20. Since there are more than one candidate for the maximum value, the subquery having the minimum w0 (qi ; sj ) where sj 2 f1; 2; 3g is chosen between the candidate subqueries qi 2 f1; 3g. Since w0 (3; 3) = 6, subquery is set to 3 and site becomes 3, that is, the subquery 3 will be executed at site 3 with cost 6. Consequently, subquery 3 is deleted and w0 (1; 3) is incremented by the value w(3; 3) = 6. Finally at the last iteration, subquery 1 is assigned trivially to site 1. The algorithm provides the optimal solution for this example. 2

5. Optimizing Intersite Joins in Multidatabases In minimizing the total execution time of queries, the critical part of the process is the scheduling of intersite joins. Due to the distributed nature of the problem, the appearance times and the communication costs of global subqueries should be taken into account in addition to the cost of the join operation. Another parameter which is accounted in the appearance time, is the conversion cost arising from the heterogeneity of local DBMSs. In order to estimate the appearance times, we used the approach followed in Pegasus [7]. The DBMSs that participate in a multidatabase system are classi ed into three categories in [7]. Proprietary DBMSs provide all the relevant information on cost functions and database statistics. Conforming DBMSs provide database statistics but are incapable of divulging the cost functions. Non-Conforming DBMSs are incapable of divulging either the database statistics or the cost functions. It is possible to deduce the coecients of the logical cost model of a conforming DBMS through synthetic database calibration as described in [7]. Furthermore, it is argued in [7] that, it is possible to extend this approach to non-conforming DBMSs.

14


Thus, it is assumed that the appearance times of global subqueries from heterogeneous DBMSs can be estimated. The problem that remains to be solved is the optimization of the intersite joins, given the estimates of the appearance times of the subqueries. Since the classical join ordering problem is NP-Complete [15], the intersite join ordering problem of MDBSs is also an NP-Complete problem7. In optimizing the intersite joins, therefore, we propose a two step heuristic algorithm that tries to maximize the parallelism in execution of the join pairs in the global query. In the rst step, a linear join order is obtained from the global query graph to minimize the overall cost of the global query. The second step of the algorithm exploits this order to nd a schedule for query execution, considering the available parallelism in executing the subqueries. A global query graph is a representation of the query submitted to the multidatabase system where nodes represent the global subquery results and edges represent the intersite operations. A schedule is the execution plan of a global query depicting the execution order and execution sites of the operations. It should be noted that the local sites may be assigned more than one global subquery of a global query with dierent appearance times by a GQMO. Using a linear join order in query optimization is an accepted practice and there are heuristics developed for this purpose. For example, System R [33] and R [18] consider execution plans in which a single table is joined at each step with the results of previous joins, in a pipelined way, producing a left deep join tree. In [15], the spanning tree of a query graph is used to nd a linear order. An observation related to these heuristics is that the linear join orders produced are not necessarily the best candidates for parallel execution [20]. Restricting the schedules to synchronized query execution plans as in [20], however, may result in missing an optimal solution which often happens to be an asynchronous bushy plan. A linear order to be used in multidatabase query optimization, therefore, should allow the optimization algorithms to consider promising asynchronous bushy inners so that optimal or near optimal solutions can be found.

5.1. A Heuristic for Generating a Linear Order for a Query Before presenting a heuristic to obtain a linear order from a given global query graph, we make the following observations. The cost and the selectivity of an intersite join operation directly aect the join order. In other words, in nding a linear order from a global query graph, it is reasonable to favor the least costly and most selective intersite join at each iteration. Therefore, we label the edges of a global query graph with a weight function that favors less costly and more selective intersite join operations. Another observation is that the eect of cost and selectivity on the intersite join order varies depending upon their actual values. To be able to take this into account, we suggest a number of weight functions, all favoring less costly and more selective intersite joins, but the degree of the eect of cost and selectivity on the weight function is dierent in each of them.


15

Before proceeding any further, we will provide some de nitions to be used in de ning the weight functions:

De nition 5.1 Size Selectivity. The size selectivity of a join operation, C =

L ./ R where L and R are two intermediate results, is denoted by size (L,R), and de ned as

size (L; R) = nbpages(C )=(nbpages(L) + nbpages(R)) where nbpages(C) is the estimated number of pages of the intermediate result C. 2 This de nition of size selectivity is slightly dierent from the one based on the cartesian product, in that it is the proportion of the estimated size of the result to the input size. It has been observed in [26, 25, 27] that size selectivity is more eective.

De nition 5.2 Per-Unit Cost. The per-unit cost of a join operation, C = L ./

R where L and R are intermediate results and Jcost is the minimum possible cost of performing this join operation using dierent join implementation techniques, is denoted by Pcost(L,R), and de ned as

Pcost (L; R) = Jcost (L; R)=(jLj + jRj) where jLj denotes the cardinality of L. 2

De nition 5.3. Weight Function. It is a measure of estimating the cost of

processing a join. We de ne four weight functions 1 , 2 , 3 , and 4 as shown below: 1 (L,R) = Jcost (L,R)/(1- size (L,R)) 2 (L,R)= Jcost(L,R)*size (L,R) 3 (L,R)= Pcost(L,R)*size (L,R) 4 (L,R)= ln Jcost(L,R)*esize (L;R) 2 In de ning the weight function, some observations have been made. First, the cost and the selectivity of each join operation in a join sequence directly aect the join order. As an example, consider A./B./C with costs Jcost (A,B), Jcost(B,C) and selectivities size (A,B), size (B,C) and assume Jcost (A; B ) > Jcost (B; C ) and size (A; B ) < size (B; C ). If we only consider cost, we will execute B./C rst, but since size (A; B ) < size (B; C ), executing the joins in the order A./(B./C) may be more costly than executing them in the order (A./B)./C, depending upon the values of cost and selectivity. Therefore, both less costly and more selective join operations must be favored at the same time. Second point to be noted is that depending upon the actual cost and selectivity values, it may be bene cial to increase the eect of cost or selectivity in the weight function. With these observations in mind and after some experimentation8 , we have de ned four weight functions 1 , 2 , 3 , and 4 . In each of these functions, low cost and high selectivity are favored, however, from


16

1 (L,R) to 4 (L,R), the eect of cost in the weight function is reduced while the eect of the selectivity is ampli ed. Algorithm 5.1 nds a linear order for a given global query graph. In this algorithm, global query graph QGk , k 2 f1; 2; 3; 4g where the cost of the edges are calculated with respect to k are formed. Then, the algorithm starts contracting nodes favoring the minimum cost edges at each iteration. The cost of the edges emanating from the contracted node are recalculated. Contraction of nodes are performed with respect to Rule 5.1 where in an intermediate contraction, the labels of the corresponding nodes are concatanated taking into account the neighbouring conditions. Rule 5.1 is a heuristic to avoid cartesian products where possible. A linear order obtained through Algorithm 5.1, has the property of having profitable join pairs juxtaposed, since the algorithm places the minimum cost intersite joins together. This property allows the second step of the optimization process called parenthesization to consider promising bushy inners, resulting in faster asynchronous execution plans.

Rule 5.1

8 > > < RL rL LR => R r LR > : LR

if LL RR else if LL RL else if LR RR else

where L and R are the labels of the nodes of the global query graph as shown in Figure 2 and Rr denotes the reverse order of label R. LL RR indicates that the leftmost node in L (LL ) and the rightmost node in R (RR ) are neighbors in the global query graph (i.e. there is a join condition between them). 2

Algorithm 5.1 (Obtaining a Linear Order) begin for k=1 to 4 do f costk =0;

Construct a global query graph QGk ; Calculate k (L,R) for each edge (L,R) 2 QGk ; while (There are edges to be processed) f Select the edge (L,R) with minimum weight; costk =costk + Jcost (L,R); Contract nodes L and R to (LR) by using Rule 5.1; Recalculate k ((LR),), 8 2 QGk adjacent to (LR);

g

g

Choose the linear order with the minimum cost mink2f1::4g costk ; end. 2 In Algorithm 5.1, QGk denotes the global query graph constructed using the weight function k . It is assumed that the decomposition of the global query has


17

Q1

Q1,Q4

5

1

Q2

7

3

Q3

6

4

Q5

3

Q3

Q5

2

2

Q7

Q7

a) Initial Query Graph

Q1,Q4 7

Q6

9

Q2

4

Q4

7

Q6

8

b) Step 1

Q1,Q4

8

4

Q6

5

8

4

Q6

7

Q2

Q5

2

Q2,Q3,Q7

Q5

Q3,Q7 d) Step 3

c) Step 2

Q1,Q4,Q5

Q5,Q4,Q1,Q2,Q3,Q7

5

4

3

Q6

Q6

Q2,Q3,Q7 e) Step 4

f) Step 5

Q5,Q4,Q1,Q2,Q3,Q7,Q6 g) Final linear order

Figure 2. Finding a Linear Order from a Global Query Graph.

been performed before constructing the global query graph. If a min-heap structure is used for maintaining the weights of the edges in the global query graph, it is observed that at least one node diminishes at each iteration. At each iteration, the minimum cost edge is found, this edge is deleted, the two involved nodes are contracted and the cost of the edges emanating from the contracted node are recalculated and the heap is adjusted. If a locate array is used pointing to the elements in the heap, the complexity of Algorithm 5.1 is O(n2 loge) [11] where e denotes the number of edges and n denotes the number of nodes in the global query graph. The complexity would be O(ne + n2 loge) without the locate array [11]. Example 5.1 illustrates how Algorithm 5.1 works.


18

Example 5.1 In the global query graph initially given in Figure 2a, the labels of

the edges denote the weights, i.e., the cost obtained through weight function 1 . At each step of Algorithm 5.1 as shown in Figure 2b-f, the nodes which have the minimum cost edge in between are contracted by Rule 5.1. To demonstrate the need for Rule 5.1, consider the nodes L= Q2 Q3 Q7 and R= Q1 Q4 Q5 in Figure 2e. Since the minimum cost edge at step 4 is between these nodes, they are quali ed to be contracted. The result of the contraction is Q5 Q4 Q1 Q2Q3 Q7 as shown in Figure 2f since (Q2 Q3Q7 )L = Q2 and (Q1 Q4 Q5 )L = Q1 are neighbors. After contraction, the cardinality and unique cardinality of node Q5 Q4Q1 Q2 Q3 Q7 are estimated given the unique cardinalities and cardinalities of the nodes Q2Q3 Q7 and Q1Q4 Q5 [26], [25]. Then, 1 (Q5 Q4Q1 Q2 Q3 Q7 ,Q6 ) is recalculated. In general, cost of all the edges emanating from a contracted node are recalculated. Note that if Rule 5.1 was not applied, then the sequence Q1 Q4Q5 Q2 Q3 Q7 could be obtained, where Q5 and Q2 would be juxtaposed in the linear order. In such a case the cost of join between Q5 and Q2 would be that of a cartesian product since there is no join condition between Q5 and Q2 which would prevent the possibility of considering Q1 Q2 for parallel execution. It should be noted that cartesian products do not prevent parallel execution, but, the algorithm tries to avoid them as a heuristic through Rule 5.1. This process is repeated for 2 , 3 , 4 and the linear order with the minimum join cost is selected. 2

5.2. A Heuristic for Determining Bushy Inners Given a linear order for a global query Q obtained by Algorithm 5.1, the problem now is to minimize the response time by increasing the parallelism of intersite joins. For this purpose, we apply the recurrence relation of Formula 5.1, given below, on the linear order Q(1) Q(2):::Q(n) of query Q where (1)(2):::(n) denotes a permutation of integers 1 i n dictated by Algorithm 5.1.

Formula 5.1

cost(i; i) = t(i) ; 1 i n cost(i; i + 1) = minfjc (Q(i) ; Q(i+1) )site(Q i ) + t(i) ; if t(i+1) + com cost(Q(i+1) ) t(i) ; t(i+1) + com cost (Q(i+1) ); otherwise t(i+1) ; if t(i) + com cost(Q(i) ) t(i+1) t(i) + com cost(Q(i) ); otherwise g; 1 i < n ( )


19

cost(1; n) = min1i