AN INTERACTIVE AGENT-BASED SYSTEM FOR CONCEPT-BASED WEB SEARCH Tsung-Che Tsai
Wei-Po Lee
Department of Management Information Systems National Pingtung University of Science and Technology, Taiwan E-mail:
[email protected]
ABSTRACT Search engines are useful tools in looking for information from the Internet. However, due to the difficulties of specifying appropriate queries and the problems of keyword-based similarity ranking presently encountered by search engines, general users are still not satisfied with the results retrieved. To remedy the above difficulties and problems, in this paper we present a multi-agent framework in which an interactive approach is proposed to iteratively collect a user’s feedback from the pages he has identified. By analyzing the pages gathered, the system can then gradually formulate queries to efficiently describe the content a user is looking for. In our framework, the Evolution Strategies are employed to evolve critical feature words for concept modeling in query formulation. The experimental results show that the framework developed is efficient and useful to enhance the quality of Web search, and the concept-based semantic search can thus be achieved. 1. INTRODUCTION Following the continuously developing of Internet and the World Wide Web, anyone can easily provide information to the Internet. Meanwhile, though it is now convenient to obtain information from the Internet, the rapidly expanded World Wide Web causes the problem of information overload. In order to help users efficiently find the information needed, different types of search engines, such as Yahoo, Google, and Infoseek, have been developed during the last few years. Search engines are useful tools to collect and index Web pages. After receiving a user-specified query, a search engine can use its internal strategy to narrow down the vast Internet information to a certain range, and can retrieve Web pages that match the query for the user. However, because most conventional search engines use the methods of keyword-based similarity ranking, people may spend a lot of time to evaluate results retrieved but eventually could not find the specific information they really needs. This is mainly because in the common keyword-match design, the search engines rank the documents according to their relevancy to the given query in which the relevance is measured by the similarity between a document and a query. The methods used to determine the similarity are normally based on probability models, such as the
well-known vector space model [12], in which the frequency of the query keywords appearing in the documents is the decisive factor. By such a method, search engines may retrieve many irrelevant documents that include the keywords used, while they are not able to retrieve the relevant documents not containing the keywords. The situation becomes worse when a user is looking for some specific concepts but he is not able to accurately specify useful keywords as a query for the search engines. To remedy the above problem, in this paper we present an agent-based system in which an interactive approach is proposed to gradually capture the concept a user has in mind. In our system, an interface agent is developed to receive a user’s query and redirect the query to existing search engines as the general metasearch engines do. An information agent then analyzes the Web pages chosen by the user and derives a temporary profile for him. Most importantly, based on the profile a discovery agent can perform query expansion and modification by the evolutionary strategies for more accurate results. A filtering agent uses the profile to rank the retrieved Web pages for a new query and recommends the most relevant ones to the user. After the user indicates the pages he really needs, those pages are further analyzed and then the profile is updated. The above procedure continues iteratively until a user terminates the search. By this interactive way, a user’s feedback is taken into account and the user’s initial request can be expanded with good query terms, therefore the task of Web search no longer completely relies on a precise query. With the evolution-based query formulation, the concept-based semantic search can be achieved. The prototype system has been evaluated and the preliminary results show the promise and efficiency of our approach. 2. WEB SEARCH As is mentioned, the search engines generally employ a probability model to measure the similarity between a document and a query to represent their corresponding relevance. Often the most popular method used is the vector space model widely accepted in the research of information retrieval. In this model, both documents and queries are represented as word vectors. The similarity of any two vectors is then determined by the angle between them in which the angle can be obtained by their inner product [12]. Though this method can measure the syntactic closeness of a document and a query, it does not mean they are relevant
in context. Hence, the search results by this approach are not satisfactory. In order to improve the search quality, it becomes important for a searching system to automatically form effective queries to capture what the user really needs. A possible way to achieve the concept-based semantic search is to develop a profile acting as a filter to further classify the results retrieved. In fact, this method has been advocated in Web page recommendations in which a profile contains some feature words extracted from the pages a user is interested in and each word has a weight indicating its relative importance. The similarity between a new page and the personal profile is then used to predict whether a user is interested in this page. This approach has been successfully used to develop systems for Web page recommendations (e.g., [7][11]), and it is generally believed that better semantic relevance can thus be captured in this way. It should be noted that the above personalization technique used in Web page recommendation can not directly be applied to the query-driven Web search. In the former case, the user is passive; he normally remains his interests for a period of time, so feature words in his personal profile can be accumulated over time. Yet in the latter case, a user may have his specific targets in each single search, therefore previously collected feature words are not necessarily related to a new search. In addition, the task of concept-based semantic search is subjective; whether a certain page is what the user needs for the concept depends on his own judgment. A Web page relevant to a query or a user’s profile may still not be what a user means for that query, because even the user himself is not capable of specifying the most appropriate query for the contents he is looking for. Under such circumstances, how to generate efficient queries becomes the major problem to solve in this kind of search [3][5]. As the above analysis shows, the concept-based search can not be achieved immediately after a query is specified; the query must be iteratively modified to capture what a user really needs. Also a profile similar to that used in the Web recommendation work is needed to characterize the concept in the user’s mind during the process of query formulation. Some methods have been proposed to create effective queries (e.g., [6][10]), and the most related work is the GA approach used in [10]. Yet, the nested design for evolving queries in their work is very time-consuming to be a practical application in the real world Web search task. As is indicated, the concept-based semantic search is user-centric, therefore in the iterative steps the profile should be developed from the user’s feedback for the Web pages retrieved in an interactive manner. In this way, additional words extracted from the pages newly retrieved can be gradually integrated into the profile and used to generate new queries that can further explore other area of the Web space. By the exploitation of the profile and exploration of Web space, the search quality does not completely rely on the preciseness of the original query any more; the interactive approach can lead the user to what he is targeting and provide the user abundant results. In this work, we take such design principles to develop a system and show that it can greatly improve the quality of traditional Web search.
3. THE SYSTEM FRAMEWORK 3.1 System Framework Developing intelligent agents for Internet-based applications has been advocated in recent years [8][14]. Hence in this work an agent-based methodology is adopted to construct a framework for interactive concept-based Web search. Each agent here autonomously performs a specific sub-task, and different agents work simultaneously to achieve the overall task. To achieve concept-based search, the system we develop includes four agents: an interface agent for interacting with the user and collecting his feedback, an information agent for extracting feature words from the pages gathered, a discovery agent for query formulation, and a filtering agent for re-ranking and recommending pages to the user. The agents work continuously until the user is satisfied with the results and then terminates the search. During the search process, a profile is used to record critical feature words for automatic query generation. Figure 1 illustrates the system framework. In Figure 1, the interface agent receives a user’s initial query and directly sends it to the discovery agent responsible for interacting with the search engines and for organizing the retrieved results. The interface agent also displays the search results from the filtering agent to the user and then collects his feedback that indicates which pages he needs. With the user’s feedback, the system can thus gradually know what he means by the query submitted. Figure 2 shows the typical search results presented by the interface agent. In this figure, the user can follow the hyperlinks to evaluate the Web pages as he usually does with the traditional search engines. He then marks the hyperlinks for identification. After gathering the hyperlinks with positive feedback, the interface agent delivers the corresponding Web pages to the information agent for further analysis. The concept-based semantic search is a subjective task: the same query by different users could mean different contents to them, or different users may use their own queries to search the same concept. It completely depends on a user’s personal opinions. Therefore, the user-system interaction described is an important feature to enable the system to capture what a user means.
user
Interface Agent
Information Agent
Filtering Agent
Discovery Agent
Profile word list
Search Engines
Figure 1: The proposed agent-based framework. The information agent takes the responsibility of analyzing the Web pages selected, and maintaining a word profile that includes useful information to describe the concept a user is looking for. The discovery agent described below will use the profile for query formulation. For a selected Web page, the information agent first removes the HTML tags and those common but not important terms
such as pronouns and prepositions, and then performs a stemming procedure to strip off word endings for statistic purpose. The words left are used to represent this page; and their appearing frequencies in the Web page indicate the corresponding importance. By this way, a page is transferred into a word vector. Consequently the similarity between two pages can be derived from the angle between them. Then the vectors for the pages collected are combined into a profile in which the words from different pages are sorted according to their accumulated frequencies. As is shown in Figure 1 and Figure 2, the system operates in an iterative way and a user can indicate the pages he needs for the search system at each iteration. Hence, the system has to continuously update the profile to take more user feedback into account, and the word frequencies are accumulated from iteration to iteration. As some pages may contain only a small number of words (for example the main page of a specific Web site), the words within them thus have relatively small impacts. In order to prevent bias caused by the page length, for those pages with words less than a pre-defined threshold, the information agent explores the hyperlinks within the pages to extend the page content. The profile derived from the above procedure is then used to generate new queries and to re-rank the retrieved pages later.
because a user tends to find some specific contents within limited time when he is operating the search engine. With the above considerations, a special kind of evolutionary approach is employed in our work to create new queries from the word profile to model what the user is seeking. The details are described in the following section. To evaluate a certain query, the discovery agent sends it to a search mechanism (e.g., an ordinary search engine or a metasearch engine), collects the Web pages retrieved, and then delivers them to the filtering agent for pages re-ranking. After receiving the Web pages, the filtering agent activates the information agent to analyze and transfer them into vector forms, and re-ranks these pages according to their similarities with the accumulated word list in the profile. In other words, the newly retrieved Web pages are re-ranked based on how they are similar to the ones previously indicated as user-needed. To save the processing time, the filtering agent only deals with the pages with top rankings (e.g., the top l pages given by the search engine) by conducting the inner product between each of the page vectors and the profile. According to the measuring results, the pages most similar to the word profile (e.g., the top k pages after re-ranking, k < l) are presented to the user through the interface agent. The user can then select the Web pages with the contents he needs as examples to show the system what he is looking for. As is mentioned, the interface agent will send the pages selected to the information agent, which will then analyze these pages and extract feature words to update the profile. By the above interactive and iterative way, our system can gradually exploit what the user provides and explore new Web regions to retrieve what he needs. 3.2 Query Formulation by Evolution Strategies
Figure 2: The typical results presented by the interface agent; the user can mark the hyperlinks to express the results he is expecting. The discovery agent is the kernel of our system; it is expected to generate new queries to better capture the concept in the user’s mind, based on the relevance feedback provided by the user. Different from the recommendation work described in Section 2, the task of concept-based semantic search is more abstract and restrictive; it is difficult to achieve by the direct measurement of lexical similarity. A more efficient approach is needed to derive sets of words to model what a user means from the pages he has selected. The sets of words produced can then form sensitive queries to explore different regions of the Web space for more accurate results. Also, the method used to construct the discovery agent for query formulation must be time-efficient,
As is analyzed, the query expansion is the major problem to solve in semantic search, especially for the situations that the user is looking for Web contents relevant to some abstract concept. In this work, a special kind of evolutionary algorithm, namely evolutionary strategies (ES) [1], is used to evolve queries for concept modeling. ES model is a kind of evolutionary algorithms that simulate the process of natural evolution to search for the fittest through selection and re-creation over the generations. An evolutionary algorithm operates as a cyclic mechanism which includes a sequence of selecting parents and creating children, after the initialization phase. Selection involves probabilistically choosing individuals from the current population to form a new population, in which the survival probability for an individual is generally based on how fit it is for the specific problem to be solved. The recreation is to apply genetic operators on the parent individuals to generate new ones. In the ES model, a member of the population is typically considered as an individual which is constituted by a set of parameters (components) to be optimized, and it is then a fixed-length real valued string. It is assumed that whatever genetic transformations happen, the resulting change in each trait will follow a Gaussian distribution with zero mean difference and some standard deviation. In ES,
mutation is the primary operator for creating offspring. It is applied to all components simultaneously, and is generally implemented by adding normally distributed random numbers to all components of an individual. The key concept of the ES is that it maintains one global step size (i.e., the standard deviation σ in the equations below) for each individual, and the step size is self-adaptive by the logarithm of normal random numbers [1]. Therefore, ES are allowed to self-adapt to different fitness landscapes. It has been shown that with its special way of recreation, ES are better alternatives to Genetic Algorithms for some problems in which the epistatis exists among the parameters to be optimized [13]. Epistatis describes the interaction (or dependency) of parameters with respect to the fitness of an individual. If it appears, all parameters involved have to be adapted simultaneously so that the overall fitness of an individual can be improved. Hence, with the genetic operators used, GAs are very time consuming for the above kind of problems. Also a mutation-based ES can reduce the negative impact of the permutation problem, and the evolutionary process can therefore become more efficient [1]. In the Web search task, the word frequencies are not mutually independent, and due to the limit of time (the time a user can wait for response), only small numbers of population size and generations are allowed for evolving queries. Therefore the ES method is a better choice than GA here. In this work, the multi-member ES model (μ+λ)-ES is used, where μis the number of individuals in each population; λis the number of offspring created from the parents; and the bestμindividuals selected from the parents and offspring form the next population. As in the traditional ES model, an individual in our system is represented as a vector I = (f1, f2, …, fn, σ1, σ2, …, σn) consisting of n components fi (1 ≤ i ≤ n) and their corresponding n standard deviations σi (1 ≤ i ≤ n) for individual Gaussian mutation. In this representation, each component fi denotes the mutated result of the appearing frequency of feature word wi in the profile and n is a user-specified parameter meaning the number of words with top frequency ranks in the profile to be considered. At each iteration of user-system interaction, the initial population is derived from the updated word profile. Then the evolutionary process continues for a pre-defined number of generations and the best query obtained from the last generation is used to represent the final query of this iteration. During the evolution, in order to create an offspring each individual I is mutated to I′ = (f1′, f2′, …, fn′, σ1′, σ2′, …, σn′) by the following operations: σi′ = σi × exp (τ′ N(0, 1) + τ Ni(0, 1))
and
fi′ = fi + σi′ × Ni(0, 1) Here, N(0, 1) is a normally distributed one-dimension random number with mean zero and variance one; Ni(0, 1) is a random number for component fi ; τ and τ′ are set to the common used values ( 2 n )-1 and ( 2n )-1, respectively. To evaluate an individual I, the discovery agent shown in Figure 1 first collects a set of m words (m < n) with largest fi (i.e., the mutated frequency values) from the
corresponding chromosome. Then the discovery agent uses the conjunction form of these m words to form a query and sends it to the search engine. Though it may be more flexible to use the combination of conjunction and disjunction of different words, the determination of the appropriate combination for a set of words nevertheless implies much more computational cost. It is thus not suitable for the interactive and iterative search task here. In order to avoid the over-constrained search situations in which the number of pages retrieved is less than a threshold, the search condition can be relaxed from the conjunction form by iteratively removing one word from the above query form. The top l pages retrieved by the search engine are then delivered to the filtering agent for re-ranking. The average of the similarity coefficients of the top k pages (k < l) after the re-ranking is thus defined as the fitness of the individual I. Once the fitness values of all individuals have been determined, the bestμindividuals selected from the (μ+λ) competitors are propagated to the next generation. For each iteration, after a pre-defined number of generations, the k pages obtained from the best query of the last generation are presented to the user. During the evolutionary process, the fitness of an individual is estimated in an indirect way by the discovery agent, rather than a direct evaluation by the user. The user only evaluates the results at the end of each iteration. With the user’s feedback, the word profile is updated, and then a new initial population is created and the evolutionary process starts again. The above procedure is repeated until the user terminates his search. 4. EXPERIMENTS AND RESULTS In order to access the proposed framework and the learning approach, this section describes the experiments conducted. In the following experiments, we concentrate on evaluating the effect of the user-system interaction and the performance of the evolutionary mechanism. The former is to investigate whether the user’s feedback is useful in modeling what he means by the query; and the latter, how the learning approach presented can further improve the search quality. 4.1 Evaluating the Effectiveness of a User’s Feedback In this experiments, three volunteers submitted their initial queries to the system (the well-known search engine Yahoo was used as the background search mechanism) to look for the same concept of “research work related to intelligent agent-based systems that involve machine learning techniques and with downloadable publications.” The three queries sent were “computer science”, “artificial intelligence”, and “intelligent agent” representing different views from the individual experimenters for the concept to be searched. In the experiments, the semantic relevance of the results retrieved was determined subjectively by the experimenters rather than objectively by the domain experts. It completely depended on their personal understanding for the concept described. The three values 5, 5, and 7 corresponding to the iteration 0 (i.e., i0) in Table 1 are the results for the above three queries respectively.
They represent the numbers of Web pages a user identified as relevant in semantic from the top 20 pages retrieved by the search engine for each original query. As can be seen, only a small portion of the pages retrieved was chosen in each case. This shows the inefficiency of the traditional search engines. To evaluate the usefulness of user’s feedback, the interactive search procedure was then activated. For each query, the pages selected were analyzed and the feature words extracted were used to modify the profile. The words with top 5 appearing frequencies (determined by preliminary testing) in the profile were combined by the logical operator “and” to form a new query, and it was submitted to the search engine again. In order to save the processing time, only the top 50 pages provided by the search engine were collected and re-ranked by the cosine similarity measurement as described in section 3.1. If the total number of pages retrieved by the search engine was less than 50, the search condition was relaxed by removing the keyword with the smallest frequency value in the query, and then the new combination was re-sent to the search engine. The above procedure continued until there were 50 different pages collected. After that, the top 20 pages most similar to the newly updated word profile were presented to the user for further identification. The user’s feedback was then analyzed and used to update the word profile for the next iteration. In our experiments, 5 consecutive iterations were performed for each query, and the results are shown in Table 1. In this table, each value in the first row (i.e., s) for a query represents the number of pages identified as relevant by the user from the 20 pages presented by the interface agent at each iteration; and a value in the second row (i.e., a), the number of different and relevant pages accumulated so far. As can be seen, the numbers of pages selected tend to gradually increase from iteration to iteration. It shows that with the exploitation of the user feedback collected from the interactive procedure, the search system can more and more accurately capture what a user is searching in semantic, and therefore it can efficiently and continuously explore different regions of the Web space. It can also be observed that some exceptions happen in which only 4 or 5 pages were selected. After further examinations, we have found that this is because at certain iterations, the user has chosen some specific pages which were newly retrieved by the search engine and different from previous ones selected. This situation is similar to the effect called query drift ([9] [4]) in which the focus of the query is altered by query expansion. In our cases, some new words appeared very often in the newly selected pages and thus dominated the profile. These words then directed the search engine to find pages with new topics but still related to the concept originally specified. For example, in one of the cases with only 5 pages selected, the experimenter has indicated some indexing pages of knowledge-based systems as relevant, and which were different from the previously selected pages with contents mainly related to intelligent computation. As is shown in Table 1, though during the topic-transition phase, the pages retrieved could not meet the user’s need
immediately, and the performance of the system thus declined temporarily, it can soon be recovered.
iter.
i0
i1
i2
i3
i4
i5
computer science
s a
5 5
8 12
8 18
12 28
12 39
17 49
artificial intelligence
s a
5 5
16 20
11 29
5 32
16 45
5 46
intelligence agent
s a
7 7
4 10
4 14
10 19
9 28
13 41
search keywords
Table 1: The search results by an interactive approach.
4.2 The Performance of the Evolution Mechanism After evaluating the validity of the above interactive approach, in the second set of experiments we investigate whether the evolution strategies described can further enhance the search quality. As in the above experiments, 5 iterations were performed for each query. In each iteration, the initial population was derived from the word profile by employing the mutation operation, and the evolution procedure continued for 5 generations. Here, a (5+5)-ES model was used: each population contains 5 individuals that were selected from the 5 parents and their corresponding 5 offspring. In the experiments, to evaluate an individual involved picking up the 5 words with top mutated frequency values from the chromosome, and sending the conjunction form of these words to the search engine Yahoo. Then the top 50 Web pages retrieved were re-ranked by calculating their cosine similarities to the profile, and the average of the similarity coefficients of the top 20 pages after re-ranking was used to determine the fitness of the individual. The 20 pages collected from the best individual obtained in the last generation were presented to the user for his personal feedback. Once the user has identified the pages, the ones indicated as relevant were analyzed and used to update the profile, and then a new evolutionary iteration started. Table 2 shows the results in which values in the rows s and a for each query are the numbers of pages selected by the experimenters at the end of each iteration and the different pages accumulated so far, respectively. Comparing with Table 1, with the evolutionary mechanism, better search results can be obtained. For the case with initial query “intelligent agent”, our system can even achieve 100% accuracy of search. It shows the success of our evolution mechanism. As in Table 1, there is a small value 3 in the third iteration for the query “computer science”. After the examination, it is found that the reason for resulting in this value is the same as the specific cases in the first set of experiments, and this situation can also be recovered soon. It should be noted that, the fitness of an individual is determined in accordance with the measure of cosine similarity between the Web pages retrieved and the word profile. The user only evaluated the results produced by the best individual at the end of each iteration to correct the search direction. Though this indirect evaluation may cause some
deviations in user modeling and the system performance may thus slightly decline, this method used for fitness measurement can largely save the processing time. It is an important factor for the interactive and iterative search task here.
The other important issue is to examine the possibility of applying the collaborative technique in recommendation work (e.g., [2]) to enhance the effectiveness of the word profile used. REFERENCES
search keywords
iter
i0
i1
i2
i3
i4
i5
computer science
s a
5 5
9 14
10 21
3 24
15 37
15 37
artificial intelligence
s a
5 5
13 17
16 31
18 38
16 43
17 50
intelligence agent
s a
7 7
12 16
16 29
13 32
14 34
20 43
Table 2: The search results by a search engine with an interactive evolution mechanism.
5. CONCLUSION AND FUTURE WORK In this paper, we have emphasized the importance of user feedback and concept modeling in Web search. To achieve the concept-based semantic search, we present an agent-based framework to overcome the difficulty of specifying appropriate queries to retrieve Web contents semantically relevant to a user’s need. In our multi-agent framework, each agent is responsible for a specific subtask, including information presentation and gathering, Web content analyzing, query formulation, and information filtering. Especially, a special kind of evolutionary algorithms, evolution strategies, is used to develop a discovery agent for concept modeling in query formulation. As is analyzed, due to the characteristics of simultaneous mutations and self-adaptation, ES are better alternatives than GAs for the search task, and consequently chosen here to evolve queries. In addition, because the search task here is user-centric, the process of query formulation must then be under the user’s guidance. Therefore in this work, an interactive approach is employed to iteratively collect a user’s feedback from the pages he has identified for the query submitted. By analyzing the pages gathered, the system can then gradually create queries that are able to efficiently describe the content a user is looking for. To access our framework, different sets of experiments have been conducted and the preliminary results have shown that the search quality can be improved by our approach. Our work presented here points to some prospects for future research. In the experiments of evolving queries, instead of asking a user to manually evaluate the results of each generation, we have used an indirect method to define the fitness of an individual in which the fitness was estimated by the average of the similarity coefficients of the pages with top ranks. Though this method greatly saves the effort of user-system interaction, it inevitably causes some deviations because the common method of cosine similarity measurement in information retrieval can not truly reflect the similarity of documents in semantic. Therefore, it is worthwhile to investigate other ways to guide the evolution.
[1] Th. Bäck. Evolutionary Algorithms in Theory and Practice. Oxford University Press, New York, 1996. [2] C. Basu, H. Hirsh, and W. Cohen. Recommendation as Classification: Using Social and Content-Based Information in Recommendation. Proceedings of National Conference on Artificial Intelligence, pp.714-720, 1998. [3] L. Chen and K. Sycara. WebMate: A Personal Agent for Browsing and Searching. Proceedings of International Conference on Autonomous Agents, pp132-139, 1998. [4] C. J. Crouch, D. B. Crouch, Q. Chen, and S. J. Holtz. Improving the Retrieval Effectiveness of Very Short Queries. Information Processing and Management, 38(1): 1-36, 2002. [5] C. V. Goldman, A. Langer, and J. S. Rosenchein. Musag: an agent that learn what you mean. Applied Artificial Intelligence, 11: 413-435, 1997. [6] C.-C. Hsu and C.-H. Chang. WebYacht: A Concept-Based Search Tools for WWW. International Journal of Artificial Intelligence Tools, 8(2): 137-156, 1999. [7] T. Joachims, D. Freitag and T. Mitchell. WebWatcher: A Tour Guide for the World Wide Web. Proceedings of International Joint Conference on Artificial Intelligence, 1997. [8] P. Maes. Agents that Reduce Work and Information Overload. Communications of the ACM, 37(7): 30-40, 1994. [9] M. Mitra, A. Singhal, and C. Buckley. Improving Automatic Query Expansion. Proceedings of the 21 Annual International Conference on Research and Development in Information Retrieval, pp.206-214, 1998. [10] Z. Z. Nick and P. Themis. Web Search Using a Genetic Algorithm. IEEE Internet Computing, 5(2): 18-26, 2001. [11] M. Pazzani and D. Billsus. Learning and Revising User Profiles: the Identification of Interesting Web Sites. Machine Learning, 27: 313-331, 1997. [12] G. Salton. Automatic Text Processing. Addison-Wesley, Reading, MA, 1989. [13] R. Saloman. Reevaluating Genetic Algorithm Performance under Coordinate Rotation of Benchmark Functions: A Survey of Some Theoretical and Practical Aspects of Genetic Algorithms. BioSystems, 39(3): 263-278, 1996. [14] K. Sycara, A. Pannu, M. Williamson, D. Zeng, and K. Decker. Distributed Intelligent Agents. IEEE Expert, 11(6): 36-46, 1996.
