A Neuro-Fuzzy (NF) system is a combination of Artificial Neural Network (ANN) and Fuzzy Inference System. (FIS) in such a way that ANN learning algorithms ...
Evolutionary Design of Neuro-Fuzzy Systems – A Generic Framework Ajith Abraham and Baikunth Nath School of Computing & Information Technology Monash University (Gippsland Campus), Churchill 3842, Australia Email: {Ajith.Abraham, Baikunth.Nath}@infotech.monash.edu.au
Abstract A Neuro-Fuzzy (NF) system is a combination of Artificial Neural Network (ANN) and Fuzzy Inference System (FIS) in such a way that ANN learning algorithms are used to determine the parameters of FIS. Potential interactions between connectionist learning systems and evolutionary search procedures have attracted lots of research work recently. There is no guarantee that the learning algorithm converges in ANN and the tuning of FIS will be successful. Success of evolutionary search procedures for optimal design of ANNs and FIS are well proven and established in many application areas. However, the evolutionary design of a NF system is yet to be explored. In this paper, we attempt to formulate a 5-tier hierarchical evolutionary search procedure for the optimal design of NF systems. We formulate each of the evolutionary search procedures in detail and the interactions among them. Keywords: Neuro-fuzzy systems, hybrid system, evolutionary algorithm, genetic algorithm.
1. Introduction Artificial Neural Networks (ANNs) [4] and Fuzzy inference Systems (FIS) [5] are both very powerful soft computing tools for solving a problem without having to analyze the problem itself in detail. Figure 1 depicts the interaction of ANN, FIS and Evolutionary Algorithms (EA) for the optimal design of intelligent systems. A FIS utilize human expertise by storing its essential components in the knowledge base, and perform fuzzy reasoning to infer the overall output value. Knowledge base of a FIS is usually implemented by applying heuristic techniques and there is no automatic way to adapt the same. There is also a need for adaptability or some learning algorithms to produce outputs within the required error rate.
Figure 1. A general framework for evolving intelligent systems On the other hand, the most prominent feature of ANN is their ability to learn from examples. Due to the homogenous structure of ANN, it is hard to extract structured knowledge from either the weights or the configuration of the network. It is not easy to encode priori knowledge into an ANN. Table 1 summarizes the comparison of ANN, FIS, EA, symbolic Artificial Intelligence (AI) and control theory [15]. Table 1. Comparison of FIS, ANN, EA, Symbolic AI and Control Theory †. Mathematical model Learning ability Knowledge representation Expert knowledge Nonlinearity Optimization ability Fault tolerance Uncertainty tolerance Real time operation
FIS SG B G G G B G G G
ANN B G B B G SG G G SG
EA B SG SB B G G G G SB
Symbolic AI SB B G G SB B B B B
Control Theory G B SB SB B SB B B G
† The fuzzy term used for grading are good(G), slightly good (SG), slightly bad (SB) and bad (B).
To a large extent, the drawbacks pertaining to these ANN and FIS seem complementary. Therefore it seems natural to consider building an integrated system combining the concepts of ANN and FIS. A common way to apply a learning algorithm to a fuzzy system is to represent it in a special ANN like architecture. However the conventional ANN learning algorithms (gradient descent) cannot be applied directly to such a system as the functions used in the inference process are usually non differentiable. This problem can be tackled by using differentiable functions in the inference system or by not using the standard neural learning algorithm. We shall present the evolutionary design of fused NF model, which make use of the complementarity of ANN and FIS to form a better intelligent system. The integrated architecture share data structures and knowledge representations [14].
2. Fuzzy Inference Systems FIS is a popular computing framework based on the concepts of fuzzy set theory, fuzzy if-then rules, and fuzzy reasoning. The basic structure of the fuzzy inference system consists of three conceptual components: a rule base, which contains a selection of fuzzy rules; a database, which defines the membership functions used in the fuzzy rule and a reasoning mechanism, which performs the inference procedure upon the rules and given facts to derive a reasonable output or conclusion. Figure 2 shows the basic architecture of FIS with crisp inputs and outputs implementing a nonlinear mapping from its input space to output space.
Figure 2. Block diagram of a fuzzy inference system We shall introduce 2 fuzzy inference systems that have been widely employed in various applications. The differences between these three fuzzy inference systems lie in the consequents of their fuzzy rules, and thus their aggregation and defuzzification procedures differ accordingly. Most fuzzy systems employ the inference method proposed by Mamdani in which the rule consequence is defined by fuzzy sets and has the following structure [1]:
if x is A 1 and y is B 1 then z 1= C 1
Figure 3. Mamdani fuzzy inference system
Takagi, Sugeno and Kang (TSK) proposed an inference scheme in which the conclusion of a fuzzy rule is constituted by a weighted linear combination of the crisp inputs rather than a fuzzy set and has the following structure [2].
if x is A 1 and y is B 1, then z 1 = p 1 x + q 1 y + r
2
Figure 4. Takagi Sugeno fuzzy inference system
TSK fuzzy controller usually needs a smaller number of rules, because their output is already a linear function of the inputs rather than a constant fuzzy set.
3. Parameterization of the inference system Evolutionary search of optimal inference procedure could only be formulated if all the node functions are parameterized. During the training process, suitable learning algorithms can be applied for fine-tuning of the parameters. •
Membership Functions (MFs)
FIS is completely characterized by its MF. A generalized bell MF is specified by three parameters (p, q, r) and is given by: 1
Bell (x, p, q, r) = 1+
x -r p
2q
Figure 5 (a-d) shows the effects of changing p, q and r in bell MF. Similar parameterization can be done with most of the membership function used in FIS.
(a) (b) (c) (d) Figure 5. (a) Changing parameter p (b) changing parameter q (c) changing parameter r (d) changing p and q simultaneously but keeping their ratio constant. •
T-norm and T-conorm operators
T-norm is a fuzzy intersection operator, which aggregates the intersection of two fuzzy sets A and B. The most frequently used T-norm operators are given by (1). Similarly, T-conorm operators compute fuzzy union and are given by (2).
Minimum : Tmin (a, b) = min(a, b) = a ∧ b A lg ebraic product : Tap (a, b) = ab Bounded product : Tbp (a, b) = 0 ∨ (a + b − 1)
(1)
ìa, if b = 1. ï Drastic product : Tdp (a, b) = íb, if a = 1 ï0, if a, b F 1 î
3
Maximum : S (a, b) = max(a, b) = a ∨ b A lg ebraic sum : S (a, b) = a + b − ab Bounded sum : S (a, b) = 1 ∧ (a + b)
(2)
ìa, if b = 0 ï Drastic sum : S (a, b) = íb, if a = 0 ï0, if a, b B 0 î
Several parameterized T-norms and T-conorms have been proposed by Yager, Dubois and Prade, Schweizer and Sklar and Sugeno. For instance, Schweizer and Sklar's T-norm operator can be expressed as [6]: 1 − p p − − é ù p T (a, b, p) = max 0, (a +b − 1) êë úû 1 − − − p p é ù p S (a, b, p) = 1 − max 0, ((1 − a ) + (1 − b ) − 1) êë úû
{
}
{
}
(a)
(b)
(c) Figure 6.(a) Bell Membership functions for fuzzy set A and B (b) effects of changing parameters of T-norm operators (c) effects of changing parameters of T-conorm operators.
Tano et al [3] proposed a new aggregation operator "and" by adding a synergic affect to Dombi's T-norm operator. Basic Dombi operator = TD ( x, y, p) =
1 1+
p
(( x − 1 − 1) p + (( y − 1 − 1) p
Modified T-norm operator = w × synergy (x, y) + (1-w) × TD ( x, y, p ) w= equal (x, y) × high (x, y) × γ equal (x, y) = almost (0, x-y, α) high (x, y) = almost (1,x,β) × almost (1,y,β) x−a 2 almost (a, x, b) = exp (ln 0.5 × ( ) ) b synergy (x, y) = 1
4
This operator has four parameters representing the strength of the synergic effect (γ), the area where this affect is required (α, β), and the Dombi's T-norm parameter (p). When synergic effect is set to zero, the operator behaves like Dombi's T-norm.
4. Generic Architecture of Evolving Neuro-Fuzzy Systems In this section, we define the architecture of the evolving neuro fuzzy model, which can adapt to Mamdani or Takagi Sugeno type fuzzy inference system. Proposed architecture and the evolving mechanism can be considered as general framework for adaptive neuro-fuzzy systems, that is systems that can change their node functions, architecture and learning parameters according to different environments without human intervention. Figure 7 depicts the basic architecture of the self-constructing adaptive neuro-fuzzy system and the function of the nodes and interconnections are described below.
Figure 6. Architecture of adaptive self-constructing optimal neuro-fuzzy system.
•
Mamdani Fuzzy Inference System
Layer -1(input layer): No computation is done in this layer. Each node in this layer, which corresponds to one input variable, only transmits input values to the next layer directly. The link weight in layer 1 is unity. Layer-2 (fuzzification layer): Each node in this layer corresponds to one linguistic label (excellent, good, etc.) to one of the input variables in layer 1. In other words, the output link represent the membership value, which specifies the degree to which an input value belongs to a fuzzy set, is calculated in layer2. A clustering algorithm will decide the initial number and type of membership functions to be allocated to each of the input variable. The optimal number and type of parameterized membership functions (MFs) will be further decided by an evolutionary search procedure. The final shapes of the MFs will be fine tuned during network learning. Layer-3 (rule antecedent layer): A node in this layer represents the antecedent part of a rule. We propose to use the parameterized "and" operation given in section 3. The output of a layer 3 node represents the firing strength of the corresponding fuzzy rule. Layer-4 (rule consequent layer): This node basically has two tasks. To combine the incoming rule antecedents and determine the degree to which they belong to the output linguistic label (high, medium, low, etc.). The number of nodes in this layer will be equal to the number of rules. The type of the parameterized MF is to be determined using an evolutionary search procedure. The shape of the MF will be fine tuned during the evolutionary learning process.
5
Layer-5 (Combination and defuzzification layer): This node does the combination of all the rules consequents using a parameterized T-conorm operator and finally computes the crisp output after defuzzification. There are various defuzzification approaches available such as the max criterion, mean of maximum and the center of area. The max criterion method finds the point at which the membership function is a maximum. The mean of maximum takes the mean of those points where the membership function is at a maximum. The most common method is the center of area method, which finds the center of gravity of the solution fuzzy sets.
Defuzzification plays a great role in FIS. It is very difficult to convert a fuzzy set into a numeric value without losing some information during defuzzification also it's very hard to find a number that best represents a fuzzy set. Most of the defuzzification methods were problem dependent. Search of a defuzzification process can be formulated as a part of evolutionary search and if the defuzzification function is parameterized, we can apply learning methods to modify the function during training. •
Takagi Sugeno Fuzzy Inference System
Layers 1,2 and 3 functions the same way as Mamdani FIS. The complete functioning of the network is as follows. Layer 4 (rule strength normalization): Every node in this layer calculates the ratio of the i-th rule’s firing strength to the sum of all rules firing strength.
wi =
wi , i = 1,2.... . w1 + w 2
Layer-5 (rule consequent layer): Every node i in this layer is with a node function
wi f i = wi ( pi x 1 + qi x 2 + ri ) , where wi is the output of layer 4, and {pi , qi , ri } is the parameter set. Parameters in this layer will be referred to as consequent parameters. A well-established way is to determine the consequent parameters using the least means squares algorithm as used in ANFIS [6] and SONFIN [8]. Layer-6 (rule inference layer) The single node in this layer computes the overall output as the summation of all åw f incoming signals: Overall output = å wi f i = i i i . å iwi i
5
Framework for evolutionary design of neuro-fuzzy systems
Solving multiobjective scientific and engineering problems is, generally, a very difficult goal. In these particular optimization problems, the objectives often conflict across a high-dimension problem space and may also require extensive computational resources. Evolutionary algorithms (EAs) are a class of stochastic search algorithms applicable to a wide range of problems in learning and optimization [9]. Different representation or encoding schemes, selection schemes, and search operators will define different EAs. While Genetic Algorithms (GAs) use crossover and mutation as search operators, Evolutionary Programming (EP) only uses mutation. Figure 7, illustrates the flow chart of a canonical GA.
Figure 7. Flowchart of a GA iteration
Literature survey reveals that most of the neuro-fuzzy models use Back-Propagation (BP) algorithm, which is based on gradient descent technique to minimize error E for a particular training pattern [12]. Empirical research has shown that the BP algorithm often gets trapped in local minima where the error surface becomes complicated. Designing ANNs [10][11] and FIS [13] using EAs has been successful in many application areas. We propose to use a 5-tier evolutionary search procedure wherein the membership functions, rules (architecture), fuzzy inference mechanism (T-norm and Tconorm operators), learning parameters and finally the type of inference system (Mamdani, Takagi Sugeno etc.) are adapted according to the environment.
6
Figure 8. General framework for hierarchical evolution of optimal neuro-fuzzy system
Figure 8 illustrates the interaction of various evolutionary search procedures. For every learning parameter, there is the global search of inference mechanisms that proceeds on a faster time scale in an environment decided by the inference system and the problem. For every inference mechanism there is the global search of fuzzy rules (architecture) that proceeds on a faster time scale in an environment decided by the learning parameters, inference system and the problem. Similarly, for every architecture, evolution of membership function parameters proceeds at a faster time scale in an environment decided by the architecture, inference mechanism, learning rule, type of inference system and the problem. Hierarchy of the different adaptation procedures will rely on the prior knowledge. For example, if there is more prior knowledge about the architecture than the inference mechanism then it is better to implement the architecture at a higher level. Sugeno-type fuzzy systems are high performers (less Root Mean Square Error - RMSE) but often requires complicated learning procedures and computational expensive. However, Mamdani-type fuzzy systems can be modeled using faster heuristics but with a compromise on the performance (high RMSE). Hence there is always a compromise between performance and computational time. An optimal design of a neuro-fuzzy system can only be achieved by the adaptive evolution of membership functions, rule base (architecture) and learning rules, which progress on different time scales as illustrated in Figure 8. Most of the neuro-fuzzy systems are either based on Mamdani-type or Takagi-Sugeno-type. As shown in Figure 8, the fuzzy inference procedure (Mamdani FIS, Takagi Sugeno FIS, etc.) will evolve at the highest level on the slowest time scale to adapt the inference system according to the problem.
6
Chromosome modeling and representation issues
The antecedent of a fuzzy rule defines a local region, while the consequent describes the behavior within the region via various constituents. Basically the antecedent part remains the same regardless of the inference system used. Different consequent constituents result in different fuzzy inference systems. We propose to partition the input space (data) using a grid partition or scatter partition to form a base architecture (e.g. ANFIS [[6]) and fine-tune the architecture and other network parameters using the evolutionary learning process. For applying evolutionary algorithms, problem representation (chromosome) is very important as it directly affects the proposed algorithm. Referring to figure 8, each layer (from fastest to slowest) of the hierarchical evolutionary search process has to be represented in a chromosome for successful modeling of an optimal neuro-fuzzy system and we briefly explain the modeling process as follows. Layer 1: The simplest way is to encode the number of membership functions per input variable and the parameters of the membership functions. Figure 9 depicts the chromosome representation of n asymmetric membership functions specified by its center, left and right base bandwidths. The optimal parameters of the membership functions located by the search procedure will be later fine tuned by the learning algorithm. Similar strategy could be used for the output membership functions in the case of a Mamdani FIS.
Figure 9. Chromosome representing n membership functions Layer 2. Fine tuning of the architectures may be explored by adding / deleting nodes and connections especially in the antecedent and consequent parts. A connection be represented by "1" and "0" otherwise. Easiest way is to represent a similar coding scheme used for neural network architecture representation [10]
7
Layer 3. This chromosome will represent different T-norm and T-conorm operators and their parameters. The parameters will be fine tuned by an optimal learning algorithm formulated by the layer4. Layer 5. This layer is responsible for the selection of optimal learning parameters. Deciding the learning rate and momentum can be considered as the first attempt of learning rules. The optimal learning parameters will be used to tune the membership functions and the inference mechanism. Layer 6. This layer basically interacts with the environment and decides which inference procedure is the optimal according to the environment.
7
Conclusions and further work
Neuro-fuzzy systems have attracted the growing interest of researchers in various scientific and engineering areas due to the growing need of intelligent systems. Neuro-fuzzy systems aim at providing fuzzy systems with automatic tuning abilities. From the fully trained, NF system users can understand the acquired rules after learning. With reference to learning speed, NF systems are sometimes faster than neural networks. As a guideline, for NF systems to be highly intelligent some of the major requirements are fast learning (memory based - efficient storage and retrieval capacities), on-line adaptability (accommodating new features like inputs, outputs, nodes, connections etc), achieve a global error rate and computationally inexpensive. The data acquisition and preprocessing training data is also quite important for the success of neuro-fuzzy systems. Many NF models use gradient descent techniques to learn the membership function parameters. The success of the learning process is not guaranteed, as the designed network might not be optimal. Empirical research has shown that gradient descent techniques gets trapped in local optima especially for complicated problems. In this paper we have presented how the optimal design of NF systems could be achieved using a 5-tier global search process based on an evolutionary search process. However, the real success in modeling such systems will directly depend on the genotype representation of the different layers. All a priori information available is to be encoded into the system to minimize the calculation procedure and the evolutionary search procedures are to be formulated accordingly. Hierarchical evolutionary search processes attract considerable computational effort. Fortunately evolutionary algorithms work with a population of independent solutions, which makes it easy to distribute the computational load among several processors using parallel algorithms.
8 [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15]
References Mamdani E H and Assilian S, An experiment in Linguistic Synthesis with a Fuzzy Logic Controller, International Journal of Man-Machine Studies, Vol. 7, No.1, pp. 1-13, 1975. Sugeno M, Industrial Applications of Fuzzy Control, Elsevier Science Pub Co., 1985. Tano S, Oyama T and Arnould T, Deep Combination of Fuzzy Inference and Neural Network in Fuzzy Inference Software – FINEST, International Journal of Fuzzy Sets and Systems, Elsevier, Vol (82), pp. 151-160, 1996. Zurada J M, Introduction to Artificial Neural Systems, PWS Pub Co, 1992. Cherkassky V, Fuzzy Inference Systems: A Critical Review, Computational Intelligence: Soft Computing and Fuzzy-Neuro Integration with Applications, Kayak O, Zadeh LA et al (Eds.), Springer, pp.177-197, 1998. Jang R, Sun C T and Mizutani E, Neuro-Fuzzy and Soft Computing – A computational Approach to Learning and Machine Intelligence, Prentice Hall, 1997. Jang R, Neuro-Fuzzy Modeling: Architectures, Analyses and Applications, PhD Thesis, University of California, Berkeley, July 1992. Feng J C and Teng L C, An Online Self Constructing Neural Fuzzy Inference Network and its Applications, IEEE Transactions on Fuzzy Systems, Vol 6, No.1, pp. 12-32, 1998. Back T, Evolutionary Algorithms in Theory and Practice, Oxford University Press, 1996. X Yao, Evolving Artificial Neural Networks, Proceedings of the IEEE, 87(9): pp. 1423-1447, September 1999. Abraham A and Nath B, Optimal Design of Neural Nets Using Hybrid Algorithms, Sixth Pacific Rim International Conference on Artificial Intelligence (PRICAI2000), Australia, August 2000. Abraham A and Nath B, Designing Optimal Neuro-Fuzzy Systems for Intelligent Control, The Sixth International Conference on Control, Automation, Robotics and Vision, (ICARCV 2000), December 2000. Abraham A and Nath B, Evolutionary Design of Fuzzy Control Systems - An Hybrid Approach, The Sixth International Conference on Control, Automation, Robotics and Vision, (ICARCV 2000), December 2000. Nauk D, Klawonn F and Kruse R, Foundations of Neuro-Fuzzy Systems, John Wiley & Sons,1997. Fukuda T and Shibata M, Fuzzy-Neuro-GA Based Intelligent Robotics, In Zurada J M et al (Eds), Computational Intelligence Imitating Life, pp. 352-362, IEEE press, 1994. 8
