Evolving Complex-Valued Interval Type-2 Fuzzy Inference System

Transcription

1 Evolving Complex-Valued Interval Type-2 Fuzzy Inference System K. Subramanian Air Traffic Management Research Institute Nanyang Technological University Singapore, S. Suresh School of Computer Engineering Nanyang Technological University Singapore, Abstract Interval Type-2 fuzzy systems have been shown to be extremely capable of handling vagueness as well as uncertainty in data, while complex-valued fuzzy sets have been demonstrated to be capable of solving classification problems efficiently. This paper combines their collective advantage to propose a complexvalued Interval Type-2 Fuzzy Inference System (referred to as CIT2FIS). To derive the fuzzy rules, a Recursive Least Squares based algorithm is proposed. The proposed algorithm evolves (add/ prune) and adapts the rules in an evolving online fashion. During sequential learning, the networ monitors the error and nowledge contained in the current sample and either rules are evolved (added, pruned) to capture the nowledge in the sample, or the rule parameter updated. Upon rule addition, the centers are determined based on the current input and the output weights are analytically determined such that a least squares fit is obtained. This ensures that the rule retain its interpretability and accuracy. Parameter update is based on recursive least squares based approach. In order to maintain the parsimony of the networ, a data-driven rule pruning scheme is employed. To further enhance the generalization ability of the networ, wellnown meta-cognitive learning mechanism is employed in this wor. The performance of the proposed CIT2FIS is evaluated on a set of real-valued classification problems. The performance comparison with other state-of-the-art complex-valued as well as fuzzy classifiers clearly highlights the advantage of the proposed wor. Index Terms Complex-valued fuzzy system, recursive least squares, data driven, evolving system, self-regulation I. INTRODUCTION Fuzzy rule based systems have long been employed in soft-computing due to their interpretability as well as their intrinsic ability to handle vagueness. To enhance their ability in accurate solution formulation, fuzzy inference systems are combined with neural networ based approaches, nown as neuro-fuzzy inference systems. These systems have found use in various applications such as system identification [12], video analytics [17], and prediction [18]. In various real-world problems, where fuzzy/neural algorithms are employed, the data is noisy and the above said neuro-fuzzy inference systems cannot effectively handle these noise due to the use of precise Type-1 fuzzy sets. To overcome this issue, Zadeh [22] proposed Type-2 fuzzy sets. These Type- 2 fuzzy sets have been demonstrated to model noisy data efficiently, but at cost of increased computational load. To address this, Interval Type-2 fuzzy sets [6] as computational simplification of Type-2 fuzzy sets have been proposed. These Interval Type-2 fuzzy rule based systems and neuro-fuzzy inference systems have found immense research interest [7], [8], [14]. In the domain of neural networs, studies on the abilities of complex-valued neurons have revealed that they possess better computational power than their real-valued counterpart [9]. Further, the networs employing these neurons have been shown to have dual orthogonal decision boundaries which enable them to solve real-valued classification tass with ease [10]. These advantages have encouraged various researchers to develop complex-valued classifiers. A review about the same is available in [17]. Similar to real-valued neural networs, the complex-valued networs also lac interpretability and ability to handle noisy data. Combination of the properties of complex-valued neurons with Interval Type-2 fuzzy rules could result in complexvalued fuzzy rules being capable of handling uncertainty. Hence, in this wor, we propose a Complex-Valued Interval Type-2 Fuzzy Inference System. In literature such a complexvalued Interval Type-2 Fuzzy Inference System has earlier been proposed in [17]. This wor employs a projection based learning algorithm to grow and adapt the networ parameters. However, the learning technique proposed herein is not purely online as it requires nowledge about entire training samples. In order to facilitate fast learning in evolving/adaptive fashion, we propose a least squares/recursive least squares [5] based algorithm, referred to as Complex-Valued Interval Type-2 Fuzzy Inference System (CIT2FIS), wherein the rules are evolved and adapted in an online sequential fashion. The inference system employs Gaussian-lie rule antecedent capable of handling uncertainty in standard deviation, while the consequents are modeled based on Taagi-Sugeno-Kang inference mechanism. As an input is presented to the system, initially it is fuzzified and its footprint of uncertainty is determined. The resulting Interval Type-2 fuzzy membership is converted to corresponding Type-1 fuzzy membership by employing a computationally fast interval reduction technique proposed in [8]. Finally, the crisp output is determined by center of mass defuzzification. During learning, initially CIT2FIS is assumed to contain

2 zero rules. During learning, as each sample is presented to the learning algorithm, the system determines the prediction error and nowledge content in the sample to determine effective learning strategy. The nowledge contained in the current sample is learnt by either adding a new rule or updating the parameters of the existing rules. During rule addition, the rule center is initialized based on the input data. This helps the system preserve its interpretability (partially deterministic). The output weights are determined analytically by least squares based approach. In online learning scenario with streaming data, it is impractical to store and process input for weight estimation. To overcome this issue, we propose to use existing samples as pseudo-samples during rule addition. Since the rule addition is data-driven, the existing rules can act as representative of training samples. The parameter update employs a recursive least squares based approach. In online learning, the distribution of data might change and certain rules might become redundant. To remove such rules and preserve the parsimony of the networ, we also employ a data-driven rule pruning scheme, where consistently non-performing rules are pruned from the networ. Further, we also employ the well-nown meta-cognitive mechanism [17] to improve the generalization of the proposed system. The performance of the proposed CIT2FIS is evaluated on a set of real-valued classification problems from UCI machine learning repository [1]. The real-valued features are converted to complex-valued features by employing circular transformation [13]. Performance comparison with other stateof-the-art classifier clearly highlights the superior performance of the proposed method. To summarize, the main contributions of this paper are: Proposal of a Complex-valued Interval Type-2 fuzzy inference system (CIT2FIS). An adaptive and evolving algorithm proposed for CIT2FIS. Developed algorithm can learn in purely online fashion. Application to real-world classification problems. Rest of the paper is organized as follows. In the next Section, we briefly describe the proposed algorithm. In Section II, we present the CIT2FIS inference mechanism and least squares based learning algorithm. Then we conduct performance evaluation on a set of benchmar real-valued classification problems. Finally, the paper concludes in Section IV. II. COMPLEX-VALUED INTERVAL TYPE-2 FUZZY INFERENCE SYSTEM AND ITS LEARNING ALGORITHM In this section, we first present the problem definition, followed by the inference mechanism and learning algorithm of the proposed CIT2FIS. A. Problem Definition Let us assume the training data arrives sequentially and be given by [x t, c t ] t=1,2,, where the input x t = [x t 1,, x t m] C m is the complex-valued input features and c t [1,, n] is its corresponding class label, with n being the total number of classes. For solving classification problems, the class label c t is converted to complex-valued coded class label y t = [y1, t, yn] t C n as { yl t 1 + 1i if l = c t = l = 1,, n. (1) 1 1i otherwise The aim of any classifier is to find the decision surface mapping the input x to coded class label y (x y). CIT2FIS employs a set of rules to approximate the decision surface, based on the given input. We first describe the output inference mechanism of CIT2FIS, followed by its sequential/evolving learning algorithm. B. Inference Mechanism CIT2FIS employs Gaussian-lie membership to realize Interval Type-2 fuzzy rule antecedents with uncertain standard deviation, and the consequents realize Taagi-Sugeno-Kang inference process. Each rule consists of five parameters which maps the given input to the corresponding output. These parameters are: rule center µ, lower and upper limit of widths [σ L, σ U ], type-reduction parameter α, and output weight W. Let us assume the system consists of K rules. The detailed inference mechanism upon presenting t-th sample is detailed below. Initially, the input is fuzzified and lower, upper limits of membership, ([φ L, φu ]) for th rule is determined as ( ) = exp (xt µ )(x t µ ) H ( ) σ L 2 (2) φ L,t φ U,t = exp ( ) (xt µ )(x t µ ) H ( ) σ U 2, (3) where (.) H represent the Hermitian operation on the complexnumber. The resulting lower and upper limits of membership (called the foot print of uncertainty) are real-valued numbers in the range [0, 1]. It should be noted that the center µ C m is complex-valued while σ R 1 is real-valued in nature. As the implication of complex-valued spread of a fuzzy rule is yet to be fully understood, σ is chosen to be a real-valued quantity. Next, the Interval Type-2 fuzzy membership is converted to Type-1 fuzzy membership (interval reduction) and it is given by, φ t = α φ L,t + (1 α ) φ U,t. (4) In this wor, we employ the interval reduction technique employed in Nie-Tan [8]. There are various other typereduction/interval-reduction technique proposed in literature, and a review is available in [21]. However, it has been shown that Nie-Tan technique is fast and is capable of producing a closed-loop solution formulation. Finally, the membership is mapped bac to complex-valued space and the crisp output of the system (ŷ t ) is determined

3 by employing center of mass defuzzification technique, K ŷ t =1 = W φ t K. (5) p=1 φt p W C n is the output weight which maps the real-valued membership to complex-valued output. C. Learning Mechanism The objective of the learning mechanism is to minimize the error (e t ) between actual output y t and predicted output ŷ t. The error e t = [e t 1,, e t n] T C n is calculated as hinge-loss error [20] and is given by e t l = { y t l ŷ t l if R(y t l ) R(ŷt l ) < 1 0 otherwise l = 1,, n. (6) During learning, as each sample x t is presented to CIT2FIS, it infers the predicted output as described in Section II-B. Based on the predicted output, two monitory signals are calculated: Spherical Potential (ψ): Average distance of current sample from existing rules in an hyper dimensional feature space. It is calculated as ψ t = K =1 φt Maximum absolute error (E M ): It is calculated as K (7) E t M = max e t (8) Based on the above nowledge measures, the CIT2FIS decides on its learning strategies. In literature [4], [3], [14], [15], it has been shown that meta-cognitive learning mechanism helps a networ attain improved generalization performance. In this wor, we employ such a meta-cognitive learning scheme to improve the generalization performance of the networ. The meta-cognitive learning mechanism employed is a self-regulatory learning mechanism which monitors the nowledge content in the networ to efficiently decide on what-to-learn, when-to-learn and how-to-learn. These three learning mechanisms are realized using sample deletion strategy, sample reserve strategy and sample learning strategy, respectively. We shall describe each of these three strategies in detail, next. Sample Deletion Strategy: A sample is deleted without being learnt, if its prediction error is significantly low. This strategy helps the networ avoid over-training and as a result reduce the computational cost. The condition for sample deletion is given by E t M < E D (9) where, E D is sample delete threshold. In this wor, this threshold is chosen in the range [0.01, 0.1]. Sample Learning Strategy: According to this strategy, the fuzzy system evolves/adapts its structure and parameters based on the nowledge contained in the training sample. There are three different ways in which the system learns: rule addition, parameter update and rule pruning. Rule addition: CIT2FIS begins as batch learning algorithm. The first n samples belonging to n classes are initialized as rules and their membership covariance matrix, H is determined as per the equation H = φ 1 1 φ 1 K. φ t.. (10) φ n 1 φ n K The parameters σ L,U and α are randomly initialized. φ t for the sample x t as determined by the Eqn. (4). The output weights are estimated as W = H T, (11) where, T = [ y 1,, y t] T is the actual class labels of t samples and represents pseudo-inverse. For the subsequent samples, if the existing rules in the system cannot cover the current sample (as measured by spherical potential) and the prediction error is significantly high (measured by maximum absolute error), a new rule is added to the system to capture the nowledge in the current sample. The rule is added to the system if ψ t < E S AND E t M > E A (12) where, E S is the novelty threshold set in the range [0.3, 0.8] and E A is rule addition threshold, which is set in the range [1.2, 1.7]. The rule addition threshold is selfadapted based on the equation E A = γe A + (1 γ)e t M, (13) where, γ is set close to 1. Assuming the networ consists K rules upon learning t samples, the new rule is added as, µ K+1 = x t+1. (14) The coded-class labels of the class which rule represents (T) is set as T K+1 = y t (15) The other parameters (σ L,U K+1, α K+1) are randomly assigned. The data-based center initialization allows the system to retain its interpretability while allowing the data-clustering in an fuzzy framewor. During least squares based weight estimation, the dimension of H is dynamically adapted from K t to K + 1 t + 1. The weight matrix is re-estimated as given in Eqn. (11). Such a rule initialization however suffers from a set-bac. This adaptation of H matrix upon rule addition requires nowledge about all the samples, which is impractical in an online learning scenario. In order to overcome this issue, we propose the use of existing centers as pseudo-samples for adapting the H matrix. Since this wor employs data-based rule initialization, the pseudosamples can be considered as representatives of actual training samples. Also, since the centers are initialized

4 based on training samples, the class to which the rule belongs to is nown apriori (T matrix). This pseudoclass label along with H calculated by pseudo-samples could help in approximating the output weights, as given by Eqn.(11). Rule update: If the prediction error is not significantly high, then the parameters of the rules are updated by recursive least squares based approach. The rule is updated if E t M > E U (16) where, E U is the rule update threshold which is set in the range [0.5, 0.8]. The rule update threshold is self-adapted based on the equation E U = γe U (1 γ)e t M. (17) When rule update criterion is satisfied, the weights are updated as, W = W + P t H ( y t ŷ t), (18) where, P = ( H T H ) 1. P is updated as, P = P PT H T HP 1 + H T PH (19) Rule pruning: In an online learning system, as the data distribution varies, certain rules might become irrelevant. These rules might be removed the system to maintain its parsimony as well as improve the generalization performance. In this wor, we propose a data-driven rule pruning scheme. If the membership of a rule is consistently low for a window E W of training samples, these rules are assumed to be irrelevant to the system and are pruned from the networ. The rule pruning criterion is given as min φ < E P (20) =1,,K where E P is the pruning threshold. In this wor, E P is set in the range [0.01, 0.05]. The window size is recommended to be set as multiple of the number of classes. Sample Reserve Strategy: If none of the above sample deletion strategy or sample learning strategy is satisfied, the sample is reserved to be considered for learning at a later point in time. The self-regulatory nature of rule addition as well as rule update threshold will ensure that these samples are used for learning later. To summarize, the pseudo-code of the proposed CIT2FIS is provided in Algorithm 1. III. PERFORMANCE EVALUATION In this section, we evaluate the performance of CIT2FIS on a set of real-valued binary/multi-class classification problems from UCI machine learning repository [1]. The performance of the system is compared against existing real-valued as well as complex-valued fuzzy and neural classifiers, including, Online Sequential Fuzzy Extreme Learning Machine for First n training samples do Initialize the rules and learning parameters in batch learning mode using Eqs. 14, 10 and 11. end while Rest of the training samples are learnt do for each input x t do Calculate the predicted output ŷ t, prediction error E t and spherical potential ψ t for a given input x t. if Eqn. 9 is satisfied, then Sample is deleted from training sequence without learning. else if Eqn. 12 is satisfied, then A new rule is added to the system based on Eqn. 14. The H matrix is updated based on Eqn. 10 and output weight re-estimated based on Eqn. 11. Upon addition of a new-rule, the rule addition threshold is self-adapted based on Eqn. 13. The H matrix is updated by employing existing rules as pseudo-samples. else if Eqn. 16 is satisfied, then The output weights of the fuzzy rules are adapted based on the Eqs. 18 and 19. The rule update threshold is adapted based of the Eqn. 17. if Eqn. 20 is satisfied for E W contiguous samples, then The corresponding rule is pruned from the networ without being learnt. else Push the sample to the rear-end of the sample pool to be learnt later. if a sample is not learnt after few processing then remove it from the data-stream. end end Algorithm 1: Pseudo-code for CIT2FIS. (OS-F-ELM) [11], Meta-Cognitive Interval Type-2 Neuro- Fuzzy Inference System (McIT2FIS) [14], Complex-Valued Neuro-Fuzzy Inference System (CNFIS) [16], Meta-Cognitive Complex-Valued Interval Type-2 Neuro-Fuzzy Inference System (PBL-McCIT2FIS) [17] and also standard Support Vector Machine (SVM) [2]. For the complex-valued classifiers, the real-valued inputs are converted to complex-valued features by employing circular transformation [13]. The circular transform is given as x C p = sin ( a p x R p + 1i a p x R p + c p ) (21) where x C p is the complex-valued feature corresponding to realvalued features x R p. a, b are random real-valued quantities in the range [0, 1] and c is in the range [0, 2π]. This helps to shift the origin to enable effective usage of the four quadrants of the complex plane. In all the problems considered, the performance is measured based on three performance measures: percentage of training samples employed (P.S), overall

5 classification accuracy (η o ) and average classification accuracy (η a ). These measures are defined as, Number of samples used in training P.S. = T otal number of training samples (22) n l=1 η o = q ll 100% (23) N η a = 1 n q ll 100% (24) n N l l=1 wheren, N is the total number of samples, N l is the total number of samples in class l, and q ll is the number of correctly classified samples in class l. All the results reported in this study have been conducted on a Matlab R2011b runtime environment on a Windows system with 8Gb memory. The best solution for all the algorithms is report upon conducting grid search. The parameters of OS-F-ELM is obtained by employing constructive-destructive approach [19]. Three binary classification and three multi-class classification problems are considered in this study. These problems have been selected based on their class-wise imbalance (I.F.), which is given by I.F. = 1 n N min N l. (25) l=1,,n A tabulated description of the problems considered is provided in Table I. TABLE I REAL-VALUED CLASSIFICATION DATASET Dataset # Features # Classes # Samples I.F. BC PIMA LIVER Iris IS GI The results for all the algorithms for binary classification problems is given in Table II and for multi-class classification problems, its given in Table III. From the tables, it could be seen that overall CIT2FIS performs better than other algorithms employed. In case of binary classification problems, the performance of CIT2FIS is at par with the next best performing algorithm: PBL-McCIT2FIS. While in multi-class classification problems, CIT2FIS attains significantly better performance in comparison to other state-of-the-art complexvalued as well as Interval Type-2 fuzzy system. It could however be observed that CIT2FIS requires more number of rules to generalize the networ efficiently in comparison to PBL-McCIT2FIS. This might be due to the use of rules as pseudo-samples during learning. More number of rules might be required for attaining this high efficiency. For more parsimonious networ, one can substitute the pseudosamples with actual training samples. This might in turn cost computationally more. PBL-McCIT2FIS is also a complexvalued Interval Type-2 fuzzy inference system. However, the TABLE II PERFORMANCE COMPARISON FOR BINARY CLASSIFICATION PROBLEMS Data Algorithm P.S. Rules Testing Set η o η a SVM OS-F-ELM Liver McIT2FIS CNFIS PBL-McCIT2FIS CIT2FIS SVM OS-F-ELM PIMA McIT2FIS CNFIS PBL-McCIT2FIS CIT2FIS SVM OS-F-ELM BC McIT2FIS CNFIS PBL-McCIT2FIS CIT2FIS TABLE III PERFORMANCE COMPARISON FOR MULTI-CATEGORY CLASSIFICATION PROBLEMS Data Algorithm P.S. Rules Testing Set η o η a SVM OS-F-ELM Iris McIT2FIS CNFIS PBL-McCIT2FIS CIT2FIS SVM OS-F-ELM IS McIT2FIS CNFIS PBL-McCIT2FIS CIT2FIS SVM OS-F-ELM GI McIT2FIS CNFIS PBL-McCIT2FIS CIT2FIS membership function employed therein is q-gaussian and it realizes both uncertainty in mean as well as standard deviation, which helps it generalize the data well. Employing such an inference mechanism might help CIT2FIS attain better results, at increased computational cost. IV. CONCLUSION In this paper, we have proposed a purely online data-driven learning algorithm for Complex-valued Interval Type-2 Fuzzy Inference System (CIT2FIS). The proposed system evolves (add/prune) rules and adapts rule parameters based on the nowledge content in the samples. Moreover, the use of metacognitive learning mechanism helps it generalize the training data, efficiently. The classification ability of the CIT2FIS was evaluated on a set of real-valued classification problems. The performance comparison with other real-valued as well as

6 complex-valued state-of-the-art networs clearly indicates that the proposed networ can achieve more robust performance and better generalize the training data. In the future, this wor will be enhanced to efficiently predict/classify weather state for profiling. ACKNOWLEDGEMENT The authors wish to than the Air Traffic Management Research Institute (ATMRI), Nanyang Technological University, Singapore, for providing financial support (M ) to conduct this study. REFERENCES [1] B. C. and C. Merz, UCI repository of machine learning databases, 1998, department of Information and Computer Sciences, University of California, Irvine. [Online]. Available: [2] C. Chang and C. Lin, LIBSVM:a library for support vector machines, ACM Transactions on Intelligent Systems and Technology, vol. 2, no. 27, pp. 1 27, [3] A. K. Das, K. Subramanian, and S. Suresh, A computationally fast interval type-2 neuro-fuzzy inference system and its meta-cognitive projection based learning algorithm, in IEEE International Joint Conference on Neural Networs, 2014, pp [4], An evolving interval type-2 neuro-fuzzy inference system and its meta-cognitive sequential learning algorithm, IEEE Trans. Fuzzy Systems, vol. -, no. -, pp., 2015, doi: /tfuzz [5] C. Lawson and R. Hanson, Solving least squares problems. Prentice- Hall, 1974, vol [6] Q. Liang and J. Mendel, Interval type-2 fuzzy logic systems: Theory and design, IEEE Trans. Fuzzy Systems, vol. 8, no. 5, pp , [7] Y. Lin, S. Liao, J. Chang, and C. Lin, Simplified interval type-2 fuzzy neural networs, IEEE Trans. Neural Networs and Learning Systems, vol. 25, no. 5, pp , [8] M. Nie and W. Tan, Towards an efficient type-reduction method for interval type-2 fuzzy logic systems, in IEEE Intl. Conf. Fuzzy Systems, 2008, pp [9] T. Nitta, The computational power of complex-valued neuron, in Artificial Neural Networs and Neural Information Processing, ser. Lecture Notes in Computer Science, vol. 2714, 2003, pp [10], Orthogonality of decision boundaries of complex-valued neural networs, Neural Computation, vol. 16, no. 1, pp , [11] H. Rong, G. Huang, N. Sundararajan, and P. Saratchandran, Online sequential fuzzy extreme learning machine for function approximation and classification problems, IEEE Trans. System, Man and Cybern, Part B: Cybern, vol. 39, no. 4, pp , [12] H. Rong, N. Sundararajan, G. Huang, and P. Saratchandran, Sequential adaptive fuzzy inference system (SAFIS) for non-linear system identification and prediction, Fuzzy Sets and Systems, vol. 157, no. 9, pp , [13] R. Savitha, S. Suresh, and N. Sundararajan, Fast learning circular complex-valued extreme learning machine (cc-elm) for real-valued classification problems, Information Sciences, vol. 187, no. 1, pp , [14] K. Subramanian, A. K. Das, S. Sundaram, and R. Savitha, A metacognitive interval type-2 fuzzy inference system and its projection based learning algorithm, Evolving Systems, vol. 5, no. 4, pp , [15] K. Subramanian, R. Savitha, and S. Suresh, A meta-cognitive interval type-2 fuzzy inference system classifier and its projection based learning algorithm, in IEEE Conference on Evolving and Adaptive Intelligent Systems, 2013, pp [16], A complex-valued neuro-fuzzy inference system and its learning mechanism, Neurocomputing, vol. 123, no. 0, pp , [17], A metacognitive complex-valued interval type-2 fuzzy inference system, Neural Networs and Learning Systems, IEEE Transactions on, vol. 25, no. 9, pp , [18] K. Subramanian, S. Sundaram, and N. Sundararajan, A meta-cognitive neuro-fuzzy inference system (McFIS) for sequential classification problems, IEEE Transactions on Fuzzy Systems, vol. 21, no. 6, pp , [19] S. Suresh, S. Omar, V. Mani, and T. Gurupraash, Lift coefficient predictiona t high angle of attac using recurrent neural networs, Aerospace Science and Technology, vol. 7, no. 8, pp , [20] S. Suresh, N. Sundararajan, and P. Saratchandran, Ris sensitive loss functions for sparse multi-category classification problems, Information Sciences, vol. 179, no. 21, pp , [21] D. Wu, An overview of alternative type-reduction approaches for reducing the comptational cost of interval type-2 fuzzy logic controllers, in IEEE Intl. Conf. Fuzzy Systems, 2012, pp [22] L. Zadeh, The concept of a linguistic variable and its application to approximate reasoning, Information Sciences, vol. 8, no. 3, pp , 1975.