Chaotic Complex-Valued Multidirectional Associative Memory with Adaptive Scaling Factor

Transcription

1 Proceedings of International Joint Conference on Neural Networks, Dallas, Texas, USA, August 4-9, 2013 Chaotic Complex-Valued Multidirectional Associative Memory with Adaptive Scaling Factor Takuya Chino and Yuko Osana Abstract In this paper, we propose a Chaotic Complexvalued Multidirectional Associative Memory (CCMAM with adaptive scaling factor. The proposed model is based on the conventional CCMAM with variable scaling factor. In the conventional CCMAM with variable scaling factor, the scaling factor of refractoriness is determined based on the time. In contrast, in the proposed model, the scaling factor of refractoriness is determined based on not only the time but also the internal states of neurons. The proposed model is composed of complex-valued neurons and chaotic complexvalued neurons, and can realize one-to-many associations of M-tuple multi-valued patterns as similar as the conventional CCMAM with variable scaling factor. We carried out a series of computer experiments and confirmed that the proposed model can determine the scaling factor of refractoriness automatically and its one-to-many association ability almost equals to that of the well-turned CCMAM with variable scaling factor. I. INTRODUCTION As the model which can deal with multi-valued patterns, the complex-valued neural network[1] has been proposed. Moreover, we modified the complex-valued neural network by introducing chaotic complex-valued neurons[2] and some models which can realize one-to-many associations of multivalued patterns[3] [8]. In these models, the association of multi-valued patterns is realized by complex-valued neurons, and one-to-many association is realized by chaotic complexvalued neurons. Moreover, in the Chaotic Complex-valued Multidirectional Associative Memory (CCMAM with variable scaling factor[6][7], one-to-many association ability is improved by introducing variable scaling factor. However, in these models, their property is very sensitive to chaotic complex-valued neuron parameters, and in most cases, these parameters are determined based on the designer s experiments or trial and errors. In this paper, we propose the Chaotic Complex-valued Multidirectional Associative Memory (CCMAM with adaptive scaling factor. In the proposed model, the appropriate parameter (scaling factor of refractoriness can be determined based on the internal states of neurons automatically. II. CHAOTIC COMPLEX-VALUED NEURON MODEL The chaotic complex-valued neuron model[2] is the extended chaotic neuron model[9] in order to deal with complex-value as internal states and output of neurons. Since the chaotic complex-valued neuron model is based on the complex-valued neuron model[1], the network composed of the chaotic complex-valued neuron models can realize association of multi-valued patterns. Moreover, since the chaotic Takuya Chino and Yuko Osana are with the School of Computer Science, Tokyo University of Technology ( osana@stf.teu.ac.jp. complex-valued neuron model is based on the chaotic neuron model, the network composed of the chaotic complex-valued neuron model can realize dynamic associations of stored patterns. The output of the chaotic complex-valued neuron is given by ( t x(t + 1 = f A(t α k d x(t d θ (1 d=0 (A(t, x(t, θ C k, α R where x(t is the output of the neuron at the time t, A(t is the external input at the time t, α is the scaling factor of refractoriness, k is the damping factor (0 < k < 1, and θ is the threshold of the neuron. And, f( is the output function which is given by ηu f(u = (η R (2 η u where η is the constant (η > 1. In Eq.(1, if the external input A(t is constant (that is, A(t = A, the output of the neuron x(t+1 can be expressed by x(t + 1 = f(u(t + 1 = f(ku(t αf(u(t + (A θ(1 k = f(ku(t αf(u(t + a (3 where u(t is the internal state of the neuron at the time t. In Eq.(3, the parameter a is the bifurcation parameter. The bifurcation parameter a corresponds (A θ(1 k, and the threshold θ and the damping factor k are constants, a varies depending on only the external input A. Figure 1 shows the bifurcation of the real part of the internal state of the chaotic complex-valued neuron. In Fig.1, the parameter of the output function η is set to 1.1, and a R is set to equal to a I. As shown in Fig. 1, area where chaos occurs are different according to the scaling factor α, and whether chaos occurs or not is determined based on the bifurcation parameter a (that is, external input. III. RELATION BETWEEN ONE-TO-MANY ASSOCIATION ABILITY AND INTERNAL STATES Here, we examined the relation between one-to-many association ability and internal states in the conventional Chaotic Complex-Valued Multidirectional Associative Memory (CCMAM with variable scaling factor[6][7] in order to decide the method how to determine the scaling factor of refractoriness. As mentioned in II, whether chaos occurs /13/$ IEEE 991

2 (a α = 1.0 (a S=4 (b α = 2.0 Fig. 1. Bifurcation Diagram of Chaotic Complex-valued Neuron Model. or not is determined based on the external input. In the CCMAM with variable scaling factor, the internal state of each neuron is calculated based on the input from other layers and the refractoriness of the neuron. And the input from other layers can be considered as the external input to the neuron. (b S=8 In the conventional CCMAM with variable scaling factor, the scaling factor of refractoriness at the time t, α(t is given by ( α(t = a + b sin c π 12 t (4 and it is known that the combination of the parameters a and b affects one-to-many association ability[7][8]. Figure 2 shows average internal states without refractoriness in various combination of a and b in the 3-layered CCMAM with variable scaling factor. As shown in Fig.2, average internal states without refractoriness is small when the number of stored patterns N is small. And it does not depends on the number of states S. Figure 3 shows the recall rate in the various combination of a and b in the same experiments. As shown in the Fig.3, Fig. 2. (c S=16 Average Internal States (M=3. 992

3 (a 1-to-2 Patterns (d 1-to-5 Patterns (b 1-to-3 Patterns (e 1-to-6 Patterns (c 1-to-4 Patterns (f 1-to-7 Patterns 993

4 TABLE I PARAMETER COMBINATION OF a AND b WHEN RECALL RATE IS HIGHEST. M S a b (g 1-to-8 Patterns Fig. 4. Relation between Internal States and Parameter a when Recall Rate is High. (h 1-to-9 Patterns Fig. 3. (i 1-to-10 Patterns Recall Rate in CCMAM with Variable Scaling Factor (M=3. the combination of a and b when the recall rate is highest in each condition. Figure 4 shows the relation between internal state without refractoriness and the appropriate parameter a. And the green line shows the function which determines the parameter a in the proposed model. This function is determined based on these experiments. IV. CHAOTIC COMPLEX-VALUED MULTIDIRECTIONAL ASSOCIATIVE MEMORY WITH ADAPTIVE SCALING FACTOR Here, I explain the proposed Chaotic Complex-Valued Multidirectional Associative Memory (CCMAM with adaptive scaling factor. The proposed model is based on the conventional Chaotic Complex-Valued Multidirectional Associative Memory with variable scaling factor[6][7]. in the network which stores 1-to-2 patterns, the recall rate is 1.0 in almost every combination of a and b. In the network which stores 1-to-3 or more patterns, the recall rate is high in a part of the combination of a and b. And, the recall rate is low when the number of the stored patterns N and the number of states S are large. We carried out same experiments in the 4 and 5-layered networks. Table I shows A. Structure The proposed Chaotic Complex-Valued Multidirectional Associative Memory with adaptive scaling factor has three or more layers as similar as the conventional Chaotic Complex- Valued Multidirectional Associative Memory with Variable Scaling Factor. Figure 5 shows the structure of the proposed model which has three layers (X Layer, Y Layer and Z Layer. Each layer consists of two parts; (1 Key Input 994

5 the context part change their states by chaos, plural patterns corresponding to the input common pattern can be recalled. Step 1 : Input to Layer x The input pattern is given to the key input part in the layer x. Step 2 : Propagation from Layer x to Other Layers Fig. 5. Structure of Chaotic Complex-Valued Multidirectional Associative Memory with Adaptive Scaling Factor Part composed of complex-valued neuron models[1] and (2 Context Part composed of chaotic complex-valued neuron models[2]. B. Learning Process In the proposed model, pattern sets are memorized by the orthogonal learning. In the proposed model which has M layers, the connection weights from the layer x to the layer y is given by w yx = X x (X yx y 1 X y. (5 In the same way, the connection weights from the layer y to the layer x is given by w xy = X y (X xx x 1 X x. (6 In Eqs.(5 and (6, * shows the conjugate transpose, 1 shows the inverse, and X x and X y are the training pattern matrixes which are memorized in the layer x and the layer y, and are given by X x = {X (1 x,, X (p x,, X (P x } (7 X y = {X (1 y,, X (p y,, X (P y } (8 where X (p x is the pattern p which is stored in the layer x, X (p y is the pattern p which is stored in the layer y and P is the number of the training pattern sets. In the orthogonal learning, the pattern sets including oneto-many relation can not be memorized. This is because the stored common pattern cause superimposed pattern in the recall process. In the proposed model, each learning pattern is memorized together with its own contextual information in order to memorize the training set including one-to-many relations as similar as the conventional CCMAM with variable scaling factor. Here, the contextual information patterns are generated randomly. C. Recall Process In the recall process, only neurons in the key input part receives input in the first step. This is because we assume that contextual information is usually unknown for users. In the proposed model, since the chaotic complex-valued neurons in The information in the layer x is propagated to the key input part in other layers. The output of the neuron k in the key input part of the layer y (y x at the time t, x y k (t is calculated by N x x y k (t = f w yx (9 j=1 kj xx j (t where N x is the number of neurons in the layer x, w yx kj is the connection weight from the neuron j in the layer x to the neuron k in the layer y, and x x j (t is the output of the neuron j in the layer x at the time t. And f( is the output function which is given by Eq.(2. Step 3 : Propagation from Other Layers to Layer x The information in other layers is propagated to the layer x. The output of the neuron j in the key input part of the layer x, x x j (t + 1, is given by M x x j (t + 1 = f y x ( n y k=1 w xy jk xy k (t + va x j (10 where M is the number of layers, n y is the number of neurons in the key input part of the layer y, w xy jk is the connection weight from the neuron k in the layer y to the neuron j in the layer x, and v is the connection weight from the external input. A x j is the external input to the neuron j in the layer x and is given by { A x 0, t < t in j = ˆx x j (t (11 in, t in t t in = min t n x (ˆx x j (t ˆx x j (t 1 = 0 (12 j=1 ˆx x j (t = argmin(ω s x x j (t (ω s x x j (t (13 s (s = 0, 1,..., S 1 ω = exp ( i 2π S (14 where ˆx x j (t is the quantized output of the neuron j in the layer x at the time t, S is the number of states and i is the imaginary unit. 995

6 The output of the neuron j of the context part in the layer x, x x j (t + 1 is given by M x x j (t + 1 = f y x α(t, I ( n y k=1 w xy jk t kmx d d k(t d d=0 t kr d x x j (t d d=0 (15 where k m and k r are damping factors. And, α(t, I is the scaling factor of refractoriness at the time t and the average internal state without refractoriness I. The average internal state without refractoriness I is given by I = u j = 1 N x n x M y x ( n y k=1 N x j=n x +1 w xy jk Re u j + Im u j 2 (16 t kmx d y k (t d. (17 d=0 In Eq.(15, α(t, I is given by ( α(t, I = a(i + b(a(i, S sin c π 12 t (18 a(i = (I (19 b(a(i, S = { a(i, S = 4, 8 a(i/2, S = 16 (20 Eqs.(19 and (20 are determined based on the experiment shown in Sec. 2. Step 4 : Repeat Steps 2 and 3 are repeated. V. COMPUTER EXPERIMENT RESULTS Here, we show the computer experiment results in order to demonstrate of effectiveness of the proposed model. The experiments were carried out under the conditions shown in Tab.II. TABLE II EXPERIMENTAL CONDITIONS Parameter in Output Function η 1.01 Parameters in Scaling Factor of Refractoriness α(t, I c 5 The Number of Neurons in Key Input Part 100 The Number of Neurons in Context Part 50 Damping Factors k m 0.89 k r 0.92 Connection Weight to External Input v 5000 The Number of States S 4, 8, 16 A. One-to-Many Association Ability in 3-Layered Network Here, we compared the one-to-many association ability in the 3-layered proposed model with the 3-layered conventional CCMAM with variable scaling factor[6][7]. Figure 6 shows the one-to-many association ability of the proposed model and the conventional model. As shown in this figure, the one-to-many association ability of the proposed model almost equals to that of the conventional model. B. One-to-Many Association Ability in 4-Layered Network Here, we compared the one-to-many association ability in the 4-layered proposed model with the 4-layered conventional CCMAM with variable scaling factor. Figure 7 shows the one-to-many association ability of the proposed model and the conventional model. As shown in this figure, the oneto-many association ability of the proposed model is superior to that of the conventional model in almost every cases. C. One-to-Many Association Ability in 5-Layered Network Here, we compared the one-to-many association ability in the 5-layered proposed model with the 5-layered conventional CCMAM with variable scaling factor. Figure 8 shows the one-to-many association ability of the proposed model and the conventional model. As shown in this figure, the oneto-many association ability of the proposed model is superior to that of the conventional model in almost every cases. D. Discussion From the results in V-A V-C, we confirmed the following features. (1 The one-to-many association ability of the proposed model almost equals to that the conventional CCMAM with variable scaling factor. (2 Parameters can be determined appropriately in various conditions (the number of layers, the number of states. (3 The recall rate decreases if N (in 1-to-N patterns becomes large. VI. CONCLUSION In this paper, we have proposed the Chaotic Complexvalued Multidirectional Associative Memory (CCMAM with adaptive scaling factor. The proposed model is based on the conventional CCMAM with variable scaling factor[6][7]. In the conventional CCMAM with variable scaling factor, the scaling factor of refractoriness is determined based on the time. In contrast, in the proposed model, the scaling factor of refractoriness is determined based on not only the time but also the internal states of neurons. The proposed model is composed of complex-valued neurons and chaotic complexvalued neurons, and can realize one-to-many associations of M-tuple multi-valued patterns as similar as the conventional CCMAM with variable scaling factor. We carried out a series of computer experiments and confirmed that the proposed model can determine the scaling factor of refractoriness automatically and its one-to-many association ability almost equals to that of the well-turned CCMAM with variable scaling factor. 996

7 (a S=4 (a S=4 (b S=8 (b S=8 (c S=16 (c S=16 Fig. 6. One-to-Many Association Ability (M=3. Fig. 7. One-to-Many Association Ability (M=4. 997

8 (a S=4 REFERENCES [1] S. Jankowski, A. Lozowski and J. M. Zurada: Complex-valued multistate neural associative memory, IEEE Trans. Neural Networks, Vol.7, No.6, pp , [2] M. Nakada and Y. Osana: Chaotic complex-valued associative memory, Proceedings of International Symposium on Nonlinear Theory and its Applications, Vancouver, [3] Y. Yano and Y. Osana : Chaotic complex-valued bidirectional associative memory, Proceedings of IEEE and INNS International Joint Conference on Neural Networks, Atlanta, [4] Y. Yano and Y. Osana : Chaotic complex-valued bidirectional associative memory one-to-many association ability, Proceedings of International Symposium on Nonlinear Theory and its Applications, Sapporo, [5] Y. Shimizu and Y.Osana : Chaotic complex-valued multidirectional associative memory, Proceedings of IASTED Artificial Intelligence and Applications, Innsbruck, [6] A. Yoshida and Y. Osana : Chaotic complex-valued multidirectional associative memory with variable scaling factor, Proceedings of International Conference on Artificial Neural Networks, Espoo, [7] A. Yoshida and Y. Osana : Chaotic complex-valued multidirectional associative memory with variable scaling factor One-to-many association ability, Proceedings of IEEE and INNS International Joint Conference on Neural Networks, [8] M. Onagi and Y. Osana : Pattern dependency of one-to-many association ability in chaotic complex-valued multidirectional associative memory with variable scaling factor, Proceedings of IEEE and INNS International Joint Conference on Neural Networks, Brisbane, [9] K. Aihara, T. Takabe and M. Toyoda: Chaotic neural networks, Physics Letter A, 144, No.6, 7, pp , (b S=8 (c S=16 Fig. 8. One-to-Many Association Ability (M=5. 998