Experimental investigation of the performance of the optical two-layer neural network

Transcription

1 Experimental investigation of the performance of the optical two-layer neural network Nikolay N. Evtihiev, Rostislav S. Starikov, Boris N. Onyky, Vadim V. Perepelitsa, Igor B. Scherbakov Moscow State Engineering Physics Institute (Technical University), Department of Quantum Electronics, Kashirskoe shosse 31, Moscow, , Russia ABSTRACT The paper presents the obtained results of learning of the two-layer (64x8) neural network (TLNN), the results of tolerant noising of weight matrixes and results of hardware implementation. The imperfectness of optics satisfies the margin requirements of TLNN model 1. INTRODUCTION Different types of neural networks (NN) are under careful investigation now. Creating of the working models of NN, including optical models, is most useful and promising'. One of the most interesting is the two-layer NN (TLNN), which can be easily made on the basis of proposed optical matrix-vector multiplier (OVMM). OVMM has extremely high information processing capabilities. Computing in such TLNN is performed sequentially in OVMM. Large time-bandwidth product acoustooptic devices are the fastest up-to-date spatial light modulators. As a base devices for OVMM we propose to implement high-speed high dynamic range spatial light modulator namely the multichannelmultifrequency acoustooptic Bragg cell (MAOM), laser array and CCD (charge-coupled device) array in well-known orientation of OVMM. 2. NEURAL NETWORKS MATHEMATICAL MODELS SUITABLE FOR OPTOELECTRONIC IMPLEMENTATION Neural network mathematical models are determined by three characteristics: (1) models of used in the network neurones, (2) architecture of the interconnections in the network, (3) algorithm of determining of the neural network parameters. The main operations performed by neuron are the weighted addition and the non-linear transformation. In the simplest case a non-linear transformation is applied to the weighted sum and the result of transformation is transmitted to the neuron's output. The neuron does not perform other operation. It is possible to prove that the network which consists of such simple neurones can solve any problem if algorithm for it's solution exist. The meanings of the synaptic weights are different for the different neurones. Non-linear transformations may be different or the same for all the neurones of the network. Neural networks with the same transformations for all the neurones are called homogeneous. Due to the fact that the optoelectronic devices are the most effective for realization of the vector-matrix multiplication, it is very useful to put the main calculation load on them and to simplify the non-linear transformations as much as possible. That is why it is possible to assume that homogeneous neural networks are the most suitable for the optoelectronic realizations. O /941$6.QQ SPIE Vol Optical Neural Networks (1994) / 189

2 In theory the neural networks can have arbitrary interconnection architecture. But from the viewpoint of hardware implementation there is the most perspective architecture. It is the multilayer neural networks with sequential interconnections. As it was proved2 such networks may solve any task of image recognition. The amount of interconnection in such networks is less then in completely-connected ones. It reduces the complexity of the hardware realization. Thus, the multilayer homogeneous neural networks with sequential connections are the most suitable for the optoelectronic realization (see Fig. 1). Fig. 1. Multi-layer neural network. 190 /SP!E Vol Optical Neural Networks (1994)

3 3. TLNN. NOISE TOLERANCE The TLNN simulation software program was designed. Simulated NN contains 64 neurones in the first layer and 8 neurones in the second one. The NN was trained with gradient and stochastic algorithms. The ability of trained network allows to recognize binary images of all 26 alphabet letters. Each image consists of 8x8 pixel For the experiment the binary images were used. After the contrasting procedure the weight coefficients became the meanings of {-1;O;1}. The homogeneous TLNN performance is defmed by the following parameters: (1) synaptic weights of first-layer neurones, (2) input data of TLNN, (3) bias of first-layer neurones, (4) sum's outputs of first-layer neurones, (5)outputs of first-layer neurones, (6) synaptic weights of second-layer neurones, (7) bias of second-layer neurones, (8) sums outputs of second-layer neurones. As a result of learning parameters N 1, 3, 6, 7 are defined. The meanings of other parameters are defmed during the TLNN performance process. In the hardware implementation of TLNN each one of the eight parameters could has meaning that is different from the calculated one. This could happen because the technology of optoelectronic NN fabrication does not allow the exact representation of the calculated meanings of parameters. The aim of statistic experiments on noising was the defmition of tolerant deviation of meanings of the parameters from the calculated values. The criteria was the ability to recognize all the learned images. The modelling program allows to carry out experiments on noising with step of noise ratio (NR) 5%.The term NR could be illustrated by the following: p0=px(1+sx(2xrnd-1)/1oo) (1) where p is initial meaning of the parameter, p0 is the meaning of the parameter after noising, s -NR percentage, RND - uniformly spreaded random value, 0< RND <1. As a result of modelling the mean value of misrecognition and the mean data of not recognized images are define. The number of iteration for meaning was 100. The results of the experiments are presented in the table. No NN's parameters to be changed Allowed % of deviation 1 Synaptic weights of first-layer neurones 35% 2 Input data of TLNN 10% 3 Bias of first-layer neurones 105% 4 Sums outputs of first-layer neurones 80% S Outputs of first-layer neurones 25% 6 Synaptic weights of second-layer neurones 30% 7 Bias of second-layer neurones 130% SPIE Vol Optical Neural Networks (1994) 1191

4 No NN's parameters to be changed Allowed % of deviation 8 Sums outputs of second-layer neurones 85% 9 Synaptic weights of first- and second-layer neurones (NR of all other parameters 5%) 15% 10 All parameters of TLNN 10% The most important information is contained in the rows 9 and 10 of the table. The TLNN retains the ability of correct recognition of all the 26 alphabet letters in two cases: 1. The NR of synaptic weights of first- and second-layer neurones is less then 15%. 2. The NR of all other parameters is less then 5%. The NR of all the parameters is less then 10%. So, it proves that the technology of optoelectronic NN fabrication which allows to present all calculated parameters in hardware with accuracy better then 10% could be implemented for hardware realization of considering TLNN. The fig. 2 presents the plot of noise tolerant of all the parameters of TLNN. It is important to note the linear decreasing of the number of correctly recognized images against the NR of TLNN's parameters. While NR=60% the TLNN retains the ability of correct recognition of 50% images. For computer systems of previous generations another type of plot is inherent, namely avalanche dependence: 100% recognition at weak noise and 0% recognition after the increasing of some noise level N NR Fig. 2. Noise toleration of the TLNN. The computer experiment shows that the TLNN has significant noise resistance. This fact becomes the TLNN attractive for on-board implementations. 192 ISPIE Vol Optical Neural Networks (1994)

5 4. TLNN. HARDWARE Optical realization of the TLNN is an example of the effective application of MAOM. The main part of the calculations in this case is vector-matrix multiplication: A*B =C. The proposed OVMM architecture (fig. 3) allows one to perform this in one step. Traditionally the DMAC (digital multiplication via analogue convolution)3 algorithm is used for obtaining high accuracy in optical calculations. 11 ni "k' MAOM fi f.. Pp 4P 1 Isuk... Fig. 3. Vector-matrix multiplier with time integration. The algorithm we propose is similar to DMAC, but there is one significant difference concerned with the ability to use frequency coding in MAOM. Due to this fact OVMM allows one to multiply (convolve) 2D and 3D data flows and obtain the whole vector-matrix multiplication in one step with speeds about the frame rate of MAOM. In this case the vector as a 2D array is inputed into the laser array (LA) and the matrix as a 3D array is inputed into MAOM. In the proposed architecture vectora has 8-components (ar) and each component is represented by an 8-bit binary number (P 1,0), i.e. by a byte: k= 1 is the most (MSB), k=8 is the least significant bit (LSB). The matrixb dimension is 8x8 and each element b13 is represented by byte too (bs, s = 1..8): s =1 is the MSB, s =8 is the LSB. The elements of the 3D array b1s (i.e. matrix B) are inputed into the MAOM: i is the pulse position in the channel aperture at the exposure moment,j is the channel number, s is the acoustic wave frequency number (f'). When the MAOS aperture is filled the LA executes eight (according to the number of bits in each vector element) light pulses (expositions). Each vector component is applied to a corresponding "window", as is shown in fig. 3. The LSB of components are transmitted first, the MSB - last. Radiation from each laser uniformiy illuminates the corresponding "window" of every channel. The optical system transfers the light diffracted in the MAOM onto a set of eight linear CCD array. Each linear CCD contain eight (according to number of frequencies) points. The CCD operates in the shift-and-add mode. The shift velocity corresponds to the bit loading speed of the lasers. The charge is accumulated in a point of a linear CCD while first exposition is shifted to the next point of this CCD for a second exposition. Each linear CCD represents a component of the output vector C SPIE Vol Optical Neural Networks (1994)! 193

6 in a mixed representation. The supporting electronic control system (CS) is responsible for converting the presentation of this data from mixed to hex format, the tresholding, the synchronization, etc. For checking of opportunity of completion DMAC by described way the breadboard model performing multiplication 8 digit numbers was assembled. The rate of bitby-bit entering was limited only by existed maximal achievable for given CCD array transfer frequency. The typical result signal of DMAC-multiplication with implementation of multiplexing in the MAOM registered directly at the CCD output is presented on fig. 4. Taking into account the utmost parameters Te02 for MAOM as follows: 100 "windows", 8 frequencies, the period of laser pulse repetition is about 10 ns and the CCD array transfer frequency is about 120 MHz. The productivity of MAOM at calculations with "constant" matrix is: P(2xMxN-M)xM/2L, (2) where: M, N- dimension, L-number of bits, M -MAOM band of transmittance. In general case such multiplier is L-times slower than the analog one, using the same elements for inputing (LA and MAOM ). Moreover, considerable defect, affecting efficiency of system, is the necessity of using the CCD for completion DMAC with time-integration. Existing in present CCDs have comparatively low detecting ability. As it shown4 this fact on present stage puts under the question the competitiness of such device not only in comparison with analog AO processors, but also with existing electronic means. The presented below architectures of the digital vector-matrix AO multipliers with space integration seams to be more perspective. I I a) b) c) Fig. 4. Example of DMAC multiplication with implementation of multiplexing: a) lazer pulses, b) multiplexing signal in acoustooptic cell, c) CCD output signal in a mixed representation. The processor architecture based on the multifrequency MAOM and realized DMAC with space integration is presented on fig. 5. The architecture (similar to SAOBIC5 and analogous) consists of the source of collimated light (SCL), two MAOM (MAOM1 has N channels, MAOM2 has L channels), the projecting (OS1) and the summing (0S2) optical systems, which perform also the optical filtration, and M-photodetector array (PDA). The processor executes the multiplication of MxN matrix onto the N-vector using L-bit representation of numbers. The elements of vector b are loaded in parallel in first MAOM (each in the appropriate channel) in serial code. The elements of matrix ajj are loaded 194 / SPIE Vol Optical Neural Networks (1994)

7 i_fl second MAOM in following order: the elements of matrix with indexes j applies multiplexually - the i-th element at the i-th frequency and occupy position in the channel of the second MAOM, corresponding to the projection of the j-th channel of the first MAOM. In the l-th channel of the second MAOM only l-th bits of matrix elements applies. The materials of modulators and the scale characteristics of the projecting optical system OS1 are chosen so, that the ratio of the time for passing of the optical image, created by the first MAOM, to the time for passing the pixel of the second MAOM by sound is equal to the word length L. The whole multiplication process requires 2xL steps of the first ("fast" one) MAOM. The optical system 0S2 performs the DMAC summation and summation of the i-th components on the i-th photodetector. The following processing of the resulting vector elements is analog-to-digital transformation of triangular pulses of mixed code, arrived from the photodiodes after amplification, and summation with shift of the results of the transformation. The postprocessing as a whole requires M ADC (analog to digital convertor) and M shift-adders. SeL MAOM1 OS1 MAOM2 0S2 IDA Fig. 5. OVMM architecture with space integration. The utmost productivity at continuous calculations with "constant" matrix can reach: P(2xMxN-M)xM/2 (3) where: M, N - dimensions, L - number of bits, M - second MAOM band of transmittance. The given architecture, after corresponding alteration of postprocessing devices can be used for operations with mixed-sign arrays, represented in additional binary code6. Registration of the results of convolution begins only after completion of filling of first modulator and requires only L "fast" steps. The source of light synchronize so that the light arrived on modulators only on period between two moments: when the channels of the first MAOM are completely filled by elements of vector b and moment when last bits of elements b leave the channel The postprocessing device in this case requires N dividers and N shift-adders. The maximal dimension of vectors and matrixes that could be processed in such system is, on the one hand, limited by the maximum number of resolvable outputs of MAOM2 channels and, on the other hand, defines necessary at given word length of representation, dynamic range (DR) of the system (in turn, DR can provide discrimination of no more than levels of accepted signals); so at L=8 at processing of square matrixes their dimensions can be about 30. In general case, n-time increasing of the word length of representation reduces the maximal dimension of processable arrays in n times and accordingly lowers the efficiency. The increasing of the dimension of processable matrix SPIE Vol Optical Neural Networks (1994) / 195

8 (simultaneously the efficiency is increasing) is possible by rn-time increasing of the temporary aperture of MAOM1 and by rn-time increasing the number of PDA and channels of MAOM2. In the system performing multiplication of the matrix of m xm xn-dimension and vector of N-dimension at word length L the channels of the "fast" MAOM1 are divided on m parts, providing L discriminated outputs each, and projected onto appropriate L channels of MAOM2 (with total number of channels m XL ).The loading of elements of vector in MAOM1 and elements of matrix in each of m groups of channels MAOM2 (with appropriate delays ),the registration of results from each of m groups of FDA and clocking of sources of light at processing of mixed-sign arrays is executed similarly to architecture on fig. S. Total matrix-vector multiplication requires (m + 1) XL steps of the first MAOM, however it is possible to begin following multiplication after 2 XL steps. The speed limit will be: Pmx(2xMxN-M)xM/2 (4) It is possible to increase the efficiency of such multiplier by using instead of first MAOM the matrixes of light sources, providing the similar "movement" of elements of vector b. It is expected the increasing of the efficiency at the expense of considerable reduction of the required light and UHF power. However because of imperfectness of technology of modem light source matrixes the practical realization of such system is seems to be rather difficult. The considerable defect of systems with the space integration are the relative complexity and high speed required for read-out and postprocessing devices (in general case L-times higher than in analog devices, used such MAOM for inputing of matrix). The valuations of throughput various acoustooptic vector-matrix multipliers with Te02 modulators as matrix inputing devices are presented on fig. 6 and show potentially high competitiness acoustooptics in comparison with electronics. MOPS 102 mathx diüoi Fig. 6. Variations of throughput: 1-TRW TMC bit multiplier4, 2-analog multiplier with updatable matrix4, 3-multiplier with time integration, 4-analog multiplier with constant matrix4, 5-space integration multiplier with constant matrix (with LINbO3 "fast" MAOM). 196 ISPIE Vol Optical Neural Networks (1994)

9 5. CONCLUSION Thus, at present, vector-matrix multiplicationsin neural networks can be realized at the base of acoustooptics processors. The potential of the speed and efficiency of such processors compares favorably with existing electronic means at large dimensions of processed arrays. Thus initially the low accuracy of such calculations can be increased for account of complication of representation of processed values, presentation of rigid requirements to adjusting of the system as a whole and complication of postprocessing system. The configuration of the system (architecture, accuracy of representation of values, interface and postprocessing systems) is defined by particular statement of the task. 6. REFERENCES 1. E. Barnard and D. Casasent, "New Optical Neural System Architectures and Applications," Proc. SPIE, Vol. 963, pp , B. Muller, J. Reinhardt, Neural Networks, Springer-Verlag, P. R. Beaudet, A. P. Goutzoulis et al,appl. Optics, VoL 25, No. 18, p. 3097, C. K. Gary, "Comparison of optics and electronics for the calculation of matrix-vector products," Proc. SPIE, Vol Guilfoyle P. S., "Systolic acousto-optic binary convolver", Opt. Eng., Vol. 23, pp , Jan./Feb R. P. Bocker, S. P. Clyton, K. Bromley, "Electro-optical matrix multiplication using 2's component arithmetic for improved accuracy," Appi. Optics, VoL 22, N 13, pp , July SPIE Vol Optical Neural Networks (1994) / 197