Lecture 3: Multi-layer perceptron

Transcription

1 x Fundamental Theores and Applcatons of Neural Netors Lecture 3: Mult-laer perceptron Contents of ths lecture Ree of sngle laer neural ors. Formulaton of the delta learnng rule of sngle laer neural ors. BP: an extended delta-learnng rule for multlaer neural ors. Produced b Qangfu Zhao (Sne 997, All rghts resered Lecture 3- Produced b Qangfu Zhao (Sne 997, All rghts resered Lecture 3- Ree of sngle laer neural or Reformulaton of the delta learnng rule There are J nputs and K outputs. The last nput s fxed to so that the correspondng eght s the thresh. For a gen nput ector The effecte nput of the -th neuron s The actual output of the -th neuron s o The desred output of the -th neuron s d The error to be mnmzed s E o o o K J Accordng to the gradent descent algorthm, the eght from the -th nput to the -th neuron should be updated b E Produced b Qangfu Zhao (Sne 997, All rghts resered Lecture 3-3 Produced b Qangfu Zhao (Sne 997, All rghts resered Lecture 3-4

2 Gradent of the error functon K E ( d Note that E s E here ( o rule, ecan get the partal dreate E ( ( J mplctel a functon of (, usng the chan Produced b Qangfu Zhao (Sne 997, All rghts resered Lecture 3-5 of E to as follos : Defnton of the error sgnal If e defne the error sgnal produced b the - th neuron as follos : o ehae here o E ( E o o o ( ( d o f '( Produced b Qangfu Zhao (Sne 997, All rghts resered Lecture 3-6 Equaton for updatng the eghts Thus, the eght can be updatedb If e use the unpolar sgmod functon th, f '( If e use the bpolar sgmod functon th, f '( o ( d ( o ( o o f '( / For off-lne learnng Remars The error should be defned as the total error of the or for all tranng examples. The tranng examples are used repeatedl untl the error becomes small enough. The eghts of all neurons are updated all together n a snchronous mode. Produced b Qangfu Zhao (Sne 997, All rghts resered Lecture 3-7 Produced b Qangfu Zhao (Sne 997, All rghts resered Lecture 3-8

3 x Program of delta learnng rule for sngle laer neural or Intalzaton(; hle(error>desred_error{ for(error=0,p=0; p<n_sample; p++{ FndOutput(p; for(=0;<k;++{ Error+=0.5*po(d[][p]-o[],.0; for(=0;<k;++{ delta=(d[][p]-o[]*(-o[]*o[]/; for(=0;<j;++{ [][]+=eta*delta*[p][]; Produced b Qangfu Zhao (Sne 997, All rghts resered Lecture 3-9 Results for AND/OR gates Error n the 3-th learnng ccle= Error n the 3-th learnng ccle= Error n the 33-th learnng ccle= Error n the 34-th learnng ccle= Error n the 35-th learnng ccle= Error n the 36-th learnng ccle=0.003 Error n the 37-th learnng ccle= Error n the 38-th learnng ccle= Error n the 39-th learnng ccle= Error n the 330-th learnng ccle= Error n the 33-th learnng ccle= W[0]: W[]: Produced b Qangfu Zhao (Sne 997, All rghts resered Lecture 3-0 Multlaer feedforard neural or o o o K Multlaer feedforard neural or J The or n the last slde s a three laer perceptron, also called three laer feed forard neural or. There are I nputs, J hdden neurons, and K output neurons. The last nput of each hdden neuron or each output neuron s. The nput can be output of another laer of neurons. The output can be nput of another laer. z z z I Produced b Qangfu Zhao (Sne 997, All rghts resered Lecture 3- Produced b Qangfu Zhao (Sne 997, All rghts resered Lecture 3-3

4 Defnton of notatons z : The -th nput : The output of the -th hdden neuron o : The output of the -th output neuron : The eght from the -th nput to the -th hdden neuron : The eght from the -th hdden neuron to the -th output neuron Rule for updatng the output neurons Weght update for the output neurons can be performed exactl n the same a as for sngle laer perceptron. For an nput, fnd the output of the hdden neurons, and then the output of the output neurons. The eghts of each output neuron can be updated b usng the delta learnng rule. Produced b Qangfu Zhao (Sne 997, All rghts resered Lecture 3-3 Produced b Qangfu Zhao (Sne 997, All rghts resered Lecture 3-4 Rule for updatng the hdden neurons Frst, e hae Usng the chan rule, ecan get E E E ( ( Error sgnal produced b the - th hdden neuron E ( Update the eghts of hdden neurons ( Note also that Usng the chan rule, ecan fnd the error sgnal as follos : ( E z z, e hae f '( K o Produced b Qangfu Zhao (Sne 997, All rghts resered Lecture 3-5 Produced b Qangfu Zhao (Sne 997, All rghts resered Lecture 3-6 4

5 Update the eghts of hdden neurons Thus, the eghts of the hdden neurons can be updated If e use the unpolar sgmod functon th, f '( If e use the bpolar sgmod functon th, f '( z ( ( / b Comments The learnng algorthm s usuall called the bac propagaton (BP algorthm because the error sgnal of the hdden neurons are bac propagated from the output laer to the hdden laer(s. In some context, the algorthm s also called the extended delta-learnng rule. Produced b Qangfu Zhao (Sne 997, All rghts resered Lecture 3-7 Produced b Qangfu Zhao (Sne 997, All rghts resered Lecture 3-8 Summar of the BP algorthm Step : Intalze the eghts. Step : Reset the total error. Step 3: Get a tranng example z from the tranng set, calculate the outputs of the hdden neurons and those of the output neurons, and update the total error. Step 4: Calculate the error sgnals as follos: o ( d ( o o / K o (- / Summar of the BP algorthm (cont. Step 5: Update the eghts as follos: ηδ o for,,, K;,,, J ηδ z for,,, J;,,, I Step 6: See f all tranng examples hae been used. If NOT, return to Step 3. Step 7: See f the total error s smaller than the desred alue. If NOT, return to Step ; otherse, termnate. Produced b Qangfu Zhao (Sne 997, All rghts resered Lecture 3-9 Produced b Qangfu Zhao (Sne 997, All rghts resered Lecture 3-0 5

6 A smple example: Solng the XOR problem The functon to be approxmated s a - arable bnar functon. Ths problem, although smple, cannot be soled b ANY sngle laer neural or. o The or structure XOR can be soled usng a three laer neural or. It contans three nputs, th the last one beng fxed to, to hdden neurons, and one output neuron. The problem s to fnd the correct eghts for all neurons, so that for an gen nput, the correct output can be proded. Produced b Qangfu Zhao (Sne 997, All rghts resered Lecture 3- Produced b Qangfu Zhao (Sne 997, All rghts resered Lecture 3- Results of BP Error[073]= Error[074]= Error[075]= Phscal meanng of the result L : 0.64x 0.5x 0.3 The connecton eghts n the output laer: The connecton eghts n the hdden laer: L : 0.3x 0.47x 0.06 L3 : Produced b Qangfu Zhao (Sne 997, All rghts resered Lecture 3-3 Produced b Qangfu Zhao (Sne 997, All rghts resered Lecture 3-4 6

7 To mproe the BP algorthm The learnng constant can be arable If the error s reduced greatl b the current update, the learnng rate can be ncreased. If the error s not reduced, the learnng rate can be decreased. Momentum method The ncrement of the eght can be modfed b the updatng hstor. Ho man hdden neurons to use? Accordng to Multlaer Perceptrons: Approxmaton order and Necessar Number of Hdden Unts (rtten b Stephan Trenn, IEEE TNN, Vol. 9, No. 5, 008, pp , for an MLP th one hdden laer, n 0 nputs, and smooth actaton functon, t achees approxmaton order N for all functons onl f the number of hdden unts s larger than N n0 n0 n 0 Produced b Qangfu Zhao (Sne 997, All rghts resered Lecture 3-5 Produced b Qangfu Zhao (Sne 997, All rghts resered Lecture 3-6 Ho man hdden Laers to use? For hgh approxmaton order (>, to hdden laers should be used nstead of one hdden laer. For lnear and quadratc approxmaton, onl one hdden laer s needed. Here, a functon f approxmates another functon g th order N f and onl f ther Talor polnomals are the same up to the order N. The functon to be approxmated b the MLP should be suffcentl smooth. The numbers gen here are relatel conserate because the MLP must approxmate ANY functon ell (to sole a practcal problem, e ma consder one functon or a specal set of functons onl. Team Proect II Mae a computer program for the BP algorthm. Test our program usng the 4-bt part chec problem. The number of nputs s 5 (4 plus one dumm nput and the number of output s ([0,] or [-,]. The desred output s f the number of ones n the nputs s een; otherse, the output s 0 or -. Chec the performance of the or b changng the number of hdden neurons from 4 to 0, th step-sze. Prode a summar of our results n our report (txt-fle. Produced b Qangfu Zhao (Sne 997, All rghts resered Lecture 3-7 Produced b Qangfu Zhao (Sne 997, All rghts resered Lecture 3-8 7