Networks. Backpropagation. Backpropagation. Introduction to. Backpropagation Network training. Backpropagation Learning Details 1.04.

Transcription

1 Networs Introducton to - In 1986 a method for learnng n mult-layer wor,, was nvented by Rumelhart Paper Why are what and where processed by separate cortcal vsual systems? - The algorthm s a sensble approach for dvdng the contrbuton of each weght. - Wors bascally the same as perceptrons Learnng Prncples: Hdden Layers and Gradents There are two dfferences for the updatng rule : 1 The actvaton of the hdden unt s used nstead of actvaton of the nput value. The rule contans a term for the gradent of the actvaton functon. Networ tranng 1. Intalze wor wth random weghts. For all tranng cases (called eamples: a. Present tranng nputs to wor and calculate output b. For all layers (startng wth output layer, bac to nput layer:. Compare wor output wth correct output (error functon. Adapt weghts n current layer Ths s what you want Learnng Detals Method for learnng weghts n feed-forward (FF s Can t use Perceptron Learnng Rule no teacher values are possble for hdden unts Use gradent descent to mnmze the error propagate deltas to adust for errors bacward from outputs to hdden layers forward to nputs bacward Algorthm Man Idea error n hdden layers The deas of the algorthm can be summarzed as follows : 1. Computes the error term for the output unts usng the observed error.. From output layer, repeat - propagatng the error term bac to the prevous layer and - updatng the weghts between the two layers untl the earlest hdden layer s reached. 1

2 Algorthm Intalze weghts (typcally random! eep dong epochs For each eample e n tranng set do forward pass to compute O = neural--output(wor,e mss = (T-O at each output unt bacward pass to calculate deltas to weghts update all weghts untl tunng set error stops mprovng Bacward Pass Compute deltas to weghts from hdden layer to output layer Wthout changng any weghts (yet, compute the actual contrbutons wthn the hdden layer(s and compute deltas Forward pass eplaned as a perceptron earler Bacward pass eplaned n slde Gradent Descent Error Surface Thn of the N weghts as a pont n an N- dmensonal space error Add a dmenson for the observed error Try to mnmze your poston on the error surface weghts Error as functon of weghts n multdmensonal space Compute deltas Gradent Usng Gradent Descent Tryng to mae error decrease the fastest Compute: Grad E = [de/dw 1, de/dw,..., de/dw n ] Change th weght by delta w = -alpha * de/dw Dervatves of error for weghts We need a dervatve! Actvaton functon must be contnuous, dfferentable, non-decreasng, and easy to compute Advantages Relatvely smple mplementaton Standard method and generally wors well Dsadvantages Slow and neffcent Can get stuc n local mnma resultng n suboptmal solutons

3 Local Mnmum Local Mnma Global Mnmum Learnng For nput pattern the -th nput layer node holds. Net nput to -th node n hdden layer: Output of -th node n hdden layer: Net nput to -th node n output layer: Output of -th node n output layer: Networ error for p: E p 1 ( l n 0 n S w 0 w, w,, o S w, ( d o 1 14 Learnng As E s a functon of the wor weghts, we can use gradent descent to fnd those weghts that result n mnmal error. For ndvdual weghts n the hdden and output layers, we should move aganst the error gradent (omttng nde p: w w,, E w, E w, Output layer: Dervatve easy to calculate Hdden layer: Dervatve dffcult to calculate Learnng When computng the dervatve wth regard to w,, we can dsregard any output unts ecept o : E l ( d o E ( d o o Remember that o s obtaned by applyng the sgmod functon S to, whch s computed by: w, Therefore, we need to apply the chan rule twce E w Snce We have: Learnng E o, o w, We now that: o S w,, w ' ( E w Whch gves us:, ( d o S' 17 Learnng For the dervatve wth regard to w,, notce that E deps on t through, whch nfluences each o wth = 1,, : o S S, w,, Usng the chan rule of dervatves agan: w E w E E o, 1 o w,, ( d o S' w, S'

4 Learnng Ths gves us the followng weght changes at the output layer: w, and at the nner layer: w wth ( d o S', w S 1 wth, ' 19 Learnng As you surely remember from a few mnutes ago: S' ( S( (1 S( Then we can smplfy the generalzed error terms: And: ( d o S' ( d o o (1 o w S 1 1, ' w, 1 0 Learnng The smplfed error terms and use varables that are calculated n the feedforward phase of the wor and can thus be calculated very effcently. Now let us state the fnal equatons agan and rentroduce the subscrpt p for the p-th pattern: w wth (1 o, ( d o o w wth, w, 1 1 How do we pc? 1. Tunng set, or. Cross valdaton, or 3. Small for slow, conservatve learnng How many hdden layers? Usually ust one (.e., a -layer How many hdden unts n the layer? Too few ==> can t learn Too many ==> poor generalzaton 4

5 How bg a tranng set? Determne your target error rate, e Success rate s 1- e Typcal tranng set appro. n/e, where n s the number of weghts n the Eample: e = 0.1, n = 80 weghts tranng set sze 800 traned untl 95% correct tranng set classfcaton should produce 90% correct classfcaton on testng set (typcal Stoppng Crteron? The algorthm termnates when the change n the crteron functon J(w s smaller than some preset value There are other stoppng crtera that lead to better performance than ths one So far, we have consdered the error on a sngle pattern, but we want to consder an error defned over the entrety of patterns n the tranng set The total tranng error s the sum over the errors of n ndvdual patterns A weght update may reduce the error on the sngle pattern beng presented but can ncrease the error on the full tranng set Other Ways To Mnmze Error Varyng tranng data Cycle through nput classes Randomly select from nput classes Add nose to tranng data Randomly change value of nput node (wth low probablty Retran wth epected nputs after ntal tranng E.g. Speech recognton Addng and removng neurons from layers Addng neurons speeds up learnng but may cause loss n generalzaton Removng neurons has the opposte effect The teachng process of mult-layer neural wor employng bacpropagaton algorthm. To llustrate ths process the three layer neural wor wth two nputs and one output Each neuron s composed of two unts. Frst unt adds products of weghts coeffcents and nput sgnals. The second unt realze nonlnear functon, called neuron transfer (actvaton functon. Sgnal e s adder output sgnal, and y = f(e s output sgnal of nonlnear element. Sgnal y s also output sgnal of neuron. To teach the neural wor we need tranng data set. The tranng data set conssts of nput sgnals ( 1 and assgned wth correspondng target (desred output z. The wor tranng s an teratve process. In each teraton weghts coeffcents of nodes are modfed usng new data from tranng data set. Modfcaton s calculated usng algorthm descrbed below: Each teachng step starts wth forcng both nput sgnals from tranng set. After ths stage we can determne output sgnals values for each neuron n each wor layer. 5

6 Pctures below llustrate how sgnal s propagatng through the wor, Symbols w (mn represent weghts of connectons between wor nput m and neuron n n nput layer. Symbols y n represents output sgnal of neuron n. Propagaton of sgnals through the hdden layer. Symbols w mn represent weghts of connectons between output of neuron m and nput of neuron n n the layer. 6

7 Propagaton of sgnals through the output layer. In the algorthm ste the output sgnal of the wor y s compared wth the desred output value (the target, whch s found n tranng data set. The dfference s called error sgnal d of output layer neuron The dea s to propagate error sgnal d (computed n sngle teachng step bac to all neurons, whch output sgnals were nput for dscussed neuron. The dea s to propagate error sgnal d (computed n sngle teachng step bac to all neurons, whch output sgnals were nput for dscussed neuron. The weghts' coeffcents w mn used to propagate errors bac are equal to ths used durng computng output value. Only the drecton of data flow s changed (sgnals are propagated from output to nputs one after the other. Ths technque s used for all wor layers. If propagated errors came from few neurons they are added. When the error sgnal for each neuron s computed, the weghts coeffcents of each neuron nput node may be modfed. df(e/de represents dervatve of neuron actvaton functon (whch weghts are modfed. 7

8 When the error sgnal for each neuron s computed, the weghts coeffcents of each neuron nput node may be modfed. df(e/de represents dervatve of neuron actvaton functon (whch weghts are modfed. When the error sgnal for each neuron s computed, the weghts coeffcents of each neuron nput node may be modfed. df(e/de represents dervatve of neuron actvaton functon (whch weghts are modfed. Learnng Factors Intal Weghts Learnng Constant ( Cost Functons Momentum Update Rules Tranng Data and Generalzaton Number of Layers Number of Hdden Nodes Matlab Eamples p=0:0.5:5; t = sn(p; fgure; plot(t,'+b'; as([ ]; = newff([0 0],[6,1],{'logsg','pureln'},'tranlm';.tranParam.epochs = 75;.tranParam.goal = 0.001; = tran(,t; a = sm(,p; hold on; plot(a,'.r'; % b-polar case clear all close all dsp (' '; dsp ('Bpolar Tranng'; P = [ ; ; ] T = [ ] [R, Q] = sze(p; % contanng the number of rows and columns n the matr. W = 0.001*randn(R,1; %RANDN(N returns an N-by-N matr contanng random values between -1 and 1 (normal Dstn, mean 0, Std Dev of 1 alpha = 0.15; err = 0.1; MaIter = 1000; ter = 0; MSE = []; % MSE s a wor performance functon. It measures the wor's performance accordng to the mean of squared errors. % MSE(E,X,PP taes from one to three arguments, % E - Matr or cell array of error vector(s. % X - Vector of all weght and bas values (gnored. % PP - Performance parameters (gnored. % and returns the mean squared error. whle ter < MaIter ter = ter + 1; Er = 0; tqe = 0; 8

9 for = 1:Q %q pattern, each tranng case, 4 cases n total, no of columns n P v = P(:, '*W; % v = w'( (; w= 31 matr; = 31 matr; w' = 11 e = T( - v; %desred value - v n = norm (P(:, ; % NORM(V,P = sum(abs(v.^p^(1/p. NORM(V = norm(v,. f n~=0 W = W + alpha * e * P(:, / n ^ ; %change weght Er = Er + 1/Q*e^; MSE = [MSE Er]; f Er <= err fprntf(1, 'err satsfed \n'; brea; span = 10; f ter > (span+1 de = MSE(ter - MSE(ter - span; f abs(de < 1e-7 fprntf(1, 'the wor s updatng too slow\n'; brea W ter % the rest of the program fgure; plot(mse; ttle('bpolar Tranng MSE performance'; label('epochs'; ylabel('mse'; rng default; % For reproducblty %random means X = [randn(100,*0.75+ones(100,; randn(100,*0.5-ones(100,]; fgure; plot(x(:,1,x(:,,'.'; ttle 'Randomly Generated Data'; opts = statset('dsplay','fnal'; [d,c] = means(x,,'dstance','ctybloc',... 'Replcates',5,'Optons',opts; fgure; plot(x(d==1,1,x(d==1,,'r.','marersze',1 hold on plot(x(d==,1,x(d==,,'b.','marersze',1 plot(c(:,1,c(:,,'',... 'MarerSze',15,'LneWdth',3 leg('cluster 1','Cluster ','Centrods',... 'Locaton','NW' ttle 'Cluster Assgnments and Centrods' hold off 9