NEURAL PROCESSIN G.SYSTEMS 2 INF ORM.ATIO N (Q90. ( Iq~O) DAVID S. TOURETZKY ADVANCES CARNEGIE MELLON UNIVERSITY. ..F~ k \ """ Ct... V\.

Transcription

1 ....F~ k \ """ Ct... V\. ~.Le.- b;e ve-. ( Iq~O) ADVANCES IN NEURAL INF ORM.ATIO N PROCESSIN G.SYSTEMS 2 EDITED BY DAVID S. TOURETZKY CARNEGIE MELLON UNIVERSITY (Q90.MORGAN KAUFMANN PUBLISHERS 2929 CAMPUS DRIVE SUITE 260 SAN MATEO, CALIFORNIA 94403

2 Fahlman and Lebere,~' The Cascade-Correlaton Learnng Archtecture :~ Scott E. Fahlman and Chrstan Lebere School of Computer Scence Carnege-Mellon Unversty Pttsburgh, PA ABSTRACT Cascade-Correlaton s a new archtecture and supervsed learnng algorthm for artfcal neural networks. Instead of just adjustng the weghts n a network of fxed topology, Cascade-Correlaton begns wth a mnmal network~ then automatcally trans and adds new hdden unts one. by one, creatng a mult-layer structure. Once a new hdden unt has been added to the network, ts nput-sde weghts are frozen. Ths unt then becomes a permanent feature-detector n the network, avalable for producng outputs or for creatng other, more complex feature detectors. The Cascade-Correlaton archtecture has several advantages over exstng algorthms: t learns very quckly, the network determnes ts own sze and topology, t retans the structures t has bult even f the tranng set changes, and t requres no back-propagaton of error sgnals through the connectons of the network. 1 DESCRIPTION OF CASCADE-CORRELATION The most mportant problem preventng the wdespread applcaton of artfcal neural. networks to real-world problems s the slowness of exstng learnng algorthms such as. back-propagaton (or "backprop"). One factor contrbutng to that slowness s what we. ~ - j;: ~.,,; call the movng target problem: because all of the weghts n the network are changng at once, each hdden unts sees a constantly changng envronment. Instead of movng ~ quckly to assume useful roles n the overall problem soluton, the hdden unts engage n ~ a complex dance wth much wasted moton. The Cascade-Correlaton learnng algorthm ~.~ was developed n an attempt to solve that problem. In the problems we have examned, ~.~ t learns much faster than back-propagaton and solves some other problems as well. ~;.::.-t ;: ~-;I ~., t~ HO l'";' r I,l

3 .. The Cascade-Correlaton Learnng Archtecture 525 Outputs.0 0 Hdden unt 1 Hdden Unt 2 Output Unts 0 Inputs Fgure 1: The Cascade archtecture, after two hdden unts have been added. The vertcal lnes sum all ncomng actvaton. Boxed connectons are frozen, X connectons are traned repeatedly..cascade-correlaton combnes two key deas: The frst s the cascade archtecture, n whch hdden unts are added to the network one at a tme and do not change after they have been added. The second s the learnng algorthm, whch creates and nstalls the new hdden unts. For each new hdden unt, we attempt to maxmze the magntude of the correlaton between the new unt's output and the resdual error sgnal we are tryng to elmnate..the The cascade archtecture s llustrated n Fgure 1. It begns wth some nputs and one or more output unts, but wth no hdden unts. The number of nputs and outputs s dctated by the problem and by the I/O representaton the expermenter has chosen. Every nput s connected to every output unt by a connecton wth an adjustable weght. There s also a bas nput, permanently set to +1. The output unts may just produce a lnear sum of ther weghted nputs, or they may employ some non-lnear actvaton functon. In the experments we have run so far, we use a symmetrc sgmodal actvaton functon (hyperbolc tangent) whose output range s -1.0 to For problems n whch a precse analog output s desred, nstead of a bnary classfcaton, lnear output unts mght be the best choce, but we have not yet studed any problems of ths knd. We add hdden unts to the network one by one. Each new hdden unt receves a connecton from each of the network's orgnal nputs and also from every pre-exstng hdden unt. The hdden unt's nput weghts are frozen at the tme the unt s added to net; only the output connectons are traned repeatedly. Each new unt therefore adds

4 .526 '. Fahlman and Lebere I.. A' ~ a new one-unt "layer" to the network, unless some of ts ncomng weghts happen to be :.zero. Ths leads to the creaton of very powerful hgh-order feature detectors; t also may :~j lead to very deep networks and hgh fan-n to the hdden unts. There are a number of ~ possble strateges for mnmzng the network depth and fan-n as new unts are added, :1 but we have not yet explored these strateges. j The learnng algorthm begns wth no hdden unts. The drect nput-output connectons _ I I.To are traned as well as possble over the entre tranng set. Wth no need to back-propagate through hdden unts, we can use the Wllow-Hoff or "delta" rule, the Perceptron learnng algorthm, or any of the other well-known learnng algorthms for sngle-layer networks. In our smulatons, we use Fahlman's "quckprop" algorthm [Fahlman, 1988] to tran the output weghts. Wth no hdden unts, ths acts essentally lke the delta rule, except that t converges much faster. At some pont. ths tranng wll approach an asymptote. When no sgnfcant error j reducton has occurred after a certan number of tranng cycles (controlled by a "patence" parameter set by the operator), we run the network one last tme over the entre tranng set to measure the error. If we are satsfed wth the network's performance, we stop; f not. we attempt to reduce the resdual errors further by addng a new hdden unt to the network. The unt-creaton algorthm s descrbed below. The new unt s added to the net, ts nput weghts are frozen, and all the output weghts are once agan traned usng J quckprop. Ths cycle repeats untl the error s acceptably small (or untl we gve up). 1 crea~e a new hdden unt, we begn wth a ca~ddate unt that receves tr.m~able.nput : connectons from all of the network's external Inputs and from all pre-existng hidden! unts. The output of ths canddate unt s not yet connected to the actve network. We run! a number of passes over the examples of the tranng set, adjustng the canddate unt's! nput weghts after each pass. The goal of ths adjustment s to maxmze S, the sum over! all output unts 0 of the magntude of the correlaton (or, more precsely, the covarance)., between V, the canddate unt's value, and Eo, the resdual output error observed at UnIt o. We defne S as,. as/8w S=~ ~(Vp- V) (Ep, 0 -~)! 0 p :, where 0 s the network output at whch the error s measured and p s the tranng pattern..the quanttes V and ~ are the values of V and Eo averaged over all patterns.! In order to maxmze S, we must compute as/aw, the partal dervatve of S wth respect to each of the canddate unt's ncomng weghts, W. In a manner very smlar! to the dervaton of the back-propagaton rule n [Rumelhart, 1986], we can expand and.; dfferentate the formula for S to get.:~. = L uo(ep,o -~).I;l,p :t p,o where u 0 s the sgn of the correlaton between the canddate's value and output 0,1; s f.ị I I :1 ;.. ~l ;1

5 ..The Cascade-Correlaton Learnng Archtecture 527 the dervatve for pattern p of the canddate unt's actvaton functon wth respect to the.sum of ts nputs, and l,p s the nput the canddate unt receves from unt for pattern p..can After computng 8 S / 8W for each ncomng connecton, we can perform a gradent ascent to maxmze S. Once agan we are tranng only a sngle layer of weghts. Once agan we use the quckprop update rule for faster convergence. When S stops mprovng, we nstall the new canddate as a unt n the actve network, freeze ts nput weghts, and contnue the cycle as descrbed above. Because of the absolute value n the formula for S, a canddate unt cares only about the magntude of ts correlaton wth the error at a gven output, and not about the sgn of the correlaton. As a rule, f a hdden unt correlates postvely wth the error at a gven unt, t wll develop a negatve connecton weght to that unt, attemptng to cancel some of the error; f the correlaton s negatve, the output weght wll be postve. Snce a unt's weghts to dfferent outputs may be of mxed sgn, a unt can sometmes serve two purposes by developng a postve correlaton wth the error at one output and a negatve correlaton wth the error at another. Instead of a sngle canddate unt, t s possble to use a pool of canddate unts, each wth a dfferent set of random ntal weghts. All receve the same nput sgnals and see the same resdual error for each pattern and each output. Because they do not nteract wth one another or affect the actve network durng tranng, all of these canddate unts be ttaned n parallel; whenever we decde that no further progress s beng made, we nstall the canddate whose correlaton score s the best The use of ths pool of canddates s benefcal n two ways: t greatly reduces the chance that a useless unt wll be permanently nstalled because an ndvdual canddate got stuck durng tranng, and (on a parallel machne) t can speed up the tranng because many parts of weght-space can be explored smultaneously. The hdden and canddate unts may all be of the same type, for example wth a sgmod actvaton functon. Alternatvely, we mght create a pool of canddate unts wth a mxture. of nonlnear actvaton functons-some sgmod, some Gaussan, some wth radal actvaton functons, and so on-and let them compete to be chosen for addton to the actve network. To date, we have explored the all-sgmod and all-gaussan cases, but we do not yet have extensve smulaton data on networks wth mxed unt-types. One fnal note on the mplementaton of ths algorthm: Whle the weghts n the output layer are beng ttaned, the other weghts n the actve network are frozen. Whle the canddate weghts are beng traned, none of the weghts n the actve network are changed. In a machne wth plenty of memory, t s possble to record the unt-values and the output errors for an entre epoch, and then to use these cached values repeatedly durng tranng, rather than reco'mputng them repeatedly for each tranng case. Ths can result n a.tremendous speedup as the actve network grows large.

6 528 Fahlman and Lebere Fgure 2: Tranng ponts for the two-sprals problem, and output pattern for one network ~ I traned wth Cascade-Correlaton. ~. I 2 BENCHMARK RESULTS.! 2.1 THE TWO-SPIRALS PROBLEM! The "two-sprals" benchmark was chosen as the prmary benchmark for ths study because!.! t s an extremely hard problem for algorthms of the back-propagaton famly to solve.! It was frst proposed by Alexs Weland of MImE Corp. The net has two contnuous- I1. valued nputs and a sngle output. The tranng set conssts of 194 X- Y values, half of whch are to produce a + 1 output and half a -1 output. These tranng ponts are arranged I n two nterlockng sprals that go around the orgn three tmes, as shown n Fgure 2a. I The goal s to develop a feed-forward network wth sgmod unts that properly classfes I all 194 tranng cases. Some hdden unts are obvously needed, snce a sngle lnear! separator cannot dvdc two sets twsted together n ths way. Weland (unpublshed) reported that a modfed verson of backprop n use at MITRE requred 150,000 to 200,000 epochs to solve ths problem, and that they had never obtaned a soluton usng standard backprop. Lang and Wtbrock [Lang, 1988] tred the problem usng a network (three hdden layers of fve unts each). Ther network was unusual n that t provded "shortcut" connectons: each unt receved ncomng connectons from every unt n every earler layer, not just from the mmedately precedng layer. Wth ths archtecture, standard backprop was able to solve the problem n 20,000 epochs, backprop wth a modfed error functon requred 12,000 epochs, and quckprop requred Ths was the best two-sprals perfonnance reported to date. Lang and Wtbrock also report obtanng a soluton wth a net (only ten hdden unts n all), but the soluton requred 60,000 quckprop epochs. ; We ran the problem 100 tmes wth the Cascade-Correlaton algorthm usng a sgmodal actvaton functon for both the output and hdden unts and a pool of 8 canddate unts..all trals were successful, requrng 1700 epochs on the average. {Ths number counts ~ :! J ị ~ ~

7 ~-- -- ~ The Cascade-Correlaton Learnng Archtecture 529,,..".w_"""." both the epochs used to tran output weghts and the epochs used to tran canddate unts.). The number of.hdden unts bu!t nt~ the net vared from 12 to 19, wth an average of 15.2 and a medan of 15. Here s a hstogram of the number of hdden unts created: Hdden Number of Unts Trals 12 4 #### 13 9 ######### ######################## ################### ######################## ############# 18 5 ##### 19 2 ## In terms of tranng epochs, Cascade-Correlaton beats quckprop by a factor of 5 and standard backprop by a factor of 10, whle buldng a network of about the same complexty (15 hdden W1ts). In terms of actual computaton on a seral machne, however, the speedup s much greater than these numbers suggest In backprop and quckprop, each tranng case requres a forward and a backward pass through all the connectons n the network; Cascade-Correlaton requres only a forward pass. In addton, many of the. Cascade-Correlaton epochs are run whle the network s much smaller than ts fnal sze. Fnally, the cacheng strategy descrbed above makes t possble to avod fe-computng the W1t values for parts of the network that are not changng. Suppose that nstead of epochs, we measure learnng tme n connecton crossngs, defned as the number of multply-accumulate steps necessary to propagate actvaton values forward through the network and error values backward. Ths measure leaves out some computatonal steps, but t s a more accurate measure of computatonal complexty than comparng epochs of dfferent szes or comparng runtmes on dfferent machnes. The Lang and Wtbrock result of 20,000 backprop epochs requres about 1.1 bllon connecton crossngs. Ther soluton usng 8000 quckprop epochs on the same network requres about 438 mllon crossngs. An average Cascade-Correlaton run wth a pool of 8 canddate unts requres about 19 mllon crossngs-a 23-fold speedup over quckprop and a 50-fold speedup over standard backprop. Wth a smaller pool of canddate unts the speedup (on a seral machne) would be even greater, but the resultng networks mght be somewhat larger. Fgure 2b shows the output of a 12-hdden-W1t network bult by Cascade-Correlaton as the nput s scanned over the X- Y feld. Ths network properly classfes all 194 tranng ponts. We can see that t nterpolates smoothly for about the frst 1.5 turns of the spral, but becomes a bt lumpy farther out, where the tranng ponts are farther apart. Ths "receptve feld" dagram s smlar to that obtaned by Lang and Wtbrock usng.backprop, but s somewhat smoother.

8 :30.Fahlman and Lebere ~ --1 [;,~ -" ".," "".2.2 N.INPUT PARITY.1 I Snce party has been a popular benchmark among other researchers, we ran Cascade- Correlaton on N-nput party problems wth N rangng from 2 to 8. The best results were obtaned wth a sgmod output unt and hdden unts whose output s a Gaussan functon of the sum of weghted nputs. Based on fve trals for each value of N, our results were as follows: N Cases Hdden Average Unts Epochs : ; ; : : For a rough comparson, Tesauro and Janssens [Tesauro, 1988] report that standard backprop takes about 2000 epochs for 8-nput party. In ther study, they used 2N hdden unts. Cascade-Correlaton can solve the problem wth fewer than N hdden unts because t uses short-cut connectons. As a test of generalzaton, we ran a few trals of Cascade-Correlaton on the 10-nput party problem, tranng on ether 50% or 25% of the 1024 patterns and testng on the rest. The number of hdden unts bult vared from 4 to 7 and tranng tme vared from 276 epochs to 551. When traned on half of the patterns, perfonnance on the test set averaged 96% correct; when traned on one quarter of the patterns, test-set perfonnance averaged 90% correct. Note that the nearest neghbor algorthm would get almost all of the test-set cases wrong. 3 DISCUSSION We beleve that that Cascade-Correlaton algorthm offers the followng advantages over network learnng algorthms currently n use:.there s no need to guess the sze, depth, and connectvty pattern of the network n advance. A reasonably small (though not optmal) net s bult automatcally, perhaps wth a mxture of unt-types..cascade-correlaton learns fast. In backprop, the hdden unts engage n a complex dance before thcy settle nto dstnct useful roles; n Cascade-Correlaton, each unt sees a fxed problem and can move decsvely to solve that problem. For the problems we have nvestgated to date, the learnng tme n epochs grows roughly as NlogN, where N s the number of hdden unts ultmately needed to solve the problem.

9 ..v 1.The Cascade-Correlaton Learnng Archtecture 531.Cascade-Correlaton can buld deep nets (hgh-order feature detectors) wthout the.dramatc slowdown we see n deep back-propagaton networks..cascade-correlaton s useful for ncremental learnng, n whch new nfonnaton s added to an already-traned net. Once bult, a feature detector s never cannbalzed. It s avalable from that tme on for producng outputs or more complex features..at any gven tme, we tran only one layer of weghts n the network. The rest of the network s constant, so results can be cached..there s never any need to propagate error sgnals backwards through network connectons. A sngle resdual error sgnal can be broadcast to all canddates. The weghted connectons transmt sgnals n only one drecton, elmnatng one dfference between these networks and bologcal synapses..the canddate unts do not nteract, except to pck a wnner. Each canddate sees the same nputs and error sgnals. Ths lmted communcaton makes the archtecture attractve for parallel mplementaton. 4 RELATION TO OTHER WORK The prncpal dfferences between Cascade-Correlaton and older learnng archtectures are the dynamc creaton of hdden unts, the way we stack the new unts n multple a!~yers (wth a fxed output layer), the freezng of unts as we add them to the net, and ~e way we tran new unts by hll-clmbng to maxmze the unt's correlaton wth the resdual error. The most nterestng dscovery s that by tranng one unt at a tme nstead of tranng the whole network at once, we can speed up the learnng process consderably, whle stll creatng a reasonably small net that generalzes well. A number of researchers [Ash, 1989,Moody, 1989] have nvestgated networks that add new unts or receptve felds wthn a sngle layer n the course of learnng. Whle sngle-layer systems are well-suted for some problems, these systems are ncapable of creatng hgher-order feature detectors that combne the outputs of exstng unts. The dea of buldng feature detectors and then freezng them was nspred n part by the work of Wabel on modular networks [Wabel, 1989], but n hs model the structure of the sub-networks must be fxed before learnng begns. We know of only a few attempts to buld up mult-layer networks as the learnng progresses. Our decson to look at models n whch each unt can see all pre-exstng unts was nspred to some extent by work on progressvely deepenng threshold-logc models by Merrck Furst and Jeff Jackson at Carnege Mellon. (They are not actvely pursung ths lne at present.) Gallant [Gallant, 1986] brefly mentons a progressvely deepenng perceptron model (hs "nverted pyramd" model) n whch unts are frozen after beng nstalled. However, he has concentrated most of hs research effort on models n whch new hdden unts are generated at random rather than by a delberate tranng process..the SONN model of Tenoro and Lee [Tenoro, 1989] bulds a multple-layer topology

10 Is3'!.4 "F'ahlman and Lebere.. I I~ IWjoljjj to sut the problem at hand. Ther algorthm places new two-nput unts at randomly se- J. lected ~ocatons, usng a smulated annealng search to keep only the most useful ones-a \~ very different approach from ours. Acknowledgments ~t We would lke to thank Merrck Furst, Paul Glechauf, and Davd Touretzky for askng :~,~. good questons that helped to shape ths work. Ths research was sponsored n part by ~~, the Natonal Scence Foundaton (Contract EET ) and n part by the Defense Advanced Research Projects Agency (Contract F C-1499). ~, References [Ash, 1989] [Fahlman, 1988] [Gallant, 1986] Ash, T. (1989) "Dynamc Node Creaton n Back-Propagaton Networks", Techncal Report 8901, Insttute for Cogntve Scence, Unversty of Calforna, San Dego. Fahlman, S. E. (1988) '"Faster-Learnng Varatons on Back- Propagaton: An Emprcal Study" n Proceedngs of the 1988 Connectonst Models Summer School, Morgan Kaufmann. Gallant, S. I. (1986) "Three Constructve Algorthms for Network Learnng" n Proceedngs. 8th Annual Conference of the Cogntve Scence Socety..[Lang, 1988] Lang, K. J. and Wtbrock, M. J. (1988) "Learnng to Tell Two Sprals Apart" n Proceedngs of the 1988 Connectonst Models Summer School, Morgan Kaufmann. [Moody, 1989] Moody, J. (1989) "Fast Learnng n Mult-Resoluton Herarches" n D. S. Touretzky (ed.), Advances n Neural Informaton Processng Systems 1, Morgan Kaufmann. [Rumelhart, 1986] Rumelhart, D. E., Hnton, G. E., and Wllams, R. J. (1986) "Learnng Internal Representatons by Error Propagaton" n Rumelhart, D. E. and McClelland, J. L.,Parallel Dstrbuted Processng: Exploratons n the Mcrostructure of Cognton, MIT Press. [Tenoro, 1989] [Tesauro. 1988] [Wabel, 1989].(ed.), Tenoro, M. F., and Lee. W. T. (1989) "Self-Organzng Neural Nets for the Identfcaton Problem" n D. S. Touretzky (ed.), Advances n Neural Informaton Processng Systems 1. Morgan Kaufmann. Tesauro. G. and Janssens, B. (1988) "Scalng Relatons n Back- Propagaton Learnng" n Complex Systems Wabel. A. (1989) "Consonant Recognton by Modular Constructon of Large Phonemc Tme-Delay Neural Networks" n D. S. Touretzky Advances n Neural Informaton Processng Systt ms 1, Morgan Kaufmann.