Connectivity Learning in Multi-Branch Networks

Transcription

1 onnectivity Learnin in Multi-Branch Networs Karim Ahmed Department omputer Science Dartmouth ollee Lorenzo Torresani Department omputer Science Dartmouth ollee Abstract Recent studies in desin convolutional networs have shown that branchin, i.e., splittin computation alon parallel but distinct threads and n areatin ir outputs, represents a new promisin dimension for sinificant in performance. To combat complexity desin choices in architectures, prior wor has adopted simple strateies, such as a fixed branchin factor, same input bein fed to all parallel branches, and an combination outputs produced by all branches at areation points. In this wor we remove se predefined choices and propose an alorithm to learn connections between branches in networ. Instead bein chosen a priori by human desiner, connectivity is learned simultaneously with weihts networ by optimizin a sinle loss function defined with respect to end tas. We demonstrate our approach on problem multi-class imae classification where it yields consistently hiher accuracy compared to state---art ResNet networ iven same learnin Introduction While deep learnin has recently enabled dramatic performance in many application domains, desin deep architectures is still a challenin and time-consumin endeavor. The difficulty lies in many architecture choices that impact ten sinificantly performance system. In specific domain imae cateorization, which is focus this paper, several authors have proposed to simplify architecture desin by definin convolutional neural networs (NNs) in terms combinations basic buildin blocs. This idea modularized desin was adopted in residual networs (ResNets) [3]. While in ResNets residual blocs are staced one on top each or to form very deep networs, recently introduced ResNet models [5] have shown that it is also beneficial to arrane se buildin blocs in parallel to build convolutional networs. The modular component ResNet n consists parallel branches, correspondin to residual blocs with identical topoloy but distinct parameters. Networ built by stacin se components have been shown to lead to better results than sinle-thread ResNets same While principle modularized desin has reatly simplified challene buildin effective architectures for imae analysis, choice how to combine and areate computations se buildin blocs still rests on shoulders human desiner. In order to avoid a combinatorial explosion options, prior wor has relied on simple, uniform rules areation and composition. For example, ResNet models [5] are based on followin set simplifyin assumptions: branchin factor (also referred to as cardinality) is fixed to same constant in all layers networ, all branches a module are fed same input, and outputs parallel branches are areated by a simple operation that provides input to next module. In this paper we remove se predefined choices and propose an alorithm that learns to combine and areate buildin blocs a neural networ. In this new reime, networ connectivity naturally arises as a result trainin optimization rar than bein hand-defined by human desiner. 3st onference on Neural Information Processin Systems (NIPS, Lon Beach, A, USA.

2 5 6 9 architectures, prior wor has adopted simple strateies, such as a branchin, i.e., splittin computation alon parallel but distinct threads and F(x) x5 areatin ir outputs, represents an new promisin dimension for sinificant areatin ir outputs, represents a new promisin dimension for sinificant fixed branchin factor, samechoices input bein fed to all parallel branches, and an 6 in performance. To combat complexity desin in F(x; in )performance. To combat complexity desin choices in 9 combination outputs produced by all branches at areation points. architectures, prior wor prior wor has adopted simple strateies, such ashas a adopted simple strateies, such as a F(x;architectures, ) branches, same input fed to parallelse branches, and an Inbein this weallremove predefined choices and propose an alorithm to learn fixed branchin factor, samefixed input branchin bein fed tactor, all parallel and an wor F(x; 2 ) 9 outputs combination outputs produced by all branches at areation points. combination produced by all branches at areation points. connections between branches in networ. Instead bein chosen a priori In this wor we remove se predefined andremove proposese an alorithm toby learn In this choices wor we predefined choices and propose an alorithm to learn connectivity is learned simultaneously human desiner, connections between branches in connections networ. Instead bein chosen priori between branches ina networ. Instead chosen priori with weihts bein by(iaoptimizin a sinle loss function defined with (i networ y = x is learned F(x;simultaneously ) ( by human desiner, connectivity by human desiner, connectivity learned simultaneously respect to(i endistas. demonstrate our approach on problem multi-class (iwe =function defined with with weihts networ by optimizin a sinle loss with weihts networ by optimizin losswhere defined 6,function, 6, a,sinle 6,, with over 3% with respect imae classification it yields absolute respect to end tas. We demonstrate our approach on problem multi-class respect to end tas. We demonstrate on,problem multi-class 33,, 33,, 33, toour approach state---art ResNet architecture iven same learnin imae classification where it yields over 3%, withrespect 6,, absolute 6,, 6, over classification where it yields absolute 3% with respect 6 to state---art ResNet imae architecture iven (i 6 (i,,,, same learnin,, (i 6 (i same learnin to state---art ResNet architecture iven (i, 33,, 33,, 33, (i F(x) x F(x; ) F(x; ) F(x; 2 ),, 6 F (x) x,, 6 F (x; ) F (x; ) F (x; 2 ) (b) y =x F(x; ) (i 6,, (i (i, 33, F(x) x (i,, 6 (i F(x; ) (i, 6, F(x; ),33, (i ( = (i (i (i F (x;(i ) (i (i (i 6,, (a) = (b) (i (c) (i Do not distribute. Submitted to 3st onference on Neural Information Processin Systems (NIPS. y =x, 33, (i (i (i 6,,,, 6 (i (i (i 6,, F(x; 2 ) (i y =x = Fiure : (a) The RexNet module consistin parallel residual blocs [5]. (b) Our approach replaces fixed areation points RexNet with learnable ates definin input connections for each individual residual bloc in each module i. F(x; ) (Fiure ( 2: onnectivity learned by our method on IFAR-0 for fan-in K = (left) and K = (riht). Each reen square is a residual bloc, each row = square is a module. The net consists a stac M = 9 modules. Arrows indicate pathways connectin residual blocs adacent modules. The squares without in/out edes are deemed superfluous and can be pruned at end learnin. This is achieved by means ates, i.e., learned binary parameters that act as switches determinin final connectivity in our networ. The ates are learned toer with convolutional weihts networ, as part a oint optimization via bacpropaation with respect to a traditional multi-class obective. We demonstrate that, iven same budet residual blocs Submitted to 3st classification onference on Neural Information Processin Systems (NIPS. Do not distribute. Submitted to 3st onference on Neural Information Processin Systems (NIPS. Do not distribute. (and parameters), our learned architecture consistently outperforms predefined ResNet networ in all our experiments. An interestin byproduct our approach is that it can automatically identify residual blocs that are superfluous, i.e., unnecessary or detrimental for end obective. At end optimization, se unused residual blocs can be pruned away without any impact on learned hyposis while yieldin substantial savins in number parameters to store and in test-time computation. Submitted to 3st onference on Neural Information Processin Systems (NIPS. Do not distribute Technical Approach Modular architecture The architecture ResNet. We bein by providin a brief review ResNet [5] architecture, which consists a stac modules. Each module contains residual blocs [3] that implement parallel multiple threads computation feedin from same input. The outputs parallel residual blocs are n summed up toer with oriinal input and passed on to next module. The resultin module is illustrated in Fiure (a). More formally, let F(x; θ ) be transformation implemented by -th residual bloc in module i-th networ, where =,..., and i =,..., M, with M denotin total number modules staced on top each or to form complete networ. The hyperparameter is called cardinality module and defines number parallel branches within each module. The hyperparameter M controls total depth networ: under assumption 3 layers per residual bloc (as shown in fiure), total depth networ is iven by D = 2 3M (an initial convolutional layer and an output fully-connected layers add 2 layers). Note that in ResNet all residual blocs in a module have same topoloy (F) but each bloc has its own parameters (θ denotes parameters P residualbloc in module i). Then, output i-th module is computed as y = x = F(x; θ ). Tensor y represents input to (i -th module. In [5] it was experimentally shown that iven a fixed budet parameters, ResNet networs consistently outperform sinle-branch ResNets same learnin We note, however, that in an attempt to ease networ desin, several restrictive limitations were embedded in architecture ResNet modules: each ResNet module implements parallel feature extractors that operate on same input; furrmore, number active branches is constant at all depth levels networ. Our approach removes se restrictions without addin any sinificant burden on process manual networ desin. Our ated architecture. As in ResNet, our proposed architecture consists a stac M modules, each containin parallel feature extractors. However, differently from ResNet, each branch in a module can tae a different input. The input pathway each branch is controlled by a binary ate vector that is learned ointly with weihts networ. Let = [,,,2,...,, ]> {0, } be binary ate vector definin active input connections feedin -th residual bloc in module i. If, =, n activation volume produced by -th branch in module (i is fed as input to -th residual bloc module i. If, = 0, n output from -th branch in previous module is inored by -th residual bloc current module. Thus, if we denote with y(i output tensor computed by -th branch in module (i, input x to -th residual bloc in module i will be iven by: 2

3 x = =, y(i ( Then, output this bloc will be obtained throuh usual residual computation, i.e., y = x F(x ; θ ). We note that under this model we no loner have fixed areation nodes summin up all outputs from a module. Instead, ate now determines selectively for each bloc which branches from previous module will be areated and provided as input to bloc. Under this scheme, parallel branches in a module receive different inputs. Dependin on constraints posed over imposin that, branch (since each,, different interestin models can be realized. By = for all blocs, n each residual bloc will receive input from only one must be eir 0 or. At or end spectrum, if we set, = for all blocs, in each module i, n all connections would be active and we would obtain aain fixed ResNet architecture. In our experiments we found that best results are achieved for a middle round between se two extremes, i.e., by connectin each bloc to K branches where K is an inteer-valued hyperparameter such that < K <. We refer to this hyperparameter as fan-in a bloc. Finally, we note that it may be possible for a residual bloc in networ to become unused. This happens when bloc in module (i is such that = 0 for all =,...,. In this case, at end optimization we prune bloc without affectin function computed by networ, so as to reduce number parameters to store and to speed up inference. This implies that a variable branchin factor is learned adaptively for different depths in networ. 2.2 GATEONNET: learnin to connect branches We refer to our learnin alorithm as GATEONNET. It ointly optimizes a iven loss l with respect to both weihts networ (θ) and ates (). To learn binary parameters, we adopt a modified version bacpropaation, inspired by alorithm proposed by ourbariaux et al. [] to train neural networs with binary weihts. Durin trainin we store and update a real-valued version [0, ] branch ates, with entries clipped to lie in continuous interval from 0 to. At each iteration, we stochastically binarize real-valued branch ates into binary-valued vectors {0, } subect to constraint that it contains only K active entries, where K is a predefined inteer hyperparameter with K. In or words:, {0, }, =, = K {,..., } and i {,..., M}. Forward Propaation. In forward propaation, our alorithm first normalizes Alorithm GATEONNET trainin alorithm. real-valued ates for each bloc Input: a minibatch labeled examples (x i, y i ), : cardinality to sum up to (i.e., =, = so (number branches), K: fan-in (number active branch that Mult(,,,2,...,, ) defines a proper connections), η: learnin rate, l: loss over minibatch, multinomial distribution. Then, binary ate [0, ] : real-valued branch ates for bloc in module i from previous trainin iteration. is stochastically enerated by drawin K Output: updated distinct samples a, a 2,..., a K {,..., }. Forward Propaation: from multinomial distribution over Normalize real-valued ate to sum up to : branch connections. Finally, entries correspondin to K samples are activated in binary ate vector. The input activation volume to residual bloc is n computed accordin to Eq.. Bacward Propaation. In bacward propaation step, radient / y (i with respect to each branch output is obtained via bac-propaation from / x and binary ates., Gate Update. In update step, our alorithm computes radient with respect to binary branch ates. Then, it updates real-valued branch ates via radient descent. At this time we clip updated real-valued branch ates to constrain m to remain within valid interval [0, ].,,, for =,..., =, Reset binary ate: 0 Draw K distinct samples from multinomial ate distribution: a, a 2,..., a K Mult(,,,2,...,, ) Set active binary ate based on drawn samples:,a for =,..., K ompute output x ate, iven branch activations y (i : x =, y(i 2. Bacward Propaation: ompute x from y ompute y (i from 3. Parameter Update: ompute iven, x x, clip(, η ), and, and y (i 3

4 Architecture onnectivity Params Accuracy (%) D,w, Train Test Top- {29,,} Fixed-Full, K= [5] 0.6M 0.6M 3.52 Learned, K= 0.6M 0.65M 3.9 Learned, K= 0.6M 0.M 5.2 Fixed-Random, K= 0.6M 0.5M 2.5 Fiure 3: Varyin fan-in (K) our model, i.e., number active branches provided as input to each residual bloc. The plot reports accuracy achieved on IFAR- 0 usin a networ stac M =6ResNet modules havin cardinality =and bottlenec width w =. All models have same number parameters (0.2M). The best accuracy is obtained for K =. {29,6,} Fixed-Full, K= [5] 3.M 3.M 2.23 Learned, K= 3.M.5M 2.3 Learned, K= 3.M 32.M.05 Fixed-Random, K= 3.M 3.3M.96 Table : IFAR-0 accuracies achieved by different architectures usin: ( predefined full connectivity ResNet (Fixed-Full), (2) connectivity learned by our alorithm (Learned) and (3) fixed connectivity (Fixed-Random) defined by settin K = random active connections per branch. After oint trainin over θ and, we fine-tune weihts θ with fixed binary ates, by choosin as active connections for each bloc in module i those correspondin to top K values in. Pseudocode for our trainin procedure is iven in Alorithm. 3 Experiments We tested our approach on two imae cateorization datasets: IFAR-0 [] and ImaeNet [2]. Effect fan-in (K). Fiure 3 shows effect fan-in (K) on performance. The networ in this study has M = 6 residual modules, each havin cardinality = (number branches in each module). We trained and tested this architecture on IFAR-0 usin different fan-in values: K =,..,. Note that varyin K does not affect number parameters. We can see that best accuracy is achieved by connectin each residual bloc to K = branches out total = in each module. Usin a very low or very hih fan-in yields lower accuracy. Note that when settin K =, each ate is simply replaced by an element-wise addition outputs from all branches. This renders model equivalent to ResNet [5], which has fixed connectivity. Varyin architectures. In Table we show classification accuracy achieved on IFAR-0 with different architectures. For each architecture we report results obtained usin GATEONNET with fan-in K = and K =. We also include accuracy achieved with full (as opposed to learned) connectivity, which corresponds to ResNet. These results show that learnin connectivity produces consistently hiher accuracy than usin fixed connectivity, with accuracy ains up 2.2% compared to state---art ResNet model. For each architecture we also report results usin sparse random connectivity (Fixed-Random). For se models, each ate is set to have K = randomly-chosen active connections, and connectivity is ept fixed durin learnin parameters. We can see that accuracy se nets is a lot lower compared to our models, despite havin same connectivity density (K = ). This shows that our approach over ResNet are not due to sparser connectivity but y are rar due to learned connectivity. Parameter savins. Our proposed approach provides benefit automatically identifyin durin trainin residual blocs that are unnecessary. At end trainin, unused residual blocs can be pruned away. This yields savins in number parameters to store and in test-time computation. In Table, columns Train and Test under Params show oriinal number parameters (used durin trainin) and number parameters after prunin (used at test-time). Note that for biest architecture, our approach usin K = yields a parameter savin 0% compared to ResNet with full connectivity (.5M vs 3.M), while achievin same accuracy. Thus, in summary, usin fan-in K = ives models that have same number parameters as ResNet but y yield hiher accuracy; usin fan-in K = ives a sinificant savin in number parameters and accuracy on par with ResNet. Fiure 2 provides an illustration connectivity learned by GATEONNET for model {D = 29, w =, = } usin K = (left) and K = (riht). While ResNet feeds same input to all blocs a module, our alorithm learns different input pathways for each bloc and yields a branchin factor that varies alon depth. Lare-scale evaluation on ImaeNet. Finally, we evaluate our approach on lare-scale ImaeNet dataset [2]. We train our models on trainin set (.2M imaes) and evaluate m on validation set (50K imaes). For se experiments we set K = /2. We tried two architectures on this dataset. The first architecture ({D = 50, w =, = 32}) achieves a top- accuracy.% when usin fixed connectivity (ResNet), and 9.% when usin our learned connectivity. Similarly, for second architecture ({D =, w =, = 32}) accuracy is.% with fixed connectivity, while learnin connectivity with our alorithm yields an accuracy 9.3%.

5 References [] Matthieu ourbariaux, Yoshua Benio, and Jean-Pierre David. Binaryconnect: Trainin deep neural networs with binary weihts durin propaations. In Advances in Neural Information Processin Systems 2, Montreal, Quebec, anada, paes 33 3,. [2] Jia Den, Wei Don, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imaenet: A lare-scale hierarchical imae database. In 09 IEEE omputer Society onference on omputer Vision and Pattern Reconition (VPR 09), -25 June 09, Miami, Florida, USA, paes 2 255, 09. [3] Kaimin He, ianyu Zhan, Shaoqin Ren, and Jian Sun. Deep residual learnin for imae reconition. In omputer Vision and Pattern Reconition (VPR), IEEE onference on,. [] Alex Krizhesvsy. Learnin multiple layers features from tiny imaes, 09. Technical Report [5] Sainin ie, Ross B. Girshic, Piotr Dollár, Zhuowen Tu, and Kaimin He. Areated residual transformations for deep neural networs. In IEEE onference on omputer Vision and Pattern Reconition, VPR,. 5