arxiv: v1 [cs.lg] 22 Jan 2016 Abstract

Transcription

1 Mne Km Department of Computer Scence, Unversty of Illnos at Urbana-Champagn, Urbana, IL USA Pars Smaragds Unversty of Illnos at Urbana-Champagn, Urbana, IL USA Adobe Research, Adobe Systems Inc., San Francsco, CA 94103, USA arxv: v1 [cs.lg] 22 Jan 2016 Abstract Based on the assumpton that there exsts a neural network that effcently represents a set of Boolean functons between all bnary nputs and outputs, we propose a process for developng and deployng neural networks whose weght parameters, bas terms, nput, and ntermedate hdden layer output sgnals, are all bnary-valued, and requre only basc bt logc for the feedforward pass. The proposed Btwse Neural Network BNN) s especally sutable for resourceconstraned envronments, snce t replaces ether floatng or fxed-pont arthmetc wth sgnfcantly more effcent btwse operatons. Hence, the BNN requres for less spatal complexty, less memory bandwdth, and less power consumpton n hardware. In order to desgn such networks, we propose to add a few tranng schemes, such as weght compresson and nosy backpropagaton, whch result n a btwse network that performs almost as well as ts correspondng realvalued network. We test the proposed network on the MNIST dataset, represented usng bnary features, and show that BNNs result n compettve performance whle offerng dramatc computatonal savngs. 1. Introducton Accordng to the unversal approxmaton theorem, a sngle hdden layer wth a fnte number of unts can approxmate a contnuous functon wth some mld assumptons Cybenko, 1989; Hornk, 1991). Whle ths theorem mples a shallow network wth a potentally ntractable number of hdden unts when t comes to modelng a compl- Proceedngs of the 31 st Internatonal Conference on Machne Learnng, Llle, France, JMLR: W&CP volume 37. Copyrght 2015 by the authors). cated functon, Deep Neural Networks DNN) acheve the goal by learnng a herarchy of features n ther multple layers Hnton et al., 2006; Bengo, 2009). Although DNNs are extendng the state of the art results for varous tasks, such as mage classfcaton Goodfellow et al., 2013), speech recognton Hnton et al., 2012), speech enhancement Xu et al., 2014), etc, t s also the case that the relatvely bgger networks wth more parameters than before call for more resources processng power, memory, battery tme, etc), whch are sometmes crtcally constraned n applcatons runnng on embedded devces. Examples of those applcatons span from context-aware computng, collectng and analysng a varety of sensor sgnals on the devce Baldauf et al., 2007), to always-on computer vson applcatons e.g. Google glasses), to speechdrven personal assstant servces, such as Hey, Sr. A prmary concern that hnders those applcatons from beng more successful s that they assume an always-on pattern recognton engne on the devce, whch wll dran the battery fast unless t s carefully mplemented to mnmze the use of resources. Addtonally, even n an envronment wth the necessary resources beng avalable, speedng up a DNN can greatly mprove the user experence when t comes to tasks lke searchng bg databases Salakhutdnov & Hnton, 2009). In ether case, a more compact yet stll well-performng DNN s a welcome mprovement. Effcent computatonal structures for deployng artfcal neural networks have long been studed n the lterature. Most of the effort s focused on tranng networks whose weghts can be transformed nto some quantzed representatons wth a mnmal loss of performance Fesler et al., 1990; Hwang & Sung, 2014). They typcally use the quantzed weghts n the feedforward step at every tranng teraton, so that the traned weghts are robust to the known quantzaton nose caused by a lmted precson. It was also shown that 10 bts and 12 bts are enough to represent gradents and storng weghts for mplementng the stateof-the-art maxout networks even for tranng the network Courbaraux et al., 2014). However, n those quantzed

2 networks one stll needs to employ arthmetc operatons, such as multplcaton and addton, on fxed-pont values. Even though faster than floatng pont, they stll requre relatvely complex logc and can consume a lot of power. Wth the proposed Btwse Neural Networks BNN), we take a more extreme vew that every nput node, output node, and weght, s represented by a sngle bt. For example, a weght matrx between two hdden layers of 1024 unts s a matrx of bnary values rather than quantzed real values ncludng the bas). Although learnng those btwse weghts as a Boolean concept s an NPcomplete problem Ptt & Valant, 1988), the btwse networks have been studed n the lmted settng, such as µ- perceptron networks where an nput node s allowed to be connected to one and only one hdden node and ts fnal layer s a unon of those hdden nodes Golea et al., 1992). A more practcal network was proposed n Soudry et al., 2014) recently, where the posteror probabltes of the bnary weghts were sought usng the Expectaton Back Propagaton EBP) scheme, whch s smlar to backpropagaton n ts form, but has some advantages, such as parameterfree learnng and a straghtforward dscretzaton of the weghts. Its promsng results on bnary text classfcaton tasks however, rely on the real-valued bas terms and averagng of predctons from dfferently sampled parameters. Ths paper presents a completely btwse network where all partcpatng varables are bpolar bnares. Therefore, n ts feedforward only XNOR and bt countng operatons are used nstead of multplcaton, addton, and a nonlnear actvaton on floatng or fxed-pont varables. For tranng, we propose a two-stage approach, whose frst part s typcal network tranng wth a weght compresson technque that helps the real-valued model to easly be converted nto a BNN. To tran the actual BNN, we use those compressed weghts to ntalze the BNN parameters, and do nosy backpropagaton based on the tentatve btwse parameters. To bnarze the nput sgnals, we can adapt any bnarzaton technques, e.g. fxed-pont representatons and hash codes. Regardless of the bnarzaton scheme, each nput node s gven only a sngle bt at a tme, as opposed to a bt packet representng a fxed-pont number. Ths s sgnfcantly dfferent from the networks wth quantzed nputs, where a real-valued sgnal s quantzed nto a set of bts, and then all those bts are fed to an nput node n place of ther correspondng sngle real value. Lastly, we apply the sgn functon as our actvaton functon nstead of a sgmod to make sure the nput to the next layer s bpolar bnary as well. We compare the performance of the proposed BNN wth ts correspondng ordnary real-valued networks on hand-wrtten dgt recognton tasks, and show that the btwse operatons can do the ob wth a very small performance loss, whle provdng a large margn of mprovement n terms of the necessary computatonal resources. 2. Feedforward n Btwse Neural Networks It has long been known that any Boolean functon, whch takes bnary values as nput and produces bnary outputs as well, can be represented as a btwse network wth one hdden layer McCulloch & Ptts, 1943), for example, by merely memorzng all the possble mappngs between nput and output patterns. We defne the forward propagaton procedure as follows based on the assumpton that we have traned such a network wth bpolar bnary parameters: l 1 a l = b l + w, l z l 1, 1) z l = sgn a l ), 2) z l B Kl, W l B Kl K l 1, b l B Kl, 3) where B s the set of bpolar bnares,.e. ±1 1, and stands for the btwse XNOR operaton see Fgure 1 a)). l,, and ndcate a layer, nput and output unts of the layer, respectvely. We use bold characters for a vector or a matrx f capcalzed). K l s the number of nput unts at l-th layer. Therefore, z 0 equals to an nput vector, where we omt the sample ndex for the notatonal convenence. We use the sgn actvaton functon to generate the bpolar outputs. We can check the predcton error E by measurng the btwse agreement of target vector t and the output unts of L-th layer usng XNOR as a multplcaton operator, E = L 1 t z L ) /2, 4) but ths error functon can be tentatvely replaced by nvolvng a softmax layer durng the tranng phase. The XNOR operaton s a faster substtute of bnary multplcaton. Therefore, 1) and 2) can be seen as a specal verson of the ordnary feedforward step that only works when the nputs, weghts, and bas are all bpolar bnares. Note that these bpolar bts wll n practce be mplemented usng 0/1 bnary values, where 2) actvaton s equvalent to countng the number of 1 s and then checkng f the accumulaton s bgger than the half of the number of nput unts plus 1. Wth no loss of generalty, n ths paper we wll use the ±1 bpolar representaton snce t s more flexble n defnng hyperplanes and examnng the network behavor. Sometmes a BNN can solve the same problem as a realvalued network wthout any sze modfcatons, but n general we should expect that a BNN could requre larger network structures than a real-valued one. For example, the XOR problem n Fgure 1 b) can have an nfnte number of solutons wth real-valued parameters once a par 1 In the bpolar bnary representaton, stands for the TRUE status, whle 1 s for FALSE.

3 x 1 x 2 Input Output A B a),1) 1,1) -x 1 +x 2 >0 x 1 -x 2 >0,) 1,) b),1) 1,1) y x 1 -x 2 <0 x 1 +x 2 >0 tons of a real-valued system, such as the power consumpton of multplers and adders for the floatng-pont operatons, varous dynamc ranges of the fxed-pont representatons, erroneous flps of the most sgnfcant bts, etc. Note that f the btwse parameters are sparse, we can further reduce the number of hyperplanes. For example, for an nactve element n the weght matrx W due to the sparsty, we can smply gnore the computaton for t smlarly to the operatons on the sparse representatons. Conceptually, we can say that those nactve weghts serve as zero weghts, so that a BNN can solve the problem n Fgure 1 d) by usng only one hyperplane as n e). From now on, we wll use ths extended verson of BNN wth nactve weghts, yet there are some cases where BNN needs more hyperplanes than a real-valued network even wth the sparsty. c),),1) 1,1) d) 1,) 3. Tranng Btwse Neural Networks We frst tran some compressed network parameters, and then retran them usng nosy backpropagaton for BNNs.,) e) 0x 1 +x 2 +0>0 1,) Fgure 1. a) An XNOR table. b) The XOR problem that needs two hyperplanes. c) A mult-layer perceptron that solves the XOR problem. d) A lnearly separable problem whle btwse networks need two hyperplanes to solve t y = x 2). e) A btwse network wth zero weghts that solves the y = x 2 problem. of hyperplanes can successfully dscrmnate 1, 1) and 1, 1) from 1, 1) and 1, 1). Among all the possble solutons, we can see that bnary weghts and bas are enough to defne the hyperplanes, x 1 x > 0 and x 1 + x > 0 dashes). Lkewse, the separaton performance of the partcular BNN defned n c) has the same classfcaton power once the nputs are bnary as well. Fgure 1 d) shows another example where BNN requres more hyperplanes than a real-valued network. Ths lnearly separable problem s solvable wth only one hyperplane, such as 0.1x 1 + x > 0, but t s mpossble to descrbe such a hyperplane wth bnary coeffcents. We can nstead come up wth a soluton by combnng multple bnary hyperplanes that wll eventually ncrease the perceved complexty of the model. However, even wth a larger number of nodes, the BNN s not necessarly more complex than the smaller real-valued network. Ths s because a parameter or a node of BNN requres only one bt to represent whle a real-valued node generally requres more than that, up to 64 bts. Moreover, the smple XNOR and bt countng operatons of BNN bypass the computatonal complca Real-valued Networks wth Weght Compresson Frst, we tran a real-valued network that takes ether btwse nputs or real-valued nputs ranged between 1 and. A specal part of ths network s that we constran the weghts to have values between 1 and as well by wrappng them wth tanh. Smlarly, f we choose tanh for the actvaton, we can say that the network s a relaxed verson of the correspondng bpolar BNN. Wth ths weght compresson technque, the relaxed forward pass durng tranng s defned as follows: l 1 a l = tanh b l ) + tanh w,) z l l 1, 5) z l = tanh a l ), 6) where all the bnary values n 1) and 2) are real for the tme beng: Wl R Kl K l 1, b l R Kl, and z l R Kl. The bars on top of the notatons are for the dstncton. Weght compresson needs some changes n the backpropagaton procedure. In a hdden layer we calculate the error, δ l n) = l tanh w l, )δl ) n) 1 tanh 2 a l ) ). Note that the errors fron the next layer are multpled wth the compressed versons of the weghts. Hence, the gradents of the parameters n the case of batch learnng are w, l = n n δ l n) z l 1 ) 1 tanh 2 w l,) ), ) b l = δn) l 1 tanh 2 bl ) ),

4 wth the addtonal term from the chan rule on the compressed weghts. Table 1. Classfcaton errors for real-valued and btwse networks on dfferent types of btwse features 3.2. Tranng BNN wth Nosy Backpropagaton Snce we have traned a real-valued network wth a proper range of weghts, what we do next s to tran the actual btwse network. The tranng procedure s smlar to the ones wth quantzed weghts Fesler et al., 1990; Hwang & Sung, 2014), except that the values we deal wth are all bts, and the operatons on them are btwse. To ths end, we frst ntalze all the real-valued parameters, W and b, wth the ones learned from the prevous secton. Then, we setup a sparsty parameter λ whch says the proporton of the zeros after the bnarzaton. Then, we dvde the parameters nto three groups:, 0, or 1. Therefore, λ decdes the boundares β, e.g. w l = 1 f wl < β. Note that the number of zero weghts w l < β equals to λkl K l 1. The man dea of ths second tranng phase s to feedforward usng the bnarzed weghts and the bt operatons as n 1) and 2). Then, durng nosy backpropagaton the errors and gradents are calculated usng those bnarzed weghts and sgnals as well: δ l n) = l w l, = n w l, δl n), δ l n)z l 1, b l = n δ l n). 7) In ths way, the gradents and errors properly take the bnarzaton of the weghts and the sgnals nto account. Snce the gradents can get too small to update the bnary parameters W and b, we nstead update ther correspondng realvalued parameters, w l, w l, η w l,, bl, b l η b l, 8) wth η as a learnng rate parameter. Fnally, at the end of each update we bnarze them agan wth β. We repeat ths procedure at every epoch. 4. Experments In ths secton we go over the detals and results of the hand-wrtten dgt recognton task on the MNIST data set LeCun et al., 1998) usng the proposed BNN system. Throughout the tranng, we adopt the softmax output layer for these multclass classfcaton cases. All the networks have three hdden layers wth 1024 unts per layer. From the frst round of tranng, we get a regular dropout network wth the same settng suggested n Srvastava et al., 2014), except the fact that we used the hyperbolc tangent for both weght compresson and actvaton to make NETWORKS BIPOLAR 0 OR 1 FLOATING-POINT NETWORKS 64BITS) FIXED-POINT 2BITS) 1.17% 1.32% 1.36% BNN 1.33% 1.36% 1.47% the network sutable for ntalzng the followng bpolar btwse network. The number of teratons from 500 to 1, 000 was enough to buld a baselne. The frst row of Table 1 shows the performance of the baselne real-valued network wth 64bts floatng-pont. As for the nput to the real-valued networks, we rescale the pxel ntenstes nto the bpolar range,.e. from 1 to, for the bpolar case the frst column). In the second column, we use the orgnal nput between 0 and 1 as t s. For the thrd column, we encode the four equally spaced regons between 0 to 1 nto two bts, and feed each bt nto each nput node. Hence, the baselne network for the thrd nput type has 1, 568 bnary nput nodes rather than 784 as n the other cases. Once we learn the real-valued parameters, now we tran the BNN, but wth bnarzed nputs. For nstance, nstead of real values between 1 and n the bpolar case, we take ther sgn as the bpolar bnary features. As for the 0/1 bnares, we smply round the pxel ntensty. Fxed-pont nputs are already bnarzed. Now we tran the new BNN wth the nosy backpropagaton technque as descrbed n 3.2. The second row of Table 1 shows the BNN results. We see that the btwse networks perform well wth very small addtonal errors. Note that the performance of the orgnal real-valued dropout network wth smlar network topology logstc unts wthout max-norm constrant) s 1.35%. 5. Concluson In ths work we propose a btwse verson of artfcal neural networks, where all the nputs, weghts, bases, hdden unts, and outputs can be represented wth sngle bts and operated on usng smple btwse logc. Such a network s very computatonally effcent and can be valuable for resource-constraned stuatons, partcularly n cases where floatng-pont / fxed-pont varables and operatons are prohbtvely expensve. In the future we plan to nvestgate a btwse verson of convolutve neural networks, where effcent computng s more desrable.

5 References Baldauf, M., Dustdar, S., and Rosenberg, F. A survey on context-aware systems. Internatonal Journal of Ad Hoc and Ubqutous Computng, 24): , January Bengo, Y. Learnng deep archtectures for AI. Foundatons and Trends n Machne Learnng, 21):1 127, Courbaraux, M., Bengo, Y., and Davd, J.-P. Low precson arthmetc for deep learnng. arxv preprnt arxv: , Cybenko, G. Approxmatons by superpostons of sgmodal functons. Mathematcs of Control, Sgnals, and Systems, 24): , Fesler, E., Choudry, A., and Caulfeld, H. J. Weght dscretzaton paradgm for optcal neural networks. In The Hague 90, 126 Aprl, pp Internatonal Socety for Optcs and Photoncs, Golea, M., Marchand, M., and Hancock, T. R. On learnng µ-perceptron networks wth bnary weghts. In Advances n Neural Informaton Processng Systems NIPS), pp , McCulloch, W. S. and Ptts, W. H. A logcal calculus of the deas mmanent n nervous actvty. The Bulletn of Mathematcal Bophyscs, 54): , Ptt, L. and Valant, L. G. Computatonal lmtatons on learnng from examples. Journal of the Assocaton for Computng Machnery, 35: , Salakhutdnov, R. and Hnton, G. Semantc hashng. Internatonal Journal of Approxmate Reasonng, 507): , Soudry, D., Hubara, I., and Mer, R. Expectaton backpropagaton: Parameter-free tranng of multlayer neural networks wth contnuous or dscrete weghts. In Advances n Neural Informaton Processng Systems NIPS), Srvastava, N., Hnton, G., Krzhevsky, A., Sutskever, I., and Salakhutdnov, R. Dropout: A smple way to prevent neural networks from overfttng. Journal of Machne Learnng Research, 151): , January Xu, Y., Du, J., Da, L.-R., and Lee, C.-H. An expermental study on speech enhancement based on deep neural networks. IEEE Sgnal Processng Letters, 211):65 68, Goodfellow, I. J., Warde-Farley, D., Mrza, M., Courvlle, A., and Bengo, Y. Maxout networks. In Proceedngs of the Internatonal Conference on Machne Learnng ICML), Hnton, G., Deng, L., Yu, D., Dahl, G. E., M, Abdelrahman, Jatly, N., Senor, A., Vanhoucke, V., Nguyen, P., Sanath, T., and Kngsbury, B. Deep neural networks for acoustc modelng n speech recognton: The shared vews of four research groups. IEEE Sgnal Processng Magazne, 296):82 97, Hnton, G. E., Osndero, S., and Teh, Y. A fast learnng algorthm for deep belef nets. Neural Computaton, 18 7): , Hornk, K. Approxmaton capabltes of multlayer feedforward networks. Neural Networks, 42): , Hwang, K. and Sung, W. Fxed-pont feedforward deep neural network desgn usng weghts, 0, and. In 2014 IEEE Workshop on Sgnal Processng Systems SPS), Oct LeCun, Y., Bottou, L., Bengo, Y., and Haffner, P. Gradentbased learnng appled to document recognton. Proceedngs of the IEEE, 8611): , November 1998.