Towards Acceleration of Deep Convolutional Neural Networks using Stochastic Computing

Transcription

1 Towards Acceleratio of Deep Covolutioal Neural Networks usig Stochastic Computig Ji Li, Ao Re, Zhe Li, Caiwe Dig, Bo Yua 3, Qiru Qiu ad Yazhi Wag Departmet of Electrical Egieerig, Uiversity of Souther Califoria, Los Ageles, CA, USA College of Egieerig ad Computer Sciece, Syracuse Uiversity, Syracuse, NY, USA 3 Departmet of Electrical Egieerig, City Uiversity of New York, NY, USA jli74@usc.edu, {are,zli89,cadig@syr.edu, byua@ccy.cuy.edu, {qiqiu,ywag393@syr.edu Abstract I recet years, Deep Covolutioal Neural Network (DCNN) has become the domiat approach for almost all recogitio ad detectio tasks ad outperformed humas o certai tasks. Nevertheless, the high power cosumptios ad complex topologies have hidered the widespread deploymet of DCNNs, particularly i wearable devices ad embedded systems with limited area ad power budget. This paper presets a fully parallel ad scalable hardware-based DCNN desig usig Stochastic Computig (SC), which leverages the eergy-accuracy trade-off through optimizig SC compoets i differet s. We first coduct a detailed ivestigatio of the Approximate Parallel Couter (APC) based euro ad multiplexer-based euro usig SC, ad aalyze the impacts of various desig parameters, such as bit stream legth ad iput umber, o the eergy/power/area/accuracy of the euro cell. The, from a architecture perspective, the ifluece of iaccuracy of euros i differet s o the overall DCNN accuracy (i.e., software accuracy of the etire DCNN) is studied. Accordigly, a structure optimizatio method is proposed for a geeral DCNN architecture, i which euros i differet s are implemeted with optimized SC compoets, so as to reduce the area, power, ad eergy of the DCNN while maitaiig the overall etwork performace i terms of accuracy. Experimetal results show that the proposed approach ca fid a satisfactory DCNN cofiguratio, which achieves 55, 5, ad improvemet i terms of area, power ad eergy, respectively, while the error is icreased by.86%, compared with the covetioal biary ASIC implemetatio. I. INTRODUCTION Machie learig techology is icreasigly preset i the Iteret ad cosumer products, ad it powers may fudametal applicatios such as speech-to-text trascriptio, selectio of relevat search results, atural laguage processig, ad objects idetificatio i images or videos [], []. Covetioal machie learig techiques are limited i its ability to process data i their raw form (e.g., the pixel values of a image) []. As a result, cosiderable huma egieerig efforts ad domai expertise are required to trasform raw data ito suitable iteral represetatios that ca be uderstood ad processed by the learig system []. With the fast growig amout of data ad rage of applicatios to machie learig methods, the ability to automatically extract powerful features is becomig icreasigly importat []. May represetatio learig methods have bee proposed to automatically lear ad orgaize the discrimiative iformatio from raw data []. Deep learig is oe of the most promisig represetatio learig methods, which eables a system to extract represetatios automatically at multiple levels of abstractio ad lear complex fuctios directly from data with very little egieerig by had [], []. With the self-learig ability to cofigure its itricate structure, a deep learig architecture ca easily take advatage of icreases i the amout of available computatio ad data []. Recetly, Deep Covolutioal Neural Network (DCNN), which is oe of most widely used types of deep eural etworks, has achieved tremedous success i may machie learig applicatios, such as speech recogitio [3], image classificatio [4], ad video classificatio [5]. DCNN is ow the domiat approach for almost all recogitio ad detectio tasks ad approach huma performace o some tasks []. Nevertheless, compared with other machie learig techiques, DCNNs require more computatios due to the deep architecture. Furthermore, the idustrial ad academic demads for better quality of results also ted to icrease the depth ad/or width of DCNNs [6], leadig to complicated topologies ad icreased computatio resources required for implemetatio. Therefore, a practical implemetatio of large-scale DCNNs is to use high performace server clusters with accelerators such as GPUs ad FPGAs [7], [8]. A otable tred is that with the astoishig advaces o wearable devices ad Iteret-of-Thigs (IoT), machie learig has also bee rapidly adopted i the widespread mobile ad embedded systems. I order to brig the success of DCNNs to these resource costraied systems, desigers must overcome the challeges of implemetig resource-hugry DCNNs i embedded systems with limited area ad power budget. Stochastic Computig (SC), which is the paradigm of logical computatio o stochastic bit streams [9], has the potetial to eable fully parallel ad scalable hardware-based DCNNs. Sice SC provides several key advatages compared to covetioal biary arithmetic, icludig low hardware area cost ad tolerace to soft errors [9 ], cosiderable research efforts have bee ivested i the cotext of desigig eural etworks usig SC i recet years [ 6]. Nevertheless, there lacks a comprehesive ivestigatio of eergy-accuracy trade-off for DCNN desigs usig differet SC compoets. I this paper, two hardware-based euro structures usig SC are itroduced, i.e., Accumulative Parallel Couter (APC) based euro ad multiplexer (MU) based euro. We further ivestigate the trade-off amog area, power, eergy ad (euro cell) accuracy for these euro structures usig differet iput sizes ad stochastic bit stream legths. The, from a architecture perspective, the ifluece of iaccuracy of euros i differet s o the overall DCNN accuracy is studied. Based o the results, a structure optimizatio method is proposed for a geeral DCNN architecture, i which euros i differet s are implemeted with the optimized SC compoets such that the overall DCNN area, power, ad eergy cosumptio are miimized while the DCNN accuracy

2 is preserved. The cotributios of this work are threefold. First, we itroduce SC ito the DCNNs, i order to make the footprits of DCNNs small eough for successful implemetatios i today s wearable devices ad embedded systems. Secod, we carry out a detailed aalysis o the eergy-accuracy trade-off for differet SC-based euro desigs. Third, based o the aalysis of the results, we propose a structure optimizatio method for a geeral DCNN architecture usig SC, which joitly optimizes the area, power, eergy, ad accuracy for the etire DCNN. Experimetal results o a LeNet 5 DCNN architecture demostrate that compared with the covetioal 8-bit biary implemetatio, the preseted hardware-based DCNN usig SC achieves 55, 5, ad improvemet i terms of area, power ad eergy, respectively, while the error is icreased by.86%. II. R ELATED W ORK DCNNs have bee recogized as oe of the most effective patter recogitio techiques. I order to improve the area, power ad eergy performace, may hardware-based eural etworks have come ito existece. The authors i [7] proposed a FPGA-based accelerator to leverage the sources of parallelism. A efficiet 3D euro topology was developed i [8], which improved the utilizatio of FPGA resources for differet covolutioal shapes. I additio to acceleratig techiques, SC becomes a very attractive cadidate for implemetig hardware-based eural etworks, sice SC buildig blocks ca greatly reduce the hardware footprits, compared to covetioal biary arithmetic compoets [7]. The authors i [4] applied SC to a radial basis fuctio artificial eural etwork ad sigificatly reduced the required hardware. Referece work [] preseted a euro cell desig usig SC compoets, where the progressive precisio characteristics of SC was exploited. Recofigurable SC based euros were developed i [5]. I additio, the authors i [6] explored the hardware-orieted poolig i DCNNs usig SC. The above-metioed works have proposed several eural etwork desigs usig SC, i order to satisfy the resource costraits i embedded systems whilig meetig the specific fuctios ad performace eeds of ed users. However, there lacks a detailed ivestigatio of the eergy-accuracy trade-offs for DCNNs usig differet SC compoets. Moreover, withi a DCNN architecture, euros i differet s have various coectio patters ad exhibit differet degrees of ifluece o the overall system performace, which idicates a structure optimizatio ca be applied to achieve further improvemet. III. OVERVIEW OF THE P ROPOSED DCNN AND S TOCHASTIC C OMPUTING A. DCNN Architecture I this paper, we cosider a geeral DCNN architecture, which cosists of a stack of covolutioal s, poolig s, ad fully coected s. By arragig the topology of above s, powerful architectures (e.g., LeNet [8]) ca be built for specific applicatios. Without the loss of geerality, we coduct the ivestigatio o the LeNet-5 architecture usig SC, which is comprised of two pairs of covolutioal ad poolig s, oe fully coected, ad oe output, as show i Figure. Note that the proposed methodology ca accommodate other DCNN architectures as well. A covolutioal is associated with a set of learable filters (or kerels), which are activated whe specific types of features are foud at some spatial positios i the iputs. After obtaiig features usig covolutio, a subsamplig step ca be applied to aggregate statistics of these features to reduce the dimesios of data ad mitigate over-fittig issues. This subsamplig operatio is realized by a poolig i hardware-based DCNNs, where differet o-liear fuctios ca be applied, such as max poolig, average poolig ad L-orm poolig. The activatio fuctios i euros are o-liear trasformatio fuctios, such as Rectified Liear Uits (ReLU) f (x) = max(0, x), hyperbolic taget (tah) f (x) = tah(x) or f (x) = tah(x), ad sigmoid fuctio f (x) = +e x. I this paper, we adopt the tah activatio fuctio sice it ca be implemeted efficietly as a fiite state machie (FSM) i SC usig a stochastic approximatio method. Fully coected is a ormal eural etwork with its iputs fully coected with its previous. The loss fuctio of DCNN that specifies how the etwork traiig pealizes the deviatio betwee the predicted ad true labels, ad typical loss fuctios are softmax loss, sigmoid crossetropy loss or Euclidea loss. B. Stochastic Computig I SC, the value of a (uipolar) stochastic umber is represeted by the probability of s i a radom bit stream, e.g., the value of a 4-bit sequece = 000 is x = P ( = ) = 4 = 0.5. Obviously, the represetatio of a stochastic umber is ot uique, e.g., to represet the value 0.5 usig a 4-bit stream, there are four differet ways: 000, 000, 000, ad 000. Besides, a m-bit sequece ca oly represet 0 m umbers i the set { m, m, m,, m, idicatig that oly a small subset of the real umbers i the iterval [0, ] ca be expressed exactly i SC. Clearly, the precisio ad accuracy of SC is depedet o the legth of the stream. The two most popular represetatios for stochastic umbers are uipolar ad bipolar formats, which iterpret values i the itervals [0, ] ad [, ], respectively. Uipolar codig is commoly used i usiged arithmetic operatios, whereas bipolar format is used i siged arithmetic calculatios. More specifically, i uipolar codig, the iformatio carried i a stochastic stream of bits is x = P ( = ) = P (), whereas i the bipolar format, x = P ( = ) = P (). The major arithmetic operatios icluded i DCNN are multiplicatio, additio, ad tah. With SC, these operatios ca be implemeted with extremely small circuits as follows. ) Multiplicatio: Stochastic multiplicatio i uipolar ad bipolar is performed by a AND gate ad a NOR gate, respectively. We deote the probabilities of o the iput bit streams by P (A) ad P (B), ad the probability of at the output of the AND gate is P (A) P (B), i.e., the product of uipolar multiplicatio. As for the bipolar codig, the output Feature maps Output Iput Covolutioal Poolig Covolutioal Poolig Fully coected Fig.. The fifth geeratio of LeNet DCNN architecture.

3 (a),,,,0,0,0,0 (4/8) A,,0,,,,,0 (6/8) B Z,,0,,0,0,0,0 (3/8) (b),,0,,0,0,,0 (0/8) A,0,,,,,,0 (4/8) B Z,0,0,,0,0,, (0/8) Fig.. Stochastic multiplicatio: (a) uipolar ad (b) bipolar. of the NOR gate is P (Z) = P (A) P (B) + P (A) P (B). Therefore, the stochastic umber of Z is calculated as z = P (Z) = 4 P (A) P (B) P (A) P (B) + = ( P (A) ) ( P (B) ) = a b. Note that the iput bit streams are assumed to be suitably ucorrelated or idepedet i the above calculatios. A example of multiplyig two bit streams A ad B is illustrated i Figure. ) Additio: Stochastic additio ca be implemeted by a OR gate, a multiplexer (MU), or a Accumulative Parallel Couter (APC) [9], as illustrated i Figure 3 (a), (b), ad (c), respectively. Whe both iputs a ad b are small, the output of the OR gate is a approximate sum that is expressed as z = P (Z) = P (A) + P (B) P (A B) a + b. A MU performs scaled additio by radomly selectig iput i Poe amog iputs with probability pi such that i= pi =. For example, addig a ad b usig MU with p = p = 50% geerates a output z = P (Z) = P (A) + P (B). This MU ca also perform scaled additio i bipolar codig, i.e., z = P (Z) = ( P (A) ) + ( P (B) ) = (a + b). However, the MU has the drawback of losig iputs iformatio, sice oly oe bit is selected ad the remaiig bits are igored at a time. I order to achieve better accuracy, APC is proposed to compute the total umber of s preset i all the iputs usig a parallel couter. Note that the output of APC is i biary, so additioal steps may be eeded to trasform this. I coclusio, OR gate is the most area efficiet but the accuracy is too low to be used i DCNN. MU is area efficiet with limited accuracy, whereas APC achieves better accuracy at the cost of a larger footprit. 3) Hyperbolic Taget: The hyperbolic taget fuctio (i.e., tah( )) is implemeted usig a K-state FSM, as show i Figure 4, where half of the states geerate output 0 ad the other half states geerate. Accordig to [0], for a give Z,,,0,0,,0, (5/8),0,,0,,,, (6/8) A (b) 0,0,0,0,0,0,,0 (/8) B Z,0,,0,,0,,0 (4/8),0,0,,0,,0, (4/8) S (c) 3 x x x3 x4 x w S SK/ SK- Z= SK- Z Fig. 4. Stochastic hyperbolic taget. biary umber (a) x x x3 x4 w w w3 w4 m x w to Stah tah( K x) Parallel Couter SK/- oe stochastic bit-stream with m legth log Up/Dow Couter stochastic bit-streams with m legth Z=0 m w w w3 w4 S0 stochastic bit-streams with m legth oe colum of products Fig. 3. Stochastic adders: (a) OR gate, (b) multiplexer (MU) for scaled additio, ad (c) approximate parallel couters (APC). A. APC-Based Neuro Figure 5 (a) illustrates the APC-based hardware euro desig, where the ier product is calculated usig NOR gates (for multiplicatio) ad a APC (for additio). To be more specific, we deote the umber of bipolar iputs ad stochastic stream legth by ad m, respectively. Accordigly, NOR gates are used to geerate products of iputs (x0i s) ad weights (wi0 s), ad the the APC accumulates the sum of s i each colum of the products. Sice the sum geerated by APC is a biary umber, the K-state FSM desig metioed i Sectio III-B3 caot be applied here directly. Istead of a FSM, a saturated up/dow couter is used to perform the scaled hyperbolic taget activatio fuctio Btah( ) for biary iputs. Details ad optimizatio of the Btah( ) activatio fuctio usig a saturated up/dow couter for biary iputs ca be foud i referece work []. For a APC-based euro with the fixed bit stream legth 04, the accuracy, area, power, ad eergy performace with respect to the iput size are show i Figure 6 (a), (b), (c), ad (d), respectively. To be more specific, as illustrated i Figure 6 (a), APC-based euro shows a very slow accuracy degradatio as iput size icreases. However, the area, power, ad eergy of the etire APC-based euro cell icreases ear liearly as the iput size grows, as show i Figure 6 (b), (c), ad (d), respectively. The reaso is as follows: With the efficiet implemetatio of Btah( ) fuctio, the hardware A0 B0 A B A B A3 B3 A4 B4 A5 B5 A6 B6 A7 B7 IV. H ARDWARE -BASED DCNN D ESIGN AND O PTIMIZATION USING S TOCHASTIC C IRCUITS I this sectio, we first coduct a detailed ivestigatio of the eergy-accuracy trade-off amog two hardware euro desigs usig SC, i.e., APC-based euro ad MUbased euro, as show i Figure 5 (a) ad (b), respectively. Hardware-based poolig is provided afterward, ad fially we preset the structure optimizatio method for the overall DCNN architecture.,0,,0,0,0,0,0 (/8) A 0,,0,0,0,,0, (3/8) B K x ) () Therefore, additioal coversio steps are eeded to calculate the tah(x). The accuracy of the K-state FSM tah fuctio is determied by state umber K ad iput stream legth. Stah(K, x) = tah( (a) bipolar stochastic umber x, the result of the FSM desig is a stochastic approximatio to the tah fuctio as follows, (b) stochastic bit-streams Fig. 5. Various hardware euro desigs. (a) APC-based euro, ad (b) MU-based euro.

4 of Btah( ) icreases logarithmically as the iput icreases, sice the iput width of Btah( ) is log. O the other had, the umber of NOR gates ad the size of the APC grow liearly as the iput size icreases. Hece, the ier product calculatio part, i.e., NOR array ad APC, is domiat i a APC-based euro, ad the area, power, ad eergy of the etire APC-based euro cell also icrease at the same rate as the ier product part whe the iput size icreases. Sice the legth of the stochastic bit stream is importat, we ivestigate the accuracy of APC-based euros usig differet stream legths uder differet iput sizes. As show i Figure 7, loger bit stream legth cosistetly outperforms lower bit stream legth i terms of accuracy i APC-based euros with differet iput sizes. However, desigers should cosider the latecy ad eergy overhead caused by log bit streams. B. MU-Based Neuro As show i Figure 5 (b), a MU-based euro is comprised of NOR gates, a MU, ad a K-state FSM, i order to compute the products of bipolar iputs (x is) ad weights (w i s), the stochastic sum of all products, ad the hyperbolic taget activatio fuctio, respectively. As the ier product calculated by a MU is a stochastic umber, the K-state FSM desig metioed i Sectio III-B3 ca be used here to implemet the activatio fuctio Stah( ). Nevertheless, two problems must be take ito cosideratio: (i) the ier product calculated by a iput MU is scaled to z, assumig the correct result is z, ad (ii) with z K z the iput, the K-state FSM calculates tah( ) istead of the desired value tah(z). Hece, i order to get the correct activatio, we eed to scale up the results of MU by times ad multiply the stream by K (or multiply by K directly). As opposed to the relatively simple ad efficiet data coversios o a software platform, such coversios i a hardware-based euro icurs sigificat hardware overhead, because the liear gai trasformatio eeds oe more FSM [0], ad the multiplicatio requires oe NOR gate as well as the geeratio of the other bipolar stochastic stream. I this paper, cosiderig a iputs euro with ier product deoted by z, we select the state umber K such that =, ad the fial output of the FSM is calculated as K Stah(K, z ) = tah(k z ) = tah(z) () I this way, we achieve the desired activatio result with o additioal bit stream coversio (i.e., o hardware overhead). Fig. 6. Usig the fixed bit stream legth 04, the umber of iputs versus (a) accuracy, (b) area, (c) power ad (d) eergy for a APC-based euro. Fig. 7. The legth of bit stream versus accuracy uder differet iput umbers for a APC-based euro. Fig. 8. Usig the fixed bit stream legth 04, the umber of iputs versus (a) accuracy, (b) area, (c) power ad (d) eergy for a MU-based euro. We first ivestigate the performace of the MU-based euro with respect to its iput size. Figure 8 (a), (b), (c), ad (d) show the results of the umber of iputs versus accuracy, area, power, ad eergy, respectively, for a MU-based euro usig a fixed bit stream legth which is equal to 04. It is importat to achieve a high accuracy of a euro cell, however, as show i Figure 8 (a), the accuracy of a MU-based euro sigificatly degrades as the iput size icreases. The reaso is that MU additio selects oly oe bit at a time ad igores the rest of the bits, leadig to low accuracy whe iput size is large. I additio, oe ca observe from Figure 8 (b), (c), ad (d) that as the umber of iputs icreases, area, power, ad eergy of the MU-based euro all ted to icrease. This is because a MU-based euro with more iputs requires more NOR gates ad MUes for ier product calculatio, ad more states i the FSM (K = ) to compute the activatio fuctio. Hece, the icreased hardware compoets result i more area, power, ad eergy of the euro cell. Next, we ivestigate the relatioship betwee bit stream legth ad accuracy uder differet umbers of iputs. As show i Figure 9, for a certai iput size, loger bit stream results i higher accuracy, ad the improvemet of accuracy is more sigificat whe iput size is larger. Hece, whe desigig a MU-based euro, log bit stream ca be applied to compesate the accuracy degradatio for large iput size. C. Poolig Operatio I a DCNN, dow samplig steps are performed by the poolig s, which summarize the outputs of eighborig groups of euros i the same kerel map. Poolig operatio achieves the ivariace to iput data (i.e., image, video, etc.) trasformatios ad better robustess to oise ad clutter. Moreover, the iter- coectios ca be sigificatly reduced for a hardware DCNN by usig poolig s. Cosiderig a poolig regio cosistig of k euros: {a,, a k i a feature map, where a i deotes the activatio result of the i-th euro, the poolig selects oe activatio a out at a time. I this paper, we adopt the average poolig,

5 Fig. 9. The legth of bit stream versus accuracy uder differet iput umbers for a MU-based euro. where each activatio result ai has the same probability to be selected as output, i.e., aout = mea(a ak ). For example, the stochastic arithmetic mea over a regio is provided i Figure 0, where three -to- MUes are eeded to implemet the average poolig. D. Structure Optimizatio for the Etire DCNN Architecture There are four performace metrics for the DCNN desig, i.e., accuracy, area, power, ad eergy. I this paper, we cosider a geeral DCNN optimizatio problem, where the objective fuctio is comprised of oe or multiple metrics ad the rest of the metrics are cosidered as costraits, e.g., eergy, power, ad accuracy as objective fuctio with area as costrait. I additio, we itroduce oe more costrait that the accuracy of hardware-based DCNN caot be sigificatly lower tha the accuracy of software-based DCNN, so as to make the accuracy of the hardware-based DCNN competitive. The DCNN architecture of iterest show i Figure cosists of two poolig s, two covolutioal s, ad oe fully-coected. The two poolig s are implemeted usig MU trees, as described i Sectio IV-C. As for the remaiig two covolutioal s (referred to as 0 ad ) ad oe fully-coected (referred to as ), they ca be built usig either APC-based euros or MUbased euros with a certai bit stream legth. We further ivestigate the iflueces of errors i 0, ad o the overall test error of the etire DCNN, as show i Figure, where the data values i each follow a ormal distributio (as observed i the test beches) with various stadard deviatios represetig the errors of the euros i that. It is observed that a closer to the iputs has more impact o the overall accuracy of the DCNN tha a closer to the output. The explaatio is that iaccurate features captured ear the iputs may affect all the followig s, whereas the errors occurrig ear the output ca oly disturb a few subsequet s. Therefore, the ituitio is that accurate euro structures should be applied to the s ear iputs, ad less accurate euros ca be used i the s closer to the output to achieve better eergy/power/area performace. Next, we compare the performace betwee APC-based euro ad MU-based euro usig a fixed bit stream legth equal to 04 uder differet iput sizes, as show i Table I. Clearly, APC-based euro is more accurate but occupies a a / bit stream a3 a4 / bit stream aout Fig. 0. A 4-to- poolig example. Fig.. The impact of errors i differet s o the overall DCNN test error. TABLE I C OMPARISON BETWEEN APC-BASED N EURON AND MU-BASED N EURON USING 04 B IT S TREAM Iput size Absolute error Area (µm ) Power (µw ) Eergy (f J) APC-based euro MU-based euro Ratio of APC/MU (%) more area tha MU-based euro. Besides, as APC is much slower tha MU, the latecy of APC-based euro is larger tha MU-based euro, which causes APC-based euro to cosume more eergy tha MU-based euro for oe calculatio. As for the power performace, a APC-based euro has less switchig (due to the log latecy) ad larger area tha the MU-based euro, resultig i less dyamic power, more leakage power, ad less overall power. The proposed structure optimizatio method for the overall DCNN architecture is give i Figure. As the bit stream legth sigificatly affects the eergy cosumptio ad accuracy of the etire DCNN, the first step is to apply biary search to choose a suitable bit stream legth for a DCNN cofiguratio (i.e., euro structure cofiguratio i each ). Note that the DCNN cofiguratio used i step is ot importat as the results will be refied i the followig steps. I step, uder the fixed bit stream legth, all the promisig cofiguratios are explored, where some cofiguratios ca be ruled out, e.g., all s usig MU-base euros is highly iaccurate ad ca be ruled out. Based o the results of step, the cofiguratios with desirable performace will be selected, ad i the followig step 3, for each cofiguratio, we try other bit stream legths to see if better performace ca be achieved. The fial cofiguratio of the DCNN is decided based o the result of step 3, ad several more iteratios may be eeded to further refie the result by explorig more cofiguratios. V. E PERIMENTAL R ESULTS The LeNet5 DCNN used i this experimet is built with a cofiguratio. The MNIST hadwritte digit image dataset [] cosists of 60,000 traiig data ad 0,000 testig data with 8x8 grayscale image ad 0 classes is used i the experimets, ad the etwork is traied with 0 epochs (batch size =500). We use Syopsys Desig Compiler to sythesize the DCNNs with the 45m Nagate Ope Cell Library []. Table II cocludes the cofiguratios ad performace for all the explored hardware-based DCNNs (No. 5) usig the proposed structure optimizatio method, the 8 bit covetioal biary pipelied baselie (No. 6) ad software-based DCNNs usig CPU (No. 7) or GPU (No. 8) for compariso. Note that the power for software is estimated usig Thermal Desig Power (TDP), ad the eergy is calculated by multiplyig the ru time ad TDP.

6 Step : fid a suitable bit stream legth usig biary search Step : explore cofiguratios usig the fixed bit stream Step 3: explore other bit streams for the promisig cofiguratios No Costraits satisfied? Performace satisfactory? Yes Fig.. Structure optimizatio method for the etire DCNN. Without ay loss of geerality, we set the desired accuracy to be 4.5% error rate. I the first step of the proposed structure optimizatio method, the bit stream legth is set to 04 usig biary search. I step, usig the fixed bit stream legth, all cofiguratios are explored, as show i Table II (No. 8). DCNNs i No. 5 are ruled out due to the low accuracy, ad i step 3, the remaiig promisig DCNNs i No. 6 8 are explored usig the decreased bit stream legth 5 bits, where the results are give as DCNNs i No. 9. Sice DCNNs No. 0 satisfy the accuracy costrait, we further reduce the bit stream to 56 to improve eergy performace, where DCNNs i No. 3 provide the results. This time, oly DCNNs i No. 3 (all APC-based euros) meet the accuracy costrait ( 4.5%). Hece, the bit stream legth is further reduced for DCNN i No. 3 so as to fid the cofiguratio that achieves the miimum eergy while satisfyig the accuracy costrait. The DCNNs that use more MU-based euros provide smaller footprits, which are suitable for area-costrait embedded systems, whereas the DCNNs with more APC-based euros achieve better accuracy, eergy ad power, which are good for power/eergy-costrait embedded systems. Give the costrait(s), the proposed structure optimizatio method ca provide the DCNN cofiguratios with satisfactory performace. For istace, DCNNs i No. 0,, 3 5 are all promisig cofiguratios foud by the proposed method, give the accuracy costrait. Compared with the covetioal 8-bit biary implemetatio, the preseted hardware-based DCNN usig SC (No. 5) achieves 55, 5, ad improvemet i terms of area, power ad eergy, respectively, while the error is icreased by.86%. VI. CONCLUSION I this paper, two hardware-based euro structures usig SC were aalyzed, ad the ifluece of iaccuracy of euros i differet s o the overall DCNN accuracy was studied. A structure optimizatio method was proposed for a geeral DCNN architecture, which joitly optimized the accuracy, area, power, ad eergy. Experimetal results demostrated that compared with the biary ASIC DCNNs, the area, power ad eergy of the hardware-based DCNN geerated by the proposed structure optimizatio were sigificatly improved, whereas the accuracy performace was slightly degraded. REFERENCES [] Y. Begio, Learig deep architectures for ai, Foudatios ad treds R i Machie Learig, vol., o., pp. 7, 009. [] Y. LeCu, Y. Begio, ad G. Hito, Deep learig, Nature, vol. 5, o. 7553, pp , 05. [3] T. N. Saiath, A.-r. Mohamed, B. Kigsbury, ad B. Ramabhadra, Deep covolutioal eural etworks for lvcsr, i 03 IEEE Iteratioal Coferece o Acoustics, Speech ad Sigal Processig. IEEE, 03, pp [4] K. Simoya ad A. Zisserma, Very deep covolutioal etworks for large-scale image recogitio, ariv preprit ariv: , 04. [5] A. Karpathy et al., Large-scale video classificatio with covolutioal eural etworks, i Proceedigs of the IEEE coferece o Computer Visio ad Patter Recogitio, 04, pp Ed TABLE II COMPARISON AMONG VARIOUS HARDWARE-BASED DCNNS AND SOFTWARE-BASED DCNNS No. Bit Error Area Power Eergy Layer 0,, Stream (%) (mm ) (W ) (µj) 04 MU, MU, MU MU, MU, APC MU, APC, MU MU, APC, APC APC, MU, MU APC, MU, APC APC, APC, MU APC, APC, APC APC, MU, APC APC, APC, MU APC, APC, APC APC, APC, MU APC, APC, APC APC, APC, APC APC, APC, APC bit fixed poit biary(pipelied) CPU: two Itel eo W GPU: NVIDIA Tesla C [6] C. Szegedy et al., Goig deeper with covolutios, i Proceedigs of the IEEE Coferece o Computer Visio ad Patter Recogitio, 05, pp. 9. [7] M. Motamedi, P. Gysel, V. Akella, ad S. Ghiasi, Desig space exploratio of fpga-based deep covolutioal eural etworks, i 06 st Asia ad South Pacific Desig Automatio Coferece (ASP-DAC). IEEE, 06, pp [8] A. Rahma, J. Lee, ad K. Choi, Efficiet fpga acceleratio of covolutioal eural etworks usig logical-3d compute array, i 06 Desig, Automatio & Test i Europe Coferece & Exhibitio (DATE). IEEE, 06, pp [9] P. Li et al., The sythesis of complex arithmetic computatio o stochastic bit streams usig sequetial logic, i Proceedigs of the Iteratioal Coferece o Computer-Aided Desig. ACM, 0, pp [0] J. Li ad J. Draper, Acceleratig soft-error-rate (ser) estimatio i the presece of sigle evet trasiets, i Proceedigs of the 53rd Aual Desig Automatio Coferece. ACM, 06, p. 55. [] J. Li ad J. Draper, Joit soft-error-rate (ser) estimatio for combiatioal logic ad sequetial elemets, i VLSI (ISVLSI), 06 IEEE Computer Society Aual Symposium o. IEEE, 06, pp [] K. Kim et al., Dyamic eergy-accuracy trade-off usig stochastic computig i deep eural etworks, i Proceedigs of the 53rd Aual Desig Automatio Coferece. ACM, 06, p. 4. [3] K. Sai, G. Garreau, J. L. Moli, ad A. G. Adreou, Fpga implemetatio of a deep belief etwork architecture for character recogitio usig stochastic computatio, i Iformatio Scieces ad Systems (CISS), 05 49th Aual Coferece o. IEEE, 05, pp. 5. [4] Y. Ji, F. Ra, C. Ma, ad D. J. Lilja, A hardware implemetatio of a radial basis fuctio eural etwork usig stochastic logic, i Proceedigs of the 05 Desig, Automatio & Test i Europe Coferece & Exhibitio. EDA Cosortium, 05, pp [5] A. Re et al., Desigig recofigurable large-scale deep learig systems usig stochastic computig, i 06 IEEE Iteratioal Coferece o Rebootig Computig. IEEE, 06. [6] Z. Li et al., Dsc: Hardware-orieted optimizatio for stochastic computig based deep covolutioal eural etworks, i Computer Desig (ICCD), 06 IEEE 34th Iteratioal Coferece o. IEEE, 06. [7] B. D. Brow ad H. C. Card, Stochastic eural computatio. ii. soft competitive learig, IEEE Trasactios o Computers, vol. 50, o. 9, pp , 00. [8] Y. LeCu, L. Bottou, Y. Begio, ad P. Haffer, Gradiet-based learig applied to documet recogitio, Proceedigs of the IEEE, vol. 86, o., pp , 998. [9] P.-S. Tig ad J. P. Hayes, Stochastic logic realizatio of matrix operatios, i Digital System Desig (DSD), 04 7th Euromicro Coferece o. IEEE, 04, pp [0] B. D. Brow ad H. C. Card, Stochastic eural computatio. i. computatioal elemets, IEEE Trasactios o computers, vol. 50, o. 9, pp , 00. [] L. Deg, The mist database of hadwritte digit images for machie learig research, IEEE Sigal Processig Magazie, vol. 9, o. 6, pp. 4 4, 0. [] Nagate 45m Ope Library, Nagate Ic., 009. [Olie]. Available: