LHCb Kalman filter cross architecture studies

Transcription

1 LHCb Kalman filtr cross architctur studis Danil Hugo Cámpora Pérz Univrsidad d Svilla, CERN. Gnva, Switzrland. dcampora@crn.ch Abstract. Th 2020 upgrad of th LHCb dtctor will vastly incras th rat of collisions th Onlin systm nds to procss in softwar, in ordr to filtr vnts in ral tim. 30 million collisions pr scond will pass through a slction chain, whr ach stp is xcutd conditional to its prior accptanc. Th Kalman filtr is a fit applid to all rconstructd tracks which, du to its tim charactristics and arly xcution in th slction chain, consums 40% of th whol rconstruction tim in th currnt triggr softwar. This maks th Kalman filtr a tim-critical componnt as th LHCb triggr volvs into a full softwar triggr in th Upgrad. I prsnt a nw Kalman filtr algorithm for LHCb that can fficintly mak us of any kind of SIMD procssor, and its dsign is xplaind in dpth. Prformanc bnchmarks ar compard btwn a varity of hardwar architcturs, including x86_64 and Powr8, and th Intl Xon Phi acclrator, and th suitability of said architcturs to fficintly prform th LHCb Rconstruction procss is dtrmind. LHCb-PROC /11/ Introduction Th LHCb dtctor will b upgradd in 2020 [1] to acquir data at an instantanous luminosity of cm 2 s 1 and to collct a datast of at last 50 fb 1. At th sam tim th first stag of filtring in th Data Acquisition procss, currntly known as hardwar lvl triggr, will b discontinud in favor of a full softwar triggr. Consquntly th throughput that th softwar lvl triggr will nd to sustain in ordr to maintain a stady triggring rat will dramatically incras, du to both th incras in rat of vnts procssd in softwar, and th influx of largr vnts. To b abl to cop with th incrasd data rat, svral hardwar architcturs ar currntly bing xplord. Whil th currnt LHCb High Lvl Triggr farm is composd solly of Intl Xon procssors, in th last fw yars many High Prformanc Computing sits ar adopting othr altrnativ hardwar architcturs, such as IBM Powr X, FPGAs, or manycor architcturs lik GPUs or Intl Xon Phi. This has provokd an intrst in th High Enrgy Physics community, and mor concrtly within th LHCb collaboration, and th qustion is whthr ths architcturs ar suitabl for prforming th High Lvl Triggr in a sustainabl way, taking into account th conomical, powr consumption and softwar maintainability aspcts. Th Kalman filtr [2] is a linar quadratic stimator, which is applid to stimat th trajctory of th particls as thy travl through th LHCb dtctor. Tabl 1 shows a tim charactrization of th first stag of th LHCb High Lvl Triggr (HLT1). Th Kalman filtr algorithm is th singl largst tim contributor in th LHCb softwar chain, taking about 60 % of th HLT1 rconstruction tim. Following Amdahl s law [3], th achivabl prformanc gain of an algorithm is boundd by its paralllizabl portion. Du to th natur of th LHCb xprimnt, many particls

2 Algorithm Tim contribution (%) Kalman filtr Hlt1UpgradForwardHPT PrVloHlt 6.38 PrVloUTHlt 3.25 HltUpgradPV3D 2.96 Hlt1UpgradTwoTrackMVAUnit 2.82 PrPixlStorClustrs 1.36 Tabl 1: Tim charactrization of most tim consuming algorithms in th first stag of LHCb High Lvl Triggr (HLT1). Only algorithms with a tim contribution of mor than 1% ar shown. travl through th dtctor simultanously and indpndntly, which maks th Kalman filtr a massivly paralllizabl problm in this contxt. In this papr, I prsnt a hardwar architctur indpndnt dsign of a Kalman filtr algorithm, Cross Kalman 1. A focus is givn to th LHCb us cas, taking advantag of th fact that many indpndnt particls can b procssd simultanously. I xplor possibl prformanc gains ovr th currnt LHCb Rconstruction softwar, and compar th spdup obtaind ovr a varity of architcturs. 2. Th LHCb Kalman filtr Th Kalman filtr algorithm is typically implmntd following a two-stag dsign, consisting of a Prdict and an Updat stag, prformd squntially. Th Prdict stag uss th built trajctory to xtrapolat whr th particl is xpctd to b on th nxt timstamp. Updat uss th dtctor information (a hit in th dtctor) to corrct th forming trajctory. In th LHCb us cas, a Kalman filtr is applid to a particl trajctory throughout th dtctor, corrcting it along th way using th dtctd hits of th particl. In hindsight, rconstructing on particl trajctory consists in applying rpatdly Prdict and Updat squntially, hit aftr hit. A schmatic of this procss is shown in Figur 1. Furthrmor, in LHCb th trajctory rconstruction for a particl is prformd in thr stps: First, a Forward Kalman filtr is prformd, in th upstram dirction of th dtctor (right in figur 1). Thn, a Backward Kalman filtr is prformd, in th downstram dirction of th dtctor. Finally, a smoothd trajctory, consisting of th avrag of th Forward and Backward trajctory, is calculatd by th Smoothr componnt. Du to th squntial natur of th Kalman filtr procss, thr is no room for paralllization of th rconstruction of a singl particl trajctory. Howvr, O(100) particls ar rconstructd within ach collision indpndntly. This fact poss th motivation for th algorithm dsign. 3. Cross Kalman algorithm dsign Th dsign of th Cross Kalman algorithm has bn carrid out with a Singl Instruction Multipl Data (SIMD) architctur in mind, focusing on thr dsign aspcts: th control flow, th undrlying data structurs and a fin-graind optimization of th most timconsuming parts Control flow As mntiond abov, particls ar procssd hit by hit in th Forward dirction, followd by th Backward dirction and th Smoothr. Concptually, th procssing of hits of diffrnt 1

3 R f r n c n o d Si g n a ln o d x z Ma g n t Figur 1: Schmatic of LHCb Kalman filtr procss. Two particls collid insid th VELO subdtctor (in rd). Two particls ar rconstructd (in blu), laving signal nods in th VELO, TT and tracking stations. Rfrnc nods ar placd at crtain points in th trajctory to triggr stat calculations for postrior usag in th rconstruction. particls is indpndnt, and so th Kalman filtr can b paralllizd horizontally, which is nvisiond in thr stags as wll: Forward, Backward and Smoothr. In ach of th stags, th hits of diffrnt particls can b procssd in paralll. Givn that diffrnt particls may hav a diffrnt numbr of hits, a static schdulr is ndd to guarant a maximum rsourc utilization in ach computation stp. Th schdulr accpts a width paramtr to dcid how many hits ar to b procssd in on stp of th computation. Th width of th schdulr should fit with th SIMD width, and th choic of th most prformant width dpnds on th architctur. Furthrmor, th dsign of th algorithm is not tid to a particular prcision, but rathr it accpts a configurabl prcision lngth to choos th according floating point standard at compilation tim. Ths two factors will dtrmin how many lmnts ar xcutd in paralll at ach computation stp: For a CPU supporting Advancd Vctor Extnsions (AVX) and choosing doubl floating point prcision, up to four lmnts will b procssabl at a tim, whras for a CPU supporting AVX-512 and singl prcision, up to sixtn lmnts would b procssabl at a tim. Sinc particl trajctoris contain varying numbr of hits, som stps will not fully fill th rqustd schdulr width. In ordr to minimiz th numbr of stps, th particls ar prordrd by dcrasing numbr of hits. Whn considring th typ of schdulr to implmnt, static and dynamic schdulrs wr considrd. A static schdulr dtrmins its ordr of xcution prior to start xcuting itrations. On th othr hand, a dynamic schdulr chooss th lmnts to xcut on-th-fly. Th rsults of th Forward and th Backward stags will b usd by th Smoothr. If on wr to us a dynamic schdulr, th output of th first two stags would not b guarantd to b alignd, and in ordr for th Smoothr to produc th corrct output, it would hav to construct alignd data structurs, rquiring additional mmory copy oprations. Th static schdulr circumvnts this by dsign: th schdulr is calculatd onc, itratd to procss th Forward, and itratd in rvrs for th Backward. That way, th Smoothr will not rquir any additional mmory oprations. Figur 2 dpicts som itrations computd with th dscribd schdulr, with a width st to four. Th last column dpicts th particl-hit bing procssd at th momnt. This schdulr can b itratd forwards or backwards, maintaining th squntiality nforcd by th Kalman filtr procss. Du to its flxibl dsign, xcution can b optimizd on multi and manycor SIMD architcturs, with varying vctor widths.

4 it in out act vctor (#particl-#hit) #540: { } #541: { } #542: { } #543: { } #544: { } #545: { } #546: { } #547: { } #548: { } #549: { } Figur 2: Static schdulr itrations. Th first column shows th itration numbr. Th scond and third show th input and output mask, usd to notify a chang of particl. Th fourth column is th action mask. Th last column shows th hits bing procssd in paralll. This schdulr was run with a width of four Data structurs In ordr to improv th arithmtic intnsity 2 of th algorithm, th data is structurd following an AOSOA (Array of Structurs of Arrays) dsign. For instanc, for ach stp of th Forward and Backward Kalman filtr, a stat, covarianc ( and χ 2 fit ar calculatd. For q ) a singl particl position, a stat is a fiv-lmnt vctor x y tx ty, a covarianc σ p is a 15-lmnt matrix (5 5 symmtric matrix), and th χ 2 is a singl lmnt [4]. Figur 3 shows th AOSOA data structur for a schdulr with width four. x 0 x 1 x 2 x 3 y 0 y 1 y 2 y 3 tx 0 tx 1 tx 2 tx 3 ty 0 ty 1 ty 2 ty 3 q q q q p 0 p 1 p 2 p 3 σ 0,0 σ 1,0 σ 2,0 σ 3,0.... σ 0,14 σ 1,14 σ 2,14 σ 3,14 χ 2 0 χ 2 1 χ 2 2 χ 2 3 Figur 3: AOSOA data structur for schdulr with width four. Ths kind of data structurs ar highly fficint for SIMD procssors, as thy xploit locality, minimizing cach misss, and allow for horizontal paralllization in a natural way. In th prsntd algorithm, ths data structurs ar gnratd with th xcution of th schdulr, and th data ar dirctly populatd in thm, avoiding xtra mmory copis. In addition, th data structur is allocatd alignd to th rquird mmory alignmnt by th spcific architctur undr xcution Fin-graind optimization In ordr to targt spcific Instruction St Architcturs (ISAs) of various architcturs, th arithmtic bulk of th Kalman filtr is writtn using a layr of abstraction, providd by a 2 Th arithmtic intnsity of an application is a mtric of how many arithmtic oprations ar prformd pr rquird byt transfr. W ar intrstd in th FLOP arithmtic intnsity, masurd in F LOP Byt.

5 vctorization library. Bindings to VCL [5] and UMESIMD [6] ar providd, laving it to th library providr to add compatibility for nw architcturs. Furthrmor, thr is no concptual imposition in th SIMD abstraction usd. For instanc, it would b possibl to dvlop an OpnCL implmntation taking advantag of th aformntiond control flow and data abstractions, targtting an arbitrary width and a manycor architctur or an FPGA. Finally, a scalar implmntation is also providd, in cas th architctur undr study dos not support any SIMD xtnsions. 4. Rsults Figur 4 shows th prformanc gain across architcturs, using th hr prsntd Kalman filtr implmntation. Th figur of mrit to compar th rsults is th combind throughput of th Forward and Backward Kalman filtr and th Smoothr: #Forward + #Backward + #Smoothr tim All th tsts wr run undr th following sttings: (1) Th program was compild with gcc 6.2.0, with options -O2 -march=nativ. Turbo Boost was on, whr applicabl. KNL was using flat mmory mod, and pinnd against th MCDRAM. On procss was spawnd pr Non-Uniform Mmory Accss (NUMA) domain, with as many TBB thrads as cors in domain and pinnd to its mmory. Ran vnts, ach vnt is a Thrading Building Blocks (TBB) task. Usd Mont Carlo vnts from th LHCb Upgrad. Doubl prcision rsults ar validatd. Singl prcision rsults show a dviation in 1% of th rsults Spdup (x) "lab21" Intl(R) Xon(R) CPU E GHz Gaudi prformanc "lab21" Intl(R) Xon(R) CPU E GHz doubl prcision, vctorisd "lab21" Intl(R) Xon(R) CPU E GHz singl prcision, vctorisd "ibmfs1" 2061 MHz doubl prcision "ibmfs1" 2061 MHz singl prcision "olninja023" Intl(R) Xon Phi(TM) 1.30 GHz Quadrant, doubl prcision, vctorisd "olninja023" Intl(R) Xon Phi(TM) 1.30 GHz Quadrant, singl prcision, vctorisd "olninja024" Intl(R) Xon Phi(TM) 1.30 GHz SNC4, doubl prcision, vctorisd "olninja024" Intl(R) Xon Phi(TM) 1.30 GHz SNC4, singl prcision, vctorisd Figur 4: Spdup across svral architcturs and prcisions. Th original implmntation is shown as th lftmost bar. It is usd as a baslin against which to compar prformanc. Using th sam prcision, th Xon machin shows a 2.01

6 spdup. It is worth noting that moving to th Intl Knights Landing (Xon Phi) is costffctiv, whn taking into considration th diffrnc in cost and prformanc vrsus th baslin undr study. Transitioning to singl prcision has a variabl impact on prformanc, dpnding on th architctur undr analysis, as shown in Tabl 2. It is particularly ffctiv on currnt HLT hardwar, whr a prformanc boost of 1.68 is obsrvd. Hardwar architctur Effct of moving to singl prcision Intl(R) Xon(R) CPU E GHz 1.68 PowrNV 8335-GCA (Powr8) 1.5 Intl(R) Xon Phi(TM) 1.30 GHz 1.27 Tabl 2: Effct of moving to singl prcision across architcturs. Th scalability of th application undr various architcturs is shown in th following. Th Xon architctur procssor undr analysis dos not scal with th numbr of tasks in flight, as illustratd in Figur 5. This is partially du to Turbo Boost incrasing th frquncy of th procssor for fwr cor counts, which th vndor announcs as 2.40 GHz bas frquncy vrsus 3.20 GHz max turbo frquncy. Additionally, th mmory footprint of Cross Kalman with highr cor counts is quit larg. Thr is no gain in using additional Hypr Thrads pr cor on this procssor. Throughput (#prdict, #updat and #smoothr / s) "lab21" Intl(R) Xon(R) CPU E GHz, doubl prcision, vctorisd "lab21" Intl(R) Xon(R) CPU E GHz, singl prcision, vctorisd (16, ) (16, ) Numbr of procssors in tbb schdulr Figur 5: Scalability of Kalman filtr for Intl(R) Xon(R) CPU E GHz. Th Knights Landing architctur scals bttr, as shown on th lft in Figur 6. This machin has 64 physical cors, and th scalability until that point is almost linar. Using two Hypr Thrads pr cor a marginal gain is obsrvd, but th prformanc flattns out whn using thr or four Hypr Thrads pr cor. It is intrsting to not th throughput of th MCDRAM is not a bottlnck, rgardlss of th numbr of tasks in flight. Th Powr8 architctur undr analysis also shows a good scaling, as shown on th right in Figur 6. Enabling Simultanous Multi Thrading (SMT) did not improv prformanc. Figur 7 dpicts a Rooflin modl [7] for an Intl Haswll procssor. A Rooflin modl rlats th arithmtic intnsity of an application and its prformanc, with th capability of th spcific hardwar platform undr considration. Th figur shows th arithmtic intnsity of two particular procsss of th Kalman filtr. In ordr to incras th arithmtic intnsity of th application, th Prdict and Updat stps (cf. Sction 2) wr combind into a singl stp calld fit, in grn in th plot. Th smooth corrsponds with th Smoothr

7 Throughput (#prdict, #updat and #smoothr / s) "olninja023" Quadrant, doubl prcision, vctorisd "olninja023" Quadrant, singl prcision, vctorisd "olninja024" SNC4, doubl prcision, vctorisd "olninja024" SNC4, singl prcision, vctorisd (256, ) (256, ) Throughput (#prdict, #updat and #smoothr / s) "ibmfs1" 2061 MHz, doubl prcision "ibmfs1" 2061 MHz, singl prcision (64, ) (64, ) Numbr of procssors in tbb schdulr Numbr of procssors in tbb schdulr Figur 6: Lft: Scalability of Kalman filtr for Intl(R) Xon Phi(TM) 1.30 GHz. Right: Scalability of Kalman filtr for PowrNV 8335-GCA (Powr8). componnt, in blu in th plot. As th figur shows, th program is clos to saturating th DDR RAM Bandwidth th CPU can support. Givn th arithmtic intnsity of this implmntation of th Kalman filtr, thr is not much mor prformanc on can xtract out of this procssor Smoothr Xon E v4 Smoothr Xon Phi 7210 Fit Xon E v4 Fit Xon Phi 7210 KNL 7210 (41.28 GFLOP/s) Broadwll E5-2683v4 (31.79 GFLOP/s) Prformanc GFLOP/s L2 KNL BW: GB/s L3 Broadwll BW: GB/s DDR Broadwll BW: GB/s DDR KNL BW: GB/s Arithmtic intnsity [FLOP/Byt] Figur 7: Rooflin modl for Intl(R) Xon(R) E GHz. 5. Conclusions and outlook I hav brifly discussd th rquirmnts of softwar throughput for th upcoming upgrad of th LHCb triggr. In ordr to cop with its stringnt ral-tim rquirmnts, a nw Kalman filtr algorithm has bn dsignd, and svral hardwar architcturs and thir amnability to procssing a substantial portion of th LHCb rconstruction hav bn xplord. Ths studis suggst th Intl Knights Landing architctur will b a strong contndr in th choic of a hardwar architctur for th Upgrad of LHCb, du to its familiar programmability and th strong scaling it shows for a highr numbr of cors. Whn

8 porting rconstruction softwar to ths architcturs howvr, on should tak into considration that thr will b a softwar framwork bhind th scns that will impact th prformanc and possibly chang mmory rquirmnts. To this nd, th Cross Kalman algorithm is bing intgratd with th framwork, and a follow up study should rlat th rsults obtaind hr with what th framwork dlivrs in th final product. It would b intrsting to xplor othr architcturs, such as ARM64, GPUs or FPGAs. As th litratur in th fild suggsts, singl prcision may hav vn a dpr impact on ths architcturs. Evn though prcision can b configurd at compil tim in Cross Kalman, intgration with th framwork will b costly in trms of dvlopmnt tim, and validation of ths rsults will rquir a ddicatd study. Nvrthlss, th rsults suggst that thr would b a high impact in trms of prformanc if a transition to singl prcision is succssfully mad. Acknowldgmnts Th author would lik to thank th HTCC at CERN for fruitful discussions through th procss of dsigning and writing th prsntd softwar, and arly accss to Intl hardwar. Spcial thanks go to O Awil for th guidanc and th cration of th Rooflin modl plot. Furthrmor, thanks to F Lmaitr for his contribution of th vctorizd transposition cod, and to O Bouizi and S Harald for th low-lvl cod discussions and for providing arly rsults and insight on th KNL architctur. In addition, thanks to C Pottrat for th Physics validation, and to W Hulsbrgn and R Aaij for th mathmatical discussions and data structur dsign. Finally, thanks to my suprvisors N Nufld and A Riscos Núñz for thir guidanc and support. Rfrncs [1] Th LHCb Collaboration (2014). LHCb Triggr and Onlin Upgrad Tchnical Dsign Rport. CERN- LHCC ; LHCB-TDR-016. [2] Kalman R E (1960). A Nw Approach to Linar Filtring and Prdiction Problms. Transactions of th ASME Journal of Basic Enginring, 82 (Sris D): [3] Amdahl G M (1967). Validity of th singl procssor approach to achiving larg scal computing capabilitis. AFIPS spring joint computr confrnc. [4] Hulsbrgn W (2008). Th Global covarianc matrix of tracks fittd with a Kalman filtr and an application in dtctor alignmnt. DOI: /j.nima [5] Fog A (2012). VCL C++ vctor class library. Gnu public licns. [6] Karpinski P (2015). UME::SIMD A library for xplicit simd vctorization. danor/umsimd. [7] Williams S W, Watrman A and Pattrson D A (2008). Rooflin: An Insightful Visual Prformanc Modl for Floating-Point Programs and Multicor Architcturs. Tchnical Rport No. UCB/EECS