MIT-CTR-III TWG-1. On-Board Very Short Interconnects Medium-term. October 12, Petre Popescu. communications technology roadmap

Transcription

1 MIT-CTR-III TWG-1 On-Bard Very Shrt Intercnnects Medium-term Octber 12, 2011 Petre Ppescu cmmunicatins technlgy radmap

2 Medium-term Based n cmmercially available cmpnents develped fr telecm and/r datacm. Minimal r n impact n system architecture. Minimal r n impact n system design methdlgies. N significant changes in current manufacturing prcesses. Limited changes fr cmpnent and system test methdlgies. Octber 12, MIT TWG1 Medium-term Restricted cmmunicatins technlgy radmap 2

3 TWG-1 Medium-term Objectives Identify dminant intercnnects within scpe and scaling. Define value prpsitin Identify candidate technlgy slutin sets expected t mature in the shrt/medium timeframe. Identify technlgy gaps that need industry attentin and pprtunities fr technlgy develpment and injectin. Develp cnceptual technlgy radmap that aligns near term technlgy effrts with lng term needs. Develp technlgy radmap that aligns technlgy develpment effrts, mnitrs prgress, and defines crrective actins. Deliver summary dcument t be shared and published. Octber 12, MIT TWG1 Medium-term Restricted cmmunicatins technlgy radmap 3

4 TWG1 Medium-term - Participants Dick Otte - manufacturability & ecnmics v. cst, structures, and will crdinate with Cu Sustainability grup Tremnt Mia - E-O, O-E Terry Bwen - identifying manufacturing prcesses, candidate technlgy slutins Rich Grzybwski - fibers, cables, O-E cnversin, technlgy knwledge, high-density fibers Brad Gaynr - electrical side f interfaces, n-chip intercnnects in extreme shrt distances Petre Ppescu and Chuck Millet - technlgy, architecture, manufacturing prcesses, electr-ptical interfaces, defining the dminant designs and challenges. YK Chen - systems architecture, E-O interspace frmat Abe Elfadel - Masdar Institute n-chip Si phtnic links- characterizatin and mdeling Octber 12, MIT TWG1 Medium-term Restricted cmmunicatins technlgy radmap 4

5 Generic Cmpute System and Very Shrt Links DRAM DRAM CPU DDR3 CPU DDR3 PCIe NIC Ethernet HT CPU CPU Chipset PCIe HD SSD DDR3 DRAM DDR3 DRAM SATA/SAS/USB GPU GDDR5 DP/HDMI Generic cmpute system and chip-t-chip n bard intercnnects (see HyperTransprt Cnsrtium fr mre details) Octber 12, MIT TWG1 Medium-term Restricted DRAM cmmunicatins technlgy radmap 5

6 PCIe TM and HT TM Intercnnect Summary PCIe and HT are high-speed full duplex differential links used in cmpute systems. Link density is critical, ne hundred full duplex links per CPU r GPU are nt uncmmn. Energy efficiency is anther critical characteristic, given the large number f links per device (CPU r GPU). The high-speed links must be capable f perating at any data rate frm 400Mb/s t 8Gb/s tday (PCIe Gen3) and pssible t 16Gb/s in the next three r fur years (32Gb/s will fllw). Pwer dwn capability and fast restart is required fr energy efficiency (n energy needed when n lad). Flexible link width (X4, X8, X16) with single surce and multiple destinatins capability. Adpting cmmn HT and PCIe physical layer (signaling type, electrical specificatin, cnnectrs, etc.) will have a psitive impact n ptical intercnnects adptin. This full-duplex cmmn high-speed interface perating at a lane data-rate f 400Mb/s t 32Gb/s can be adpted fr all n-bard very-shrt intercnnect. Octber 12, MIT TWG1 Medium-term Restricted cmmunicatins technlgy radmap 6

7 DDR3/GDDR5 DRAM Interface Summary DRAM interfaces are standard based (JEDEC) and are critical fr cmpute system perfrmance. Current DDR3 interfaces (data bus) have data rates frm 666Mb/s t 2.133Gb/s and GDDR5 data bus data rate can be as high as 5.5Gb/s. Mst cmmn DRAM interface type is 64 bit data interface. This interface is single-ended and cnsists f 64 half-duplex data links, 16 t 20 address links and up t 54 cntrl (including clcks) links, few f them half-duplex. The cmmn high-speed interface will require a multiplexing factr f 4 fr the data bus such that the data bus becmes a full-duplex 16 lanes link perating at a data rate f 2.6Gb/s t 22Gb/s. The address bus and cntrl/clck bus can be multiplexed t have the same lane data rate as the data bus. Sme f the cntrl signals can be full-duplex. Adpting the prpsed slutin will reduce the 64 bit data bus width frm ~130 signals t ~40 Cmmn High-Speed Interface lanes. Is cmmn fr a CPU/GPU t have 4 t 6 64bit data DRAM interfaces. Adpting the prpsed slutin will significantly reduce the width h the interface and will be based n the unique signaling standard. Octber 12, MIT TWG1 Medium-term Restricted cmmunicatins technlgy radmap 7

8 TWG1 Medium-term - Next Steps 1. Define n-bard very-shrt cmmn high-speed ptical intercnnects and radmap: Signaling type and WDM cmpatibility Lane data rate range radmap Intercnnect width and scaling Energy efficiency definitin (what is included and hw is measured) Cmpatibility with n-die ptical intercnnects (lng-term) 2. Electrical interface specificatin. 3. EO and OE cmpnents radmap. 4. DRAM interface definitin based n cmmn high-speed intercnnects. 5. Pssible radmap fr energy efficiency (cmmn high-speed ptical intercnnects and n chip interfaces). 6. Optical intercnnects density (waveguides and cnnectrs) radmap. 7. SC packaging technlgy supprting ptical intercnnects radmap. Octber 12, MIT TWG1 Medium-term Restricted cmmunicatins technlgy radmap 8

9 Thank Yu! Trademark Attributin PCIe is a registered trademark f PCI-SIG. HyperTransprt is a trademark f the HyperTransprt Technlgy Cnsrtium. Other names used in this presentatin are fr identificatin purpses nly and may be trademarks f their respective wners. Octber 12, MIT TWG1 Medium-term Restricted cmmunicatins technlgy radmap 9

10 Challenges fr the lnger term Limits f silicn phtnics Pwer handling, density, speed Limits f VCSELs Speed, density, reliability, pwer New Func?nality and Architectural Flexibility Bradcast Clck synchrniza?n Enabling new architectures Use f mul?ple wavelengths Path t System in Package netwrks Panelist: Madeleine Glick

11 Mul5ple Array Intercnnect methds With Varied Density Are Available fr The Varius Electrnic Packaging Levels Flipchip bump pitch f 70 µm and up Chip TSV Pitch f 20 µm? and up Package BGA pitch f 0.5 mm and up Interpser TSV Pitch f 100 µm?? and up MIT CTR September 8,

12 Density Decreases frm Chip t Circuit Bard Chip ~24 t 1 density reduc5n frm die t printed circuit bard. Frm ~ 100 micrns ff- chip t ~ 2.4 mm (0.100 ) pitch BGA. MIT CTR September 8,

13 Wire Bnd Density Density t 35 micrn pitch and up in a single rw is available. Mul5ple rws with higher density (< 25 micrns) is in use. MIT CTR September 8,

14 Optics fr Prgrammability Armand Slar-Lezama

15 What FLIP brings t the table Shared memry - the prmise f shared memry all data is glbal, accessible by all threads n need t wrry abut wh has the data details f shuttling data frm ne cre t the ther are handled by cache cherence prtcl - many architectures claim t ffer this but the cst f cache cherence makes it expensive t use in practice, prgrammers d have t wrry abut where the data lives they als wrry abut minimizing data mvement this requires significant redesign f data-structures, restructuring f the cmputatin - FLIP will bring the prmise f shared memry clser t reality

16 ACKwise vs Electrical Netwrk [PACT 2010]

17 What FLIP brings t the table Explicit bradcast with virtual channels - what des bradcast buy us? make it easy t maintain cnsistency amng distributed datastructures we can think f this as custm cache cherence at the level f a single data-structure can help the prgrammer take advantage f knwn access patterns fr example, we can avid sme cache cherence verhead if we knw that ne thread will be the single writer fr a particular data item. - dn t MPI r UPC ffer bradcasts as well? yes, but... it s a very high verhead peratin. they bth cnsider bradcast a cllective peratin requiring matching calls acrss all prcessrs bradcast implicitly

18 Shared Memry with Glbal Messages Gal: Leverage glbal bradcast t cmbine the advantages f message passing and shared memry Prgramming mdel cmbines 3 ingredients - Efficient supprt fr shared memry - Virtual bradcast channels - Hardware based message filtering

19 Shared Memry vs. Message passing Shared Memry Message Passing Advantages Advantages - Prgrammer desn t wrry abut where the data lives - Gd lcality cmes naturally - Data mvement is managed by cache cherence prtcl - Easier t avid data-races Prblems Prblems - Need t crdinate cncurrent accesses t shared state Otherwise easy t get data races - Gd perfrmance requires careful reasning abut lcality - Prgrammer needs t reasn abut where data lives - Transferring data requires marshaling and unmarshaling f ptentially cmplex state

20 Bradcast based prgramming mdel Basic mdel is like message passing - Each thread wns its data - Cmmunicatin is dne by sending messages - Enables gd lcality - Largely avids having t deal with data-races Glbal messages initialize cmmunicatin - Avids having t knw hw data is distributed - Message filtering means nt everyne needs t listen Cache cherence is used t transfer data - Avids painful marshaling and unmarshaling

21 Key design issues Bradcast vs. Unicast - Every bradcast channel knws hw many subscribers it has - As a starting pint, the decisin f whether t send ne bradcast message r multiple pint-t-pint messages will be based n a simple threshld. i.e. if channel has less than k subscribers, use pint-t-pint - A mre effective scheme wuld als take int accunt Wh the subscribers are The lad n the respective netwrks - Decisin is made at runtime

22 Key design issues Glbal Messages Vs. Shared Memry - Shared memry is better fr transferring cmplex data Avid marshaling and unmarshaling - Simple scalar values are easier t transmit with messages Avid messy hand-shakes which are needed t avid data-races when transferring simple values thrugh shared variables. Basic plicy - Use messages t cmmunicate wnership transfers - Use shared memry t mve actual data

23 Killer Applicatin: SAT This is a wrkhrse applicatin fr - EDA tls - prgram analysis and verificatin - planning, scheduling and lgistics Building blck fr new class f prgramming mdels - key enabler fr declarative prgramming - tell the machine what yu want, nt hw t d it Cmputatinally intensive prblem Very difficult t parallelize effectively

24 Applicatin: SAT Slver Sequential SAT Cnstraint database Cnstraint is a CNF Clause x 1 v x 2 v x 3 v x 4 v x 5 v x 6 x 1 - x 1 v x 7 x 5 - x 5 v x 9 v - x 4 x 7 - x 9 - x 4 - x 7 v x 6 v - x 5 - x 5 v - x 3 v x 9 - x 1 v - x 5 v x 9 x 6 - x 3 - x 6 v x 3 v x 4 x 4

25 THE END

26 Traditinal PSAT Slver Traditinal Parallelizatin Strategy - replicate the database acrss all cres - make different randm chices n each cre - share cnflict clauses

27 Shared Memry Implementatin

28 FLIP API Implementatin

29 Alternative Strategy

30 Alternative Strategy each prcessr has a fragment f the database 90% f the variables are private t the prcessr - the ther 10% are shared When a prcessr sets a shared variable, it needs t let everyne else knw This strategy is unwrkable n a traditinal architecture - each cre shares variables with 95% f the ther cres

31 New Partitined SAT Apprach