A Nanophotonic Interconnect for High- Performance Many-Core Computation

Transcription

1

2 A Nanophotonic Interconnect for High- Performance Many-Core Computation Ray Beausoleil Quantum Optics Research Group Information and Quantum Systems HP Laboratories 008 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice

3 Collaboration Ray Beausoleil, Jung Ho Ahn, Nate Binkert, Al Davis, David Fattal, Marco Fiorentino, Norm Jouppi, Ted Kamins, Moray McLaren, Bill Munro, Charlie Santori, Rob Schreiber, Sean Spillane, Tim Spiller, Dana Vantrease, Stan Williams, and Qianfan Xu HP Laboratories Bob Bicknell and Pavel Kornilovich Technology Development Operation, HP Mark Brongersma, Shanhui Fan, and David Miller Stanford University Eli Yablonovitch UC Berkeley John Bowers UCSB Alan Willner USC 3 April 9, 008 Ray Beausoleil - HP Laboratories

4 Overview Research context Architecture Nanophotonic systems and subsystems Microring resonators System performance April 9, 008 Ray Beausoleil - HP Laboratories

5 Research context

6 Multicore Many-core The multicore era is already here Core scaling gets worse with lithography n # of cores (C) increases as O(n ) Computational bandwidth increases as O(n ) linear in C Communication bandwidth is limited by interconnect physics to O(n) Parallel processing performance is severely limited by poor memory and interconnect bandwidth Nanophotonics is a credible solution in the next decade 0 00x higher communication bandwidth System performance increase of 0x over electrical interconnects ( 5 extra Moore s Law generations) Energy/bit improvements: 3x on chip, 0 30x off chip But pins are the key consideration for performance! Enables new research: all-to-all programmable architectures 6 April 9, 008 Ray Beausoleil - HP Laboratories

7 The Chip: Performance (GOPS) Moore s Gap Transistors The GOPS Gap SMT, FGMT, CGMT OOO Superscalar Pipelining Diminishing returns from single CPU mechanisms (pipelining, caching, etc.) Wire delays Power envelopes Anant Agarwal, MIT time 5

8 Possible architectures? Replace copper global interconnects w/optics Packet-switched, bus Circuit-switched: mesh/torus Supports known programming models Latency still an issue Parallelism buried by code/data locality All-to-all single-hop cross-bar Completely infeasible using copper Mandates DWDM Programming: parallelism at the highest possible level Changes the architectural research landscape 8 April 9, 008 Ray Beausoleil - HP Laboratories

9 Possible architectures? Replace copper global interconnects w/optics Packet-switched, bus Circuit-switched: mesh/torus Supports known programming models Latency still an issue Parallelism buried by code/data locality All-to-all single-hop cross-bar Completely infeasible using copper Mandates DWDM Programming: parallelism at the highest possible level Changes the architectural research landscape 9 April 9, 008 Ray Beausoleil - HP Laboratories

10 DWDM is inevitable High 7 nm technology node (07?): 5 GHz clock frequency 0 00 W dissipated in Si 0 Terabytes/sec bidirectional interconnect 50 W byte/flop from cores to DRAM How many active physical channels do we need? 0 Gbps (w/o SerDes) 56 waveguides using 6 wavelengths each -5 photonic crystal fibers w/60 optical cores each But there will be 638 x 6 channels to support reconfigurable all-to-all connections! How many layers do we need for an all-to-all crossbar w/o DWDM? Assume 6 clusters of cores on a 3 cm die Assume each electronic wire occupies cm x 3 µm (average) 638 x 6 x cm x 3 µm/3 cm = 0 layers Compromise: use a mesh/torus Large latencies lead to data locality and other programming problems Significantly increased power dissipation due to switches and repeaters 0 April 9, 008 Ray Beausoleil - HP Laboratories

11 DWDM is inevitable Resistance is futile High 7 nm technology node (07?): 5 GHz clock frequency 0 00 W dissipated in Si 0 Terabytes/sec bidirectional interconnect 50 W byte/flop from cores to DRAM How many active physical channels do we need? 0 Gbps (w/o SerDes) 56 waveguides using 6 wavelengths each -5 photonic crystal fibers w/60 optical cores each But there will be 638 x 6 channels to support reconfigurable all-to-all connections! How many layers do we need for an all-to-all crossbar w/o DWDM? Assume 6 clusters of cores on a 3 cm die Assume each electronic wire occupies cm x 3 µm (average) 638 x 6 x cm x 3 µm/3 cm = 0 layers Compromise: use a mesh/torus Large latencies lead to data locality and other programming problems Significantly increased power dissipation due to switches and repeaters April 9, 008 Ray Beausoleil - HP Laboratories

12 DWDM is inevitable Resistance is futile High 7 nm technology node (07?): 5 GHz clock frequency 0 00 W dissipated in Si 0 Terabytes/sec bidirectional interconnect 50 W byte/flop from cores to DRAM How many active physical channels do we need? 0 Gbps (w/o SerDes) 56 waveguides using 6 wavelengths each -5 photonic crystal fibers w/60 optical cores each But there will be 638 x 6 channels to support reconfigurable all-to-all connections! How many layers do we need for an all-to-all crossbar w/o DWDM? Assume 6 clusters of cores on a 3 cm die Assume each electronic wire occupies cm x 3 µm (average) 638 x 6 x cm x 3 µm/3 cm = 0 layers Compromise: use a mesh/torus Large latencies lead to data locality and other programming problems Significantly increased power dissipation due to switches and repeaters April 9, 008 Ray Beausoleil - HP Laboratories

13 DWDM is inevitable Resistance is futile High 7 nm technology node (07?): 5 GHz clock frequency 0 00 W dissipated in Si 0 Terabytes/sec bidirectional interconnect 50 W byte/flop from cores to DRAM How many active physical channels do we need? 0 Gbps (w/o SerDes) 56 waveguides using 6 wavelengths each -5 photonic crystal fibers w/60 optical cores each But there will be 638 x 6 channels to support reconfigurable all-to-all connections! How many layers do we need for an all-to-all crossbar w/o DWDM? Assume 6 clusters of cores on a 3 cm die Assume each electronic wire occupies cm x 3 µm (average) 638 x 6 x cm x 3 µm/3 cm = 0 layers Compromise: use a mesh/torus Large latencies lead to data locality and other programming problems Significantly increased power dissipation due to switches and repeaters 3 April 9, 008 Ray Beausoleil - HP Laboratories

14 DWDM is inevitable Resistance is futile High 7 nm technology node (07?): 5 GHz clock frequency 0 00 W dissipated in Si 0 Terabytes/sec bidirectional interconnect 50 W byte/flop from cores to DRAM How many active physical channels do we need? 0 Gbps (w/o SerDes) 56 waveguides using 6 wavelengths each -5 photonic crystal fibers w/60 optical cores each But there will be 638 x 6 channels to support reconfigurable all-to-all connections! How many layers do we need for an all-to-all crossbar w/o DWDM? Assume 6 clusters of cores on a 3 cm die Assume each electronic wire occupies cm x 3 µm (average) 638 x 6 x cm x 3 µm/3 cm = 0 layers Compromise: use a mesh/torus Large latencies lead to data locality and other programming problems Significantly increased power dissipation due to switches and repeaters April 9, 008 Ray Beausoleil - HP Laboratories

15 DWDM is inevitable Resistance is futile High 7 nm technology node (07?): 5 GHz clock frequency 0 00 W dissipated in Si 0 Terabytes/sec bidirectional interconnect 50 W byte/flop from cores to DRAM How many active physical channels do we need? 0 Gbps (w/o SerDes) 56 waveguides using 6 wavelengths each -5 photonic crystal fibers w/60 optical cores each But there will be 638 x 6 channels to support reconfigurable all-to-all connections! How many layers do we need for an all-to-all crossbar w/o DWDM? Assume 6 clusters of cores on a 3 cm die Assume each electronic wire occupies cm x 3 µm (average) 638 x 6 x cm x 3 µm/3 cm = 0 layers Compromise: use a mesh/torus Large latencies lead to data locality and other programming problems Significantly increased power dissipation due to switches and repeaters 5 April 9, 008 Ray Beausoleil - HP Laboratories

16 Architecture

17 Architecture and programming Multicore architectures are increasingly bandwidth constrained and difficult to program Algorithms must be adapted to achieve high levels of code and data locality Nonuniform memory latency, bandwidth, and data distribution significantly inhibits optimization Programmers must be aware of physical placement of program and data We want a completely symmetric architecture in which all memory is close to all processors Core-to-core transfer is faster and more uniform Uniformly high memory bandwidth from any core to all of memory Programmers can ignore low-level physical details Programmers can express parallelism using high abstraction levels 7 April 9, 008 Ray Beausoleil - HP Laboratories

18 High-level design considerations Clusters of cores Each core has private L cache Each cluster has private L cache No cluster-cluster or same-level cache-cache communication One cluster s L cache communicates with one memory controller at a time one or more memory controllers per optical channel No SerDes: synchronized operation Minimal broadcast requirements Low complexity scalability High-speed network reconfiguration for low latency DIMM may be off-chip Core Fiber I /O s to OCMs or Netowrk Through Silicon Vias Heat sink Processor /L Die Memory Controller /Directory /L Die Analog Electronics Die Optical Die 8GB DRAM (option ) 8GB DRAM (option ) 8GB DRAM (option ) 8GB DRAM (option ) Package D M DIMM DIMM DIMM D Core Core 3 Core Cluster s L Cache Core 5 Core 6 Core 7 Core 8 Cluster s L Cache CPU Optical Crossbar/Ring M N Memory Controller Memory Controller Memory Controller M N = C Core N 3 Face to Face Bonds Core N Face to Back Bonds Core N Core N Cluster C s L Cache 8 April 9, 008 Ray Beausoleil - HP Laboratories

19 Nanophotonic systems and subsystems

20 07 Design: 56 cores L-I L-I Core 0 Core L _ L Interface L-D L-D L-D L-D L-I L-I Core Through Silicon Via Array Core 3 L Cache L _ L Interface Direct ory Hub MC M y X-bar Connection NI P eer X-bar Connection Modulators Splitters Detectors Off-Chip Modulators Broadcast Detectors Splitters -waveguide bundles Modulators Modulators Modulators Modulators 0 3 Data & Control Modulators Detectors Modulators Detectors Modulators Modulators Modulators N- N N Arbitration Star coupler Star coupler Laser 0 April 9, 008 Ray Beausoleil - HP Laboratories 6 variable drop filters / detectors fixed drop filters

21 Data & control block 0 Modulator Modulator Modulator 3 Modulator -waveguide bundles N- N N+ Modulator Detector Detector Modulator 900um 6 Modulator 6 63 Modulator Modulator 55um 50um 5um um.5um A bundle of waveguides (56 channels) is dedicated to each computational component: data (6), control (3), application (8) Each logical channel includes a gated clock for data acquisition at the receiver 57.5 um April 9, 008 Ray Beausoleil - HP Laboratories

22 Optical arbitration example Epoch Componen t Frequency ML Laser Component 3 Comp Bid Win Component Component 3 Component 3 April 9, 008 Ray Beausoleil - HP Laboratories

23 Optical arbitration example Epoch Componen t Frequency ML Laser Component 3 Comp Bid Win Component Component 3 Component 3 3 April 9, 008 Ray Beausoleil - HP Laboratories

24 Optical arbitration example Epoch Componen t Frequency ML Laser Component 3 Comp Bid Win Component Component 3 Component 3 April 9, 008 Ray Beausoleil - HP Laboratories

25 Optical arbitration example Epoch Componen t Frequency ML Laser Component 3 Comp Bid Win Component Component 3 Component 3 5 April 9, 008 Ray Beausoleil - HP Laboratories

26 Optical arbitration example Epoch Componen t Frequency ML Laser Component 3 Comp Bid Win Component Component 3 Component 3 6 April 9, 008 Ray Beausoleil - HP Laboratories

27 Optical arbitration example Epoch Componen t Frequency ML Laser Component 3 Comp 3 Bid Win Component Component 3 Component 7 April 9, 008 Ray Beausoleil - HP Laboratories

28 Optical arbitration example Epoch Componen t Frequency ML Laser Component 3 Comp 3 Bid 3 Win Component Component 3 Component 8 April 9, 008 Ray Beausoleil - HP Laboratories

29 Si microring resonators

30 Compact Microring design Silicon-on-insulator (50 nm Si, 3 µm BOX) Target Q = 0,000 unloaded (0,000 loaded) Small-ring / waveguide coupling issues solved c i s n i r t n I R=.5 µm Traditional design Mode mismatch Poor coupling High radiation loss Radius (µm) R=.5 µm Optimized Narrower guide Mode-matched 30 April 9, 008 Ray Beausoleil - HP Laboratories

31 Microring Network 5 cascaded microring resonators, slightly different radii ~.5 µm. High Q of 9,000 (BW ~ 0 GHz) and high extinction ratio of 6 db. 0.7 nm b Q. Xu, D. Fattal, and RGB, Opt. Express 6, (008) 3 April 9, 008 Ray Beausoleil - HP Laboratories

32 Nanoimprint Lithography (NIL) Cost effective No expensive high-na UV optics R&D tool costs: $.6M High throughput: 0 9 PhC holes in 60 seconds Recent advances Liquid monomer instead of soft polymer Mold & substrate parallel to within µr Ultimate benefit: Moore s Law for integrated optics(?) Start Imprint & cure Liftoff Residual material RIE Mold/Template Polymer Substrate UV 3 April 9, 008 Ray Beausoleil - HP Laboratories

33 Nano-imprinted Rings E-beam litho: SLOW, good for design, testing Nano-imprint = Road to commercialization NIL microring resonator Direct EBL microring resonator 33 April 9, 008 Ray Beausoleil - HP Laboratories

34 NIL typical performance NIL R =.5 µm Q = 9,00 NIL R = µm Q =,500 Radius Nano-Imprint Coupled Q ER E-beam Litho Coupled Q ER Matches EBL.5 µm 9,00 0 db 9,600 6 db within 0%! µm,500 db,500 7 db 3 April 9, 008 Ray Beausoleil - HP Laboratories

35 System performance

36 Interconnect power & energy Power 7 nm (0 TB/s) 3 W in the laser 0 W in the active modulators 5 W in the inactive rings Manufacturing imperfections Heating: red shift Current: blue shift (only available for 0.7-nm CD control!) Nanoimprint lithography: could reduce power to 5 W Thermal stabilization: minimize the ring size! Energy budget 90 fj/bit (90 µw/gbps) in crossbar 90 fj/bit (90 µw/gbps) direct 36 April 9, 008 Ray Beausoleil - HP Laboratories

37 Benchmark performance at 7 nm Benchmark PTRANS (GB/s) STREAM (GB/s) GUPS DGEMM (Gflops) FFT (Gflops) MPI (GB/s) Optical Performance Multi-threaded SMP with data imperfectly placed Optical model 50 W dissipated in the entire interconnect Ring crossbar connecting clusters, and optical interconnect to off-chip memory Electrical model On-chip mesh network power-limited to 50 W Pin-limited bandwidth to memory (ITRS 7-nm technology node) 37 April 9, 008 Ray Beausoleil - HP Laboratories 7 nm Technology Node Optical Performance per Watt Electrical Performance Electrical Performance per Watt Optical/Electrical Performance 9 9 9

38 Challenges and research directions Large-scale photonic integration Fabrication, integration, and process compatibility Microring fabrication tolerances: nanoimprint lithography? Die stacking Power-efficient resonator locking Receiverless SiGe detectors with C ~ 0 ff On-chip hybrid mode-locked lasers (or off-chip comb lasers?) Thermal regulation & compensation Small devices + heat spreaders mitigate intra-device thermal gradients Analog + thermal + optical co-design essential Signal processing: Telecom rules of thumb no longer apply L. Zhang, J. -Y. Yang, M. Song, Y. Li, B. Zhang, RGB, and A. E. Willner, Opt. Express 5, 56 (007) No single group will be able to meet all of these challenges! 38 April 9, 008 Ray Beausoleil - HP Laboratories

39