Optical technologies for data communication in large parallel systems

Transcription

1 Journal of Instrumentation OPEN ACCESS Optical technologies for data communication in large parallel systems To cite this article: M B Ritter et al View the article online for updates and enhancements. Related content - Roadmap of optical communications Erik Agrell, Magnus Karlsson, A R Chraplyvy et al. - Roadmap on silicon photonics David Thomson, Aaron Zilkie, John E Bowers et al. - Silicon photonic modulators for PAM transmissions Wei Shi, Yelong Xu, Hassan Sepehrian et al. This content was downloaded from IP address on 14/08/2018 at 13:11

2 PUBLISHED BY IOP PUBLISHING FOR SISSA TOPICAL WORKSHOP ON ELECTRONICS FOR PARTICLE PHYSICS 2010, SEPTEMBER 2010, AACHEN, GERMANY RECEIVED: November 2, 2010 ACCEPTED: December 13, 2010 PUBLISHED: January 5, 2011 Optical technologies for data communication in large parallel systems M.B. Ritter, a,1 Y. Vlasov, a J.A. Kash a and A. Benner b a IBM T.J. Watson Research Center, Yorktown Heights, NY, U.S.A. b IBM Poughkeepsie, Poughkeepsie, NY, U.S.A. mritter@us.ibm.com ABSTRACT: Large, parallel systems have greatly aided scientific computation and data collection, but performance scaling now relies on chip and system-level parallelism. This has happened because power density limits have caused processor frequency growth to stagnate, driving the new multi-core architecture paradigm, which would seem to provide generations of performance increases as transistors scale. However, this paradigm will be constrained by electrical I/O bandwidth limits; first off the processor card, then off the processor module itself. We will present best-estimates of these limits, then show how optical technologies can help provide more bandwidth to allow continued system scaling. We will describe the current status of optical transceiver technology which is already being used to exceed off-board electrical bandwidth limits, then present work on silicon nanophotonic transceivers and 3D integration technologies which, taken together, promise to allow further increases in off-module and off-card bandwidth. Finally, we will show estimated limits of nanophotonic links and discuss breakthroughs that are needed for further progress, and will speculate on whether we will reach Exascale-class machine performance at affordable powers. KEYWORDS: Data acquisition circuits; Special cables; Hardware and accelerator control systems 1 Corresponding author. c 2011 IOP Publishing Ltd and SISSA doi: / /6/01/c01012

3 Contents 1 Introduction 1 2 System bandwidth scaling 2 3 Electrical bandwidth limits 2 4 Integrated optical transceivers 5 5 Integrated silicon nanophotonics 8 6 Power and cost constraints 10 7 Summary and conclusions 11 1 Introduction High performance computing (HPC) system performance scaling has benefitted large physics experiments requiring large amounts of data to be logged in real-time. System performance in the early 1990 s was driven both by architectures allowing increased instruction-level parallelism (ILP, multiple arithmetic units, for example), and by Dennard scaling [1] which resulted in improved transistor switching speed with each lithographic generation. As a consequence, more instructions could be executed per cycle, and clock speeds were increasing. ILP advances reached a plateau in the mid 90 s, but clock speeds continued to climb until the early years of the new millennia, allowing single-thread performance, hence system performance, to continue increasing. This trend could have continued if cooling had not been an issue; however, once processor chip powers began to exceed 100 Watts, a new paradigm was adopted to continue HPC performance scaling: the era of multicore. Moore s law [2] has driven performance gains for the last six years; though the processor frequency has reached a plateau to avoid power density increases, it has been possible to cram more processor cores on the chip for each new generation of CMOS technology. Core and thread-level parallelism now allow continued system scaling. [3] However, to maintain HPC performance scaling with increased parallelism requires a concomitant increase in communication bandwidth. The purpose of this paper is to look at the limits to communication bandwidth in parallel systems and show how new technologies, like integrated silicon nanophotonics, promise to extend bandwidth scaling trends and system performance. 1

4 Figure 1. Number of signal and reference pins on a processor chip given the number of cores on a chip. Differential signaling with one reference pin per differential pair is assumed. Assumptions: 1 B/Flop memory, 0.1 B/Flop processor-to-processor bus, 0.05 B/Flop I/O (Ethernet, etc.) and an instruction-level parallelism of 4 (on average) for transmitted and received data, and 3 pins/differential pair. 2 System bandwidth scaling System designers have certain rules-of-thumb for bandwidth partitioning when architecting new machines. Bandwidth allocation does depend on applications and workloads, but we take Amdahl s rule of thumb for system balance for a parallel system as a particular case. By this rule of thumb 1 byte/flop of memory bandwidth is required for each instruction per cycle, 0.1 byte/flop for processor-to-processor data bandwidth, and 0.05 byte/flop for I/O bandwidth (to disks, Ethernet, etc., and this must be both in send and receive pins). This bandwidth scaling assumption has worked well for many generations of parallel machines to balance bandwidth between processors with the bandwidth from processor to memory and I/O. If we take Amdahl s bandwidth partitioning as a concrete case, assume 4 instructions/cycle and constant core function and clock frequency (3 GHz) due to power, we can estimate the total I/O bandwidth required per processor chip given the number of cores on that chip. Assuming an (optimistic and constant) I/O bandwidth of 10 Gb/s per differential pair, it is easy to see that the number of pins required on a chip quickly exceeds practical limits of roughly 10,000 signal and reference pins for 128 cores on a die of 400 mm 2 area (figure 1; the limit is less than 10,000 because note that not all pins can be devoted to I/O). Some hope has been expressed that 3D silicon integration will allow more cores on a die by separating cache and processing functions into different planes but our simple analysis shows that adding more cores only exacerbates these off-chip bandwidth bottlenecks. In the remainder of this paper, we will show approximate off-card and off-module bandwidth limits for practical packaging technologies, and what can be done to exceed these limits. 3 Electrical bandwidth limits What determines practical bandwidth limits of processor cards in a typical HPC system? There is no definitive answer to this question because one can always spend more money on expensive packages, boards, and connectors. However, we can get a good idea of the limits by performing detailed electromagnetic modeling of present-day high-end packaging materials and combine this 2

5 Figure 2. Cross-section of on-card processor-to-processor link. with simulations of equalized I/O on our processor chips to estimate practical maximum achievable bitrates. When we combine per-pin bitrates with physical density limitations of various hierarchies of the system packaging, we can estimate the system bandwidth bottlenecks. Only when one understands where and why bottlenecks exist can solutions be proposed to provide more bandwidth. Figure 2 shows an idealized module-to-module link on a printed circuit board. We assume that there are processor chips mounted on two modules (the first-level package) which are connected to the board by ball-grid array solder balls or demountable connectors, and communicate with each other electrically. The printed circuit board is a multi-layer structure with advanced, lowloss dielectric (Megtron 6), and controlled-impedance differential striplines connecting the two modules. Commercial electromagnetic full-wave finite-element simulators exist which can solve Maxwell s equations for this structure; however, for any practical structure the size of the problem is intractable because the combination of length scales from 100 micron wire widths to 60 centimeters wire length create too fine a finite-element mesh and hence too many variables to admit simulation in a practical time. Therefore, we divide the structure into pieces, solve each separately, then concatenate the results for these different regions. Figure 3 shows the path of figure 2 chopped into regions; each section of the model is simulated separately and concatenated to find the overall response. An example of model-hardware correlation for a 60 cm link is shown in figure 4, where we demonstrate db model-hardware correlation for through loss (S21) more than adequate for our simulation purposes [4]. To compute I/O speed limits, we employ a custom simulator that includes the computed link loss (figure 4) and a behavioral model of I/O performance, including clocking jitter, amplitude noise, and crosstalk as computed from the electromagnetic modeling. This internal simulator allows exploration of different modulation schemes and amounts of equalization. As shown in figure 5, the maximum bandwidth for this particular 60 cm link is between 20 and 25 Gb/s for NRZ signaling, with duobinary and four-level (PAM4) signaling performing more poorly than NRZ signaling with feed-forward (FFE) and decision-feedback (DFE) equalization [4]. Large servers typically have many of these processor boards interconnected through a passive backplane using electrical connectors. Models for a link from one processor card to another through a passive backplane and two connectors show a loss increase due to the four added vias (two at each connector) and the backplane striplines which limits off-card bitrates to less than 10 Gb/s. Combining the estimated limits of per-pin bitrate with geometrical escape estimates gives the bandwidth escape for each level of package. We assume the pins (C4) on a chip are limited to a pitch of 150 to 200 microns, and also assume that the module has up to 8 metal levels with wiring pitch much finer than the C4 pin pitch (a range of pitches was assumed, with some beyond 3

6 Figure 3. Electromagnetic simulation of sub-elements of the packaging path is required for tractable computations. The chip solder balls (C4), module ball grid array (BGA), land-grid array connector (LGA), and printed circuit board lines (PCB) are all separately simulated, then concatenated. Figure 4. Link loss (S21) vs. frequency (GHz) for the link with 60 cm long PCB striplines shown in figure 3. present capabilities). This allows a bandwidth escape of between 56 and 112 Tb/s beneath the chip (figure 9) on the module for a 400 mm 2 chip with 200 Watt dissipation ( 400 power and ground pins for 1V supply). Transitioning to the next level of packaging, vias in the printed circuit board (PCB) are limited to a pitch of around 1 mm due to the necessity of escaping one differential pair between two vias beneath the module. Using this fact, one can compute a module bandwidth escape limit of between roughly 12 and 29 Tb/s, depending on the number of layers in the circuit board, on the use of advanced technologies which do not require the vias to go through the entire board, and 4

7 Figure 5. Throughput (Gb/s) vs. distance for various modulation schemes for the link described in the text. on the module signal pitch (which can be made slightly smaller than 1 mm perhaps 0.8 mm). Card edge connectors have mechanical constraints (insertion force) limiting practical connectors to 8 pairs/mm card edge. Using this and per-pin data rates, escape bandwidths are limited to 8 Tb/s for a 22 cm card edge length. Summarizing our estimates, the electrical escape bandwidth limits for a server chip with power dissipation of 200 Watts, area of 400 mm 2, and advanced packaging materials (not yet standard today) gives chip escape bandwidth limits of 56 Tb/s, module escape bandwidths of 12 Tb/s, and card edge escape of 8 Tb/s (For more details on these estimates, see [4]). 4 Integrated optical transceivers Optical communication technologies have been used for more than a decade at the card edge for rack-to-rack long-distance communication where the superior distance-bandwidth product of the optical media is unchallenged. Electrical communication continues to dominate for distances of less than a few meters because electrical links still provide the necessary bandwidth at a lower cost than optical technology. However, in very high-end HPC systems, like the P7 IH Blue Waters system to be installed by IBM, available electrical off-card bandwidths are inadequate, driving denser optical packaging. Shown in figure 6a shows the P7 IH node, approximately 1 meter wide by 1.7 meters long, which has all-optical off-node communication. The optical transceivers are mounted with switch chips on 8 multi-layer ceramic (MLC) carriers, each containing up to 56 optical modules having 12 lanes of 10 Gb/s data. Each module allows the escape of 4 Tb/s data from 12.5 cm card edge, which exceeds the practical electrical escape bandwidth limit. Future systems will require ever-greater escape bandwidth, hence even denser optical technologies. One approach to achieve this employs advanced packaging to integrate two-dimensional arrays of optical transmitters and receivers in a small area. The Terabus Program at IBM Research has focused on packaging integration by using flip-chip packaging in place of traditional wirebonds on the transceivers in conjunction with polymer waveguides on cards. 2D arrays of VCSELs and photodiodes are coupled to CMOS driver and receiver circuitry. In one version [5] (figure 7a), 985 nm VCSELs emit through a transparent GaAs substrate to an integrated lens array, and are 5

8 Figure 7. Schematic cross-section (a) of 985nm integrated transceiver with OE devices (pink) flip-chip attached to CMOS transceiver IC (grey) mounted on an organic module (SLC). A lens array focuses light to/from waveguides on the printed circuit board surface (blue). Assembled 985nm Optomodule (b, left and center) and 5 3mm Optochip (b, right). The bottom view (b, center) shows an optical access hole cut in the module to allow light to pass through to the waveguides. The BGA balls are cleared to allow planar polymer waveguide access. re-focused into waveguides on the board by another lens array, with the receiver packaged in a similar manner. Another version of the transceivers (not shown) flips an 850nm VCSEL array onto a silicon driver chip with etched holes allowing the light to pass through the driver chip to a lens array (not shown) [6]. Driver and receiver array circuitry has been demonstrated operating at up to 20 Gb/s per channel over 48 channels, achieving nearly 1 Tb/s aggregate data rates in a 3 5 mm silicon footprint. Figure 7b (right) shows the assembled optochip, and to the left is a BGA package with the optochip attached. The bottom view shows an optical access hole cut in the module to allow light to pass through to the optochip with BGA balls cleared to allow planar polymer waveguide access. Employing this integration technology, one could build a module with a processor chip in the 6 Figure 6. (a) P7 IH node with optical MCM (red circle), processor MCM (blue circle) and memory. (b) An enlarged view of the 8 8 mm Avago MicroPODT M optical modules, the multi-layer ceramic (MLC) module, and the completed module assembly.

9 Figure 8. Notional design for module with optical signal escape. Figure 9. Bandwidth of on-card elements in the electrical (left) and optical (right) packaging hierarchies (this comparison does not show off-card bandwidths). center and optochips arranged around the perimeter, as shown in a notional design in figure 8. Each optochip requires a hole for the light to pass through the module to waveguides in the board, so there must be spaces between the optochips to allow wiring on the module or waveguides on the board to pass, resulting in the staggered notional design. For a 50 mm organic module, one could then escape 46 Tb/s from a module with a single layer of planar waveguides at 62.5 µm spacing, and around 100 Tb/s with two-layers of planar waveguides. To summarize, we compare the escape bandwidth limit estimates for electrical and off-chip optical module technologies in figure 9. Ranges of bandwidths are given to account for likely reductions of interconnect dimensions (line widths, C4 pitches, etc), as well as the potential increase in the number of package layers. Note that the electrical card limits in this figure do not take into account card-edge connector bandwidth limitations it is rather dealing only with the potential wiring capacity between two modules on a printed circuit board. Card edge connector bandwidths, as noted earlier, limit electrical off-card bandwidths to 8 Tb/s for a 22cm board edge. On-module optical transceivers still require electrical I/O connections to the logic chip on the 7

10 Figure 10. Notional chip with 3D integration allowing separate logic, memory, and silicon nanophotonics planes. Figure 11. Silicon nanophotonics using SOI technology. Schematic cross-section (left) shows Si waveguide surrounded by SiO 2. Actual waveguide (right) is 400 nm 200 nm. module, so that a complete EOE link includes two short electrical links with an optical link in the middle. This reduces their power efficiency (table 1), and power and cost will likely favor electrical links for short-distance communications. Nevertheless, the higher module escape bandwidth afforded by optics will drive its use for on-card communications when electrical escape bandwidths create bottlenecks. 5 Integrated silicon nanophotonics Significant research effort has been expended to achieve integration of optical transceivers in CMOS-compatible silicon technology [7 12, 14]. Figure 10 shows a notional chip which assumes 3D silicon integration enabling layers of processing elements, memory, and optical I/O (photonics plane), allowing the ultimate in integration density. Integrating optical devices in SOI technology has allowed the ultimate in silicon nanophotonics miniaturization. As shown in figure 11, optical waveguides have rectangular silicon cores surrounded by lower refractive index SiO 2, allowing 400 nm wide waveguides. The large step in refractive index allows small radius bends and small ring filter structures for wavelength-division multiplexing (WDM) applications. Employing silicon as a waveguide requires modulators and receivers operating in the 1300 to 1500 nm wavelength ranges, where silicon is transparent and, fortunately, fiber loss and dispersion are at a minimum. Since silicon is an indirect band gap semiconductor, no efficient emitters are 8

11 Figure 12. Germanium photodiode (Ge PD) deposited on top of a silicon waveguide demonstrating avalanche gain and 40 Gb/s performance. Figure 13. Integrated planar grating WDM mux/demux demonstrating temperature-insensitive 8- wavelength selectivity with a 30 40µm footprint. possible, but good modulators can be made (of course, a transmitter then requires a III-V DC light source somewhere in the system). Mach-Zender interferometer (MZI) and ring resonator modulators have been fabricated and demonstrated to operate at 1 V or less drive swings and at bitrates of 10 to 20 Gb/s [7, 11]. Because we use silicon waveguides that allow tight bend radii, the length of the MZI can be made as small as 200 µm. An efficient photodetector (PD) has been designed and fabricated utilizing pure germanium deposited on top of the silicon waveguides [14] (figure 12). Because the Ge has a lower refractive index, light is coupled out of the silicon into the Ge detector. Low-noise avalanche gain and 40 Gb/s data rates have been demonstrated with the Ge PD biased at 1 V; the speed is high because the device is very small and has a capacitance on the order of 8 ff. Finally, silicon nanophotonics allows wavelength division multiplexing, wavelength add/drop multiplexers, and optical switch elements. An example of a planar grating wavelength mux/demux demonstrating 8-channel, temperature insensitive channel selection is shown in figure 13. With each channel modulated at 20 Gb/s, this allows 160 Gb/s per waveguide, greatly improving the offchip bandwidth that can be achieved. This device is also small, measuring only 30 40µm [12]. The escape bandwidth limit of integrated photonics is limited by the three surface problem [13]. The bottom surface of future chips will be consumed with power and ground pins and a 9

12 few low-frequency control signals. Barring advances in thermal technology, the top surface of the chip will be reserved for cooling, and will likely not allow perforation for escaping signals. This leaves only the edges of the 3D silicon chip to escape optical signals. Tapered-waveguide edge couplers have been demonstrated down to pitches of 20 µm, with denser pitches possible. This would allow a maximum of 4,000 optical channels to escape from a 2 2 cm chip. If each were carrying, say, 120 Gb/s WDM data, the aggregate data rate would be 480 Tb/s. It would seem that silicon photonics would solve the problem of escape bandwidth. However, we have not yet taken into account power considerations. 6 Power and cost constraints Each link, optical or electrical, must perform certain functions. Typically, wide, relatively slow on-chip electrical buses are serialized and sent off chip at faster rates to utilize a small number of off-chip channels efficiently. In most machines, the receiving clock domain is not synchronous with the sending clock; even if it were, temperature differences between sending and receiving chips over time result in slow variations in phase that must be tracked at the receiver for robust communication. The higher line speeds and phase tracking require clock generation and recovery circuits at the sending and receiving chips, respectively, as well as serializers and deserializers. In state-of-the-art short-distance (60 cm) electrical I/O, the clocking and serialization circuits consume almost half the I/O power. Optical links require all of the clocking and serialization of electrical links, so this cannot be eliminated; therefore, though today s optical links are much more efficient for long-distance transmission, they are less efficient for short links. In table 1 we present the efficiencies of equalized electrical, optical module, and integrated nanophotonics links. These are projected efficiencies for nm CMOS nodes, for state-of-theart optics and silicon nanophotonics. From the previous discussion, even if the photonic transmitters and receivers dissipate no power, it would only save half of the power of an electrical link. For off-chip optical modules, there are two short electrical links; even if we save power by good packaging, these two short links would add up to roughly 1 3 pj/bit. In the lab, VCSEL emitters and receiver amplifier links have demonstrated efficiencies around 4 5 pj/bit [5, 6], so complete links using optical modules would consume 5 8 pj/bit. Silicon nanophotonics eliminates the power of off-chip drivers, but the clocking power remains ( pj/bit), and to that one must add MZI or ring resonator driver and receive chain amplifiers, pushing the power to 2 3 mw/gb/s. Though estimates, the numbers in the table are probably realistic, and show that optical solutions will likely consume more power than short-distance (60 cm) electrical links over well-behaved channels. The power efficiencies of realistic IO circuits set a limit on bandwidth escape. With existing cooling solutions, we will not be able to escape much more than a few 10 s of Tb/s from a single chip or a single-chip module because the total I/O power becomes prohibitive, barring unforeseen advances in I/O efficiency. To underscore this, consider building an exascale system with only 0.4 Bytes/Flop optical communication bandwidth, and assume an efficiency of 3 mw/gb/s for optical links. Such a system would dissipate 12 Megawatts, a large fraction of total system power. Therefore, reducing I/O power is a key challenge for exascale systems. Cost issues are much harder to estimate, as they depend on volume adoption of optical technologies. Rather than attempt to estimate the cost of yet undeveloped technologies, we give an idea 10

13 Table 1. The efficiency of different communication link types and the total power required for 50 Tb/s off-module data bandwidth. Link Type Efficiency (pj/bit) Distance 50 Tb/s BW Power (W) Electrical 1 3 < 1 m Optical Module 5 8 < 100 m 350 Silicon Nanophotonics 2 3 km of what system designers could afford when building an exascale system. Assuming system performance trends continue, an exascale system could be built in To make such systems affordable, optics costs in 2018 would need to drop by almost two orders of magnitude from the cost today. Such aggressive cost reductions present another challenge to optical I/O technologies which silicon manufacturing techniques may help address. 7 Summary and conclusions Optical communication technologies are seeing increasing use in HPC systems where the distancebandwidth improvement, as well as improved bandwidth escape density and power efficiency for links over 5m are essential to system performance. We have reviewed electrical performance and bandwidth escape limits at various levels of the packaging hierarchy beyond which optical technologies will become necessary. Rack-to-rack links are already optical, and the Power7 IH machine has exceeded the off-card electrical bandwidth limit, forcing this machine to use all-optical off-card communication technology. The steady advance of Moore s Law, allowing more cores per processor chip, will eventually mandate the use of integrated silicon nanophotonics to breach the module electrical escape bandwidth barrier. Challenges remain for optical technologies, and these must be addressed to assure the success of future systems. The cost of optical technology must continue its rapid takedowns to be viable for exascale machines and other large-scale data communication uses. We have shown that the promise that optical technologies allow lower communication link power is true for longer communication links, but dubious for short (on-board or on-chip) links where clocking and serialization/deserialization circuitry consume half or more of the power. Therefore, to achieve the largest-possible bandwidth escape density from chips or modules will likely require cooling technology improvements and/or radical innovations to increase I/O power efficiencies. Acknowledgments The authors gratefully acknowledge the support of DARPA under contracts MDA , HR C-0074, and HR C The views, opinions, and/or findings contained in this article are those of the authors and should not be interpreted as representing the official views or policies, either expressed or implied, of DARPA or the Department of Defense. 11

14 References [1] R. Dennard et al., Design of ion-implanted MOSFET s with very small physical dimensions, IEEE J. Solid-State Circ. 9 (1974) 256. [2] G.E. Moore, Cramming More Components onto Integrated Circuits, Electronics Magazine (1965) pp. 4. [3] K. Olukotun, L. Hammond and J. Laudon, Chip Multiprocessor Architecture, Morgan and Claypool (2007). [4] D. Kam et al., Is 25 Gb/s On-board Signaling Viable?, IEEE Trans. Adv. Pack. 32 (2009) 328. [5] F.E. Doany et al., 160 Gb/s Bidirectional Polymer-Waveguide Board-Level Optical Interconnects Using CMOS-Based Transceivers, IEEE Trans. Adv. Pkg. 32 (2009) 345. [6] F.E. Doany et al., Terabit/s-class 24-channel bidirectional optical transceiver module based on TSV Si carrier for board-level interconnects, Proceedings of ECTC (2010) 58. [7] B.G. Lee, A. Biberman, P. Dong, M. Lipson and K. Bergman, All-Optical Comb Switch for Multiwavelength Message Routing in Silicon Photonic Networks, IEEE Photon. Technol. Lett. 20 (2008) 767. [8] P. Dong, S.F. Preble and M. Lipson, All-optical compact silicon comb switch, Opt. Express 15 (2007) [9] W.M.J. Green, M. J. Rooks, L. Sekaric and Y. A. Vlasov, Ultra-compact, low RF power, 10 Gb/s silicon Mach-Zehnder modulator, Opt. Express 15 (2007) [10] Y. Vlasov, W.M.J. Green and F. Xia, High-throughput silicon nanophotonic deflection switch for on-chip optical networks, Natur. Photon. 2 (2008) 242. [11] J. Van Campenhout, W. Green, S. Assefa and Y.A. Vlasov, Low-power, 2 2 silicon electro-optic switch with 110-nm bandwidth for broadband reconfigurable optical networks, Opt. Express 17 (2009) [12] S. Assefa, F. Xia and Y.A. Vlasov, Reinventing germanium avalanche photodetector for nanophotonic on-chip optical interconnects, Nature 464 (2010) 80. [13] M. Horowitz, private communication. [14] S. Assefa et al., CMOS-Integrated 40GHz Germanium Waveguide Photodetector for On-chip Optical Interconnects, in Proceedings of OFC, paper OMR4,