Practical Limitations of State of the Art Passive Printed Circuit Board Power Delivery Networks for High Performance Compute Systems Presented by Chad Smutzer Mayo Clinic Special Purpose Processor Development Group (SPPDG) +1-507-538-5462 smutzer.chad@mayo.edu 17 th IEEE Workshop on Signal and Power Integrity (SPI) Paris, France May 14, 2013 SPPDG Archive 12345-1
SPPDG Archive 12345-2 Agenda Introduction to High Performance Compute (HPC) systems Motivation for Printed Circuit Board (PCB) passive power delivery study Design and layout of hypothetical compute PCB State of the art technologies Simulation methodology and results Technology tradeoffs and impact PCB fabrication and measurement Conclusive remarks and future guidance
SPPDG Archive 12345-3 Agenda Introduction to High Performance Compute (HPC) systems Motivation for Printed Circuit Board (PCB) passive power delivery study Design and layout of hypothetical compute PCB State of the art technologies Simulation methodology and results Technology tradeoffs and impact PCB fabrication and measurement Conclusive remarks and future guidance
SPPDG Archive 12345-4 Background High Performance Compute Systems Several high-power computation nodes (processors) arranged in a reasoned interconnect matrix to solve complex problems Components often include: processors, memory, storage, clock distribution, and multi-domain voltage regulation Performance, capability and reliability sometimes limited by deleterious effects of power supply noise Large current transients in the logic or switching noise in the I/O Excessive noise causes timing and/or signaling errors Solution options Intentionally reduce performance: slower clock, code to avoid timealigned or periodic current transients, reduce computation burden Reduce power distribution network impedance
SPPDG Archive 12345-5 Agenda Introduction to High Performance Compute (HPC) systems Motivation for Printed Circuit Board (PCB) passive power delivery study Design and layout of hypothetical compute PCB State of the art technologies Simulation methodology and results Technology tradeoffs and impact PCB fabrication and measurement Conclusive remarks and future guidance
SPPDG Archive 12345-6 Motivation Passive Power Distribution Board Frequency Domain Target Impedance Method (FDTIM) dv/di = allowable voltage ripple / expected current transient = target power distribution network (PDN) impedance Z target Championed by Larry Smith (et al.) at Sun in late 1990 s Frequently cited and implemented method for HPC PDN design Z target is frequency-independent (flat across all frequencies) Diverging trends in voltage (decreasing with IC process) and current (increasing with computational requirements) Trending toward the need for sub-milliohm PDN impedance; Z target Fundamental question: What is the lowest practical PCB PDN impedance achievable, over what frequency range, given reasonable technology constraints?
Peak Current, A Voltage, V Target Impedance, mohm TRENDS IN PROCESSOR VOLTAGE, PEAK CURRENT AND RESULTING POWER DISTRIBUTION NETWORK TARGET IMPEDANCE 250 2.5 200 Vdd Peak Current [1] Vdd Voltage [2] Target Impedance: 5% Vtol., 50% ΔI Target Impedance: 3% Vtol., 70% ΔI 2 150 1.5 100 1 50 0.5 0 0 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 Time, Year [1] Stanley-Marbell, P., Cabezas, V.C., and Luijten, R.P., "Pinned to the Walls - Impact of Packaging and Application Properties on the Memory and Power Walls," International Symposium on Low Power Electronics and Design, pp. 51-56, August 2011. [2] International Technology Roadmap for Semiconductors (ITRS): Executive Summary, 2011 Edition, http://www.itrs.net/ JUL_31 / 2012 / CMS / 43526
SPPDG Archive 12345-9 Agenda Introduction to High Performance Compute (HPC) systems Motivation for Printed Circuit Board (PCB) passive power delivery study Design and layout of hypothetical compute PCB State of the art technologies Simulation methodology and results Technology tradeoffs and impact PCB fabrication and measurement Conclusive remarks and future guidance
Design Passive Power Distribution PCB (1) Elements emulating typical processor PCB in HPC system ASIC/FPGA pinfield 45 mm x 45 mm, 1 mm pin pitch, variable number of P/G pin pairs 50, 145* or 242 Voltage regulation Quantity 8, 15 mm x 15 mm Linear Technology µmodule footprints, populated with copper shorting slugs Provisioned space for auxiliary circuits (memory, passives, etc. and mechanical) Stackup and material sets Reduced spreading inductance with variable ultra-thin dielectrics for P/G plane layers (4, 6, 12*, 14 and 50 µm) Fixed 24 total metal layers, variable P/G layer pairs (2, 4* or 6) Variable P/G copper weight (1* or 2 oz.) Variable total PCB thickness (80, 100* or 120 mils) * Denotes nominal value SPPDG Archive 12345-10
Design Passive Power Distribution PCB (2) Signal vias included to emulate typical P/G plane voiding Oval antipads for differential pairs 10 mil drill, 20 mil pad, 27 mil P/G anti-pad, 32 mil signal anti-pad Pushed technology limits of PCB fabrication vendor State of the art, commercially available decoupling capacitor technology Minimize total loop inductance for highest-frequency performance Low intrinsic self inductance, multi-terminal and reverse-geometry capacitor technology Optimized via design using via-in-pad, for low mounted inductance Highest-frequency capacitors placed nearest to the ASIC/FPGA Variable total quantity (0, 210, 271* or 311) * Denotes nominal value SPPDG Archive 12345-11
SPPDG Archive 12345-13 Agenda Introduction to High Performance Compute (HPC) systems Motivation for Printed Circuit Board (PCB) passive power delivery study Design and layout of hypothetical compute PCB State of the art technologies Simulation methodology and results Technology tradeoffs and impact PCB fabrication and measurement Conclusive remarks and future guidance
SPPDG Archive 12345-14 Simulation Metrics and Methodology Traditional PCB PDN Z(f) curve V-shaped capacitive at low frequency, inductive at high frequency Perfect short at the VRM sites resembles a series resistor and inductor (R/L) circuit Multi-step and multi-tool simulation process used to extract the effective R and L values from the Z(f) curves for a variety of PCB configurations PCB physical layout imported into 2.5D electromagnetic solver Super-port placed on all top-side ASIC/FPGA power/ground pins Frequency noted where Z(f) exceeds 1 mω Low frequency impedance recorded in flat region Inductance calculated in well-behaved, linear region
SPPDG Archive 12345-16 Simulation Relative Improvement Summary PCB Variation Relative Improvement (Ratio) Low Freq. Z Inductance 1 mω Freq. With caps vs. without caps 1.01 2.44 2.75 50 vs. 242 P/G pin pairs 1.40 2.43 2.92 2 vs. 6 P/G layer pairs 2.24 1.47 1.60 4 um vs. 50.8 um P/G dielectric thickness 1.00 1.62 1.51 1 oz. vs. 2 oz. planar copper weight 1.66 1.04 1.05 80 mil vs. 120 mil PCB thickness 1.02 1.12 1.17 Values near 1.00 indicate very little change between test cases e.g. as expected, dielectric thickness had no impact on low frequency Z Results are interdependent on other factors e.g. careful stackup design, with P/G planes near surface, influences relative improvement observed with the addition of capacitors
SPPDG Archive 12345-17 Agenda Introduction to High Performance Compute (HPC) systems Motivation for Printed Circuit Board (PCB) passive power delivery study Design and layout of hypothetical compute PCB State of the art technologies Simulation methodology and results Technology tradeoffs and impact PCB fabrication and measurement Conclusive remarks and future guidance
SPPDG Archive 12345-18 Fabrication and Measurement Methodology Fabricated practical PCB attributes based on: sensitivity analysis, fabricator guidance, cost and manufacturability 242 P/G pin pairs, 100 mil thick, 311 caps, 1 oz. copper, 6 P/G plane pairs, 12 µm dielectric Assembled in-house with vapor phase solder reflow technology Significant copper in the board created solder reflow and rework challenges recommend optimizing temperature soak / ramp profile Shunt 2-Port VNA measurement methodology [1] High-frequency micro probes applied to pads in ASIC/IC pin field Extract 2-port S-parameters and calculate self-impedance Z(f) Measured with and without shorting slugs attached to VRM pads [1] Agilent, Appl. Note 5989-5935EN, Ultra-Low Impedance Measurements Using 2-Port Measurements, February 2007
SPPDG Archive 12345-22 Measurement Results Summary Reasonable measurement correlation to simulation results using equivalent 2-port method High-resolution images of PCB cross-section helped fine-tune the model geometries Low-frequency region (appearing capacitive when VRM shorts are not present, resistive with shorts attached) nearly identical to simulation Elements influencing intrinsic-capacitance and resistance were modeled with great accuracy Minimal discrepancy in modeled vs. measured inductance resulting in differences with the self resonant frequency and high-frequency impedance visible in measurements with and without decoupling capacitors Does not negatively influence report conclusions
SPPDG Archive 12345-23 Agenda Introduction to High Performance Compute (HPC) systems Motivation for Printed Circuit Board (PCB) passive power delivery study Design and layout of hypothetical compute PCB State of the art technologies Simulation methodology and results Technology tradeoffs and impact PCB fabrication and measurement Conclusive remarks and future guidance
SPPDG Archive 12345-24 Conclusions and Future Direction (1) What is the lowest practical PCB PDN impedance achievable, over what frequency range, given reasonable technology constraints? Low frequency (less than ~1 MHz): 100-200 µω VRM, DC resistance, copper weight, plane count, bulk decoupling Mid frequency (~ 1 MHz 10 MHz): 300-500 µω Vias, mid-range capacitor selection and placement, stackup High frequency (> 10 MHz): Increasing proportionally with frequency Vias, capacitor technology and placement, plane location, dielectrics Relying on package and chip decoupling for impedance reduction
SPPDG Archive 12345-25 Conclusions and Future Direction (2) Need for frequency-dependent target impedance based on predictable (or partially-predictable) load transients Seeing some progress in recent publications DesignCon, IEEE, etc. Prevent over- or under-design
SPPDG Archive 12345-26 THANK YOU! Practical Limitations of State of the Art Passive Printed Circuit Board Power Delivery Networks for High Performance Compute Systems Chad Smutzer smutzer.chad@mayo.edu 17 th IEEE Workshop on Signal and Power Integrity (SPI) May 14, 2013
SPPDG Archive 12345-27 Backup Slides