DesignCon A Tale of Long Tails. Dai Fen, Huawei Mike Harwood, HSZ Consulting, Ltd.

Transcription

1 DesignCon 2010 A Tale of Long Tails Dai Fen, Huawei daifen@huawei.com Mike Harwood, HSZ Consulting, Ltd. mike@hszconsulting.com Huang Chunxing, Huawei huangchunxing@huawei.com Mike Steinberger, SiSoft msteinb@sisoft.com

2 Abstract Serial channels over a lossy cable path can have a much higher bit error rate than channels which use the same pin electronics over a PC board path with approximately the same loss. The problem is that the internal impedance of the cable conductors causes the impulse response of the channel to decay very slowly, resulting in a long tail effect. This paper presents system measurements illustrating the long tail effect, a review of the physics involved, a description of five different methods for compensating for the long tail effect, and a performance analysis of these solutions. Authors Biographies DAI FEN joined Huawei Technologies in She is responsible for high-speed Backplane design. Her most current activities is focusing on rack to rack interconnection design. She received her Bachelor degree in Instrumentation and Optoelectronics Engineering from Hefei University of Technology in MIKE HARWOOD is a Consulting Engineer at HSZ Consulting and a visiting research fellow at Oxford Brookes University. Before co-founding HSZ Consulting, Mike Harwood was the main technical architect of TI's latest, highest speed and technically challenging SerDes - the last one being a 12.5Gb/s long-reach SerDes on a 65nm process. He has worked on over fifteen successful mixed-signal designs on CMOS processes with process geometries from 3um to 65nm. He was elected a Distinguished Member of Technical Staff at TI in Mike graduated in Physics from Trinity College, Oxford. HUANG CHUNXING joined Huawei Technologies in He is responsible for highspeed end-to-end interconnection simulation and measurement techniques within Highspeed Interconnect Research Group. His background includes realizing high-speed serial link Stateye simulation flow in ADS. His most current activities is focusing on next generation long reach serial link research work. He received his Master degree in communications and info. system from Nanjing University of Science&Technology in MIKE STEINBERGER is currently responsible for leading the development of SiSoft's serial link analysis products. He has over 30 years experience in the design and analysis of very high speed electronic circuits. Prior to joining SiSoft, Dr. Steinberger worked at Cray Inc., where he designed very high density interconnects and increased the data rate and path lengths to the state of the art. Mike holds a B.S. from the California Institute of Technology and a Ph.D. from the University of Southern California, and has been awarded 13 U.S. patents. A Tale of Long Tails SiSoft, 2010 Page 2

3 System Impact With the increasing demand for switch capacity, it is quite normal that two or more equipment enclosures (or chassis) are linked together to achieve much higher switch capacity. Chassis to chassis interconnects are becoming an important part of a signal integrity engineer s responsibilities, in addition to the more typical card to card interconnects. As a first estimate, SI engineers determine whether a high speed serial link is feasible by comparing the channel loss (db) at half the data rate to the loss compensation ability of a SerDes. This method may not be reliable, however, if SerDes performance determined for card to card interconnects is applied to chassis to chassis interconnects. Link performance was measured for three different configurations: Card to Card Interconnect: 16inch N inch FR4 + 2 ZD Connectors, as shown in Figure 1 Chassis to Chassis Interconnect: 10m QSFP Passive Cable (24AWG) + 10inch N as shown in Figure 2; Cable-only: 10m QSFP Passive Cable (24AWG) as in Figure 2, but without the card interconnect. The same SerDes was used in all three configurations, and 50mV of interference was injected to simulate the effects of crosstalk. Figure 1: Card to Card Interconnect Case A Tale of Long Tails SiSoft, 2010 Page 3

4 Figure 2: Chassis to Chassis Interconnect Case The differential insertion loss and return loss of the channels is shown in Figure 3. The differential insertion loss is on the left hand side and the differential return loss is on the right hand side. The red curves are for the backplane interconnect channel and the blue curves are for the chassis to chassis interconnect channel. The backplane channel insertion loss is 18.6dB and chassis to chassis cable channel insertion loss is 19.9dB for a data rate of 6.0 Gb/s. Figure 3: Insertion/Return Loss of Two Channels The results for all three configurations are shown in Table 1. For the backplane channel, all BER measurements met the design requirements whereas none of chassis to chassis interconnect measurements passed. It is clear that when a SerDes mainly designed for backplane applications is used in a chassis to chassis cable application, its performance is severely degraded. A Tale of Long Tails SiSoft, 2010 Page 4

5 Testing Result with 50mv Interference 6.25Gbps PRBS7 6.25Gbps PRBS31 7.5Gbps PRBS7 Card to Card Pass Pass Pass Interconnect Chassis to Chassis Fail Fail Fail Interconnect 10m QSFP cable(- Pass Fail Fail Table 1. Testing Results of Two Channels Note the degradation in performance when using patterns with long run length. This performance degradation has been observed in other SerDes testing. Two questions may be raised here: what caused this performance degradation and how could we work it out to make 10m QSFP Passive Cable (24AWG) + 10inch N application realizable? Physics The difference between a PC board path and a cable path lies in the primary loss mechanisms for the respective paths. Whereas for frequencies above about 1 GHz, the primary loss mechanism in a PC board is almost always dielectric loss, the primary loss mechanism in a cable is almost entirely conduction loss. In other words, PC board dielectrics almost always have a much higher dielectric loss tangent than the dielectrics used in cables. This difference would imply that whereas the loss of a PC board trace should be more or less linear with frequency, the loss of a cable should vary as the square root of the frequency; and this observation is consistent with the data in Figure 1. The difference between a linear loss curve and a square root loss curve is not, however, sufficient to explain the observed difference in path performance. The difference in performance is due to the internal impedance of the signal conductors. As explained in [1, section 4.5], current penetrates conductors at low frequency, and this penetration results not only in skin effect resistance, but in an additional inductance due to the magnetic fields inside the conductor. This so-called internal impedance causes the group delay of the transmission line to increase significantly at very low frequencies. In copper, at frequencies above about 100 khz, the skin resistance per unit length and the internal inductive reactance per unit length are essentially equal. That is, A Tale of Long Tails SiSoft, 2010 Page 5

6 The internal conductance can be explicitly calculated for many different conductor geometries by considering the proportion of current flowing within the bulk of the conductor. This gives rise to an internal magnetic field where and I is the current flowing within the geometry defined by the line integral. Integrating the magnetic field to calculate the inductance often becomes problematic, but [1, section 2.17] outlines a simpler method using energy methods. Alternatively, Maxwell's equations [1, section 3.17] establish that the internal impedance and internal resistance must necessarily be equal. Since the internal resistance can be accurately approximated, the internal inductance as a function of frequency can be similarly approximated. For a co-axial cable the internal inductance can easily contribute 15% towards the total lowfrequency inductance and results in a significant increase in low-frequency group delay. [1, section 4.5] also gives the equations which demonstrate that at low frequencies, the resistance converges to its DC value and the inductance goes to zero. [2] provides a useful approximation for the transition between the low frequency and high frequency approximations. Define the internal impedance per unit length as Then given the DC resistance r dc per unit length, the internal impedance per unit length is approximately While the above equations are all that s required to include the internal impedance in a transmission line model, they don t give a great deal of insight into the physical principles that cause the skin resistance and the internal inductive reactance to be approximately equal. [3] provides another point of view. In [3], the skin depth of a conductor is derived as a corollary to the calculation of the dielectric constant of an insulator. In an insulator, the dielectric constant is caused by resonant modes of electrons bound to molecules. Each such mode has a resonant frequency and a quality factor or Q. The combined interaction with these modes is sufficient to explain both the dielectric loss tangent and the frequency dependence of the dielectric constant. This model of the physics of a dielectric is also used in a widely referenced paper on lossy dielectrics [4]. When in [3] the treatment is applied to a good conductor, the only change is that the conduction electrons are not bound to any one molecule. Therefore, there is only one resonant mode that s important, and its resonant frequency is zero. The Q of this mode can only be zero, and so the loss tangent of the conductor is 1.0 and the real and A Tale of Long Tails SiSoft, 2010 Page 6

7 imaginary parts of the conductor s dielectric constant are equal. The imaginary part of this dielectric constant causes the skin resistance and the real part causes the internal reactance. Baseline Results Using SiSoft Quantum Channel Designer high speed serial channel simulator, the equations in [1] through [4] were applied to transmission line models with lengths equal to those depicted in Figures 1 and 2 in such a way as to reproduce the transmission loss shown in Figure 3. Connectors and other discontinuities were not included, and so the conductor dimensions were probably a little smaller and the dielectric loss tangent a little larger than those in the actual system. The resulting transmission loss is shown in Figure 4. Figure 4: Idealized transmission path losses Figure 5 compares the impulse responses of the two transmission paths. In this figure, the impulse response of the chassis to chassis (to be referred to as cable ) transmission path was then shifted in time so that the peak of its impulse response lines up with the peak of the impulse response from the card to card (to be referred to as PCB ) transmission path. A Tale of Long Tails SiSoft, 2010 Page 7

8 Figure 5: Transmission path impulse responses In Figure 5, the impulse response for the cable path is narrower than that of the PCB path, and so one would suppose that the cable path would provide higher performance than the PCB path. The expanded view in Figure 6, however, shows that the tail of the cable path s impulse response has a greater amplitude of the that for the PCB path. It is actually the cumulative effect of this tail over many bit times that is the dominant effect. This is the origin of the term long tail effect. Figure 6: Transmission path impulse response tails One way to demonstrate the long tail effect is to stimulate the path with data with a long run length. In Figures 7 and 8, a data pattern consisting of a repeating sequence of seven ones and one zero is followed by a repeating sequence of seven zeros and a single one. Figure 7 is measured data and Figure 8 is a simulation for the cable and PCB paths. A Tale of Long Tails SiSoft, 2010 Page 8

9 Figure 7: Measured long run length data pattern Figure 8: Simulated long run length data pattern for the cable and PCB paths Because of the long tail effect, a long run of identical bits will cause the voltage at the end of the path to drift toward its DC value. When a single bit of the opposite value is inserted into the bit stream, this bit will have the same amplitude as any other isolated bit, but it will be starting from a baseline which is further away from the decision threshold, making it more difficult for the signal to cross the decision threshold and cause the bit to be detected correctly. Because of this behavior, this phenomenon was known for many years at Cray Research as the lonely pulse effect. A Tale of Long Tails SiSoft, 2010 Page 9

10 To determine whether the long tail effect could explain the system level results, statistical and time domain simulations were run using transmit de-emphasis tap settings of (0.7, , ). Figure 9 is a comparison of the equalized pulse responses. Figure 9: Equalized pulse responses for the cable and PCB paths From Figure 9, it is clear that while the equalization is enough for the PCB path, it is not quite enough for the cable path. Time domain simulations were run with two different PRBS patterns: PRBS7 and PRBS22. While a simulation with the complete PRBS31 pattern would be desirable, the PRBS22 pattern was chosen as having a run length that is enough longer than the PRBS7 pattern to illustrate the effect of long run lengths while requiring only a reasonable simulation execution time. Figure 10 shows the persistent eyes for the PRBS7 pattern and Figure 11 shows the persistent eyes for the PRBS22 pattern. Figure 12 shows the statistical eyes for the two cases. This represents a uniform sampling of the intersymbol interference population over all possible messages of length 256. Figure 10: Eye patterns for PRBS7 data pattern. PCB path (left), Cable path (right) A Tale of Long Tails SiSoft, 2010 Page 10

11 Figure 11: Eye patterns for PRBS22 data pattern. PCB path (left), Cable path (right) Figure 12: Statistical eyes. PCB path (left), Cable path (right) As a baseline measurement, the eye heights and eye widths in Figures are summarized in Table 2. Eye Height (V) Eye Width (ps) PCB path Cable path PCB path Cable path PRBS PRBS Statistical Table 2: Eye heights and eye widths for baseline case The simulated results in Table 2 are in at least qualitative agreement with the measured system results in Table 1. Solutions There are a number of ways to compensate for the long tail effect. Some are based on the nature of the impulse response as described above, some are based on the equivalent observation that the loss decreases rapidly at low frequencies, and one is based directly on the physical source of the phenomenon. In the following sections, the performance of each solution will be characterized by the eye diagram for both the PCB and cable paths, for a PRBS22 pattern. The eye heights and eye widths are summarized in Table 3 at the end of the paper. 8B10B Encoding There are some forms of data encoding, notably 8B10B encoding [5], that truly limit the run length of the data. Data encoded in this way avoids the long tail effect since there are in fact no long runs to excite the low frequency behavior in the transmission path. A Tale of Long Tails SiSoft, 2010 Page 11

12 Figure 13: PRBS22 eye diagrams with 8B10B encoding. PCB path (left), Cable path (right). This solution has the advantage that it is already implemented on a number of transmission paths as a way to enable AC coupling. Its primary disadvantage is that it reduces data throughput by 20%. Transmit Over-Equalization Most systems include a form of transmit equalization that some call de-emphasis. That is, the transmitter implements a synchronously spaced tapped delay line, with the net result that the spectral content at frequencies below one half of the bit rate is reduced, or in other words de-emphasized. If the de-emphasis is increased past the level needed to equalize the channel, then in particular the spectral content at the lowest frequencies will be reduced the most. If a receive equalizer such as a decision feedback equalizer (DFE) is then used to compensate for the general level of over-equalization, the frequency response at the lowest frequencies can be de-emphasized enough to compensate for the long tail effect. One way to understand this solution is to note that the long tail on the channel impulse response resembles an exponential decay, and that as seen at the receiver pad, what the transmit post cursor taps do is to subtract a delayed, attenuated version of the channel impulse response from the channel impulse response due to the main tap. If the ratio of the magnitude of the first post cursor tap to the magnitude of the main tap is equal to the exponential decay that occurs in the long tail over the course of one bit time, then the contribution of the first post cursor tap will cancel out the long tail. This effect is most easily seen in the step response, since the step response puts more emphasis on the low frequency response. Figure 14 compares the step response at the receiver pad for the cable path with and without (0.6, -0.4) transmit equalization. Note that in the equalized response, the long tail effect has been essentially canceled out, leaving an over-equalized response in the first post cursor tap position. A Tale of Long Tails SiSoft, 2010 Page 12

13 Figure 14: Cable path step response with and without transmit over-equalization To evaluate this solution, the transmit de-emphasis was increased from (0.7, , ) to (0.6, -0.4, -0.0) and DFE was enabled at the receiver. The resulting eye diagrams are shown in Figure 15. Figure 15: PRBS22 eye diagrams with transmit over-equalization and DFE. PCB path (left), Cable path (right). While this solution is readily available, it comes at a cost. Consider that de-emphasis has the general effect of reducing the signal amplitude delivered to the receiver. Since receivers have limited gain, this reduction in signal amplitude represents an impairment in itself. The power dissipation and complexity of the DFE must also be considered as part of the cost of the solution. Self-Equalizing Cable There is commercially available cable which uses a silver plated steel center conductor rather than the more typical silver plated copper. This cable has the characteristic that the current is constrained to flow in the silver plating and there is negligible current flow in the steel core. At frequencies for which the silver plating is less than one skin depth thick, the internal impedance of the conductor is therefore essentially constant. This structure was originally designed to create a cable which has constant loss as a function of frequency; however, it also has the effect of creating a cable which does not exhibit the long tail effect. A Tale of Long Tails SiSoft, 2010 Page 13

14 The exact plating thickness for these cables is not published; however, calculations using a plating thickness of 3um (120 micro-inches) and a center conductor diameter of 0.01 inches seems to reproduce the published characteristics of the cable. Figure 16 compares the transmission loss of the cable path without and with self-equalizing cable. Figure 16: Cable path transmission loss without (RED) and with (BLUE) self equalizing cable Figure 17 shows the impulse responses for the cable path without and with selfequalizing cable. From this Figure, the reduction in the long tail effect is readily apparent. Figure 17: Cable path impulse response without (RED) and with (BLUE) self equalizing cable With the self equalizing cable, somewhat less equalization is required in the transmitter, with the optimum transmit equalization being approximately (0.725, -0.25, ). This A Tale of Long Tails SiSoft, 2010 Page 14

15 equalization was applied to both the PCB path (which has no cable at all, self-equalizing or otherwise) and the cable path with self-equalizing cable. The resulting eye diagrams are shown in Figure 18. Figure 18: PRBS22 eye diagrams with self-equalizing cable. PCB path (left), Cable path (right) The advantages of this solution are that it does automatically and effectively eliminate the long tail effect for any cable length and is readily available commercially. The disadvantages are that the cable media is more expensive, cable assembly is more difficult/expensive, and the cable is stiffer than conventional cable and therefore more difficult to route in a chassis. It should also be noted that the higher DC resistance results in lower signal amplitude at lower frequencies, and could also affect the DC operating point for CML type electrical interfaces. Passive Network In the demonstration of the lonely pulse effect in Figure 8, it appears as though the signal baseline has an exponentially rising shape for a long string of ones and an exponentially falling shape for a long string of zeros. One way to look at long tail compensation, therefore, is to apply an exponentially falling shape for a long string of ones and vice versa for a long string of zeros. AC coupling exhibits exactly such a response, and therefore if the AC coupling time constant were carefully chosen, that might reduce the long tail effect. The idea of AC coupling also makes sense when viewed in the frequency domain. It is clear from Figure 4 that the loss changes much more rapidly at lower frequencies than it does at higher frequencies, and so maybe the response would be better behaved if it changed at a more uniform rate. This line of thinking suggests that maybe some additional loss should be inserted at low frequencies and DC, but that maybe that loss should not be as extreme as would be introduced by AC coupling. Perhaps a series resistor could be used to increase the loss at low frequencies, but then an AC coupling capacitor in parallel with it could remove the resistor from the circuit at higher frequencies. In 2007, Cray Inc, filed a patent application on the passive network approach [6]. The passive network used is very simple consisting, in its most basic form, of a twocomponent RC high-pass filter. It was incorporated into a SerDes used in a Cray Inc. supercomputer; and the resulting performance was acceptable for a wide range of cable A Tale of Long Tails SiSoft, 2010 Page 15

16 lengths. It could also be easily incorporated into a PC board design, so it could be used with existing ICs. Figure 19 is the schematic for the PCB path with passive equalization. The network values were chosen to optimize the performance for the cable path; however the same element values were used for the PCB path to show how the performance might vary for a range of paths. Figure 19: Passive lonely pulse compensation network Figure 20 is an expanded view of the way the passive equalizer network modifies the path transfer function. From Figure 20, it would appear that the equalization for the cable path is probably about right, but the PCB path may be over-equalized. A Tale of Long Tails SiSoft, 2010 Page 16

17 Figure 20: Path loss without and with passive equalization network RED: PCB path baseline PURPLE: PCB path with equalization BLUE: Cable path baseline BLACK: Cable path with equalization The degree of compensation is easier to determine by looking at the impulse responses of the paths. Figure 21 shows the impulse response without and with the passive equalization network for the PCB path, while Figure 22 shows the same information for the cable path. Figure 21: PCB path impulse response without and with passive equalization network. RED: Baseline PURPLE: With equalization A Tale of Long Tails SiSoft, 2010 Page 17

18 Figure 22: Cable path impulse response without and with passive equalization network BLUE: Baseline BLACK: With equalization From these impulse responses, it appears that the PCB path might be slightly overequalized, but that the cable path might benefit from a little more equalization. In either case, however, it s clear that the long tail effect has been substantially reduced. Both the PCB path and the cable path were simulated with the same (0.725, -0.25, ) transmit equalization as was used for the self-equalized cable. The resulting eye diagrams are shown in Figure 23. Figure 23: PRBS22 eye diagrams with fixed compensation network. PDB path (left), Cable path (right). The advantages of the passive network are that it is simple, inexpensive, and effective. Furthermore, it can be applied to PCB as well as cable paths. While this network will typically be built using fixed component values, a single set of values seems to offer useful compensation over a wide range of cable lengths. The compensation with fixed component values is only optimal for one cable length, however. In addition, the parasitics of the equalization network must be considered. If the solution is integrated within a SerDes, the signal amplitude is reduced by capacitive division; and the capacitor s bottom-plate adds a parasitic pole to the receiver s input frequencyresponse. Neither of these consequences is desirable. Similarly, if the network is mounted A Tale of Long Tails SiSoft, 2010 Page 18

19 on a PC board, the vias to and from the network should be designed carefully to avoid transmission line discontinuities. Finally, as with the self-equalizing cable, the series resistance of the equalization network will affect the DC operating point of CML type interfaces. Active Compensation Other compensation techniques are available from commercial suppliers, and have been shipped in products. Details on these techniques could not, however, be shared at the time this paper was written. Performance Summary Table 3 summarizes the performance for the various solutions described above. Eye Height (V) Eye Width (ps) PCB path Cable path PCB path Cable path Baseline B10B Encoding Transmit Over-equalize Self-equalizing Cable NA NA 102 Passive Equalization Table 3: Performance summary for long tail compensation techniques and PRBS22 data sequence Conclusions This paper has reported that the low frequency characteristics of a transmission path can have a substantial effect on the system level performance of high speed serial channels. In particular, it has demonstrated that not only the skin effect resistance, but the internal inductance of the signal conductors causes the path impulse response to have an exponential decay with a relatively long time constant, a so-called long tail or lonely pulse effect. Four solutions to this problem have been described in detail and a fifth one alluded to. Table 4 summarizes the advantages and disadvantages of each approach A Tale of Long Tails SiSoft, 2010 Page 19

20 Advantages Disadvantages 8B10B Encoding Already used in many 20% loss of data throughput channels. Transmit Over-equalize Readily available Requires DFE Reduced transmit amplitude Self-equalizing Cable Automatically compensates all cable lengths. Helps equalization overall. Commercially available Expensive Cable only Mechanical challenge DC resistance may affect Passive Equalization Inexpensive Use on PCB or silicon operating point. Fixed element values Passive parasitics DC resistance may affect operating point. Table 4: Advantages and disadvantages of long tail compensation techniques The results presented have also consistently demonstrated that, for equivalent overall loss characteristics, the long tail effect makes it more difficult to obtain good performance from a cable path than from a PC board path, Thus, the long tail effect must be considered more carefully when designing high speed serial channels over long cables. References [1] Ramo, Whinnery and Van Duzer, Fields and Waves in Communication Electronics, third edition, John Wiley and Sons, Inc, copyright [2] Howard Johnson and Martin Graham, High-Speed Signal Propagation, Advanced Black Magic, Prentice Hall, pg 71-2, copyright [3] Feynman, Leighton and Sands, The Feynman Lectures on Physics, vol II, ch. 32, Addison Wesley, copyright [4] Djordjevic, Biljic, Likar-Smiljanic and Sarkar, Wideband Frequency-Domain Characterization of FR-4 and Time-Domain Causality, IEEE Transactions on Electromagnetic Compatibility, Vol. 43, No. 4, pg , November [5] Al X. Widmer, Peter A Franaszek, A DC Balanced, Partitioned-Block, 8B10B Transmission Code, IBM Journal of Research and Development 27 (5):440, [6] Michael Steinberger, Ricky Hakes, Chris White, Lonely Pulse Compensation, U.S. patent application , September 24, A Tale of Long Tails SiSoft, 2010 Page 20