SKA CENTRAL BEAMFORMER CONCEPT DESCRIPTION

Transcription

1 SKA CENTRAL BEAMFORMER CONCEPT DESCRIPTION Document number... WP TD 001 Revision... 1 Author... B Carlson Date Status... Approved for release Name Designation Affiliation Date Signature Additional Authors Submitted by: B.Carlson DRAO Approved by: W. Turner Signal Processing Domain Specialist SPDO

2 DOCUMENT HISTORY Revision Date Of Issue Engineering Change Number Comments DRAFT March 9, 2011 First draft release for internal review A March 11, 2011 First release to SPDO 1 29 th March 2011 First Issue DOCUMENT SOFTWARE Package Version Filename Wordprocessor MsWord Word h wp td centralbfm concept description Block diagrams Other ORGANISATION DETAILS Name National Research Council Canada Physical/Postal Herzberg Institute of Astrophysics Address Dominion Radio Astrophysical Observatory P.O. Box White Lake Rd Penticton, BC, Canada, V2A 6J9 Tel: Fax Website cnrc.gc.ca/eng/ibp/hia.html Page 2 of 24

3 TABLE OF CONTENTS 1 TRODUCTION Purpose of the document Assumptions REFERENCES HIERARCHY Hierarchical Lifecycle ELEMENT LEVEL: SIGNAL PROCESSG Fundamental Geometry: Offset Beam Generation Antenna Beam Generator General Central Beamformer GSA Specific Implementation PAF Central Beamformer Band/Slice Beamformer Board (BSBB) Location BSBB Board Psuedo Layout WBSPF Central Beamformer SAA and DAA Beamformer Phase I SKA TERFACES GSA Central Beamformer COST AND POWER ESTIMATES GSA Central Beamformer RISKS GSA Central Beamformer APPENDICES Central Beamforming Stair Step Phase Delay Fitting MathCad Spreadsheet Page 3 of 24

4 LIST OF FIGURES Figure 4 1 Geometry for an offset beam from the main correlator delay center beam Figure 4 2 Stair step tracking of beam offset Δτ Figure 4 3 Method 1 ABG. General linear phase delay and fringe phase offset beam generation Figure 4 4 Method 2 ABG. A single, simple dual port RAM LUT is used to generate sine and cosine values required by the complex multiplier, from the spectral channel number. The coefficients of the LUT are calculated by a CPU Figure 4 5 Method 3 ABG, using minimal register hardware, and 8 bit sine+cosine LUTs Figure 4 6 Finite sized phase and phase rate register error quantities Figure 4 7 Central Beamformer general structure Figure MHz/polarization PAF GSA central beamforming pseudo mechanical drawing Figure 4 9 One section of the rear edge of an 8 rack dish PAF bay showing mounting of BS BEAMFORMER boards (BSBBs) in adjacently mounted Pony Crates Figure 4 10 Possible 1024 polarized beam per band/slice simplified rear rack arrangement Figure rack bay, with BSBB Pony Crates, centralized air cooling concept. The dual racksupport columns between the Pony Crate and the adjacent set of correlator boards is needed to allow for sufficient airflow in this scheme. (Note that this cooling arrangement might be ideally suited to cooling rear rack installed vertical boards as in Figure 4 10.) Figure 4 12 Pseudo layout of the Band/Slice Beamformer Board (BSBB). The drawing shows a depth of ~12.5, however the depth is restricted only by needing to roughly match the depth of the X part GSA correlator board, which itself is something of a free parameter No table of figures entries found. LIST OF TABLES Page 4 of 24

5 LIST OF ABBREVIATIONS ABG... Antenna Beam Generator. ASIC...Application Specific Integrated Circuit. BSBB... Band/Slice Beamformer Board. BSM... Band/Slice Merger. COTS... Commercial Off-The-Shelf. CPU... Central Processing Unit. DAA...Dense Aperture Array. DRAM... Dynamic Random Access Memory. edram... embedded DRAM. FPGA... Field Programmable Gate Array. GSA... Giant Systolic Array. HPB... Half-Power Beam. HPBW... Half-Power Beam Width. LC... Logic Cell. LE... Logic Element. LUT... Look Up Table. MAMBG... Multi-Antenna Multi-Beam Generator. MGT... Multi-Gigabit Transceiver. NVP... Non-Visibility Processing. PAF...Phased-Array Feed. ROM... Read Only Memory. SAA...Sparse Aperture Array. SRAM... Static RAM Page 5 of 24

6 1 Introduction 1.1 Purpose of the document The purpose of this document is to provide a concept level description of the Phase II (full scale SKA) Central Beamformer for the signal processing Conceptual Design Review to be held in April 2011 in Manchester. This document also briefly describes beamforming for Phase I. Many specific aspects of the Central Beamformer are not yet defined, including the exact nature of the correlator which the beamformer must tap into 1, the number of beams that must be generated, and in conjunction with that point, the data rate that the system the beamformer feeds, known as the Non Visibility Processing (NVP) system, can handle. Nevertheless, an attempt is made to describe the basic beamformer structure in general terms, and then in specific terms how it might attach or tap into one concept of the correlator, namely the (GSA) Giant Systolic Array [1]. 1.2 Assumptions The concept description provided here makes the following assumptions: Coherent beams are formed. What this means is that beams are formed on data already coherently corrected to the delay center on the sky by the F part of the correlator. Any delay and phase corrections needed for coherent delay center beamforming are determined from correlator residuals via a route outside the Central Beamformer proper, and applied to the correlator to achieve coherence. (This not a strict requirement, but makes the simplification that the beamformer does not have to somehow come up with these numbers itself). Central Beamformer beams will have a much smaller angular extent on the sky than the element beam as the aperture of the beam formed array is much larger than the element diameter D; it is therefore necessary for the beamformer to produce multiple beams, each offset by some different epsilon from the correlator delay center, to provide full element beam coverage. How many beams should be provided to accomplish this is defined in [2]. All necessary beamformer beam offset corrections can be applied by approximating the resulting offset delay function as a frequency dependent step wise complex phase rotation function applied to frequency channels in real time. (This seems like a reasonable assumption; for the GSA X421 ASIC standard spectral channel resolution of ~36 khz, the worst case channel edge coherence loss as a consequence of doing this is <~0.2% for ν LO =700 MHz, dish diameter=15 m, at the edge of the dish HPBW, and a 10 km baseline; a 5 km core is indicated in [2]). What this means is that the Central Beamformer does not have to further channelize the data, and can use it as produced by the F part of the correlator. 1 Not strictly necessary, however likely prudent to avoid duplication of effort, in particular the F-part of the correlator Page 6 of 24

7 This necessarily assumes that only the central core of the SKA, within ~<20 km diameter, will be beamformed. The output of the beamformer is in the same frequency domain form, although possibly a different format and arrangement, as that produced by the F part of the correlator. i.e. there is no need to produce re constructed non channelized real time data. Furthermore, it is acceptable to produce complex, rather than real data. Due to the bandwidths and data rates involved, it is not necessary for a single data stream to contain the entirety of a single beam, all channels, at full bandwidth. Nevertheless, it is necessary for the beamformer to collate the data so that it is possible for all beam data, present on one or more physical communication links, to be directed towards a single NVP. 2 References [1] Carlson, B., Giant Systolic Array (GSA) Correlator Concept Description, Document No. WP TD 001, Rev. A, February 28, [2] Turner, W., et. al., High Level SKA Signal Processing Description, Document No. WP TD 001, Rev. D, November 24, [3] Thompson, A.R., Moran, J.M., & Swenson, G.W. 1986, Interferometry and Synthesis in Radio Astronomy, Wiley, New York. [4] v/stx5_51001.pdf [5] tcs.com/search?n=crossbow Page 7 of 24

8 3 Hierarchy 3.1 Hierarchical Lifecycle 4 Element Level: Signal Processing 4.1 Fundamental Geometry: Offset Beam Generation In the case where the phased array (a.k.a. tied array ) beam has the same delay center on the sky as the F part of the correlator produces, central beamforming is a relatively trivial operation. It is simply, for each frequency channel, the complex addition of the data from all of the antennas that go into that beamformer. As the radio source tracks across the sky, the F part of the correlator tracks delay and fringe phase, applies these corrections to the frequency channels to achieve coherence, and the beamformer simply uses (sums) the data. If, however, the beamformer is to generate a coherent tied array beam, which is offset somewhat from the main correlator delay center beam, an epsilon delay must be applied to each antenna s data before summing to achieve coherence. This is a well known concept in radio interferometry [3] and is reproduced in the simple diagram of the figure below: 1 α correlator delay-center beam Θ 2 offset beam B1 B2 Θ α Ant-1 Ant-2 b Figure 4 1 Geometry for an offset beam from the main correlator delay center beam. The quantity τ=(τ 1 τ 2 ) is the offset delay function, which must be applied in the Central Beamformer, to each antenna, before complex addition of all antenna data to produce each resulting offset beam. Note that an antenna based correction is applied; in Figure 4 1, Ant 1 is a real antenna, and Ant 2 is (or can be) the array phase center, a common reference point for all antennas in the array. Δτ increases as the baseline b increases, and as the beam offset from the phase center Θ increases. The extent of Θ is strictly governed by the element (e.g. dish) beamwidth there is no sense in producing a tied array beam which is outside of the main beam of the element.

9 The equations, not necessarily in their simplest form, are as follows: b 1 2 sin( ) sin( ) c HPBW 2 c LO 2D Where b is the baseline from the antenna to the array phase center, c is the speed of light, Θ is the worst case beam offset at the edge of the HPBW (assuming the correlator delay center is the center of the beam), D is the element diameter, and ν LO is the observing frequency. The phase step Δψ that must be applied to each frequency channel of width Δf, to achieve a delay slope of Δτ is then: 2 f Stair step tracking of Δτ is shown in the following figure: Ψ (phase) f Ψ Figure 4 2 Stair step tracking of beam offset Δτ. freq channel There is an additional term that must be accounted for. Since delay is being applied at baseband, there is an earth rotation fringe phase term that causes the phase for every frequency channel to vary according to: ( t) 2 ( t) f LO When many antennas are corrected in this way, each with their own Δτ, and assuming uniform distribution of phase across Δψ max, the resulting coherence [3] at the edge of the HPBW, at the edge of the spectral channel is: sin( ) Page 9 of 24

10 As mentioned in the introduction, for D=15 m dishes, ν LO =700 MHz (the approximate lowest operating frequency of the dish, resulting in the worst case beam offset requirement), b= 10 km, f~=36 khz 2, and worst case pointing (zenith, α=0), the coherence loss (1 η) is ~0.2%. At b=20 km the worst case coherence loss is ~0.8%. For this case (b=20 km) as well, the worst case diurnal beam differential delay phase step rate ( d ( ) dt d( d df ) dt ) is ~10 3 deg/sec per 36 khz spectral channel (x2048 channels ~= 2 deg/sec, last channel), and the worst case diurnal beamdifferential fringe rate ( dt ) is ~18 deg/sec (both at α=90 o ). d f For SAAs with ν LO =70 MHz, D=250 m, b=10 km, and the same Δf, the coherence loss is ~0.07%. For DAAs under the same conditions except ν LO =300 MHz, the coherence loss is proportionately less. At b=20 km, the coherence loss is <0.3%. For this case (b=20 km, ν LO =70 MHz) as well, the worst case diurnal beam differential delay phase step rate is ~ deg/sec per 36 khz spectral channel, and the worst case diurnal beam differential fringe rate is also ~1 deg/sec (both at α=90 o ). Suffice to say, that as long as the Central Beamformer forms beams from the central ~20 km core of the SKA, stair step phase corrections to approximate delay beam offset corrections are sufficient for beamforming. Also, the worst case beam differential phase rates noted above are important considerations when it comes to the design of the Antenna Beam Generator (ABM), as the ABM consumes the bulk of Central Beamformer signal processing hardware, and must be optimized to maximize the number of beams that can be generated, whilst minimizing cost and power dissipation. 4.2 Antenna Beam Generator The smallest, most elemental part of Central Beamformer signal processing is the circuit which implements offset delay tracking and the fringe phase term Φ f (t), as depicted in Figure 4 2. In this document, this circuit will be known as the Antenna Beam Generator (ABG). For each output tiedarray beam, for each antenna, there must be one of these. One method ( method 1 ) of implementing the ABG is shown in Figure 4 3 below: Figure 4 3 Method 1 ABG. General linear phase delay and fringe phase offset beam generation. 2 GSA X421 ASIC spectral resolution, 75 MHz bandwidth Page 10 of 24

11 The circuit of Figure 4 3 contains phase synthesizers to implement the stair step phase delay function, which varies linearly with frequency channel, and the fringe phase function, which is the same for each frequency channel for the same time epoch. This implementation necessarily requires that frequency channel data get presented to the ABG in a sequential fashion, with or without channel burst depending on the fringe phase rate. Not shown are the additional 4 registers normally used to double buffer the coefficients so as to avoid collisions between the CPU writing the registers and the circuit that uses them. However, the circuit of Figure 4 3 requires multi bit registers for each of the phase synthesizers, and for full scale SKA beamforming consisting of thousands of antennas with the ultimate desire of providing enough beams to fill the entire antenna primary beam, significant quantities of hardware are required for implementation. Given the very low diurnal phase rates noted in the previous section, it is likely more hardware efficient to implement the phase synthesizers and sine+cosine LUTs in a single LUT, which is periodically updated by a controlling CPU. The bulk of the registers in the ABG then resides in the complex multiplier ( c mult block of Figure 4 3). A second method ( method 2 ) of implementing the ABG is shown in Figure 4 4 below. In this implementation, a single simple dual port RAM LUT is used to directly convert the channel number to the sine and cosine functions used by the complex mixer. As the diurnal phase rates are very small, an update rate of the LUT of once every 100 msec should be sufficient, and it doesn t seem like it should particularly matter if the entire LUT is updated between sets of complete channels. However, LUT contents must be updated such that there isn t a collision between the read of a memory location, and the writing of the same memory location, which could generate corrupt values 3. channel packets ch# ch-data sin_cos(t,beam) phase+ sin_cos LUT sin(ch) cos(ch) c-mult Q I offset-beam output LUT ABG Figure 4 4 Method 2 ABG. A single, simple dual port RAM LUT is used to generate sine and cosine values required by the complex multiplier, from the spectral channel number. The coefficients of the LUT are calculated by a CPU. In addition to reduced register usage, this circuit has a couple of additional advantages. The first is that the phase vs frequency function does not have to be linear. The second is that the incoming channel packets don t have to have spectral channels in any particular order, which eliminates the 3 It might be the case that simple dual-port RAMs in FPGAs already have this exclusion mechanism built in Page 11 of 24

12 need for any upstream processing to have to sort spectral channels. The disadvantage is that the LUT is more memory intensive than method 1, as it must be a K channel by n bit memory. For example, if there are 2048 channels, and 48 db of dynamic range is required, a 2k x 16 = 32k bit memory per ABG is required. Unfortunately it would seem not possible to lump all of the memory for many antennas and many beams into a single high capacity external chip memory, and so this solution might be too distributed RAM intensive as to be practical (e.g. for 256 antennas, 8.3 Mbits/beam are required; for 128 beams, this is 1 Gbit), particularly for FPGAs. Although, one can imagine a design using distributed edram, with short SRAM buffers in front of each one, which could be implemented in an ASIC 4, although updating all of the LUT coefficients contained in edram every 100 msec by a CPU could be problematic from a compute and write bandwidth perspective. Yet a third method ( method 3 ) is a variation of method 1 whereby the fringe phase and rate is absorbed into the Φ o register of Figure 4 3, the registers are made as small as possible, and 8 bit sine and cosine LUTs are used. The critical question then is how large do the registers have to be? The method 3 ABG is shown in Figure 4 5 below: channel packets ch phasestep ch-0 next-ch en ch-data ᶲmodel(t,beam) S LUT COS LUT c-mult Q I offset-beam output ABG phase(ch-0) Figure 4 5 Method 3 ABG, using minimal register hardware, and 8 bit sine+cosine LUTs. If we set the maximum error due to the finite size registers of Figure 4 5 to be one half the phasestep size as shown in Figure 4 6, then the error equations are: cum _ max nchan 16 n 2 2 Where nchan is the number of spectral channels being beamformed, n is the number of bits in the phase rate register of Figure 4 5, and the register is sized to allow for up to +/ o (+/- π/16) of phase per frequency channel. The 0 th order error offset equation is: 4 The X421 ASIC design, briefly described in [1] contains about the same amount of on-chip RAM, but 32,768 4-bit x 8-bit CMACs to implement the c-mult functions would be required (the X421 has 1024 CMACs) Page 12 of 24

13 2 offset m 2 Where m is the number of bits in the accumulator register. Note also that m=n+5, since the phase accumulator must encompass all bits of each phase rate addition. Ψ (phase) req cum_max offset actual central spectral channel 1024 freq channel Figure 4 6 Finite sized phase and phase rate register error quantities. The total error is the sum of both error quantities, and setting m=n+5: 16 2 nchan total nchan radians n1 n5 n Setting the peak ε total equal to o (which is slightly less than ΔΨ/2 at zenith pointing, ν LO =700 MHz, D=15 m, b=10 km), yields n=11 bits (for each increase in the number of bits n, the peak phase error is reduced by a factor of 2). Therefore, the phase rate register is 11 bits, representing phase of up to +/ o per phase step for 2048 phase steps (set for minimum error at the central spectral channel), and accumulator register is 16 bits. The o register need only adequately represent the initial phase, and likely ~1 o of initial phase precision is enough, requiring 9 bits over 360 o. Earthrotation phase, because it is changing very slowly, is subsumed into the o register, provided updates to these registers occur about every 100 msec. In this case ( method 3 ), each ABG requires = 36 FPGA LEs 5, a 256 x 16 bit LUT, and a 4 bit by 8 bit complex multiplier, assuming double buffering of phase registers is not required. 4.3 General Central Beamformer Figure 4 7 is a diagram showing a general Central Beamformer structure: 5 Where each FPGA LE (Logic Element; a.k.a. LC Logic Cell ) is a D-flip flop, fed by an x-input LUT, and accompanying fast-carry-lookahead logic Page 13 of 24

14 1 2 ABG ABG BEAM-1 BS-BEAMFORMER Ants 1-j; band/slice 1 Antenna band/slice De-constructor/Buffer j ABG ABG BEAM-2 BEAM-1 ABG BEAM-M BEAM-2 BEAM-1 band/slice 1 ABG MAMBG BEAM-2 band/slice 1 Ants j+1-2j; band/slice 1 MAMBG BEAM-M BSM-1 BEAM-1 Ants wj-j; band/slice 1 Ants 1-J; band/slice 2 MAMBG Ants 1-J, band/slice 1, BEAMs 1-M Beamformer BS-BEAMFORMER - 2 BEAM-1 band/slice 2 BEAM-1 band/slice B BSM-2 BSM-M BEAM-2 BEAM-M Ants 1-J; band/slice B BS-BEAMFORMER - B Figure 4 7 Central Beamformer general structure. The MAMBG element is a Multi Antenna Multi Beam Generator. The MAMBG block produces all required beams for some sub set of array elements, for some sub set of element bandwidth ( band/slice ). Multiple MAMBGs constitute a BS BEAMFORMER, which is a Band/Slice Beamformer producing all of the beams for all array elements for some band/slice, or fractional part of the bandwidth. Multiple BS BEAMFORMERs are required to produce all beams for the full element bandwidth, for all of the antennas in the array. Subsequently, the Band/Slice Merge (BSM) boxes merge all of the beamformed packet streams from all of the BS BEAMFORMERs to produce final, full bandwidth, beam output ( M beams) for all the elements in the array. The outputs of BSM boxes then go to downstream NVP (Non Visibility Processing) equipment. The specific dimensions in each of these blocks are dependent on the content of the packets the F part of the correlator produces, but any Central Beamformer likely requires some implementation of this general structure Page 14 of 24

15 4.4 GSA Specific Implementation This section presents some concepts on how the Central Beamformer could be implemented within the context of the GSA (Giant Systolic Array) correlator implementation PAF Central Beamformer In the GSA implementation, all of the real time F part channel burst packet data streams are already all available at the outside edges of the band/slice triangles, just waiting to be beamformed. Thus, all that central beamforming requires is to patch these signals into one or more beamforming boards, which produce band/slice beams for all array (or selected array) elements, which are subsequently routed to NVP equipment. A simplified pseudo mechanical diagram, illustrating how this might be done is shown in Figure below: BSMs: Band/slice Mergers BSBB-2 PAF beam-n Band/slice-2 PAF beam-n Band/slice-1 BSBB-1 COTS Switch COTS Switch COTS Switch BSBB-4 PAF beam-n Band/slice-4 PAF beam-n Band/slice-3 BSBB-3 COTS Switch BSBB-6 PAF beam-n Band/slice-6 PAF beam-n Band/slice-5 BSBB-5 to non-vis processors, PAF beam-n BSBB-8 PAF beam-n Band/slice-8 PAF beam-n Band/slice-7 BSBB-7 BSBB-2 BSBB-10 PAF beam-n Band/slice-10 PAF beam-n+1 Band/slice-2 PAF beam-n Band/slice-9 PAF beam-n+1 Band/slice-1 BSBB-1 BSBB-9 BS-BEAMFORMER boards Figure MHz/polarization PAF GSA central beamforming pseudo mechanical drawing. The central part of the diagram represents the 8, 19 racks of the X part of the correlator. F part band/slice packet data, traversing through the X part systolic array exits out the edges, where it is fed into multiple BSBB (Band/Slice Beamformer Boards), one BSBB for every band/slice. A 6 Note that the very bottom two band/slice correlators in the figure are for a different beam, so as to fully utilize the 48-U space in the rack. It may be logistically helpful to have only one PAF-beam per 8-rack-bay, but does increase the number of 8-rack bays from 25 to 30, for 30 beams Page 15 of 24

16 packetized beam output from each BSBB (i.e. for every band/slice, or the entire bandwidth) then routes to a COTS network switch (or portion of a COTS network switch), which performs the BSM ( Band/Slice Merge ) function. One or more merged switch output spigots then route to NVPs. Depending on how many beams are produced, the output of each BSBB can be less than, equal to, or (much) greater than the input. As the tied array beam angular extent is inversely proportional to the array aperture, it is advantageous to use the smallest array aperture possible to get the largest sky coverage, possibly even coverage of the entire element beam. Thus, for a fixed number of beams output, there is likely some advantage in a design which allows for dynamic selection of which antennas to include in the tied array output Band/Slice Beamformer Board (BSBB) Location The physical attributes and location of the BSBBs will greatly impact the cost and complexity of implementation. If connections from the band/slice correlators to the BSBBs must be established with cable, then the quantity (and probably cost) of cable, either copper or fiber, will be equal to that used for transferring signals from the F part to the X part of the correlator, no doubt a significant cost, given the data rates involved. Clearly then, there is some incentive to eliminate such cabling, and only require cable where absolutely necessary, and hopefully at much reduced data rates. Figure 4 9 is a simplified mechanical drawing showing the back of one end of an X part correlator rack bay. The BSBB board mounts vertically in a single slot Pony Crate, which is attached to the right edge of the rack in the figure. By doing this, X part to beamformer patch boards (printed wiring boards specifically for this purpose) can be installed to make the necessary connections without having to use cable. The BSBB board therefore has 8, 40 pair 0XT connectors, to tie into the 8 boards that feed it (note that the figure shows only 5 an abbreviation for compactness of the concept drawing). Vertical stiffening bar Input from F-part 48 V power studs Remote power M&C studs Rack Support Columns Beamformer Pony Crate Direction turner patch board HMZd Midplane Vertical patch board Horizontal patch board to beamformer BS-BEAMFORMER board Beamformer midplane Figure 4 9 One section of the rear edge of an 8 rack dish PAF bay showing mounting of BS BEAMFORMER boards (BSBBs) in adjacently mounted Pony Crates Page 16 of 24

17 This physical implementation matches the pseudo mechanical drawing of Figure 4 8; there are 6 such Pony Crates and BSBBs mounted on either end of the rack bay. These Pony Crates would be designed to be an integral part of the end racks, and would take up negligible additional floor space. It should be noted that the physical arrangement shown in Figure 4 9 is a minimum board/minimum cost implementation arrangement. It is entirely possible to install an entire rack on either side of a rack bay, and use 8 horizontally mounted boards, each one fed data from a single 40 pair 0XT connector via different patch boards, and daisy chain fed to multiple chips on each board to yield many more beams than proposed in Figure 4 9, using existing and emerging technology. All of these boards partial array outputs are then finally summed in orthogonally mounted boards at the rear of the rack, using Crossbow orthogonal connectors [5], with final output beams exiting the front of each rear mounted orthogonal board. Using this arrangement and 32, 32 polarized beam ASICs 7 on each horizontal board, it is likely feasible to produce 1024 polarized beams per band/slice, requiring a total of 256, 10G outputs (per band/slice) to transmit to NVP equipment. The simplified rear rackview arrangement of horizontal and vertical boards, with patch boards feeding the array from the X part of the correlator is shown in Figure 4 10: X-part Correlator Boards Rack Support Columns Partial-Array Band/ Slice Beamformer Horizontal Boards (256 Ants, 1024 polarized beams ea) Final Band/Slice Vertical Beamformer Boards (128 beams, all Ants ea) Figure 4 10 Possible 1024 polarized beam per band/slice simplified rear rack arrangement. Indeed, many of these types of racks could be adjacently mounted and daisy chained (in this case the far right connector of Figure 4 10 would have to be changed to contain real time data to go to the next adjacent rack) to produce a virtually unlimited number of output beams, limited only by power, cost, NVP processing capacity, and the number of beams fundamentally required. Although 7 See following pages for current 28 nm FPGA beamformer capacity limitations Page 17 of 24

18 the approach shown in Figure 4 10 is entirely feasible, the rest of this section will concentrate on the baseline approach shown in Figure 4 9. A possible centralized air cooling arrangement (following that outlined in [1]) for an entire 8 ray bay, with Pony Crates and BSBB boards as per Figure 4 9 is shown in Figure 4 11 below: Warm air out Warm air out Boards Boards Beamformer Boards Boards Boards Boards Beamformer Boards Cold air in Pressurized floor cavity Cold air in Figure rack bay, with BSBB Pony Crates, centralized air cooling concept. The dual rack support columns between the Pony Crate and the adjacent set of correlator boards is needed to allow for sufficient airflow in this scheme. (Note that this cooling arrangement might be ideally suited to cooling rearrack installed vertical boards as in Figure 4 10.) A similar cooling arrangement would be used for the configuration of Figure 4 10; the horizontallymounted boards get cooled as in the above figure, and the rear vertically mounted boards get cooled with direct vertical airflow BSBB Board Psuedo Layout A pseudo layout of the Band/Slice Beamformer Board (BSBB) is shown in Figure Each MAMBG (Multi Antenna Multi Beam Generator) device (matching the similarly named block of Figure 4 7), in this drawing has 32, 10G inputs, and 32, 10G outputs. If data is formatted according to the GSA description [1], each 10G signal contains 8 antennas, 75 MHz/polarization, 2048 frequency channels per polarization. Therefore, each MAMBG device beam forms 256 antennas, 75 MHz/polarization, M beams. If the output data rate matches the input data rate (except with 8 bit I and 8 bit Q beam formed samples), then M=256 beams (128 beams, dual polarization) are possible. Each ABG of Figure 4 5 requires 36 LEs 8, a 256x16 bit ROM for the sine and cosine LUTs (providing 48 db dynamic range), and a 4x8 bit complex multiplier for each polarization. For 256 antennas, 256x36=9.2k LEs are required for the phase synthesizers, x8 multipliers + 256x8 12 bit adders (24,576 LEs) for the dual polarization complex multiplier logic, plus 255x2 (avg=) 14 bit adders for the adder tree (7124 LEs), and formatting/de formating glue logic. Thus, ~41k LEs plus 256x16x256=1 Mbits of ROM, plus x8 multipliers are required for each polarized output beam. As the data is only coming in at a 75 MHz sample rate, it might be possible to time multiplex the use of some hardware (such as the LUTs and complex multipliers) in the ABG for more compact operation. 8 Assuming the same phase corrections are applied to both polarizations Page 18 of 24

19 Additional RAM might be required for de channel bursting the data; at a channel burst factor of 64, 2048 channels, 16 bits per polarized complex sample, and 8 antennas requires 16 Mbits per 10G stream way too much for any internal FPGA with 32 streams active! The only requirement for not de channel bursting the data is that within a channel burst factor time (in the X421 ASIC 9 case the channel burst factor J is 64) the beam differential fringe phase has not changed significantly. For 36.6 khz channel bandwidth, and channel burst of 64, this time is 64/36.6 khz ~= 1.7 msec, not a problem since the fringe phase rate is small (see page 10). ~ pair 0XT () 40-pair 0XT () 40-pair 0XT () -48 PWR+C 40-pair 0XT () -48 PWR+M 40-pair 0XT () 40-pair 0XT () 40-pair 0XT () 40-pair 0XT () 32 x 10G 32 x 10G 32 x 10G 32 x 10G MAMBG FPGA 32 x 10G Power Supply MAMBG FPGA 32 x 10G Power Supply MAMBG FPGA 32 x 10G Power Supply MAMBG FPGA 32 x 10G ~ x 10 G: up to 32 x 8 = 256 beams MAMBG FPGA 32 x 10 G: up to 32 x 8 = 256 beams MAMBG FPGA 32 x 10 G: up to 32 x 8 = 256 beams MAMBG FPGA Power Supply Power Supply Power Supply 32 x 10 G: up to 32 x 8 = 256 beams MAMBG FPGA Final Summing FPGAs; Quantity depends on number of beams being formed. Power Supply Power Supply Output to COTS network: each 75 MHz dual-polarization beam (8-bit I, 8-bit Q) requires ~2.4 Gbps. So, 4 dual-pol beams per 10G link, requires 32 x 10G links for 128 beams. Power Supply Networked, embedded CPU with RTOS for real-time frequency-dependent complex beam coefficient generation. Power Supply RJ- 45 Figure 4 12 Pseudo layout of the Band/Slice Beamformer Board (BSBB). The drawing shows a depth of ~12.5, however the depth is restricted only by needing to roughly match the depth of the X part GSA correlator board, which itself is something of a free parameter. If the full M=128 polarized beam output capacity is filled, then 41k x 128 = 5.2 MLEs, 128 Mbits of ROM, and 128 x x8 multipliers (262,144) are required. This far exceeds the capability of the latest generation 28 nm FPGAs [4]. 9 i.e. the baseline ASIC proposed for the GSA concept Page 19 of 24

20 If we imagine a factor of 4 time multiplexing is possible into the LUTs, and out through the complex multipliers and adder trees, the 128 polarized beam requirement drops to 128/4 ~= 32 Mbits ROM, 17kx128=2.2 MLEs, and 64k 4x8 multipliers. The Altera Stratix V [4] second largest GS FPGA (the 5SGSD6) contains 583k LEs, 48 Mbits of RAM, 48 MGTs, and x9 multipliers. Using this device, at 75% LE utilization, would allow for 25 beams in a single MAMBG FPGA, without considering that ~12,800 4x8 multipliers are still required. More likely, 12 polarized beams would fit in this device, reduced because of multiplier requirements. Approximately 2.4 Gbps of output bandwidth (16 bit output samples, 8 bit I, 8 bit Q, at 75 MHz) is required for each polarized beam, for a total of 28.8 Gbps. Thus, 3x10G output links are required (4 polarized beams per 10G channel) for all 12 beams. Clearly the MAMBG beamformer chip is compute, rather than I/O dominated, and as with the X part of the correlator, the most power and cost effective implementation of the beamformer chip is likely an ASIC. Further study is required to determine how many beams could be formed in an ASIC. I ll leave it up to others to speculate as to future FPGA capacity improvements, to increase the number of beams per device even more than described. Note that additional daisy chained devices, each operating on the same data, could be added to the board to increase the number of beams by the number of devices in the daisy chain. For example, if 4 MAMBG devices are added in each daisy chain (so there are 32 MAMBG devices on the board), then 48 beams per polarization could be realized, within the present example FPGA implementation. The rest of the devices on the board perform final summing operations for all (12) beams. For 12 beams, (8x3 10G inputs, 3x10G outputs) 27 MGTs in the device are required, and all of the required operations can likely fit in a single mid sized FPGA such as an Altera Stratix V 5SGXA7 FPGA, since only simple summing operations are required. This FPGA can also format and re arrange the data for output to the COTS network. If de channel bursting of the data is required before transmission to the NVP, each beam de channel burster requires 64 (burst length) x 2048 (channels) x 32 bits (8 bit I and 8 bit Q samples per polarization)=4 Mbits per beam or 48 Mbits for all 12 beams, fitting within the FPGA s 50 Mb capacity. Each of the 3, 10G output links from each BSBB board contains 4 polarized beams, 8 bit I and 8 bit Q samples, with each beam in its own Ethernet packet to facilitate packet switching. As shown in Figure 4 8, these outputs route to COTS network switches, which, based on destination addresses, route single (or multiple) beam data, all band slices to final destination NVPs. If located in the same rack bay 10 (and ignoring the extra band/slice correlator at the bottom of Figure 4 8), a 24 port 10G switch can handle 10 inputs (4 beams, all band/slices), switched to 3x10G links per beam (all band/slices) = 12x10G outputs. Three such 24 port switches are required and could be deployed so as to occupy only 1U vertical space in each rack of the 8 bay rack. Or, these switches could be located in another rack or room as desired as there is no data compression or cable savings occurring. 10 Or elsewhere, doesn t matter Page 20 of 24

21 A close alternative is to locate the NVP processing equipment in an adjacent rack, and internal to that equipment is also included any band/slice packet routing. This would eliminate all of the output cabling from the switches, assuming the output data rates of the NVPs are negligible WBSPF Central Beamformer The PAF Central Beamformer concepts presented in the previous sections directly apply to the WBSPF correlator, if it is assumed that only 2048 of the possible 3072 antennas are beamformed. The same beamformer board (BSBB) is used, with identical location as in Figure 4 9, except that only 8 of the available 12, 40 pair 0XT connectors feed into the board. In this case it is necessary that the 4x256 ~= 1024 out of central core antennas are fed into the WBSPF correlator strategically so that at either end of the band/slice correlator the central 8x256=2048 antennas feed into the BSBB via patch boards. A similar arrangement as shown in Figure 4 10 could be used, indeed with boards fed by all 3072 antennas to produce many more output beams SAA and DAA Beamformer All correlations for two band slices (2 boards required for 300 MHz/polarization, 1 beam), and up to 256 antennas are performed on a single GSA correlator board for the SAA and DAA application. Thus, it only makes sense to put dual MAMBG devices on each correlator board, with each one forming 12 complete beams for a band/slice (see section for emerging technology limitations). In this case, the output of each correlator board also includes formed beams, which route, via 6x10G outputs for each MAMBG device, to NVP equipment contained in the same racks, or different racks accordingly. Alternatively, if many more beams are required, the output of each correlator board could be daisy chain fed to one or more daisy chain multi chip beamforming boards, limited in the number of output beams only by one s imagination and pocketbook Phase I SKA In the GSA Phase I SKA implementation discussed in [1], all cross correlations for a band/slice (and there can be multiple band/slices on a board), whether for WBSPFs or SAAs occur on a single board. So, in this case, one or more MAMGB chips are on the same board, similar to the scheme described above for the SAA and DAA beamformer Page 21 of 24

22 5 Interfaces 5.1 GSA Central Beamformer The X part of the correlator feeds the Central Beamformer directly. Final packet format is TBD, however, each 10G link nominally contains 8 antennas, 75 MHz/polarization, 2048 channels/polarization, with a minimum channel burst factor J of 64. Within a Packet Group (defined as a set of channel burst packets containing all spectral channels), there is an FO Packet, which contains antenna based information and the channel burst packet time centroid timestamp. This FO Packet is used by the Central Beamformer to label output packets destined for NVP equipment. The output format and granularity of the Central Beamformer generated packets is TBD, although likely generally follows the input channel burst format, but with more bit resolution. 8 bits per I sample and 8 bits per Q sample are proposed, for each polarization, for each channel. The channelization of the output mirrors the input, and outputs are complex. Each packet contains identification information (timestamp, channels, band/slice etc.), which might be required by downstream NVP equipment. 6 Cost and Power Estimates 6.1 GSA Central Beamformer If the minimum configuration of Figure 4 8 and Figure 4 9 is used, then the Central Beamformer could be roughly 1/32 nd (i.e. for every 32 X part correlator boards there is 1 beamformer board of similar ish power and complexity) the cost and power of the X part of the correlator, depending on the number of beams. If many more beams are required (possibly full primary beam coverage if dense enough technology is used), with the configuration of Figure 4 10 (1 adjacent orthogonalboard rack on either end of the rack bay), then the Central Beamformer could be roughly ½ the cost and power of the X part of the correlator. Refer to section 9 of [1] for more information on correlator cost and power estimates. 7 Risks 7.1 GSA Central Beamformer As the Central Beamformer concept closely ties into the correlator, the risks for it are essentially the same as for the correlator. Refer to Table 5 1 of [1] for more information Page 22 of 24

23 8 Appendices WP TD Central Beamforming Stair Step Phase Delay Fitting MathCad Spreadsheet Page 23 of 24

24 Page 24 of 24