xNTDSKAMPLFD Correlator

1
xNTD/SKAMP/LFD Correlator

• 4th RadioNet Engineering Forum Workshop
• Next Generation Correlators for Radio Astronomy
  and Geodesy
• 27-29 June 2006, Groningen, The Netherlands,
• John Bunton
• CSIRO ICT Centre, Sydney

2
Common Design

• Developing a set of technologies that will be
  used in xNTD, SKAMP3 and LFD
• Common correlator, VHDL code and hardware
  elements
• This talk mainly the correlator and xNTD
  beamformer
• Team (SKAMP3, xNTD and LFD)
• Uni Sydney/ ATNF - Ludi de Souza,
• Uni Sydney Duncan Campbell-Wilson, John Russel,
  Chris Weimann, Adrian Blake
• ATNF/ ICT Centre John Bunton, Jaysri Joseph
• MIT Roger Cappallo, Brian Fanous
• ANU Frank Briggs
• and growing

3
xNTD specification

• Connected element telescope with 20-30 antennas
  within 10km area
• Each antenna generates 32 dual pol. beams giving
  a field of view of 30 square degrees frequency
  independent
• Dynamic range 50dB (8-bit A/D) aided by low RFI
  environment at Mileura WA, gt60dB filterbanks
• Frequency range below 1.7 GHz with 300 MHz BW

4
xNTD Correlator

• FX correlator, FPGA based using polyphase
  filterbank
• Large design with 20-40 antennas x 32 beams
• Input to beamformers 1.5 THz, to correlator 500
  GHz (16bits/Hz)
• 6,000-15,000 baselines, each full Stokes
• 1000 Tops/sec (mainly beamformer/filterbanks)
• Output data rate 1-2 Gcorrelation/visability set
• Full 300 MHz processed to full frequency
  resolution
• 50,000 channel 6 kHz resolution (no options)
• BUT
• Low resolution and continuum by averaging
  channels
• Daisy chain data for special processing
• Later can trade bandwidth for higher
  resolution or time sampling

5
Design

• Beamformers and filterbank
• 200 inputs generating 32 dual pol. beams
• Signal transport from feeds optical analogue
  (TBD)
• Beamformer location TBD
• Design similar to SKAMP3 filterbank (Ludis talk)
• Correlator based on correlation cell
• Highly configurable as to number inputs
• If the beamformer is at the antenna crossconnect
  in fibre network and O/E interface
• Needs data re-ordering (Ludis talk)

6
xNTD Beamformer

• Beamformer input
• 96 Dual polarisation inputs
• 300 MHz bandwidth
• About 1 Terabit/sec input data rate per antenna
• Each of 32 output beams is the sum of 20 to 96
  inputs depending on frequency
• Would like to have data from all inputs to each
  unit that generates a beam
• Instead Divide and Conquer
• Use the same topology as SKAMP3 filterbanks
  (Ludis talk)

7
Filterbank to Beamformer
8
Correlator

• The challenge was to develop a correlator concept
  the could be used in xNTD, SKAMP3 and LFD -
  20-30, 192, and 512 antennas
• Systolic array too inflexible
• xNTD 760 correlations/beam
• LFD 500,000 correlations
• Approach developed Correlation Cell
• Combination of multiply-accumulate and storage
• Each cell handles 256 correlations at a time
• 49,000 correlations per FPGA simultaneously
• 512 time sample short integration on chip

9
Correlation Cell

• Input 16 pairs of data
• 4bit complex multiply in 18-bit multiplier
• Accumulation to block RAM
• Calculate 256 correlation, 512 successive time
  samples
• Data reordering in filterbank
• xNTD 4-7 cells for all correlations
• 30-70 MHz BW per FPGA
• All baselines LFD 12, SKAMP3 1.5 FPGAs
• 1.2-1.5 MHz of bandwidth

10
22 antenna xNTD configuration

• Store 22 or 22 pairs per correlation cells
• Within group 231 correlation
• Between groups 484 correlations
• Need two cells

• 48 sets of 22-antenna correlators per FPGA
  (Virtex4 SX35)
• Clock cycles 242 for 44 inputs 5.5 clocks per
  input
• Input data rate 48/5.5 8.7 bytes/clock
• As systolic array 192 multipliers gives 13x13
  array
• Input data rate 3 times higher at 26 bytes/clock

11
Board Manufacture Simplification (1)

• Manufacture of correlator board a major task
• Examples SKAMP1, EVLA
• Correlation cell reduces input data rate into
  correlation chip
• For xNTD correlation cell data rate 3 times less
  compared to systolic array
• For individual correlation cell 2 sets of 16
  inputs requires 256 clock cycles to process.
• Data rate reduction up to a factor of 16
• This value approached for SKAMP3 and LFD
• For xNTD need 5 byte wide busses into each
  correlator FPGA. (input data rate higher than
  correlator clock)

12
Board Manufacture Simplification (2)

• Correlation cell also reduces data duplication
• SKAMP1 4x4 systolic array, EVLA 8x8 systolic
  array
• Data duplication 8 in EVLA, higher in SKAMP1 due
  to array reuse
• Each Correlation Cell process 256 correlations at
  once
• Can reduce size of systolic array by sqrt(256)16
• No data duplication on board for up to 150
  antennas
• Data duplication none for xNTD, 1.5 for SKAMP3,
  and 3.5 for LFD, LFD 12 FPGAs data input
  1/sqrt(12) of total
• Correlation cell leads to a large simplification
  of correlator board

13
Putting it Together The SKAMP protoype
Correlator interface
Correlations
Routing
Long Term Accumulations
Autocorrelations
14
Input, Daisy Chain, Route, Autocorrelate

• Two FX20
• Interconnection for high antenna number designs
• Input 16 rocket I/O on unidirectional Infinband
• Output 16 Rocket I/O unidirectional Infiniband
• Can daisy chain modules for reuse of data in
  further processing modules or conform to
  standards, Infiniband

Two FX20
15
Compute Engine

• Six SX35 FPGAs
• Input 24, Output 18 LVDS per FPGA
• 1152 Correlation cells total
• Up to 294k correlations on board at a time (256
  per cell)
• Data re-ordering in filterbank to achieve
• Process of 512 time values for each frequency
  channel
• Then dump to LTA

16
Long Term Accumulator

• Number of Correlation require DRAM for storage
• Data rate requires two DIMM modules for three
  SX35

Three SX35 Two LX15 DDR2
DIMM
Half of Correlation and LTA hardware
17
Estimated Performance

• Correlator board clock rate 330MHz, 192
  cells/FPGA, 6 FPGAs
• Board processing rate 400GCMACs/s 2.8Tops/s
• Power consumption 100W
• Power efficiency 0.25W/GCMAC (4bit FX)
• Filterbank board 3.2Gsample/s, two polyphase
  filterbanks 32 operations per sample
• Board processing rate 100Gops/s (18 bit)
• Power consumption 60W
• Power efficiency 0.6W/Gop

18
Conclusion

• Common hardware, hardware modules and VHDL for
  xNTD, SKAMP3 and LFD
• SKAMP3 in the lead with filterbank and correlator
  hardware well on the way
• Initial Manufacture this year
• Correlator common to all using correlation cell
  to gain required flexibility
• xNTD beamformer hardware starts July 06
• LFD receiver will piggy back on this
• Developing international project - distributed
  design team