JavaSeis Parallel Arrays

1
JavaSeis Parallel Arrays

• JavaSeis data structures
• Synchronous parallel model
• Arrays in Java
• mpiJava
• Parallel Distributed Arrays
• Examples

2
JavaSeis Data Structures

• View data as N-dimensional array
• Most common view is a series of 3D volumes
• Use generic names for each axis Sample, Trace,
  Frame, Volume, Hypercube,
• Associate LogicalNames with each axis
• Time, Offset, X-Line, InLine
• Time, Channel, Shot, Swath

3
JavaSeis Dataset Logical View
Frame
Sample
Volume
Trace
4
JavaSeis Bounding Box
X-Line
InLine
5
Parallel Distributed Objects
Node 0
Node 1
Node 2
Node 3
pdoRead
pdoRead
pdoRead
pdoRead
pdoExec
pdoExec
pdoExec
pdoExec
pdoWrite
pdoWrite
pdoWrite
pdoWrite
6
The Transpose Problem
CPU 1
CPU 0
CPU 2
CPU 3
Local Access
Remote Access
7
Data Parallel Transpose
CPU 1
CPU 0
CPU 2
CPU 3
Local Tile Transpose
8
Arrays in Java

• 1D arrays or arrays of arrays
• Subarrays and multi-dimensional views of a 1D
  array are not supported by the language
• Subarrays are constructed by passing the full
  array and an upper and lower bound
• Example reference from Matuszkek, University of
  Pennsylvania

9
Design Sources

• NCAR / UCAR NetCDF
• University Center for Atmospheric Research
• High Performance Fortran
• Ken Kennedy, Rice University 1993
• Colorado School of Mines
• Dave Hale, Mines Java Toolkit
• Landmark ProWESS / SeisSpace
• ARCO Parallel Seismic Environment

10
DistributedArray Class Structure
mpiJava
java.lang.Array
TransposeType
IMultiArray MultiArray
ITranspose Transpose
IParallelContext MPIContext
DistributedArray
Decomposition
11
parallel and array packages

• org.javaseis.parallel
• IParallelContext, ParallelContext
• Message passing support
• Decomposition
• Define decompositions for array dimensions across
  processors
• org.javaseis.array
• IMultiArray, MultiArray
• Containers for Fortran style multidimensional
  arrays
• ITranspose, Transpose, TransposeType
• Transpose operations for Fortran style arrays
• DistributedArray
• Extends MultiArray to distribute across processors

12
MultiArray Design Targets

• 1D Java arrays of primitive elements or Objects
• A superimposed "shape" that follows Fortran
  conventions
• Access via "range" triplets (start,end,increment)
  
• Ranges for Java zero based indexing or Fortran 1
  to N based indexing
• Access to the "native" storage array for more
  arbitrary access
• Array "elements" can have multiple values (i.e.
  complex, multi-component)
• Designed to be extended to provide JavaSeis
  DistributedArrays
• Allow use of other array utility classes
  (java.util.Arrays, edu.mines.jtk.dsp.Array)

13
Transpose Operations

• TransposeType
• Java enum that defines the set of available
  transpose operations (i.e. T312, T1243, T21)
• ITranspose, Transpose
• Interface and pure java implementation
• In-place 2D transpose is the basic operation
• Extended to 132 transpose for 3D arrays
• Combinations yield full set of 3D transposes
• A single 1243 transpose provided for 4D

14
Message Passing

• IParallelContext
• Interface for the minimal set of message passing
  needed to support JavaSeis Parallel Arrays
• Send, Receive, getSize, getRank
• Barrier, Broadcast (optional)
• Shift, Transpose, BinaryTree built from the above
• Init and Finish

15
MPI for Java

• mpiJava from Syracuse University (NPAC) selected
  for SeisSpace
• Java wrappers for native MPI calls
• Support for sending serialized objects
• MPIContext implements IParallelContext
• MPICH for native methods
• Mpirun np 16 machinefile machines.txt java
  ClassName arguments

16
DistributedArray

• Extends MultiArray
• Requires IParallelContext for constructor
• Adds distributed tiled transpose (ttran)
• Last dimension is spread across processors
  (Decomposition, BLOCK or CIRCULAR)
• Transpose operations support arbitrary
  distribution of a single dimension
• Multiple decompositions possible but not
  currently supported

17
Decomposition

• Design concept from High Performance Fortran
• Default decomposition is BLOCK
• Allocates a fixed number of array indices per
  node
• Remainder is pushed to the edge, NOT evenly
  allocated
• May result in zero elements on high rank nodes
• Simple start,end indexing with stride 1
• CIRCULAR decomposition
• Round robin allocation
• Remainder spread across nodes
• Good for load balancing
• Permutation logic required to keep track of
  indices

18
BLOCK vs CIRCULAR Decomposition 13 array indices
on 4 nodes
141 581 9121 13131
1134 2104 3114 4124
19
Transform - Transpose Pattern
0
1
2
3
Time
0
1
2
3
X-Line
X-Line
InLine
InLine
Frequency
20
Transform - Transpose Pattern

• // Create a 3D distributed array
• DistributedArray a new DistributedArray(
  Seis.getParallelContext(), 3, float.class, new
  int 512,256,128, Decomposition.BLOCK )
• // Transform x axis of an array in xyz order
• computeTransform1D( a )
• // Transpose to yxz
• a.tran213()
• // Transform y axis
• computeTransform1D( a )
• // Transpose to zyx
• a.tran132()
• // Transform z axis
• computeTransform1D( a )
• // Transpose back to xyz
• a.tran321()

21
Distributed Array Padding

• Decomposition will likely have a remainder that
  requires padding
• Constructor allocates an array that accounts for
  padding
• Use constructor with an array of Decompositions
  if transpose operations will be used
• Index and range methods only traverse the live
  section of the array

22
Distributed Array Padding
Padded Array
Decomposition Padding
Partial Array Section
Live Section
23
Planned Additions

• Support for other patterns
• Transpose-Reduce
• Transpose-Overlap
• Arrays of Arrays optional variable length
• floata new float10
• for (int i0 ilt10 i)
• ai new floati
• Parallel Sorting
• Requires variable length array of arrays

24
Reduce - Transpose Pattern
PDO ( x, y f )
PDO ( x, y n )
PDO ( x, n y )
25
The Overlap-Transpose Pattern
Overlap-Expand Locally
Transpose to Distributed Overlap
Distributed array
0
1
2
3
0
1
2
3
0
1
2
3
26
Parallel Data Sort
Sort
27
Parallel Data Sort
Block parallel output
Variable length Transpose and resort within tile