Parallel MxN Communication Using MPIIO

1
Parallel MxN Communication Using MPI-I/O

• Felipe Bertrand, Yongquan Yuan,
• Kenneth Chiu, Randy Bramley
• Department Of Computer Science, Indiana
  University
• Supported by
• NSF Grants 0116050 and EIA-0202048
• Department of Energy's Office of Science
  SciDAC grants

2
Motivation

• Interesting problems span multiple regimes of
  time, space, and physics. Multiphysics,
  multidiscipline
• Climate models
• Combustion models
• Fusion simulation
• Protein simulation

3
Community Climate System Model (CCSM)

• CCSM models are parallel but the inter-model
  communication is serial

4
Existing Approaches

• Refactor codes integrate all components into a
  single, much larger program
• More efficient data sharing
• Large time investment in rewriting program
• Closer coordination between teams more costly in
  development, testing and deployment
• Component models
• Simplify the overall application development.
• Complicate the efficient coordination and sharing
  of data between components (MxN problem).

5
Goals

• Scalably connect codes to create new multiphysics
  simulations
• Target existing and currently evolving codes
• Created by teams with disparate research
  interests
• Spanning multiple time scales, spatial domains,
  and disciplines
• Rapid prototyping/testing without extensive
  rewriting
• Use standard APIs and paradigms familiar to most
  application area scientists (MPI I/O)

6
The MxN Problem

• Transfer data from a parallel program running on
  M processors to another running on N processors.
• M and N may differ
• May require complex all-to-all communications,
  data redistribution

7
Solving the MxN Problem

• Existing solutions
• Use process 0 on all components
• Used by CCSM model
• Not scalable
• Read/Write through files
• Scalable if parallel I/O used
• Slow because involves hard drive read/write
• Our solution
• Use MPI I/O interface, create middleware to
  transfer data via network
• Treat application codes as software components
• Provide easy migration path for existing
  applications

8
Solving the MxN Problem

• MPI-I/O defines an API for parallel I/O using
  file-like semantics.
• ROMIO is an implementation of MPI-IO.
• Provides an abstract device interface (ADIO) that
  allows different physical I/O mechanisms to be
  plugged in.

9
MxN MPI-IO Communication

• Application level
• Components live in different MPI instances
• Transparency
• Not aware of the MxN communication
• Reads and writes data through regular MPI
  interface.
• No change in the source code is required
• Switch to MxN backend with filename prefix
  mxn
• Communication can be established between
  different MPI ROMIO-based implementations.

10
MxN MPI-IO Communication

• MxN backend
• Logical serialization intuitive paradigm.
• Parallel implementation high performance.

11
MxN MPI-IO Communication

• MxN backend

12
MxN MPI-IO Communication

• Timing first MxN connection between discretizer
  and solver components.
• 4 discretizer processes, 16 solver processes

13
Thor Results Time vs. Bytes

• Socket serialized through process 0
• File PFVS parallel file I/O
• MxN new device

14
Future work

• Incorporate MxN communication system into a CCA
  component
• Explore standard API for MxN components
• Identify current computations challenges in
• areas of scientific application and design
  supporting middleware

15
References

• Ian Foster, David Kohr, Jr., Rakesh Krishnaiyer,
  Jace Mogill. Remote I/O Fast Access to Distant
  Storage. Proceedings of the Fifth Workshop on
  Input/Output in Parallel and Distributed Systems,
  1997.
• Climate and UCAR Global Dynamic Division.
  Community Climate System Model,
  http//www.cgd.ucar.edu/csm
• Message Passing Interface Forum.
• http//www.mpi-forum.org
• R..Thakur, W..Gropp, E. Lusk. An abstract-device
  interface for implementing portable parallel-I/O
  interfaces. In Proceedings of the Sixth Symposium
  on the Frontiers of Massively Parallel
  /computation, p.180, 1996
• S.A. Hutchinson, J.N. Shadid, R.S. Tuminaro.
  Aztec Users Guide Version 2.30. Sandia National
  Laboratories. http//www.cs.sandia.gov/CRF/aztec1.
  html, 1998

16
Extra
17
Goals

• Large-scale scientific computations span
  multiple scales, domains, disciplines developed
  by large and diverse teams (multi-physics
  simulations)
• Create multi-physics simulations using existing
  community parallel codes
• Rapid prototype/testing without rewriting codes

18
MxN Problem Defined

• The transfer of data from
• a parallel program running
• on M processors to another
• parallel program running on
• N processors. Ideally
• neither program knows
• the number of processes
• on the other one.

19
Solving MxN Problem

• Existing approaches
• Use process 0 on all components
• example CCSM models
• Read/Write through files
• Our Approaches
• Decouple application into components
• Provide easy migration path for existing
  application
• Enable an intuitive model
• Use MPI I/O interface

20
RI Support

• Critical to have cluster where we can install
  variant file systems, modified middleware such as
  ROMIO with new abstract devices
• Next phase components on different clusters
  fast network connection to university clusters
  critical for testing
• Storage updates allow ability to switch between
  MxN and standard file I/O.