Opportunities in Parallel I/O for Scientific Data Management

1
Opportunities in Parallel I/O for Scientific Data
Management

• Rajeev Thakur and Rob Ross
• Mathematics and Computer Science Division
• Argonne National Laboratory

2
Outline

• Brief review of our accomplishments so far
• Thoughts on component coupling
• Topics for future work

3
PVFS2

• Collaborative effort between ANL, Clemson,
  Northwestern, Ohio State, etc.
• Very successful as a freely available parallel
  file system for Linux clusters
• Also deployed on IBM BG/L
• Used by many groups as a research vehicle for
  implementing new parallel file system concepts
• True open source software
• Open source development
• Tightly coupled MPI-IO implementation (ROMIO)
• Forms the basis for higher layers to deliver high
  performance to applications

4
PVFS2 Performance
5
PVFS2 Performance
Time to Create Files Through MPI-IO
6
PnetCDF

• Parallel version of the popular netCDF library
• Major contribution of the SDM SciDAC (funded
  solely by it)
• Collaboration between Argonne and Northwestern
• Main implementers Jianwei Li (NW) and Rob Latham
  (ANL)
• Addresses lack of parallelism in serial netCDF
  without the difficulty of parallelism in HDF
• Only minor changes to the standard netCDF API
• Being used in many applications

7
MPI-IO over Logistical Networking (LN)

• LN is a technology that many applications are
  using to move data efficiently over the wide area
• Implementing MPI-IO over LN enables applications
  to access their data directly from parallel
  programs
• We are implementing a new ROMIO ADIO layer for LN
  (Jonghyun Lee, ANL)
• Nontrivial because the LN API is unlike a
  traditional file system API
• Collaboration between Argonne and Univ. of
  Tennessee

Application
MPI-IO
ADIO
LN
PVFS
UFS
Local Storage
Remote Storage
8
Fruitful Collaborations

• Key to our successes in this SciDAC have been
  strong collaborations with other participants in
  the Center
• Northwestern University
• PnetCDF, PVFS2
• Jianwei, Avery, Kenin, Alok, Wei-keng
• ORNL
• Nagizas group
• MPI-IO and PnetCDF for visualization (parallel
  VTK)
• LBNL
• Ekow (MPI-IO on SRM)
• Ongoing collaboration with Univ. of Tennessee for
  MPI-IO/LN

9
Component Coupling via Standard Interfaces

• We believe that well-defined standard APIs are
  the right way to couple different components of
  the software stack
• Having the right API at each level is crucial for
  performance

Application
HDF-5
PnetCDF
MPI-IO
Lustre
PVFS
GPFS
10
Topics for Future Work
11
Guiding Theme

• How can we cater better to the needs of SciDAC
  applications?

12
Make Use of Extended Attributes on Files

• PVFS2 now allows users to store extended
  attributes along with files
• Also available in local Linux file systems, so a
  standard is emerging
• This feature has many applications
• Store metadata for high-level libraries as
  extended attributes instead of directly in the
  file
• avoids the problem of unaligned file accesses
• Store MPI-IO hints for persistence
• Store provenance information

FILE
Xattr NameMesh Size Value1K x 1K
13
Next Generation High-Level Library

• HDF and netCDF were written 15-20 years ago as
  serial libraries
• Explore the possibility of designing a new
  high-level library that is explicitly built as a
  parallel library for modern times
• What features are needed?
• Can we exploit extended attributes?
• Can the data span multiple files instead of one
  file, with a directory as the object?
• What is the portable file format?
• New, more efficient implementation techniques

14
Implement Using Combination of Database and
Parallel I/O

• Use a real database to store metadata and a
  parallel file system to store actual data
• Flexible and high performance
• Powerful search and retrieval capability
• Prototype implemented in 1999-2000 (published in
  SC2000 and JPDC)
• Jaechun No, Rajeev Thakur, Alok Choudhary
• Needs more work collaboration with application
  scientists
• Serializability/portability of data is a
  challenge
• What is the right API for this?

Application
SDM
Data
Metadata
MPI-IO
Database
Berkeley DB, Oracle, DB2
Parallel file system
15
Parallel File System Improvements

• Autonomic
• Self-tuning, self-mantaining, self-healing
• Fault tolerant
• Tolerate server failures
• Scalability
• Ten to hundred-thousand clients
• Active storage
• Run operations on the server, such as data
  reduction, filtering, transformation

16
End-to-End Data and Performance Management

• Applications run and write data at one site (say
  NERSC)
• Scientists need to access the data at their home
  location, which is geographically distant
• Need high-performance and management of this
  whole process
• We intend to focus on ensuring that our local
  access tools (PVFS, MPI-IO, PnetCDF) integrate
  well with other tools that access data over the
  wide area (SRM, Logistical Networking, Gridftp)

17
Summary

• Despite progress on various fronts, managing
  scientific data continues to be a challenge for
  application scientists
• We plan to continue to tackle the important
  problems by
• focusing on our strengths in the areas of
  parallel file systems and parallel I/O
• collaborating with other groups doing
  complementary work in other areas to ensure that
  our tools integrate well with theirs