FLASH IO and Visualization March 23, 2005

1
FLASH I/O and Visualization March 23, 2005

• Katie Antypas

2
The Flash Code
Shortly Relativistic accretion onto NS
Flame-vortex interactions
Compressed turbulence
Type Ia Supernova

• The Flash code
• Parallel, adaptive-mesh simulation code
• Designed for compressible reactive flows
• Has a modern CS-influenced architecture
• Can solve a broad range of (astro)physics
  problems
• Portable- runs on many massively-parallel systems
• Scales and performs well
• Is available on the web http//flash.uchicago.edu

Gravitational collapse/Jeans instability
Wave breaking on white dwarfs
Intracluster interactions
Laser-driven shock instabilities
Nova outbursts on white dwarfs
Rayleigh-Taylor instability
Orzag/Tang MHD vortex
Helium burning on neutron stars
Cellular detonation
Magnetic Rayleigh-Taylor
Richtmyer-Meshkov instability
3
Outline

• I/O Challenges with Scientific Computing
• speed, portability, file size
• serial, parallel multi-file, MPI-IO
• I/O Libraries
• hdf5
• parallel-netcdf
• Flash I/O specifics
• parameter input file
• checkpointing and restarts
• plotfiles
• Visualization
• runtime visualization
• fidlr3
• flashView

4
I/O Challenges with Scientific Computing
5
I/O Challenges Portability
Taking a basic binary output file from one
machine and trying to use it on another often
causes problems

• Portability
• big endian, little endian
• size of types vary
• data alignment in memory

4 byte int
2 byte int
int
int
int
double
24
0
4
8
12
16
20
6
I/O challenges File size and Speed

• File size
• Files are huge --in Flash a typical checkpoint
  file typically 5 GB and up to 50 GB for a
  research simulation (plotfiles 2GB)
• Files limited to 2GB with 32 bit addressing
• Need access to large storage areas
• Data transfer takes lots of time (.5 MB/sec from
  labs)
• I/O can be a major performance bottle neck
• slow file system
• writing to disk takes time

It isnt hard to have speed, portability or
usability. It is hard to have speed, portability
and usability in the same implementation
7
Serial I/O
0
1
2
3
4
5
processors

• Each processor sends its data to the master who
  then writes the data to a file
• Advantages
• Dont need a parallel file system
• Simple
• Disadvantages
• Not scalable
• Not Efficient

File
8
Parallel I/O Multi-file
0
1
2
3
4
5
processors
File
File
File
File
File
File

• Each processor writes its own data to a separate
  file
• Advantages
• Fast!
• Disadvantages
• can quickly accumulate many files
• hard to manage
• requires post processing

9
Parallel I/O Single-file
1
2
3
4
5
0
processors
File

• Each processor writes its own data to the same
  file using MPI-IO mapping
• Advantages
• single file
• scalable
• Disadvantages
• requires MPI-IO mapping or other higher level
  libraries

10
Parallel IO single file
0
1
2
3
4
5
processors
array of data
Each processor writes to a section of a data
array. Each must know its offset from the
beginning of the array and the number of elements
to write
11
I/O Libraries
12
I/O Libraries

• Flash works with 2 different I/O libraries
• HDF5
• Parallel-NetCDF
• Use MPI-IO mappings
• Both Portable libraries
• Both can translate data between systems
• Scientific Data mostly stored in multidimensional
  arrays

13
Parallel I/O under the hood
14
HDF5 library
HDF5 is a general purpose library and file format
for storing scientific data

• Can store data structures, images, arrays of
  vectors, grids, etc
• Can use basic HDF5 types integers, floats, reals
  or user defined types such as multi-dimensional
  arrays, objects and strings
• Stores metadata necessary for portability -
  endian type, size, architechture
• Handles files greater than 2GB

15
HDF5 Examples
ATTRIBUTE elapsed time DATATYPE
H5T_IEEE_F64LE DATASPACE SIMPLE (1)/(1) DATA
29.8322
Basic hdf5 type 64 bit floating point little
endian
ATTRIBUTE geometry name DATATYPE H5T_STRING
STRSIZE 80 STRPAD H5T_STR_NULLTERM CSET
H5T_CSET_ASCII CTYPE H5T_C_S1 DATASPACE
SIMPLE (1)/(1) DATA Cartesian
User defined string Length 80 Null terminated
16
HDF5 Virtual File Layer
HDF5
Virtual File Layer
stdio
File Family
Split Files
MPI-IO
Custom
Memory
Network

?
sequence of files
file
split metadata and raw data files
single file on parallel file system
user defined device
Memory
Network
schematic adapted from HDF5 documents
17
Parallel-NetCDF

• Parallel NetCDF isnt as flexible as HDF5 but our
  initial tests indicate it may be significantly
  faster
• Developed at Argonne National Lab
• Attempting to make PNetCDF sit closer to MPI-IO
• PNetCDF does not have a virtual file layer
• Like HDF5, PNetCDF can store data in multi
  dimensional arrays
• Currently cant write out non-contiguous data of
  varying sizes - coming soon with the ability to
  do MPI Datatypes

18
HDF5 vs. Parallel NetCDF
19
Flash I/O Specifics
20
Flash I/O directory tree
hydro
hdf5_parallel
hdf5_serial
io
amr
ncmpi
hdf4 (deprecated)
source
21
Parameter Input File - flash.par

• To change parameters in a simulation edit the
  flash.par file
• Defaults are set in module Config files and any
  value in the flash.par overides the Config values
• Controls parameters for all modules (examples
  below)
• Grid parameters - lrefine_max, lrefine_min,
  geometry
• Physics parameters - flame_speed
• I/O parameters - restart, cpnumber

22
Flash Checkpoint (restart) Files

• Checkpoint file -- a file that saves all the
  data you need to restart your simulation from the
  exact state it left off
• Grid Data
• refinement level
• nodetype (parent, child)
• bound box
• coordinates
• unknown variables (density, pressure, temperature
  etc)
• Simulation Data
• time
• dt
• timestep
• MetaData
• time stamp
• build directory
• setup problem etc

For any number of reasons your simulation could
be interrupted - the machine crashes, the queue
window closes, the machine runs out of disk space
etc. Rather than starting again from the
beginning of your simulation you can restart your
problem using a flash checkpoint files
23
Checkpoint Parameters

• There are a number of ways to control the
  occurrence and spacing of checkpoint dumps
• number of timesteps - (parameter nrstrt)
• number of seconds of simulation time -
  (parameter trstrt)
• wall clock seconds between dumps - (parameter
  wall_clock_checkpoint)
• very useful when running in a queue or when only
  a certain amount of time is available on a
  machine
• creating a file called .dump_restart will force a
  checkpoint file to be dumped immediately
• rolling checkpoint ability - (parameter
  rolling_checkpoint)
• checkpoint files are big! 5 - 50GB for a large
  science run
• this feature allows you to keep only a specified
  number of checkpoints to save disk space

The Flash Users Guide gives a detailed
explanation of each of the runtime parameters
controlling checkpoint output
24
Restarting a Simulation
To restart a simulation set the runtime parameter
restart .true. and the cpnumber parameter to
the checkpoint file number you want to restart
with in your flash.par

• Because the checkpoint files save the exact state
  of the simulation you can restart your simulation
  from any flash checkpoint file
• Can start with Parallel IO if previous run was
  written in serial
• Newly added in flash 2.5 you can start with an
  HDF5 file and continue writing checkpoints in
  Parallel-NetCDF format (also a file converter)
• Future working on restarting with a different
  state
• Can restart with a different number of processors
  than the previous run

25
Restarting on a different number of processors
0
1
2
3
4
5
processors
array of data
restart
0
1
2
processors
array of data
26
Flash Plotfiles
Plotfiles store data needed for visualization

• Plotfiles are smaller than checkpoints
• Only store data needed for visualization
• Data stored at 1/2 precision (check what is left
  out)
• Control Plotfile frequency with the runtime
  parameter tplot, the amount of simulation time
  between plotfile dumps

27
Particle Plot Files in Flash2

• Similar to plotfiles, can output files based on
  simulation time passed or number of timesteps
• Can write out a subset of particles by filtering
• Flexible output style control single or multiple
  files or multiple files after N particles have
  been dumped
• Runtime parameters that control particleplotfile
  output
• pptype LOGICAL if true output to a new file
  for each particleplot call, if false write to the
  same file
• nppart INTEGER if pptype is false still create
  a new file after nppart particles have been
  dumped
• pfilter LOGICAL if true output
  particleplotfile every timestep (user defined
  filter in place), if false dump based on ptplot
• ptplot REAL time elapsed between
  particleplotfile dumps
• pstruct LOGICAL if true dump particle data as
  mixed data type structure, if false split
  structure into arrays of the same type

28
Visualization
29
Visualization

• Runtime Visualization
• Fidlr3
• FlashView

30
Runtime Visualization

• Runtime Visualization module
• not automatically included -- can add it into
  your Modules file to set up problem with Runtime
  Viz
• Allows user a quick and dirty view of the
  simulation
• Only requires the widely available png library
• Creates png files in current directory during
  runtime

31
Runtime Visualization

• User can view any number of variables - density,
  pressure, temperature etc
• Parameters set in flash.par control how often
  plots are made
• Can write png files every n timesteps or every
  n seconds
• Can define various types of color maps
• Set dimensions of domain
• View in logscale
• Rescale, zoom

The Flash Users Guide gives a full explanation of
all the runtime parameters that control runtime
visualization
32
Fidlr3 (Flash IDL Routines)
Fidlr is an application written in idl to
visualize Flash checkpoint and plotfiles

• IDL tool for 2d visualization
• Available with flash directly
• Can step through plots to view sequences or see
  movies
• Can visualize 3d data by looking at slices
• Make histograms
• Additionally, if you have access to idl and want
  to do more analysis you can use the routines to
  read in data, statistical analysis, max min vaules

33
2d Fidlr Movie
34
Example of 3d slice taken by FILDR3
35
FlashView
FlashView is a 3d visualization package developed
for Flash data at Argonne National Lab (Mike
Papka, Randy Hudson)

• Features
• Reads native Flash plotfiles natively
• XYZ cut planes features so entire range of data
  for a variable can be projected on a plane
• Renders 3d iso-surfaces of Flash Data
• Fully parallel (both reading the file and
  calculations)
• Parallelism allows for the ability to look at
  very large data sets interactively
• Displays mulitple (up to 5) variables at the same
  time
• various color maps
• zoom, rotation, translation
• shows grid structure
• movies making
• Currently only supports hdf5 format

36
FlashView
37
White Dwarf rising bubble
38
Example of an isosurface (single variable)
39
Another example with two variables
40
Another example/ isosurface with cut plane
41
Isosurface with Grid
42
Issues with Cell-centered vs. Corner Interpolated
Data
43
Issues with cell-centered vs. corner interpolated
data
44
FlashView Future

• Continuing to enable the scientist
• Deployment of volume visualization system on
  Flash cluster
• Deployment of movie planning tools
• Continue vector visualization work
• Continue volume rendering work
• Paraview (in progress by viz group)
• Techplot translator (finished)
• Translator for EnSight

45
Shock Cylinder Movie