Inner Workings of FLASH

1
Inner Workings of FLASH

• Anshu dubey
• March 21, 2005

2
Overview

• What is FLASH
• Basics of FLASH2 Architecture
• Basics behind a problem setup
• Enough of an idea of these concepts to be able
  to look at a sample setup and understand what is
  happening.

3
FLASH Results

• Compressible reactive flow
• Wide range of length of time scales
• Many interacting physical processes
• Only indirect validation possible for the
  astrophysics
• Many people in collaboration

Flame-vortex interactions
Compressible turbulence
Shocked cylinder
Nova outbursts on white dwarfs
Intracluster interactions
Cellular detonations
White Dwarf deflagration
Helium burning on neutron stars
Rayleigh-Taylor instability
4
What FLASH Provides

• Physics
• Hydrodynamics
• PPM
• MHD
• Relativistic PPM
• Nuclear Physics
• Gravity
• Cosmology
• Particles

• Infrastructure
• Setup
• AMR Paramesh
• Regular testing
• Parallel I/O
• hdf5, pnetcdf,
• Profiling
• Runtime and post-processing visualization

5
A Little FLASH History
BAM

• FLASH0
• Paramesh2, Prometheus and EOS/Burn
• FLASH1
• Smoothing out the smash
• First form of module architecture inheritance
• FLASH2
• Untangle modules from each other (Grid)
• dBase
• Concept of levels of users
• FLASH3
• Stricter interface control module architecture
• Taming the database

6
FLASH2 Audiences
FLASH Application
Developer
End User
Application Programmer

• Initialize setup
• Boundary conditions
• Basic data structure
• (Might add a kernel)

• Works on just about everything
• Grid development
• Data access
• Architecture

• Develop physics modules
• Talk to grid
• Module communication

7
FLASH2 Code Basics

• An application code, composed of units/modules.
  Particular modules are set up together to run
  different physics problems.
• Performance, Testing, Usability, Portability
• Fortran, C, Python,
• More than 560,000 lines of code
• 75 code, 25 comment
• Very portable
• Scaling to 1000s of procs

Internal Release
8
Basic Computational Unit Block

• The adaptive grid is composed of blocks
• All blocks same dimensions
• Cover different fraction of the physical domain.
• Blocks at different levels of refinement have
  different grid spacing.

9
Structure of FLASH Modules (not exact!)
Materials
Hydro
Source_terms
Gravity
init() tstep() hydro3d()
init() tstep() grav3d()
init() tstep() src_terms()
eos3d() eos1d() eos()
10
Whats a FLASH Module?

• FLASH basic architecture unit Modules
• Component of the FLASH code providing a
  particular functionality
• Different combinations of modules are used for
  particular problem setups
• Ex driver, hydro, mesh, dBase, I/O
• Fake inheritance by use of directory structure
• Modules communicate
• Driver
• Variable Database

11
Abstract FLASH2 Module

• 1. Meta-data (Configuration Info)
• Interface with driver and setup
• Variable/parameter registration
• Variable attributes
• Module Requirements

FLASH Component

• 2. Interface Wrapper
• Exchange with variable database
• Prep data for kernels

• 3. Physics Kernel(s)
• Single patch, single proc functions
• written in any language
• Can be sub-classed

FLASH Application
driver
Collection of Flash2 Modules
Database
12
Module Implementations

• FLASH2 Modules are directory trees
• source/hydro/explicit/split/ppm
• Each level might have source
• Source relevant for all directories/implementation
  s below
• Preserves interfaces
• Allows flexible implementations

13
Inheritance Through Directories Hydro
An empty hydro init, hydro, tstep are
defaults on top of the directory tree.
init
hydro
tstep
Explicit
Hydro/Explicit Replaces tstep Introduces
shock No hydro Implemented yet!
tstep
tstep
shock
Split
Hydro/Explicit/Split hydro implemented Uses
general explicit tstep Uses general shock
Replaces init
hydro
hydro
implemtation
init
implemtation
DeltaForm
14
The Module Config File

• Declare solution variables, fluxes
• Declare runtime parameters
• Sets defaults
• Lists required, exclusive modules
• Config files are additive down the directory tree
  - no replacements

15
Setup Building an Application
Configuration Tool (Setup)
Mesh
Database
16
FLASH Setup Implements Architecture

• Python code links together needed physics and
  tools for a problem
• object
• Traverses modules to get implementations
• Determines solution data storage list
• Creates list of parameters from modules
• Configures Makefiles properly

17
Accessing the Mesh

• Physics and tools access some scope of mesh
• Database
• Unifies method for module to access data out of
  its scope
• Mesh data
• Solution Data, Grid information
• Runtime parameters
• Mesh Interface
• Wrappers to the public grid routines
• initialize, guardcell, fluxconserve,
  updateRefinment

18
Simple FLASH Flow
Function
Driver Init Evolve TimeStep
Variable Database
Data
dBase
Paramesh updateGrid ghostCellFill fluxConserve
Evolve Hydro SourceTerms UpdateGrid
Mesh
Hydro Prepare data for kernels
SourceTerms Burn Heat Cool Ioniz
hydroBlock
Burn burnBlock
19
Pulling it All Together

• Choose a problem setup
• Run setup to configure that problem
• Everything is in a new top-level directory
• object
• Make
• Run
• Flash.par for runtime parameters
• Defaults already set from particular modules

20
Setups

• A basic problem setup
• Config file
• Required physics modules
• Flash.par
• Default list runtime parameter configuration
• Init_block
• Initial conditions for the problem set block by
  block
• Many other possible files
• Driver, Refinement algorithms, User defined
  boundary conditions
• Any files in setup take precedence

21
Provided Driver

• Provided
• Second order, state form, strang split

• New drivers
• Put in setups
• Welcome contributions

set time step hydro sourceTerms cosmology radiatio
n particles gravity set time step (repeat
physics) Mesh_updateGrid
evolve.F90
flash.F90
Initialize() Loop over timesteps
evolvePhysics() timestep() output()
visualize() End loop Finalize()
22
Future of Flash

• Thats the next talk
• In the meantime
• Questions ?