Combustion as a Sustained Petascale App

1
Combustion as a Sustained Petascale App
Jacqueline H. Chen Combustion Research
Facility Sandia National Laboratories Livermore,
CA jhchen_at_sandia.gov SOS11 Workshop on
Challenges of Sustained Petascale
Computation June 12-14, 2007 Key West,
Florida Supported by Division of Chemical
Sciences, Geosciences, and Biosciences, Office
of Basic Energy Sciences and ASCR Computing
ORNL NLCF, NERSC, PNNL, SNL
2
Coauthors and Collaborators

• SNL
• Evatt Hawkes (SNL, U. New South Wales) BES
  postdoc
• David Lignell (U. Utah, SNL) BES Ph. D. student
• Chunsang Yoo (SNL)
• Ramanan Sankaran (SNL, NCCS ORNL)
• External
• Chung K. Law, Princeton University
• Tianfeng Lu, Princeton University
• Rodney Fox, Iowa State University
• Mark Fahey, National Leadership Computing
  Facility
• David Skinner, National Energy Research
  Supercomputing Center
• Kwan-Liu Ma, H. Akiba, Hongfeng Yu UC Davis,
  Ultrascale Viz. Inst.
• Alok Choudhary, Wei-king Liao , Northweswtern U.
• Valerio Pascucci, LLNL, VACET
• SciDAC PERI

3
Combustion and energy security

• Combustion accounts for 85 of energy used in
  U.S.
• Transportation accounts for 2/3 of petroleum
  usage
• Potential for improvement in thermal efficiency
  by 25?50
• Diverse new fuel sources (bio-fuels, oil shale,
  oil sands, coal)
• Advanced low-temperature concepts for
  transportation require combustion operating at
  the edge
• Sound scientific understanding is necessary to
  develop predictive, validated multi-scale models!
• Savings of 3 million barrels of oil per day (out
  of 20M)

4
Turbulent combustion is a grand challenge!

• Stiffness wide range of length and time scales
• turbulence
• flame reaction zone
• Chemical complexity
• large number of species and reactions (100s of
  species, thousands of reactions)
• Multi-Physics complexity
• multiphase (liquid spray, gas phase, soot,
  surface)
• thermal radiation
• acoustics ...
• All these are tightly coupled

Diesel Engine Autoignition, Soot
Incandescence Chuck Mueller, Sandia National
Laboratories
5
Several decades O(10) of relevant scales

• Typical range of spatial scales
• Scale of combustor 10 100 cm
• Energy containing eddies 1 10 cm
• Small-scale mixing of eddies 0.1 10 mm
• Diffusive-scales, flame thickness 10 100 ?m
• Molecular interactions, chemical reactions 1
  10 nm
• Spatial and temporal dynamics inherently coupled
• All scales are relevant and must be resolved or
  modeled

Terascale computing 3 decades in scales
6
Combustion CFD approaches to tackledifferent
length scale ranges
RANS

• ReynoldsAveraged NavierStokes (RANS)
• coarse meshes, bulk approximation
• full range of dynamic scales modelled
• Large Eddy Simulation (LES)
• energetic scales resolved
• sub-grid scale dynamics modelled
• Direct Numerical Simulation (DNS)
• all scales resolved, no sub-grid model
• limited range of scales (Re)
• Canonical laboratory configurations

Premixed swirl burner
LES
Canonical DNS
7
Measurements provide partial information at best

• Combustion diagnostics point, line, planar, or
  integrated line-of-sight of select species and
  temperature (handful of reactive intermediates,
  CO, NOx, CH2O, H2O, CO2)
• Velocity measured by PIV (2D map in unburnt
  gases only)
• Diagnostics affected by noise and sampling
• Diagnostics more limited at high pressure due to
  optical access and interference of 3-body
  reactions with diagnostic methods.

High-speed Chemiluminescence Imaging in a
combustion vessel near the lift-off length, Lyle
Pickett,SNL
8
Direct Numerical Simulation (DNS)

• DNS is a tool for fundamental studies of the
  micro-physics of turbulent reacting flows
• Full access to time resolved 3D fields
• chemistry-turbulence interactions
• Develop and validate reduced model descriptions
  used in macro-scale simulations of
  engineering-level systems

9
DNS then and now

• 2003 2D DNS with detailed chemistry

2006 3D DNS with detailed chemistry at Re 9000

• Enabled by
• DOE INCITE and LCF awards of large computer
  allocations
• Customized chemical mechanisms for DNS

10
Computing Resources

• DOE Office of Science Capability Computing
• LCF Computational Combustion End Station (3.6M
  cpu-hours, 2006)
• DOE Incite Award 2005 (2.5M cpu-hours on NERSC
  IBM SP)
• DOE Incite Award 2007 (6M cpu-hours on Cray-XT3,
  ORNL)
• SNL BES clusters
• BES Opteron cluster (144 nodes)
• BES visualization cluster (32 nodes, fast
  graphics cards)

Cray XT3, ORNL
IBM SP, NERSC
11
Chemistry Models for DNS

• Usual CH4-Air mechanisms are not suitable for DNS
• Custom chemistry for DNS
• By T. Lu and C.K. Law (Princeton U.)

• Starting with GRI1.2
• 32 species, 177 reactions
• Identify species for elimination
• Directed relation graph (DRG)
• Sensitivity analysis
• Eliminate unimportant species
• Quasi-steady state assumption for CH2OH, CH2,
  CH2(s), HCO
• Explicit algebraic relations
• No costly iterations
• Ethylene-air and n-heptane-air (high pressure,
  low temperature)

12
DNS Capability at Sandia
S3D is a state-of-the-art DNS code developed with
15 years of BES sponsorship.

• Solves compressible reacting Navier-Stokes
  equations.
• High fidelity numerical methods.
• 8th order finite-difference
• 4th order explicit RK integrator
• Hierarchy of molecular transport models
• Detailed chemistry
• Multi-physics (sprays, radiation and soot)
• From SciDAC-TSTC (Terascale Simulation of
  Turbulent Combustion)
• Fortran90 and MPI
• Highly scalable and portable

13
Parallelism

• Parallelism is achieved through 3D domain
  decomposition.
• Each MPI process is in charge of a piece of the
  3D domain.
• All MPI processes have the same number of grid
  points and the same computational load
• Inter-processor communication is only between
  nearest neighbors in 3D topology
• Large message sizes. Non-blocking sends and
  receives
• All-to-all communications are only required for
  monitoring and synchronization ahead of I/O

1
1
N
N
14
Partnerships with HPC Community

• Scalar and vector optimization of S3D (Mark
  Fahey, Ramanan Sankaran, ORNL, David Skinner,
  LBNL, SciDAC PERI Institute, David Bailey LBNL)
• Collective I/O (M. Fahey, ORNL, Choudhary and
  Liao, Northwestern U.)
• Multi-core programming paradigm (Fahey, Cray)
• Parallel Viz. (Ahern, ORNL, Ma, UC Davis)
• Topological Feature Segmentation and Tracking
  (Pascucci, LLNL)
• Combustion workflow in Kepler (Scott Klasky,
  ORNL)

15
S3D Parallel Performance

• Measure of performance cost of execution per
  grid-point per time-step
• Weak scaling test with 503 grid points per MPI
  thread

16
MPI communication times from fpmpi

• Benchmark run took 877s on 2048 XT3 cores and
  1047s on 4096 XT3 cores (similarly on XT4)
• Weak scaling test with 503 grid point per mpi
  thread
• Most of the increase is from MPI_Wait on
  non-blocking sends and receives

17
Achieving quantitative predictability requires
petascale computing

• Petascale computers needed to achieve relevant
  parameters spaces for turbulent combustion N
  Re9/4 turbulence plus flame scales (3-4 decades
  of scales)
• Relevant parameter regimes of real devices and
  laboratory-scale flames Regt15,000.
• Terascale computing ReO(10,000),
  fully-developed turbulence
• Turbulence-chemistry interactions requires
  transporting 20-80 species plus turbulence

18
Demonstrated Parallel Performance

• S3D has been used to perform several hero
  simulations in the past couple of years
• Enabled by large INCITE awards
• Continuous effort in porting and optimizing code
  on evolving architectures

Details on recent S3D production simulations Details on recent S3D production simulations Details on recent S3D production simulations
Oct 05 150M grid points, 16 variables Itanium cluster at PNNL
Nov 05 350M grid points, 16 variables 600 processors on IBM SP5 at NERSC
Nov 05 500M grid points, 16 variables 512 MSPs on X1E at NCCS
Dec 05 52M grid points, 18 variables 512 MSPs on X1E at NCCS
May 06 88M grid points, 18 variables 4800 processors on XT3 at NCCS
Sep 06 194M grid points, 18 variables 7200 cores on dual-core XT3 at NCCS
Dec 06 1B grid points, 14 variables 9000 cores on dual-core XT3 at NCCS
June 07 350M grid points, 24 variables Planned run of 20K cores on dual-core XT3/4 at NCCS June 2007
19
TurbulenceChemistry Interactions Revealed by DNS

• Extinction and Reignition in CO/H2 and
  Ethylene/Air Jet Flames
• Stabilization in a Turbulent Lifted H2/air Jet
  Flame in Vitiated Coflow

20
Turbulent nonpremixed combustion

• Fuel and air segregated
• Mixing limited
• Extinction
• Reignition
• Flame stabilization

21
Extinction and Reignition Objectives

• What are the prevalent modes of reignition?
• Edge propagation
• Autoignition
• Turbulent engulfment
• Joint DNS/expt. extinction/reignition in unsteady
  laminar counterflow (Uendo, Yoo, Frank Kaiser and
  Chen, 2007)
• How does the reignition depend on flame location
  relative to jet mean shear and fuel ignition
  chemistry?
• How are the turbulent jet motions correlated with
  extinction and reignition?

• Extinction holes
• in a lifted jet
• flame
• (courtesy
• R. Schefer)

22
Extinction and Reignition in a CO/H2 Jet Flame
Hawkes, Sankaran Chen 2006, 2005 DOE INCITE
award
Burning
Extinguished

• Understanding extinction/reignition in
  non-premixed combustion is key to flame stability
  and emission control in aircraft and power
  producing gas-turbines
• Discovered dominant reignition mode is due to
  engulfment of product gases, not flame propagation

Scalar dissipation rate

• The largest ever simulations of combustion have
  been performed to advance this goal
• 500 million grid points (Re 2500-9000)
• 40CO/10H2/50N2 (11 species and 21
  reactions) Li et al. 2006
• 16 DOF per grid point
• 35 TB raw data
• 2.5M hours on IBM SP NERSC (INCITE)
• 400K hours on Cray X1E (ORNL)

Rendering by Yu and Ma
23
Description of Runs- Temporally Evolving
Non-premixed Plane Jet Flame
Streamwise BC periodic
Spanwise BC periodic

• Initial condition

• Later time

• Jet develops temporally.
• Shear-driven turbulence interacts with the flame.
• CO/H2 detailed chemistry (Li et al. 2006), Da
  0.01, 50CO 10 H2 40 N2 25O2 75N2,
  stoich. mixture fraction of 0.42.

24
Reynolds Number Effects on Mixing
Case H Re9000
Case M, Re4500
Case L Re2500

• Higher Re
• more fine-scale intermittent structure
• higher fluctuations of ?

25
How is extinction correlated with local mixing
rates?

• Scalar Dissipation (mixing rate)

• Extinguished Regions

26
Quantification of Extinction- Extinguished Flame
Area

• Reaction rate related to the conditional
  fine-grained surface-density of the
  stoichiometric surface
• Isosurface extraction from volume data through
  triangulation
• Data analysis on iso-surface and local normal
  vector
• Identify flame holes
• Scalar threshold ? (YOH gt0.0007)
• Half of steady strained extinction value
• Flame edge analysis
• Edge propagation speed, Se

Se
27
Joint PDF Reveals High Edge Speeds at High ?

• Color scale Joint PDF, Black line conditional
  mean speed.
• First, mainly negative speeds, strong negative
  correlation with ?.
• Then, broader PDF, with 2 branches
• negatively correlating branch at very high ?
• positively correlating branch at low-intermediate
  ?
• Peak positive edge speed occurs at quite high
  ?!!!

Simulation Time
28
Interpretation

• ugtgtsL indicates laminar edge flame propagation
  unimportant.
• Expect reignition by turbulent flame-folding.
• To bring burning and non-burning surfaces
  together, compressive strain is required, leading
  to high dissipation.
• Interpretation consistent with conditional edge
  speed and alignment statistics

29
Apriori Modeling of Conditional Reaction Source
in Multi-Environment Conditional PDF Model
Smith, Fox, Hawkes and Chen, 2006

• Multi-Environment Conditional PDF Model (R.
  Fox)
• Uses Gaussian quadrature to account for
  additional dimensions of probability space
• Accounts for mixing between regions of differing
  dissipation, macroscopic transport of progress
  variables, and variations in conditional
  dissipation
• DNS provides detailed scalar, reaction rate and
  dissipation data needed to validate model.

Weight
30
Community Data Sets

• Precedents for comparison of measured and modeled
  results
• Turbulent Nonpremixed Flame workshop
• http//www.ca.sandia.gov/TNF/abstract.html
• Premixed Flame workshop
• http//eetd.lbl.gov/aet/combustion/workshop/w
  orkshop.html
• High-fidelity numerical benchmarks for model
  validation and development (no noise, known
  upstream b.c.s)

31

• Extinction/ Reignition in Turbulent Ethylene-Air
  Flames

Temperature during reignition

• Extinction-reignition in C2H4
• Temporal jet configuration
• Reduced C2H4 mechanism (Lu and Law 2006) (18
  species, 15 global steps, 167 elementary
  reactions, 10 quasi-steady state species)
• Bimodal vs Monomodal scalar PDFs
• Study reignition modes
• Edge flame propagation
• Flame folding
• Auto-ignition

?st 0.17
32
Ethylene Flame Global Extinction, Reignition

• Ethylene extinction/ignition gives a bimodal
  chemical state.
• Ethylene jet experiences nearly complete
  extinction.
• Minimum 2 flame burning
• Impact on reignition mode
• timescales of edge propagation, turbulent
  transport, ignition delay time

33
Timescale Analysis for Re-ignition by Autoignition

• Eigenvalues of the reaction rate and temperature
  rate Jacobian are (inverse of) chemical
  timescales of fundamental reaction rate modes.
• Positive eigenvalues correspond to explosive
  modes.
• Compute the most positive mode in the domain to
  indicate ignition kernels.

T (K)
YHO2
Log(1/Tau)
t0.34 ms
34
CSP Analysis Participation and Importance Index
Lu, Law, Lignell and Chen, in preparation 2007

• Participation index What are the
  rate-controlling elementary reactions
• contributing to the most explosive mode?

• Importance index Which elementary reactions are
  most important
• in controlling the heat release rate?

35
Multi-resolution Feature Definition Coupled With
Parallel Feature Detection and Tracking

• Science Goal Correlate dynamics of turbulence
  with
• key scalars during extinction and reignition
• Feature Definition multi-scale representation
  of topological information based on Morse theory
  
• Detection/Tracking Algorithms
• Modular Abstract Interfaces
• Multiple steps detection, segmentation
  tracking
• multiple algorithms threshold, overlap, etc.
• Novel multiple concurrent feature definitions
  and hierarchical features
• Parallel
• Perform feature tracking on each processor
• Stitch together results off-line or on-line
• Single optimized communication cycle per time
  step with adjacent neighbors, and build global
  information.
• Collaborate with computer scientists
• Kwan-Liu Ma (UC Davis) feature tracking
• Valerio Pascucci (LLNL) Morse theory

Multi-resolution representation of topology
36
Summary of Ethylene-Air Extinction/Reignition

• Bimodal scalar pdfs
• More global quenching than CO/H2
• Primary reignition mode is by auto-ignition of
  reactant mixture with hot intermediates and
  products
• Autoignition is primarily by thermal explosion
  versus radical chain branching
• Contribution from turbulent transport of heat and
  radicals to ignition kernels (existing radical
  pool)
• CSP analysis useful in identifying rate limiting
  steps and reactions contributing to heat release
  rate and species reaction rates.

37
Stabilization of Lifted Hydrogen Jet Flame in
Vitiated Coflow
Yoo, Sankaran and Chen, 2007

• Lifted flame base
• Determine stability and characteristics of
  overall lifted flame
• Stabilization mechanism
• Effect of degree of fuel-air pre-mixing
• Premixed flame theory (Vanquickenborne and
  Tiggelen, 1966)
• Laminar flamelet theory (Peters and Williams,
  1983)
• Edge flame theory (Buckmaster, 2002 Favier and
  Vervisch, 1998)
• Effect of turbulent flow
• Turbulent intensity theory (Vanquickenborne and
  Van Tiggelen, 1966 Kalghatgi, 1984)
• Large eddy concept (Broadwell et al, 1984 Su et
  al, 2006)
• Critical scalar dissipation concept (Peters and
  Williams, 1983)
• Effect of preheating autoignition
• Another stabilization mechanism (Cabra et al,
  2002)
• Exclusively or inclusively affect stabilization
  of flame base

38
Turbulent Hydrogen Lifted Flame

• Find the stabilization mechanisms in lifted,
  vitiated flames
• Turbulent hydrogen lifted flame
• High flow velocity to lift off flame (347m/s)
• Possible auto-ignition due to high co-flow
  temperature (1100K)
• Approximately 2.5M CPU hours were used on XT3 at
  NCCS
• (Li et al. 2006) 9 species w/ 21 elementary
  reaction steps
• Lx x Ly x Lz 24 x 32 x 6.4mm3 w/ 1600 x
  1372 x 430 grid resolution ( 944M grid points)
• u?/U 0.1 (at the inlet)
• Rejet 11000, Ret 360 (at 1/4Lx at the
  centerline)

Isocontours of temperature at (a) t 0 and (b)
0.03ms
39
Instantaneous Flame Structure - OH
YOH isocontours on top of stoichiometric mixture
fraction iso-surface
40
OH mass fraction
YOH (color flood) and stoichiometric mixture
fraction line (white)
YOH (color flood) and YH2O 0.05 and 0.001
(white)
41
Scalar dissipation rate
? (color flood) and stoichiometric mixture
fraction line (white)
? (color flood) and YH2O 0.05 and 0.001 (white)
42
Structure of the Lifted Jet Flame
Isocontours of temperature, heat release rate,
YOH and YHO2 on z 0 plane at 6 flow-through time

• Flame base stabilizes on lean mixture rather than
  stoichiometric mixture (red line)
• Hydroperoxy radical (HO2)
• Precursor of auto-ignition in hydrogen-air
  chemistry
• Builds up upstream of OH and other intermediate
  radicals (H and H2O2)
• Indicates auto-ignition should be stabilization
  mechanism

43
Ignition at Flame Base
HO2
OH
Isocontours of (a)YOH and (b)YHO2 on z 0 plane
with velocity vector (arrowed white line) and ?st
(red line) from 0.37 to 0.44ms with 0.01ms
increment (left flame branch).

• Auto-ignition in lean and rich mixture
• HO2 always exists upstream of OH
• Ignition occurs in lean mixtures due to hot
  coflow and shorter ignition delay compared to
  stoichiometric and rich mixture
• Sometimes, ignition occurs in rich mixture right
  after extinction (0.410.43ms)
• Vortex assists flame base stabilization

44
Vorticity Generation

• Density in mixing layer is lower than in hot
  coflow and in cold fuel jet
• Turbulent eddies ahead of flame base
• Locally lowest pressure region in the center
• Positive vortex is generated by baroclinic torque
• Its magnitude is 12 order of magnitude larger
  than vortex generation by flow expansion or by
  vortex stretching
• Reduce the incoming axial velocity

Schematic of vortex generation by baroclinic
torque
45
Flame base in time and space averaged values

• Same as instantaneous flame characteristics
• Near flame base, slightly negative axial velocity
  is observed
• Assists stabilization of flame base.
• Flame base lies in lean mixture
• Ignition delay in lean mixture is short compared
  to stoichiometric and rich mixture with same
  temperature
• Ignition delay in lean mixture gets shorter due
  to hot coflow

Isocontours of temperature and YOH with mixture
fraction (dotted white line) and streamlines
(arrowed line) averaged in time and z-direction
46
Flame structure at different axial locations
6
7.5
9
12
18
21.5
Isocontours of heat release rate with flame index
0.005/mm2 (red line) and -0.005/mm2 (dashed
white line) at different axial locations (6, 7.5,
9, 12, 18 and 21.5mm) at 6 flow-through time

• Flame index (?YF??YO) indicates premixed and
  nonpremixed combustion
• Main region of heat release rate moves from lean
  to stoichiometric and to rich mixture as the
  flame develops
• Near domain boundary, rich premixed and diffusion
  flame dominate the main heat release

47
Summary Lifted Igniting Flames

• Stabilization mechanism is due to autoignition
  upstream of flame base and recirculation region
  generated by baroclinic torque ahead of flame
  base
• Stabilization occurs under fuel-lean conditions,
  lower ignition delay time and lower scalar
  dissipation rate.
• HO2 concentration plays an important role in the
  auto-ignition region.

48
Petascale Run n-Heptane Lifted Autoigniting Flame

• Effect of multi-stage ignition kinetics on
  stabilization of the lifted, high pressure diesel
  jet.
• Does flame stabilize on cool flame
  intermediates?
• Effect of lift-off height on soot precursors
• Custom chemistry reduction 65 species, 167
  reactions (further reduction may be possible,
  ongoing work)
• Requires 30 million cpu-hrs on a petascale
  computer!

Propane lifted flames in hot coflow (from Kim et.
al, Proc. Combust. Inst. 31, 2007 to appear)