Anthony Mezzacappa

1
TeraScale Supernova Initiative
2
TeraScale Supernova Initiative
www.tsi-scidac.org
Explosions of Massive Stars

• Relevance
• Element Production
• Cosmic Laboratories
• Driving Application

• 11 Institution, 21 Investigator, 34 Person,
  Interdisciplinary Effort
• ascertain the core collapse supernova
  mechanism(s)
• understand supernova phenomenology
• e.g. (1) element synthesis, (2) neutrino,
  gravitational wave, and gamma ray signatures
• provide theoretical foundation in support of OS
  experimental facilities
• (RHIC, RIA, NUSEL, )
• develop enabling technologies of relevance to
  many applications
• e.g. 3D, multifrequency, precision radiation
  transport
• serve as computational science testbed
• drive development of technologies in simulation
  pipeline
• (data management, networking, data analysis,
  and visualization)

With ISIC and other collaborators 89 people from
28 institutions involved.
3
Anatomy of a Supernova

• Core collapse supernovae are
• radiatively (neutrino) driven.
• Fluid instabilities, rotation, and
• magnetic fields will play a role.
• Strong (Einsteinian) rather than
• weak (Newtonian) gravity.

Bruenn, DeNisco, and Mezzacappa (2001)

• Components of a Supernova Model
• 1. Accurate (Boltzmann) Neutrino Transport
• 2. Turbulent Stellar Core Flow
• 3. Stellar Core Magnetic Fields
• 4. Gravity (Einsteinian)
• 5. Nuclear (Stellar Core) and
• Weak Interaction (Neutrino) Physics

4
TSIs 3 Paths to 2D/3D Supernova Models
1) 2D/3D Hydrodynamics/MHD with Radial Ray
Transport Use existing 1D codes along each
ray. Transport parallelized trivially (one ray
per processor). 2) 2D/3D
Hydrodynamics/MHD with 2D/3D Multigroup
Flux-Limited Diffusion (MGFLD) Fully
2D/3D. Sophisticated approximation to Boltzmann
transport. Less computationally intensive than
Boltzmann transport. Access to 3D science on
TeraScale (versus PetaScale) platforms. 3) 2D/3D
Hydrodynamics/MHD with 2D/3D Boltzmann
Transport Final word. Explosion mechanism
extremely sensitive to transport treatment. MGFLD
compares well with Boltzmann in 1D. What about
2D? 3D?
5

• TSI members first to perform 1D models with
  Boltzmann transport.
• Mezzacappa and Bruenn (1993)
• Mezzacappa et al. (2001)
• Progress on 2D (and 3D) Boltzmann transport
  progressing rapidly.
• Development of formalism for
• conservative (energy and lepton number)
• general relativistic
• neutrino transport (analytical tour de
  force).
• Cardall and Mezzacappa (2003)
• Development of finite differencing.
• Construction of GenASiS.
• Completion of test problems.
• Initiation of realistic 2D supernova studies.

Without this, supernova simulations difficult to
interpret.
What makes neutrino transport difficult?
1. Difficult to develop number- and energy-
conservative differencing for these observer
corrections (aberration, frequency shift). 2.
Difficult to handle advection terms when
neutrinos and matter are in equilibrium. 3.
Memory and CPU requirements.
6
2D Boltzmann Neutrino Transport Test Problem
Development of radiation field stationary state
in nonspherical fixed medium
Density Distribution
Radiation Field Energy Density and Flux
7
Solve for first moment of neutrino
distribution (truncation of 2N-1 moments obtained
with Boltzmann solution).
Observer Corrections
Advection Terms
2D MGFLD Equations
8

• First 2D simulations with multifrequency neutrino
  transport,
• advection terms, and observer corrections
• Scientific Target Development of Proto-Neutron
  Star Instabilities
• Close coupling of matter and neutrinos requires
  fully 2D transport
• for an accurate assessment.
• What impact do the neutrinos have on the
  development of these instabilities?

• Running on 1024
• processors at
• NERSC.
• Scaling now to 2048.
• Fully implicit solve.

Swesty and Myra (2004)
9
Initial shock location/strength depend on
knowledge of nuclear states and their
occupation during core collapse.
This is a challenge in nuclear computation being
addressed by TSIs nuclear theorists.
This challenge is exacerbated by the fact that
nuclei increase in size (neutron and proton
number) /complexity (population of states,
collective excitations) during collapse.
10
Significant change in initial shock location and
strength and stellar core profiles when state of
stellar core nuclei computed with more realistic
nuclear models and when this new nuclear physics
is included in the supernova models. Hix et al.
2003, Physical Review Letters, 91,
201102. Langanke et al. 2003, Physical Review
Letters, 90, 241102.
Merger of two fields at their respective states
of the art. (SciDAC enabled.)
11

• In addition to physics of nuclei during collapse,
  supernova mechanism depends on
• high-density physics of nuclear matter.
• Complex nuclear many-body problem.
• How sensitive is the supernova mechanism to
  changes in this physics?

Nuclei
Nuclear Matter
TSI using several equations of state (EOSs). 1.
Lattimer-Swesty (LS, Industry Standard) 2.
Wilson 3. Stone-Newton (SN, New) LS and SN EOSs
are both phenomenological at some level, but
fundamentally different. Will allow us to explore
sensitivities to high-density EOSs despite
uncertainties.
12
2 fundamentally new instabilities discovered by
TSI (computationally)
Stationary Accretion Shock Instability (SASI)

• Supernova shock wave may become unstable.
• Instability will
• help drive explosion,
• define explosions shape.
• Operates between the proto-neutron star and
• supernova shock wave.
• Blondin, Mezzacappa, and DeMarino (2003)

Lepto-Entropy Fingers
Operates in the proto-neutron star. Instability
may help boost neutrino luminosities, which power
the explosion. Bruenn, Raley, and Mezzacappa
(2004)
13
SASI Visualized with EnSight
14

• r-process Element Synthesis
• Responsible for half the elements heavier than
  iron.
• Believed to occur above proto-neutron star after
• explosion in a neutron-rich, neutrino-driven
  wind.
• In past, difficult to obtain right conditions
  in
• supernova simulations.
• Neutrinos driving wind also drive it toward
• proton richness.
• TSI Discovery An r-process can occur under such
  conditions

Near Symmetric (equal numbers of protons
and neutrons)
for rapid wind velocities
Meyer 2002, Physical Review Letters, 89, 231101
15
Correspondence between structure of integro-PDE
and underlying linear systems...

• Leads to Nonlinear Algebraic Equations
• Linearize
• Solve via Multi-D Newton-Raphson Method
• Solve Large Sparse Linear Systems

• Implicit Time Differencing
• Extremely Short Neutrino-Matter
• Coupling Time Scales
• Neutrino-Matter Equilibration
• Neutrino Transport Time Scales

Tera- to Peta-Scale Systems
16

• Progress (in conjunction with TOPS)
• 2D/3D MGFLD
• Sparse Approximate Inverse Preconditioner
• Saylor, Smolarski, and Swesty (2004)
• Successfully implemented in 2D MGFLD code (V2D).
• 2D/3D Boltzmann Transport
• ADI Preconditioner
• DAzevedo et al. (2004)
• Successfully implemented in 1D Boltzmann code
  (AGILE-BOLTZTRAN).
• Dense LU factorization was used for dense blocks
  (DAzevedo).
• Being implemented in 2D/3D Boltzmann code
  (GenASiS).
• Sparse incomplete LU factorization for dense
  blocks (DAzevedo, Eijkhout).

17
TSI Radiation Magnetohydrodynamics Code Suite

• Hydrodynamics
• VH-1 (PPM)
• ZEPHYR (Second Order Finite Difference)
• Zeus-MP (Second Order Finite Difference)
• Magnetohydrodynamics
• Zeus-MP (Second Order Finite Difference)
• Neutrino Transport
• MGFLD_TRAN 1D General Relativistic
  Hydrodynamics
• with 1D General Relativistic
  Multifrequency Flux-Limited Diffusion
• AGILE-BOLTZTRAN 1D General Relativistic
  Adaptive Mesh Hydrodynamics
• with 1D General Relativistic
  Boltzmann Transport
• V2D/V3D 2D/3D MGFLD Transport Code (Under
  Development)
• GenASiS 2D/3D Boltzmann Code (Under Development)

• Zeus-MP
• Single-CPU performance boosted by factor of 2.
• Highest stat ever seen at NERSC (30 of peak).
• AGILE-BOLTZTRAN
• Parallel port, scalable linear solve, PERC
  analysis
• reduced run times from weeks to days.

Analysis by PERC recently begun.
Extensively analyzed by PERC.
18

• As TSI enters production mode managing its
  Workflows has become a paramount issue.
• Ideally, we would like to automate these
  workflows.

Data Management Networking Visualization
These must be viewed together.

• Collaboration between
• SDM
• Arie Shoshani
• Nagiza Samatova
• Guru Kora
• Ian Watkins
• Mladen Vouk
• Networking
• Beck
• Atchley
• Moore
• Rao
• Visualization (TSI)
• Blondin
• Toedte

19
Addressing Bulk Data Transfer Needs
Current Data Generation Rate 500 Mbps (10 TB in
2 days).

• Logistical Networking
• Light Weight
• Low Level
• Deployable Solution
• New Paradigm
• Integrate storage and networking.
• Multi-source, multi-stream.
• Easy for TSI members to share data.

• Data transfer rates 200-300 Mbps using TCP/IP!
• Limit set by ORNL firewall.
• Greater rates expected
• outside firewall,
• other protocols (e.g., Sabul).
• Direct impact on TSIs workflow!

Atchley, Beck, and Moore (2003)
20
TSI serving as a testbed for an NSF-funded
network (Cheetah) designed to develop
provisioning technologies for dedicated channels.
PI Rao TSI serving as a testbed for a proposed
effort to bring together UltraNet and
Cheetah. PI Rao
TSI is testbed for a proposed effort to allow a
broad research community to access and use
UltraNet. PI Beck
21
Visualization

• TSI members played a major role in
• development of Exploratorium/Powerwall
• at ORNL.
• Successfully deployed EnSight for
• production,
• remote, and
• collaborative visualization.
• Development of representations for
• higher dimensional data for 2D
• supernova models.
• Close coupling of
• data management,
• networking, and
• visualization.

• New Rendering Techniques/Quantitative,
  Comparative Visualization
• Kwan-Liu Ma (UC Davis), Pat McCormick, Jim
  Ahrens (LANL)
• Rapid rendering.
• Rendering in hardware.
• Interactivity.
• High-resolution, high-quality rendering.
• Quantitative visualization.
• Multiple variables, gradients and functions of
  variables.
• Computation done in hardware.
• Interactivity.

Kwan-Liu Ma (UC Davis)
22
Kwan-Liu Ma (UC Davis)
23
Conclusions
A number of scientific discoveries have been made
in TSIs first two years of operation. A number
of technical and technological breakthroughs have
enabled our science. Execution of the next major
stage in our science has begun Delivery of 2D
supernova models. Continue to serve as a testbed
for enabling technologies of relevance to other
SciDAC applications.