Design and Optimization of Large Accelerator Systems through HighFidelity Electromagnetic Simulation

1
Design and Optimization of Large Accelerator
Systems through High-Fidelity Electromagnetic
Simulations

• Cho Ng
• Stanford Linear Accelerator Center
• SciDAC08 Conference
• Seattle, July 14 17, 2008

Work supported by US DOE Offices of HEP, ASCR
and BES under contract AC02-76SF00515 and grants
DE-FC02-07ER41499 and DE-FG02-04ER41317.
2
ComPASS Electromagnetic (EM) Collaborators
Stanford Linear Accelerator Center
(SLAC) Physicists A. Candel, A. Kabel, K. Ko,
Z. Li, C. Ng, L. Xiao Computatioal Scientists
V. Akcelik, S. Chen, L. Ge, L.-Q. Lee, G.
Schussman,
R. Uplenchwar TechX Corporation T. Austin,
J.R. Cary, S. Ovtchinnikov, D.N. Smith SciDAC
CETs/Institutes TOPS E. Ng, X. Li, C. Yang,
J. Demmel (LBNL), D. Keyes, B. Osting (Columbia),
O. Ghattas (UT Austin) ITAPS L. Diachin (LLNL),
X. Luo, M. Shephard (RPI), K. Devine
(SNL) CSCAPES K. Devine, E. Boman (SNL), A.
Pothen (Purdue) CScADs W. Gropp (UIUC) SIUV
K. Ma (UC Davis) Others R. Barrett, S. Hodson,
R. Kendal (ORNL), Z. Bai (UC Davis) SciDAC
support from Offices of HEP, ASCR and BES
3
EM Modeling in SciDAC1

• The Advanced Computing for 21st Century
  Accelerator Science and Technology (AST) project
  made tremendous progress under SciDAC1. In EM
  modeling
• A set of parallel high-accuracy finite-element
  codes was developed for high-fidelity accelerator
  simulations
• These codes were applied with significant impact
  to the design, optimization and analysis of
  accelerator facilities within Office of Science
  (HEP, NP and BES)
• Important advances in computational science
  research were achieved in collaboration with
  ISICs/SAPP partners under ASCR support

4
Highlights of Accomplishment in AST
Track3P
Omega3P
NLC cell design to machining accuracy
Dark current in 30-cell accelerator structure
T3P
Re-design of PEP-II interaction region leading to
new particle discoveries
MP Trajectory _at_ 29.4 MV/m
Omega3P
V3D
Track3P
Prediction of multipacting barriers in Ichiro SRF
cavity
RF studies of RIA RFQ
Discovery of mode rotation in superconducting
cavity
5
EM Modeling Objectives in ComPASS

• Based on the success of AST, ComPASS EM Modeling
  will strive for
• Tackling the most computationally challenging
  problems in design and optimization facing the
  DOE accelerator complex
• Enhancing the simulation capabilities by
  incorporating multi-physics effects for
  engineering prototyping of future facilities
• Further advancing computational science
  techniques to enable petascale accelerator
  simulations and maximize the use of ASCR resourses

6
High-Fidelity EM Modeling
0.5 mm gap
200 mm

• Complexity HOM coupler (fine features) versus
  cell
• Problem size multi-cavity structure, e.g.
  cryomodule
• Accuracy 10s kHz mode separation out of GHz
• Speed Fast turn around time to impact designs
  

7
Higher-order Finite-Element Method
LL end cell with input coupler only
dense
Error 20 kHz (1.3 GHz)

• Tetrahedral Conformal Mesh
• w/ quadratic surface
• Higher-order Finite Elements
• (p 1-6)
• Parallel Processing
• (large memory speedup)

67000 quad elements (lt1 min on 16 CPU,6 GB)
8
Parallel Finite-Element EM Codes

• Higher-Order Finite-Element (FE) Electromagnetics
  Codes
• Frequency Domain Omega3P eigensolver (mode
  damping, non-linear)
• S3P
  S-parameter
• Time Domain T3P transients
  wakefields
• Pic3P
  self-consistent particle-in-cell (PIC)
• Particle Tracking Track3P dark current
  and multipacting
• Gun3P
  space-charge beam optics
• Multi-Physics TEM3P
  EM-thermal-mechanical
• Visualization V3D meshes,
  fields and particles

9
Parallel FE Based Multi-Physics Code - TEM3P
TEM3P for design and optimization
CAD model of LCLS RF gun
Electromagnetics
Metal

Vacuum
Courtesy E. Jongewaard
Engineering prototype
Thermal
Mechanical
Courtesy D. Dowell
10
Parallel 3D FE PIC Code Pic3P
LCLS RF Gun 3D emittance calculations with Pic3P
include space-charge, wakefield and retardation
effects from first principles. Parallel
processing and conformal higher-order finite
elements allow unprecedented modeling accuracy.
Evolution of electron bunch and scattered
self-fields
Evolution of transverse phase space, starting
from SLAC measurement data
11
3D Space-Charge Beam Optics Code Gun3P

• L-band sheet beam klystron DC gun
• Gun3P includes an electrostatic and a
  magnetostatic solvers and PIC particle tracking
• Benchmarked against Michelle
• Parallel computation speeds up runtime by an
  order of magnitude

Input 115 kV Output 129 A
Gun3P
Michelle
144K tets 4.5M DOFs
Beam profile at cathode
12
Parallel Finite-Difference Time-Domain Code -
VORPAL

• Dey-Mittra embedded boundary algorithm for
  accurate representation of geometric boundaries
• Efficient frequency extraction from time-domain
  simulation that combines targeted excitation
  bandwidth with filter subspace diagonalization
• Richardson extrapolation for accuracy to next
  order

G.R. Werner and J.R. Cary, J. Comput. Phys.,
227, 5200 (2008).
Fermilab A15 Crab Cavity
Simulation shows that cavity radius is 25 mm
smaller than designed value, which was confirmed
later from measurements.
Courtesy L. Bellantoni
13
Realistic Simulation Design of Accelerators

• High Energy Physics
• LHC Impedance HOM heating in collimator,
• crab cavity design
• High Gradient Structures
• Choke-mode structure, CLIC PETS
• ILC Wakefield in cryomodule, HOM heating in
  Main Linac,
• multipacting in cavity coupler,
  damping ring impedance
• Nuclear Physics
• CEBAF Beam breakup in cryomodule of 12 GeV
  Upgrade
• Basic Energy Sciences

14
Supporting the LHC
LHC Collimator (Upgrade) Impedance and beam
heating effects are important for the
design. (Omega3P and T3P )
LHC Crab Cavity (Upgrade) The crab cavities
rotate the beams at the IP to produce head-on
collisions, improving luminosity. Design for
strong damping of SOM/LOM/HOM is needed. (Omega3P
)
15
High Gradient Structures _at_ SLAC CERN
Choke Mode Structure High gradient design with
strong HOM damping targeting external Qs of
dipole modes in the order of 10. (Omega3P )
Courtesy S. Pei
CLIC Power Extraction Transfer Structure
(PETS) A traveling wave structure for power
extraction uses absorbers along the vanes to
damp HOMs. (Omega3P and T3P )
16
Modeling an Entire RF Unit of ILC Linac
cavity
cryomodule
RF Unit of 3 cryomodules
Physics Goal Calculate wakefield effects in the
3-cryomodule RF unit (26 cavities) with realistic
3D dimensions and misalignments

• The LARGEST problem for time-domain analysis
• 80 million-element mesh, 500 million DOFs, 4096
  CPUs (Jaguar),
• 4 seconds per time-step.
• For frequency domain
• 3 million-element mesh, 20 million DOFs, 1024
  CPUs (Seaborg),
• 300GB memory, 1 hour per mode.

17
Beam Transit in Cryomodule with T3P
ILC cryomodule of 8 superconducting RF cavities
Expanded views of input and HOM couplers
T3P
Fields in beam frame moving at speed of light
18
Trapped Modes in ILC Cryomodule

• Trapped modes in 3rd dipole band

• Modes above cutoff frequency are coupled
  throughout 8 cavities
• Modes are generally x/y-tilted twisted due to
  3D end-group geometry
• Both tilted and twisted modes cause x-y coupling
  in the beam

Qext

• Trapped mode in beampipe between 2 cavities

• TM-like mode at 2.948 GHz, higher than 2.943 GHz
  TM cutoff
• R/Q 0.392 W, Q 6320
• Mode power 0.5 mW (averaged)
• (not a concern for heating in this case)

19
BBU in CEBAF 12 GeV Upgrade Cryomodule
Low-loss cavities
High-gradient cavities

• Tests show 3 abnormally high Q modes in 5 of
  the high-gradient cavities
• Beam-breakup (BBU) threshold current is
  significantly below design value
• Issues could not be resolved experimentally
• SLAC scientists have made great progress in
  finding a solution by treating it as an inverse
  problem

3 high-Q modes
Courtesy H. Wang, F. Marhauser, J. Sekutowicz,
C. Reece, R. Rimmer (TJNAF)
20
Inverse Problem for Shape Determination

• Solve an inverse problem to determine the
  deformed cavity shape
• Use measured rf parameters such as f, Qext, and
  field profile as inputs
• Parameterize shape deviations using pre-defined
  geometry variations
• Objective (function J ) - minimize weighted least
  square misfit of the computed and measured
  response
• The optimization algorithm typically converges
  within a handful of nonlinear iterations

Ref V. Akcelik et al., Shape Determination
for Deformed Electromagnetic Cavities, J.
Comput. Phys., 227, 1722 (2008).
21
CEBAF BBU Simulation Analysis
Ideal
Deformed

• Solutions to the inverse problem identified the
  main cause of the BBU instability Cavity is 8 mm
  shorter predicted and confirmed later from
  measurements
• The fields of the 3 abnormally high Q modes are
  shifted away from the coupler
• This demonstrates that Quality Control in cavity
  manufacturing is essential
• Success requires a multidisciplinary effort in
  accelerator modeling, applied mathematics and RF
  measurements

Omega3P
Field profiles in deformed cavity
HOM coupler
22
Multipacting in SNS HOM Coupler
HOM2

• SNS SCRF cavity experienced
• rf heating at HOM coupler
• 3D simulations showed MP
• barriers close to measurements
• Similar analyses for ILC cavities

Track3P
Expt. MP bands
23
SciDAC Collaborations under ASCR

• Advances in Computer Science Applied Math
  include
• Shape Determination Optimization (TOPS, ITAPS)
• Obtain cavity deformations from measured mode
  data through solving a weighted least square
  minimization problem
• Parallel Complex Nonlinear Eigensolver (TOPS)
• Develop scalable algorithms for solving LARGE,
  complex, nonlinear eigenvalue problems
• Parallel Adaptive Mesh Refinement (ITAPS)
• Optimize computing resources and increase
  solution accuracy through adaptive mesh
  refinement
• Dynamic Load Balancing (CSCAPES)
• Implement dynamic partitioning scheme to
  optimize computational load for particle-field
  simulations

24
Posters on ComPASS EM Modeling
See posters by L. Lee et al., Computational
Science Research in Support of Petascale
Electromagnetic Modeling M. Shephard et
al., Curved Mesh Correction and Adaptation Tool
to improve ComPASS electromagnetic analysis
25
Adaptive Refinement Moving with Beam

• Provide refined mesh only around the moving beam,
  thereby reducing computational resources by
  orders of magnitude

mm
X. Luo M. Shephard, ITAPS/RPI
T3P
p-refined moving window
h-refined moving window
26
Parallel Performance

• Weak scaling of VORPAL
• Normalize to 4 CPUs
• 95 efficiency on Franklin

• Strong scaling of Omega3P
• 1.5 million elements
• scale to 4000 CPUs with 95 efficiency on Jaguar

Parallel Efficiency
Number of CPUs
27
Summary

• Electromagnetic modeling, a main thrust of
  ComPASS, has a tremendous impact on the design
  and optimization of accelerator projects in DOEs
  Office of Science.
• Success is witnessed by continued development of
  the state-of-the-art parallel codes and their
  wide range of applications to important machines
  in HEP, NP and BES
• Advances in computer science and applied math
  remain to be the essential component that will
  enable realistic accelerator simulations to reach
  the petascale