Spectroscopic Simulations on the HPC Grid

1
Spectroscopic Simulations on the HPC Grid

• or -
• Life is too short to wait 5 hours
• for fake spectra

2
Grid Pipelines
The Grid

• Using grid_pipeline

3
Grid Considerations

• Simulation of large datasets is impractical on a
  single CPU
• Grid jobs require about 30-40 seconds of overhead
  time, in addition to running time
• It is difficult to debug jobs running on the grid
• Grid jobs must be compiled for grid architecture
• CPU time is almost always cheaper that
  researcher/programmer time

4
Options for Batch Grid Submissions

• Write qss format scripts for each grid batch
• grid_pipeline
• Combine local and grid jobs in one file
• grid_batch
• Run the same command on multiple input files

5
Grid Batch Submission with grid_pipeline

• Interpreter for .grid file format
• Needs !/usr/bin/env grid_pipeline
• Can specify dependencies and parallelization of
  jobs
• Can run commands locally and on grid CPUs
• Can specify machine architecture
• Allows variable interpolation and command-line
  arguments
• Outputs qss grid submission scripts

6
Running grid_pipeline

• grid_pipeline
• grid_pipeline --scriptonly
• --dumpqueue
• --variables
• file.grid argument1 argument2 ...
• --scriptonly outputs file.grid.x only and does
  not submit anything
• to the grid. Useful for debugging.
• --dumpqueue prints the queue and prompts for
  approval before executing
• --variables outputs file.grid_vars.txt which
  contains variable values
• --quiet does not print output from local
  commands

Where file.grid contains commands inside of
curly braces to indicate parallel operation.
Curly braces, and commands, must be on lines by
themselves. Commands outside of braces are run on
the local machine. Curly brace blocks may not be
nested. Comments are preceeded by . Blank
lines are ignored. Variables representing
command line arguments may be specified by 1,
2, 3, etc. and are interpolated according to
the command line arguments. grid_pipeline
currently supports the following
definitions VAR name replacement replaces
the variable name with replacement QUEUE
queue.q specifies the queue for grid
jobs ARCH architecture specifies the
architecture for grid
jobs ARGS arg1 arg2 specifies the
arguments for usage
(not used for grid submission)
7
Format of .grid executables

• !/usr/bin/env grid_pipeline
• ARGS rootname profile
• ARCH lx24-x86
• VAR variable myparfile.par
• QUEUE brain.q
• run locally with command line args
• myshortcommand 1 2
• run these on the grid, in parallel
• mylongcommand 1_sim1 2 variable
• mylongcommand 1_sim2 2 variable
• mylongcommand 1_sim3 2 variable
• when all the grid jobs are done, do this
  locally
• myshortcommand 1 2

8
grid_pipeline Output

• one .csh file for each parallel block
• Used for grid submission
• filename.grid_1.csh
• qss output and error files for each parallel
  block
• Useful for debugging
• filename.grid_1.e1234.1

9
Spectroscopic Simulations
10
Methods Described

• Nelson, Sarah J. Analysis of Volume MRI and
  MR Spectroscopic Imaging Data for the Evaluation
  of Patients With Brain Tumors. Magnetic Resonance
  in Medicine 46228 (2001)
• Fortran programs
• make_ellipsoidimage
• generate_fid
• simulate_csfidkspace
• press_3dprofile
• vss_3dprofile
• process_spec

11
Principles for Simulating 3-D Spectra

• Generate a 3-D shape (x)
• Generate a characteristic spectral FID (t)
• Make PRESS and VSS profiles (x)
• Make k-space spectra (f, t)
• Add up k-space data (f, t)
• Add noise to k-space data (f, t)
• Fourier transform into spectra (x, f)
• Quantify metabolite peaks (x, f)

12
Simulation Programs

• Making shapes
• make_ellipsoidimage_v6
• Making spectra
• generate_fid_v6
• simulate_csfidkspace_v6
• Making profiles
• press_3dprofile_v6
• vss_3dprofile_v6
• Adding Noise
• csi_add_noise_v6

13
(No Transcript)
14
Making Shapes

• Theory
• Make separate shapes for each area with differing
  spectral characteristics
• Method
• Define using qbrain ROIs
• make_ellipsoidimage_v6
• define resolution, fov, ellipsoid locations

15
Making Spectra - FIDs

• Theory
• Simulate characteristic spectra using a sum of
  lorentzian peaks, and IFT
• Method
• generate_fid_v6
• Define number of points
• Define peak locations, heights

16
Making Profiles

• Theory
• Define parameters and effectiveness of PRESS and
  VSS selection
• Method
• press_3dprofile_v6, vss_3dprofile_v6
• DDF
• Effectiveness
• Width / Overpress

17
Making Spectra k-space

• Theory
• Use the spatial distribution of a shape, PRESS
  and VSS profiles, and PE matrix to generate an
  array of k-space FIDs
• Method
• simulate_csfidkspace_v6
• FID
• DDF
• Image

18
Adding Spectra

• Theory
• Combine k-space information from fat, brain, and
  lesion by addition
• Method
• add_spec_v6
• Spectra
• Weights

19
Adding Noise in the Time Domain

• Theory
• Add Gaussian noise to k-space FIDs to create a
  desired SNR for some peak
• Method
• csi_add_noise_v6
• SNR
• Reference peak ppm
• data in (f,t)

20
Processing Spectra

• Theory
• Fourier transform in each direction to create
  (x,f) spectra
• Method
• set_parameters_v6
• process_spec_v6

21
Data Inputs to Simulations

• There are only 2 real data inputs necessary to
  simulate spectra from scratch. Everything else
  is an input value.
• Profile DDF
• Coil
• Patient name, MRN, study date, etc
• SW, DT, ref freq, etc
• Fieldstrength
• Number of points
• PE matrix
• Lesion, if desired

22
Field Strength Considerations

• Profile DDF
• Sweepwidth
• Dwelltime
• Points
• Transmitter frequency
• ppm reference
• Inputs to
• generate_fid
• simulate_csfidkspace
• set_parameters
• csi_image

23
Multiple coil considerations

• Increased parallel processing
• Easy reconstruction available with existing
  processing programs
• Sensitivity maps
• Independent noise for each coil

24
Multiple Coil Implementation

• Multiply anatomical images by coil profiles
• Simulate k-space spectra for each coil image for
  each anatomical shape
• Add k-space spectra for each coil
• process_mc

25
Multi-Coil Computational Considerations

• 8 coil profiles
• 3 anatomical shapes
• 24 coil images of anatomical shapes
• 24 k-space images to compute
• 3 x 8 k-space images to add
• 3 noise additions
• process_mc

26
Parallel Processing for Simulations
The Grid
27
(No Transcript)
28
Outline of simulation .grid file

• !/usr/bin/env grid_pipeline
• make the brain and fat images
• make the associated spectra
• put the head together
• make the press and vss profiles
• make the k-space data for lesion, brain,and
  fat
• sum the spectra
• add noise to k-space dataset
• process data

29
A Real Example pg 1

• !/usr/bin/env grid_pipeline
• ARGS rootname lesion
• ARCH lx24-x86
• QUEUE brain.q
• VAR parfile myparfile.par
• VAR profileddf test.ddf
• VAR fieldstrength 1.5
• VAR waterlw 3
• VAR metlw 2
• VAR overpress 1.3
• VAR presseffect 0.95
• VAR vsswidth 30
• VAR vsseffect 0.8
• make fids and images
• make_brain images/1_brain
• make_scfat images/1_scfat
• make_fid -w waterlw -m metlw -B fieldstrength
  lesion spectra/1_lesion_fid

30
A Real Example pg 2

• send to grid lesion_sim, brain_sim, scfat_sim
• make_csfidkspace profileddf spectra/1_press_prof
  ile.int2 spectra/1_vss_profile.int2 2
  spectra/1_lesion_fid.cmplx
• make_csfidkspace profileddf spectra/1_press_prof
  ile.int2 spectra/1_vss_profile.int2
  images/1_brainminuslesion.int2
• make_csfidkspace profileddf spectra/1_press_prof
  ile.int2 spectra/1_vss_profile.int2
  images/1_scfat.int2
• make the sum of these images
• make_headsim spectra/1_brain_sim.cmplx
  spectra/1_lesion_sim.cmplx spectra/1_scfat_sim.c
  mplx spectra/1_head_sim
• now add noise at 3 different levels
• csi_add_noise_v6.dev -s snr1 -p naappm -o
  spectra/1_head_sim_snrsnr1 -b
  spectra/1_head_sim.cmplx
• csi_add_noise_v6.dev -s snr2 -p naappm -o
  spectra/1_head_sim_snrsnr2 -b
  spectra/1_head_sim.cmplx
• csi_add_noise_v6.dev -s snr3 -p naappm -o
  spectra/1_head_sim_snrsnr3 -b
  spectra/1_head_sim.cmplx
• send to grid process all data

31
Invocation

• chmod x mysim.grid
• mysim.grid mysim lesion.byt
• Options include
• --scriptonly
• --dumpqueue
• --quiet
• -h

32
Shell Script Output

• !/bin/csh
• -sync yes
• -N simulation.grid_1
• -l archlx24-x86
• -q brain.q
• -t 1-3
• -cwd
• if (SGE_TASK_ID 1) then
• echo HOST running make_csfidkspace
• make_csfidkspace test.ddf
  spectra/test_press_profile.int2
• endif
• if (SGE_TASK_ID 2) then
• echo HOST running make_csfidkspace
• make_csfidkspace test.ddf
  spectra/test_press_profile.int2
• endif

33
What to Expect in the Future

• More wrappers for simulation programs
• Coil sensitivity map simulations
• Multiple coil simulations
• More realistic press and vss profiles

34
Two examples

• cvs co classes/grid_pipeline
• cd classes/grid_pipeline/
• cd example
• ./example.grid test
• cd ../sim
• ./simulation.grid test circle.byt

35
Simulated Spectra
36
Metabolite maps