Online Parallel Tomography

1
On-line Parallel Tomography

• Shava Smallen
• UCSD

2
Talk Outline

• I) Introduction to On-line Parallel Tomography
• II) Tunable On-line Parallel Tomography
• III) User-directed application-level scheduler
• IV) Experiments
• V) Conclusion

3
What is tomography?

• A method for reconstructing the interior of an
  object from its projections

• At the National Center for Microscopy and Imaging
  Research (NCMIR), tomography is applied to
  electron microscopy to study specimens at the
  cellular and subcellular level

4
Example
Tomogram of spiny dendrite (Images courtesy of
Steve Lamont)
5
Parallel Tomography at NCMIR

• Embarrassingly parallel

Z
specimen
slice
X
Y
projection
scanline
6
NCMIR Usage Scenarios

• Off-line parallel tomography (off-line PT)
• Data resides somewhere on secondary storage
• Single, high quality tomogram
• Reduce turnaround time
• Previous work (HCW 00)

• On-line parallel tomography (on-line PT)
• Data streamed from the electron microscope
• long makespan, configuration errors, etc.
• Iteratively computed tomogram
• Soft real-time execution

7
On-line PT

• Real-time feedback on quality of data acquisition
• ) First projection acquired from microscope
• ) Generate coarse tomogram
• ) Iteratively refine tomogram using subsequent
  projections (refresh)
• Update each voxel value
• Size of tomogram is constant

8
NCMIR Target Platform

• Multi-user, heterogenous resources
• NCMIR cluster
• SGI Indigo2, SGI Octane, SUN ULTRA, SUN
  Enterprise
• IRIX, Solaris
• Meteor cluster
• Pentium III dual proc
• Linux, PBS
• Blue Horizon
• AIX, Loadleveler, Maui Scheduler

network
9
On-line PT Architecture
writer
preprocessor
10
On-line PT Design

• 1) Frame on-line parallel tomography as a
  tunable application
• Resource limitations / dynamic
• Availability of alternate configurations
  Chang,et al
• each configuration corresponds to different
  output quality and resource usage
• 2) Coupled with user-directed application-level
  scheduler (AppLeS)
• adaptive scheduler
• promote application performance

11
On-line PT Configuration

• Triple (f, r, su)
• Reduction factor (f)
• Reduce resolution of data ? reduce both
  computation and communication
• Projections per refresh (r)
• Reduce refinement frequency ? reduce
  communication
• Service Units - (su)
• Increase cost of execution ? increase
  computational power

12
User Preferences

• Best configuration (f, r, su) (1, 1, 0 )
• Several possible configurations ? user specifies
  bounds
• projections should be at least size 256x256
• 1 ? f ? 4 or 1 ? f ? 8
• user could tolerate up to a 10 minute time wait
• 1 ? r ? 13
• reasonable upper bound
• 0 ? su ? (50 x acquisition period x c)

13
User-directed

• Feasible?
• Use dynamic load information
• if work allocation found
• Better?
• e.g.
• 1. (1, 6, 4) - best f
• 2. (2, 2, 8) - good su/r
• 3. (2, 1, 20) - best r

14
User-directed AppLeS
generate
request
adjust
process
infeasible
request
request
feasible
display
triples
review
rejects all
triples
accepts one
find
work
allocation
User-directed AppLeS
User
execute on-line PT
15
Triple Search

• Search parameter space
• If triple satisfies constraints ? feasible
• Constrained optimization problem based on soft
  real-time execution
• compute constraint
• transfer constraint
• Heuristics to reduce search space
• e.g. assume user will always choose (1,2,1) over
  (1,2,4)

16
Work Allocation
Multiple mixed-integer programs ? approx soln
17
Experiments

• Impact of dynamic information on scheduler
  performance
• Usefulness of tunability Grid environments
• Scheduling latency

18
Dynamic Information

• We fix the triple and let schedulers determine
  work allocation

19
Simulation

• Evaluate schedulers
• Repeatibility
• Long makespan
• several resource environments
• Simgrid (Casanova CCGrid2001)
• API for evaluating scheduling algorithms
• tasks
• resources modeled using traces
• E.g. Parameter sweep applications HCW00
• Simtomo

20
Performance Metric

• Relative refresh lateness

21
NCMIR experiments

• Traces (8 machines)
• 8 hour work day on March 8th, 2001
• Ran simulations throughout day at 10 minute
  intervals

22
Perfect Load Predictions
4
10
wwa
wwacpu
wwabw
AppLeS
3
10
mean relative refresh lateness
2
10
1
10
0
10
0
1
2
3
4
5
6
7
8
hours since 3/8/2001 - 800 PST
23
Imperfect Load Predictions
Student Version of MATLAB
24
Synthetic Grids

• Bandwidth predictibility
• Average prediction error
• pi ? L, M, H
• p1 p2 p3
• e.g. LMH
• 27 types
• 2510 Grids
• x 4 schedulers
• 10,040 simulations

p1
p3
p2
25
Relative Scheduler Performance
Student Version of MATLAB
26
Partial Ordering

• Performance vs. bandwidth predictability
• Grid predictibility
• Partial orders using p1 p2 p3
• Comparable/Not Comparable
• e.g. HML is comparable to HLL
• e.g. HLM is not comparable to LHM
• HHH, HHM, HMM, HLM, MLM, LLM, LLL

27
Example Partial Order
4
10
wwa
wwacpu
wwabw
AppLeS
3
10
relative refresh lateness (seconds)
2
10
1
10
0
10
HHH
HHM
HMM
HLM
MLM
LLM
LLL
.
28
Tunability Experiments

• How useful is tunability?
• variability
• Fixed topology
• categorized traces
• L, M, H
• v1 v2 v3 v4 v5
• 243 Grid types

v4
v1
v3
v5
v2
29
Tunability Experiments

• Run over a 2 day period
• back-to-back
• assume single user model
• f, r, su
• Set of triples chosen
• T 1,,61

30
Tunability Results

• Count how many times a triple changed per 2-day
  simulation
• e.g.
• 12.9
• 25.7

31
Scheduling Latency

• Time to search for feasible triples
• e.g.
• 88 under 1 sec
• 63 under 1 sec

32
Conclusions and Future Work

• Grid-enabled version of on-line parallel
  tomography
• Tunable application
• Tunability is useful in Grid environments
• User-directed AppLeS
• Importance of bandwidth predictability
• e.g. rescheduling
• Scheduling latency is nominal
• Production use

33
Search optimization