Project 4: 2D Excitable Media Simulations on an MPI Cluster

1
Project 4 2D Excitable Media Simulations on an
MPI Cluster

• Robert Dubin
• Irina Ashkenazi
• Ramazan Burus
• CMPT585
• December 4, 2003

2
Excitable Media Simulations

• - One important use is simulating electrical
  currents in normal and damaged hearts.
• Every year, approximately 300,000 people in the
  United State die due to heart attack.
• Excitable media simulations may help in
  developing treatments for this ailment.

3
Heart attacks and spiral waves

• -The heart pumps blood from the atria and then to
  the ventricles using and electrical wave signal.
• - This electrical wave becomes abnormal during
  (or as a result of) a heart attack.
• -These abnormal waves are called spiral waves
  because they look like spirals.

4
Heart attacks and spiral waves(The figure below
is copied from http//www.cam.cornell.edu/ojalvo/
papers/spiral.pdf)
5
Our role

• To model the observed, abnormal electrical heart
  waves through computer simulations

6
Our Project

• Given project contains
• - README
• - spint.cc ( this file was modified)
• - graphth.cc
• - Makefile

7

• This program is designed to be run on a
  MPI cluster with Linux OS
• This program performs a parallel simulation of a
  2D rectangular piece of excitable media, modeled
  by FitzHugh-Nagumo equations

8
Where u u(x,y,t) is the excitation
functionand function v v(x,y,t) corresponds
to the recovery process
9

• Graphical window contains a grid of size N x M,
  NM300
• When the program is run on 5 CPUs the grid is
  divided into 4 slices corresponding to 4 slave
  CPUs and one master .

10
Each slave CPU computes values of the functions
u and v for each grid point (x,y) inside its
slice, for every time step ( dt .1), and for a
total simulation time of DT 2000000
M
N
Slave1
Slave2
Slave3
Slave4
11
Part 5 Altered Conditions Generate Spiral Pairs

• Overlapping Squares
• conditions u 1 v 1
• Multiple Pairs
• Random Placement
• Generates Stable Triple Spiral

12
(No Transcript)
13
Generation of a Single Spiral

v 1 refractory
u 1 excited
u v 0 ready
N
wave front
M
14
Square Variables
(0, 0)
(x, y)
side
shift
N
shift
M
15
Overlapping Squares
v 1 refractory
u 1 excited
u v 0 ready
N
wave front
M
16
Multiple, Random Overlapping Squares
v 1 refractory
u 1 excited
u v 0 ready
N
wave front
M
17
Code
int side25 int shift10 int x, y for
(int h0 hlt50h) int x rand
()(N-side-shift) int yrand()(M-side-shift)
for(ixiltxsidei) for(jyjltysidej)
(u)(i,j)1.0 for(ixshiftiltxshiftsidei
) for(jyshiftjltysideshiftj)
(v)(i,j)1.0
50 pair
Random
Square
Overlapping
Square
18
Beginning
19
After 5 minutes
20
After 10 minutes
21
After 14 minutes
22
After 80 minutes
23
Part 6 From no-flux to periodic
(torus)boundary conditions
24
Flow Of Slaves

• Identify slaves left and right of self
• Receive from Master which portion of arrays to
  work on (MPI_Recv)
• Main Loop
• Calculate updated values for internal points
• Exchange left and right border data with
  neighboring slaves (MPI_Send and MPI_Recv)
• Calculate updated values for border points
• Send updated data to Master

25
Calculate updated values
Slave1
Slave2
Slave3
Slave4
Internal
Border
N
Data Exchange
M
26
How to generate torus topology

• Computed values of the U-function for the grid
  points
• from top were exchanged with corresponding
  values at bottom of grid
• from left side were exchanged with
  corresponding values from right side of grid.

27
c
28
Code
RIGHTLEFT CONDITIONS   MPI Send ( to right )
function was changed to make CPU with logical
rank 4 To participate in sending data to the
right neighbor   //SEND TO RIGHT csize is
of slaves if(lrank lt csize 1) // send from
cpus with ranks 1, 2, 3, 4
MPI_Send((void)uu-gtGetColumnPtr(L_min_2),
N_plus_1, MPIFPTYPE, right,
11DiscrTime, MPICOMM_WORLD)   MPI Receive
(from left) function was changed to make CPU with
logical rank 1 To participate in receiving data
from the left neighbor   // RECEIVE FROM
LEFT if(lrank lt csize 1) // receive
by cpu with ranks 1,2,3,4
MPI_Recv((void)uu-gtGetColumnPtr(0), N_plus_1,
MPIFPTYPE, left, 11DiscrTime,
MPICOMM_WORLD, status)
29
Code
RIGHTLEFT CONDITIONS (continued) MPI Send ( to
left ) function was changed to make CPU with
logical rank 1 To participate in sending data to
left cpu   //SEND TO LEFT if(lrank gt 1)
// send by cpus with ranks 1, 2, 3, 4
MPI_Send((void)uu-gtGetColumnPtr(1), N1,
MPIFPTYPE, left, 33DiscrTime,
MPICOMM_WORLD)   MPI Receive ( from right
) function was changed to make CPU with logical
rank 4 to participate in receiving data from
right cpu   //RIGHT RECEIVE if(lrank lt
csize 1) // receive by cpus with ranks 1,
2, 3, 4 MPI_Recv((void)uu-gtGetColumnPtr
(L_min_1), N_plus_1, MPIFPTYPE, right,
33DiscrTime, MPICOMM_WORLD, status)
30
Code
//TOPBOTTOM   Top bottom boundary
conditions were changed to transfer data computed
for the top to the bottom of the grid, and to
transfer data computed for the bottom to the top
of the grid. for ( j0 j lt L_min_1 j )
// (uu)(0,j) (u)(1,j) //
(v)(0,j) (v)(1,j) // (uu)(N,j)
(u)(N-1,j) // (v)(N,j) (v)(N-1,j)
(uu)(0,j) (u)(N-1,j) (v)(0,j)
(v)(1,j) (uu)(N,j) (u)(1,j) (v)(N,j)
(v)(N-1,j)
THIS CODE WAS REMOVED
AND THIS WAS INSERTED
31
Torus behavior Observation of wave
interactions performed over a 30 minute time
period


32
At first, randomly placed spirals started to
rotate
33
After some time, spirals started to actively
interact with each other
34
During first 15 minutes, snapshots were taken at
2 minute intervals
35
Spiral density increased rapidly with time
36
Spirals became parallel to each other with very
high density
37
(No Transcript)
38
During last 15 minutes of the observation
snapshots were taken at 3 - 4 minute intervals
39
(No Transcript)
40
On the last snapshots, spirals almost perfectly
parallel to each other were observed
41
Part 7 Add Probe To Measure Wave Width
Frequency

• Add probe to media
• Detect and quantitate (in counts)
• wave width
• wave frequency
• Examine beat to beat intervals for
• Single spiral
• Triplet spiral
• Torus

42
Two Parameters
Wave Width
Wave Frequency
43
Concept
Slave1
Slave2
Slave3
Slave4
Probe in Slave 2
Wave Front
44
Concept (continued)
Slave2
x
u 0.1 low
x
u gt 0.7 changing from low to high
x
x
x
x
x
x
u 0.9 high
u lt 0.3 changing from high to low
x
At this instant, probes u 0.1
45
Code
VARIABLES ADDED TO SLAVE unsigned long
counter 0 unsigned long sizeOfWave 0//from
wave front to back of wave unsigned long
waveToWaveSize 0//from one wave front to next
wave front bool priorReadingWasLow false int
slaveWhereProbeResides 2//here, we'll simply
choose slave 2 int elementAOfArrayForProbe
30 int elementBOfArrayForProbe 30//here,
we'll simply choose pixel   //arbitrarily chose
pixel 30, 30 of the 15040 elements in Slave 2
46
Code added following if(DiscrTimeTStepuv0)
block in slave
if( rank slaveWhereProbeResides ) //only do
this if you are slave 2 //must make
sure we've past at least one wave //to avoid
starting inside a wave   if( ( u )(
elementAOfArrayForProbe,
elementBOfArrayForProbe ) gt 0.7
priorReadingWasLow )
waveToWaveSize counter cout
ltlt"WaveSize " ltlt sizeOfWave ltlt "
WaveToWaveSize " ltlt waveToWaveSize
ltlt endl   counter 0
priorReadingWasLow false
u gt 0.7 changing from low to high
//hit a new wave at pixel (30, 30) //report
previously gathered data //re-initialize counter
to zero, //priorReadingWasLow false //VERY
FIRST SET OF DATA WILL LIKELY BE GARBAGE!!!!!
47
else if( ( u )( elementAOfArrayForProbe,
elementBOfArrayForProbe ) gt
0.7
!priorReadingWasLow )
counter else if(
( u )( elementAOfArrayForProbe,
elementBOfArrayForProbe ) lt 0.3
!priorReadingWasLow
) sizeOfWave counter
priorReadingWasLow true
counter
u 0.9 high
//within a wave, //so just increase counter
u lt 0.3 changing from high to low
//just hit back of wave, so //mark size of wave,
//priorReadingWasLow true //and increase
counter
48
else if( ( u )( elementAOfArrayForProbe,
elementBOfArrayForProbe ) lt
0.3
priorReadingWasLow )
counter else if( ( u )(
elementAOfArrayForProbe,
elementBOfArrayForProbe ) gt 0.3
( u )( elementAOfArrayForProb
e, elementBOfArrayForProbe
) lt 0.7 )
counter //end ifs //end main
loop
u 0.1
//within a trough so //just increase counter
No Mans Land
//within a transition region //( u gt 0.3 u lt
0.7 ) //just increase counter
49
t 80
1
t 80
t 1
2
t 80
3
50
Wave Width Avg. Std. (1) Single
Spiral 51.5 1.2 (2) Triple Spiral 52.8
2.8 (3) Torus 48.5 4.0
51
Probe Results

• Single Spiral
• Beat to beat interval remained high during the
  time period
• wave width stable at around 51
• Double spiral
• Beat to beat interval reduced during the time
  period and leveled to an average of 50
• wave width stable at around 52
• Torus
• Beat to beat interval reduced and leveled to an
  average of 30
• wave width averaged around 48, with a bit more
  variability

52
Part 8 Speed Up Analysis (DT set to 20000)
Speed up was observed with increasing CPUs, but
it was below a linear increase
53
Group 1

• Ramazan Burus (burusr1_at_mail.montclair.edu )
• Part 5
• Background to excitable media
• Irina Ashkenazi (vashkena_at_bellatlantic.net)
• Part 6
• Background to excitable media
• Robert Dubin (dubinr_at_pegasus.montclair.edu)
• Part 7
• Part 8