Ten Tusscher Propagation in a 3072 Element Biventricular Mesh

Description

Solve the monodomain equation on a 3072 element biventricular mesh with extraordinary nodes using the ten Tusscher 2003 ionic model.
Submit a job to a cluster using SGE
Leverage GPU architecture to decrease ODE solve time
Render voltage solution with time
Render activation times and display isochromes

Submit an EP Simulation to the ROCCE Cluster

Gather Files on ROCCE

Log onto your guest account on the Rocce cluster
Copy files to your local continuity directory

[guest@login-0-0 ~]$ cd continuity/
[guest@login-0-0 continuity]$ cp /share/apps/si2012/cardiac/EP_TenTusscher_Panfilov_Epi_sympy_GPU.zip EP_TenTusscher_Panfilov_Epi_sympy_GPU.zip
[guest@login-0-0 continuity]$ cp /share/apps/si2012/cardiac/SubmitSerial.qsub SubmitSerial.qsub
[guest@login-0-0 continuity]$ cp /share/apps/si2012/cardiac/EP_BiV3072.py EP_BiV3072.py

Move the CUDA ionic model binaries to the appropriate location and unzip them

[guest@login-0-0 continuity]$ unzip EP_TenTusscher_Panfilov_Epi_sympy_GPU.zip
[guest@login-0-0 continuity]$ mv  EP_TenTusscher_Panfilov_Epi_sympy_GPU/ pcty/server/problem/Electrophysiology/

Inspect the Run and Submit Scripts

Open the file EP_BiV3072.py. This file contains the commands that Continuity will execute.

[guest@login-0-0 continuity]$ vi EP_BiV3072.py

   1 #************************************************************
   2 # NBCR Summer Institute
   3 # Category: EP
   4 # Date: 8/1/2012
   5 # Description: Solve the monodomain equation on a BiV mesh with
   6 #              extrodinary nodes and the TenTusscher ionic model
   7 # 
   8 #************************************************************
   9 
  10 import os
  11 import sys
  12 import numpy
  13 
  14 #**************define parameters for the EP solve step**************
  15 
  16 #Output file name
  17 fName = 'EP_BiV3072'
  18 
  19 #Use GPU acceleration for ionic model
  20 CUDA = 1
  21 
  22 #Set simulation time and step size
  23 tstart = 0.0
  24 duration = 60
  25 stepsize = 0.05
  26 
  27 #select to save voltage renderings at given intervals
  28 #since rendering is very memory intensive, we will render
  29 #the voltage every 10 steps or 0.5ms
  30 renderfile = 1
  31 rendercount = 10
  32 
  33 #**************end define parameters for the EP solve step**************
  34 
  35 #Load the model from the database, send, and calculate mesh
  36 self.Load_File({'model_id':'1173', 'username':'guest', 'password':'guest', 'version':'1'}, log = 0)
  37 self.Send(None, log=0)
  38 self.CalcMesh([('Calculate', None), ('Do not Calculate', None), ('Do not Calculate', None), ('Angle change scale factors (for nodal derivs wrt angle change)', None)], log=0)
  39 
  40 #Perform Simulations
  41 
  42 self.SinitElectrophys(log=0)
  43 self.Send(None, log=0)
  44 self.SintElectrophys({'plicitType':'Implicit','parallelLinearSolver':0,'conductivityBasis':3, \
        'solutions':{'writeFile': renderfile, 'counter': rendercount, 'tableResult': 0, 'renderResult': 0},  \
        'stateVarInputSelections':[],'stateVarDoTable':0,'parallelODESolver':0,'tstart':tstart,'useCuda':CUDA,  \
        'stateVarOutputSelections':[],'serverKeyname':'electromech_exchange','stateVarList':'1','fileName':fName,  \
        'aps':{'writeFile': 1, 'counter': 1, 'tableResult': 0, 'node_list': 'all', 'renderResult': 0}, \
        'stateVarListType':'collocation points','useGalerkinAssembly':True,'stateVarFrequency':1,'stateVarSelections':[], \
        'dtout':stepsize,'tlen':duration,'reassemble_lhs':1,  \
        'ecgs':{'getHeartVector': False, 'writeFile': 0, 'counter': 1, 'tableResult': 0, 'renderResult': 0}}, log=0)

No changes are needed to this file
Typing :q will exit from the file
Open the submit script

[guest@login-0-0 continuity]$ vi SubmitSerial.qsub

   1 #!/bin/sh
   2 # 
   3 # EXAMPLE OPEN MPI SCRIPT FOR SGE
   4 # Modified by Basement Supercomputing 1/2/2006 DJE
   5 # Modified by cmrg 19/June/2008 FVL
   6 
   7 # Your job name 
   8 #$ -N EP_BiV_run1
   9 
  10 # Use Verbos
  11 #$ -V
  12 
  13 # Use current working directory
  14 #$ -cwd
  15 
  16 # Join stdout and stderr
  17 #$ -j y
  18 
  19 # Use our GPU queue, which uses GPU and CPU nodes
  20 # -q gpu@compute-1-5.local
  21 #$ -q gpu
  22 
  23 # To use CUDA nodes only
  24 #$ -l cuda
  25 
  26 # Set your number of processors here. 
  27 # Requests mpich environment although we actually are using openmpi
  28 #$ -pe orte 1
  29 
  30 # Run job through bash shell
  31 #$ -S /bin/bash
  32 
  33 # Export Library path
  34 export LD_LIBRARY_PATH=/opt/cuda/lib64:$LD_LIBRARY_PATH
  35 export LD_LIBRARY_PATH=/opt/openmpi-myrinet_mx/lib:$LD_LIBRARY_PATH
  36 export MX_RCACHE=0
  37 
  38 # Use full pathname to make sure we are using the right mpirun
  39 ./continuity --full --no-threads --batch /home/*yourUserName*/continuity/EP_BiV3072.py

Change the job name (line 8) or python file path (line 39) as needed
Typing :i allows you to edit files in vim

Submit EP Job to ROCCE

Submit the job using the qsub command

[guest@login-0-0 continuity]$ qsub SubmitSerial.qsub

Check the job status in the queue using qstat or monitor the Continuity stdout using tail -f jobname.o#

Inspect the Cont6 File

Load from Repository

File→Library→Search
Right-click ‘BiV_3072‘ and left-click Load
When the dialog box pops up asking you save your current session, pick the middle radio button, Reset (without save) and proceed

Inspect Mesh

Mesh→Edit→Nodes…
Mesh→Edit→Elements…
Mesh→Calculate Mesh…
- Under Calculate Global Faces, select Don’t Calculate. This step is unnecessary for EP. Leave the other options unchanged.
- Click OK to close the Calculate Mesh Form and execute mesh calculations on the server
Mesh→Render→Elements…
- Click the lines radio button
- Click Render to display mesh lines
Mesh→Render→Elements…
- Click the surfaces radio button
- Enter 1.0 in the dialog box Xi Location. Leave the Xi-direction 3 unchanged. This will render the epicardial surface.
- Click Render to display mesh lines

Inspect Electrophsyiology

Electrophysiology→Edit→Ionic Model
Electrophysiology→Edit→Conductivity Model

Render Voltage Solution

Continuity will have saved the EP solution on the cluster, which you can transfer to your computer from an SCP client
- Find the directory. You should be able to get there by typing

[guest@login-0-0 continuity]$ cd $HOME/.continuity/working

If you are running Mac or Linux, you can SCP to your current directory from the terminal:

[guest@login-0-0 continuity]$ scp -r *yourUserName*@rocce.ucsd.edu:/home/*yourUserName*/.continuity/working/Vsoln_BiV_3072_50.pickle .

The last dot in the command above copies the file to your current directory.
If you are running windows, you will need an SCP client to transfer the file to your computer.
The ’50’ at the end of the file name indicates that only the first 50 frames are rendered. If you are rendering on Windows especially, you might want to render only 50 frames at a time due to memory limitations.

Electrophysiology→Solve→Initialize
Electrophysiology→Calculate→Boundary Conditions
File→Send
Electrophysiology→Solve→Load EP Solution
When the “Load VSoln Pickle” dialog box pops up, click “Browse”, and find your pickle file Vsoln_BiV_3072_50.pickle
- Press OK
Electrophysiology→Render→Render Solution
- You should be prompted to choose a surface from your OpenMesh listing of rendered objects. You should have one object that has the name ‘surfaces’ in it, with a number.
- Leave the frames to be rendered to the default (all will be selected).
- For the ten Tusscher model, good limits for the color map are -90 for Min Value and 10 for Max Value

Render Activation Map

Activation times can be calculated from the AP_out file. The activation times for this simulation have been stored in field variable 1 in the nodes form.
Before rendering the activation times, delete the voltage rendering using the “Open Mesh Controls”
Mesh→Render→Field…
When the “Continuity Render Fields Form” dialog box pops up, select the following options
- Field Variable 3
- Color range 0.0 to 60.0
- At Xi3 location 1.0
Continuity can add isochromes from within the “Open Mesh Controls” to make the activation map more clear
In “Open Mesh Controls” select the object corresponding to the activation map (i.e. Textured Field2)
Select the Colors tab
In the Contours area change the Min,Max,Step to 0.0,60.0,5.0
Increase the Line Width to 5
Select the square Draw button
Rendering the contours may take a few minutes