PyTrilinos Cluster

Installing PyTrilinos on Continuity

The easiest way to get PyTrilinos on the Rocce cluster is to copy the binaries from someone who has already built it. Email for help with this. After obtaining the binaries, proceed to steps 4 and 5.

The purpose of this short guide is to install Trilinos framework outside the Continuity environment, in order to be able to use all its linear algebra capabilities. Installing outside of Continuity enables multiple versions of Continuity to use the same instal of Trilinos and its dependencies.

1. Configuring building environment

Suppose that svn version of Continuity is installed in ~/dev/cont and your Trilinos directory is ~/libs/PyTrilinos:

First, create your Trilinos directory if you have not already done so.
simone@rocce:~$ cd
simone@rocce:~$ mkdir lib/PyTrilinos && cd lib/PyTrilinos
simone@rocce:~PyTrilinos$ _

Next, navigate to a Continuity installation.

simone@rocce:~$ cd ~/dev/cont
simone@rocce:~/dev/cont$ source mglinit
setting PYTHONHOME environment
simone@rocce:~/dev/cont$ echo $MGL_ROOT
/home/simone/dev/cont/pcty/MglToolsLib

2. Trilinos dependencies

Remark: Trilinos has alot on dependencies, most of them not required by default. In fact, Trilinos is a development framework, with the aim of simplifying usage of complex linear algebra solvers on a common infrastructure. Each component can be enabled if required by the application.

2a. SWIG

SWIG is a multi-language wrapper for Python, and is required by PyTrilinos.

First, check if SWIG is already present and if version is greater than 2.0 (in this case no further action is required):
simone@rocce:~/libs/PyTrilinos$ which swig
simone@rocce:~/libs/PyTrilinos$ swig -version

If not, proceed as follows:

simone@rocce:~/libs/PyTrilinos$ wget http://prdownloads.sourceforge.net/swig/swig-2.0.4.tar.gz
simone@rocce:~/libs/PyTrilinos$ tar xvfz swig-2.0.4.tar.gz
simone@rocce:~/libs/PyTrilinos$ cd swig-2.0.4
simone@rocce:~/libs/PyTrilinos/swig-2.0.4$ wget ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-8.21.tar.gz
simone@rocce:~/libs/PyTrilinos/swig-2.0.4$ Tools/pcre-build.sh
simone@rocce:~/libs/PyTrilinos/swig-2.0.4$ ./configure –prefix=$MGL_ROOT
simone@rocce:~/libs/PyTrilinos/swig-2.0.4$ make
simone@rocce:~/libs/PyTrilinos/swig-2.0.4$ make install
simone@rocce:~/libs/PyTrilinos/swig-2.0.4$ cd ..
simone@rocce:~/libs/PyTrilinos$ which swig
/home/simone/dev/cont/pcty/MglToolsLib/bin/swig

2b. CMake

CMake is a build system, required by Trilinos configuration.

First, check if CMake is already present (in this case no further action is required):
simone@rocce:~/dev/cont/buildenv$ which cmake

If not, proceed as follows:

simone@rocce:~/libs/PyTrilinos$ wget http://www.cmake.org/files/v2.8/cmake-2.8.7.tar.gz
simone@rocce:~/libs/PyTrilinos$ tar xvfz cmake-2.8.7.tar.gz
simone@rocce:~/libs/PyTrilinos$ cd cmake-2.8.7
simone@rocce:~/libs/PyTrilinos/cmake-2.8.7$ ./bootstrap –prefix=$MGL_ROOT
simone@rocce:~/libs/PyTrilinos/cmake-2.8.7$ make
simone@rocce:~/libs/PyTrilinos/cmake-2.8.7$ make install
simone@rocce:~/libs/PyTrilinos/cmake-2.8.7$ cd ..
simone@rocce:~/libs/PyTrilinos$ which cmake
/home/simone/dev/cont/pcty/MglToolsLib/bin/cmake

2c. ParMETIS

ParMETIS is a parallel graph partitioner, used (but not required) by Trilinos and other software. Last major version 4.x is not well supported yet, so we install legacy 3.x version.

simone@rocce:~/libs/PyTrilinos$ wget http://glaros.dtc.umn.edu/gkhome/fetch/sw/parmetis/OLD/ParMetis-3.2.0.tar.gz
simone@rocce:~/libs/PyTrilinos$ tar vxfz ParMetis-3.2.0.tar.gz
simone@rocce:~/libs/PyTrilinos$ cd ParMetis-3.2.0

Change line 8 of Makefile.in to OPTFLAGS = -O3 -fPIC. Then

simone@rocce:~/libs/PyTrilinos/ParMetis-3.2.0$ make
simone@rocce:~/libs/PyTrilinos/ParMetis-3.2.0$ cp lib{par,}metis.a $MGL_ROOT/lib
simone@rocce:~/libs/PyTrilinos/ParMetis-3.2.0$ cp {METISLib/metis.h,parmetis.h} $MGL_ROOT/include
simone@rocce:~/libs/PyTrilinos/ParMetis-3.2.0$ cd ..

2d. SuperLU and SuperLU_Dist

SuperLU is a direct solver based on LU decomposition and implemented in C. SuperLU_Dist is the parallel version.

simone@rocce:~/libs/PyTrilinos$ wget http://crd-legacy.lbl.gov/~xiaoye/SuperLU/superlu_4.3.tar.gz
simone@rocce:~/libs/PyTrilinos$ tar xvfz superlu_4.3.tar.gz
simone@rocce:~/libs/PyTrilinos$ cd SuperLU_4.3

Now edit make.inc file as follows:

Line 24: SuperLUroot = $(HOME)/libs/PyTrilinos/SuperLU_4.3
Comment out (place a # in front of it) lines 30 and 31
Uncomment line 34
Line 49: CFLAGS = -DPRNTlevel=0 -O3 -fPIC
Line 50: NOOPTS = -fPIC
Line 51: FORTRAN = ifort
Line 52: FFLAGS = -O2 -fPIC

NOTE for building on rocce:The test scripts that run during the SuperLU make process are in csh, but rocce’s default shell is in bash. You will need to convert the scripts to bash and the Makefiles to call your new bash scripts to get the build to complete successfully. SuperLU Dist does not have this issue and should build fine following the directions specified.

simone@rocce:~/libs/PyTrilinos/SuperLU_4.3$ make
simone@rocce:~/libs/PyTrilinos/SuperLU_4.3$ cp lib/libsuperlu_4.3.a $MGL_ROOT/lib/libsuperlu.a
simone@rocce:~/libs/PyTrilinos/SuperLU_4.3$ mkdir $MGL_ROOT/include/superlu
simone@rocce:~/libs/PyTrilinos/SuperLU_4.3$ cp SRC/*.h $MGL_ROOT/include/superlu
simone@rocce:~/libs/PyTrilinos/SuperLU_4.3$ cd ..

SuperLU_Dist installation is quite similar, there are only few differences in make.inc file.

simone@rocce:~/libs/PyTrilinos$ wget http://crd-legacy.lbl.gov/~xiaoye/SuperLU/superlu_dist_2.5.tar.gz
simone@rocce:~/libs/PyTrilinos$ tar vxfz superlu_dist_2.5.tar.gz
simone@rocce:~/libs/PyTrilinos$ cd SuperLU_DIST_2.5

As before, edit make.inc file as follow:

Line 24: SuperLUroot = $(HOME)/dev/cont/buildenv/SuperLU_DIST_2.5
Line 28: BLASLIB = -lblas
Line 29: METISLIB = -L$(HOME)/dev/cont/pcty/MglToolsLib/lib -lmetis
Line 30: PARMETISLIB = -L$(HOME)/dev/cont/pcty/MglToolsLib/lib -lparmetis
Line 33: FLIBS =
Line 48: CC = mpicc
Line 50: CFLAGS = -O3 -fPIC -DDEBUGlevel=0 -DPRNTlevel=0 -DPROFlevel=0 \
Line 54: NOOPTS = -O0 -fPIC
Line 57: FORTRAN = mpif90
Line 58: F90FLAGS = -O3 -fPIC
Line 60: LOADER = $(CC)

simone@rocce:~/libs/PyTrilinos/SuperLU_DIST_2.5$ make
simone@rocce:~/libs/PyTrilinos/SuperLU_DIST_2.5$ cp lib/libsuperlu_dist_2.5.a $MGL_ROOT/lib/libsuperlu_dist.a
simone@rocce:~/libs/PyTrilinos/SuperLU_DIST_2.5$ mkdir $MGL_ROOT/include/superlu_dist
simone@rocce:~/libs/PyTrilinos/SuperLU_DIST_2.5$ cp SRC/*.h $MGL_ROOT/include/superlu_dist
simone@rocce:~/libs/PyTrilinos/SuperLU_DIST_2.5$ cd ..

2e. SuiteSparse

SuiteSparse is a set of libraries that implement direct solvers and renumbering of matrix entries. Mostly used packages are UMFPACK, AMD and CHOLMOD.

simone@rocce:~/libs/PyTrilinos$ wget http://www.cise.ufl.edu/research/sparse/SuiteSparse/SuiteSparse-3.7.0.tar.gz
simone@rocce:~/libs/PyTrilinos$ tar vxfz SuiteSparse-3.7.0.tar.gz
simone@rocce:~/libs/PyTrilinos$ cd SuiteSparse

Now edit UFconfig/UFconfig.mk file:

Line 77: INSTALL_LIB = $(MGL_ROOT)/lib
Line 78: INSTALL_INCLUDE = $(MGL_ROOT)/include

simone@rocce:~/libs/PyTrilinos/SuiteSparse$ make
simone@rocce:~/libs/PyTrilinos/SuiteSparse$ make install
simone@rocce:~/libs/PyTrilinos/SuiteSparse$ cd ..

3. Installing Trilinos

Now we should be able to install Trilinos (and PyTrilinos extension) on Continuity.

simone@rocce:~/libs/PyTrilinos$ wget http://trilinos.sandia.gov/download/files/trilinos-10.8.4-Source.tar.gz
simone@rocce:~/libs/PyTrilinos$ tar vxfz trilinos-10.8.4-Source.tar.gz

Now create a new file config-trilinos.sh inside the current directory, and put this:

#!/bin/sh

_src=${HOME}/libs/PyTrilinos/trilinos-10.8.4-Source
_dst=${MYLIBS}

cmake \

  • -DCMAKE_BUILD_TYPE:STRING=”Release” \
    -DBUILD_SHARED_LIBS:BOOL=ON \
    -DCMAKE_INSTALL_PREFIX:STRING=$_dst \
    -DTPL_ENABLE_MPI:BOOL=ON \
    -DTPL_ENABLE_BLAS:BOOL=ON \
    -DTPL_ENABLE_LAPACK:BOOL=ON \
    -DTPL_ENABLE_ParMETIS:BOOL=ON \
    -DTPL_ENABLE_Zlib:BOOL=ON \
    -DTPL_ENABLE_Pthread:BOOL=ON \
    -DTPL_ENABLE_SuperLU:BOOL=ON \
    -DTPL_ENABLE_SuperLUDist:BOOL=ON \
    -DTPL_ENABLE_UMFPACK:BOOL=ON \
    -DTPL_ENABLE_AMD:BOOL=ON \
    -DTrilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=ON \
    -DTrilinos_ENABLE_PyTrilinos:BOOL=ON \
    -DParMETIS_LIBRARY_DIR:PATH=$_dst/lib \
    -DParMETIS_INCLUDE_DIR:PATH=$_dst/include \
    -DSuperLU_LIBRARY_DIR:PATH=$_dst/lib \
    -DSuperLUDist_LIBRARY_DIR:PATH=$_dst/lib \
    -DSuperLU_INCLUDE_DIRS:PATH=$_dst/include/superlu \
    -DSuperLUDist_INCLUDE_DIRS:PATH=$_dst/include/superlu_dist \
    -DUMFPACK_INCLUDE_DIRS:PATH=$_dst/include \
    -DAMD_INCLUDE_DIRS:PATH=$_dst/include \
    -DUMFPACK_LIBRARY_DIRS:PATH=$_dst/lib \
    -DUMFPACK_LIBRARY_NAMES:STRING=”umfpack;amd;cholmod;colamd” \
    -DAMD_LIBRARY_DIRS:PATH=$_dst/lib \
    -DPYTHON_EXECUTABLE:PATH=”${MGL_ROOT}/bin/python” \
    -DSWIG_DIR:PATH=${MGL_ROOT} \
    $_src

Be sure to change _src variable at the top of the file in order to match the correct Trilinos source path.

NOTE for building 10.12.2 I had to use -DTPL_ParMETIS_INCLUDE_DIRS:PATH=$_dst/include and -DTPL_SuperLU_LIBRARIES:PATH=$_dst/lib for my configuration to find all the dependencies.

simone@rocce:~/libs/PyTrilinos$ cd trilinos-10.8.4-Source
simone@rocce:~/libs/PyTrilinos/trilinos-10.8.4-Source$ mkdir build && cd build
simone@rocce:~/libs/PyTrilinos/trilinos-10.8.4-Source/build$ sh ../../config-trilinos.sh
simone@rocce:~/libs/PyTrilinos/trilinos-10.8.4-Source/build$ make
simone@rocce:~/libs/PyTrilinos/trilinos-10.8.4-Source/build$ make install
simone@rocce:~/libs/PyTrilinos/trilinos-10.8.4-Source/build$ cd ../..

Compilation typically takes alot of time, because all available features are enabled.

4. Add PyTrilinos to your path

Next, your install of Continuity needs to be pointed to your install of PyTrilinos. There are three files used to set the path for Continuity in different contexts: mglinit, continuity, continuityserver, and continuityparallel. The following lines need to the end of mglinit, and incontinuity and continuityparallel, after the entire block of code following “deduce which python to use” ending in “fi” and before “cd $MGL_ROOT”. It should be near the end of the file

export MYLIBS=/home/simone/libs/Pytrilinos 
 export PATH=$PATH:$MYLIBS/bin
 export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$MYLIBS/lib
 export PYTHONPATH=$PYTHONPATH:$MYLIBS/lib/python2.5/site-packages

5. Testing

The final step is to check that everything goes smoothly. To do so, let’s write a simple Python script Test.py:

  • from PyTrilinos import Epetra comm = Epetra.PyComm () ilead = (comm.MyPID() == 0)

  • if ilead:
    • print “Leader!”

    else:

    • print “Slave %d of %d” % (comm.MyPID(), comm.NumProc())

Then: simone@rocce:~/dev/cont/buildenv$ python test.py
Leader!
simone@rocce:~/dev/cont/buildenv$ mpirun -np 4 python test.py
Slave 1 of 4
Leader!
Slave 2 of 4
Slave 3 of 4
. To get test.py to run with mpirun on rocce, you must submit a job through sge. Below is an example for the last line in the submit script. /opt/openmpi/bin/mpirun -np 4 python ~/libs/Pytrilinos/test.py

Finally, line 152 in Problem.py (after #Trilinos implementation”) must be changed from ‘if False:’ to ‘if True:’.

Have fun!