PyTrilinos

Installing PyTrilinos on Continuity

The purpose of this short guide is to install Trilinos framework inside Continuity environment, in order to be able to use all its linear algebra capabilities. To install Trilinos outside of the Continuity environment see the page PyTrilinosCluster. This enables multiple versions of Continuity to use the same instal of Trilinos and its dependencies.

1. Configuring building environment

Suppose that svn version of Continuity is installed in ~/dev/cont. Then:
simone@rocce:~$ cd ~/dev/cont
simone@rocce:~/dev/cont$ source mglinit
setting PYTHONHOME environment
simone@rocce:~/dev/cont$ echo $MGL_ROOT
/home/simone/dev/cont/pcty/MglToolsLib

As showed with the last command, MGL_ROOT variable points to MGLTools environment directory, and it’s where everything will be installed to.
Now create a building directory, for instance:

simone@rocce:~/dev/cont$ mkdir buildenv && cd buildenv
simone@rocce:~/dev/cont/buildenv$ _

2. Trilinos dependencies

Remark: Trilinos has alot on dependencies, most of them not required by default. In fact, Trilinos is a development framework, with the aim of simplifying usage of complex linear algebra solvers on a common infrastructure. Each component can be enabled if required by the application.

2a. SWIG

SWIG is a multi-language wrapper for Python, and is required by PyTrilinos.

First, check if SWIG is already present and if version is greater than 2.0 (in this case no further action is required):
simone@rocce:~/dev/cont/buildenv$ which swig
simone@rocce:~/dev/cont/buildenv$ swig -version

If not, proceed as follows:

simone@rocce:~/dev/cont/buildenv$ wget http://prdownloads.sourceforge.net/swig/swig-2.0.4.tar.gz
simone@rocce:~/dev/cont/buildenv$ tar xvfz swig-2.0.4.tar.gz
simone@rocce:~/dev/cont/buildenv$ cd swig-2.0.4
simone@rocce:~/dev/cont/buildenv/swig-2.0.4$ wget ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-8.21.tar.gz
simone@rocce:~/dev/cont/buildenv/swig-2.0.4$ Tools/pcre-build.sh
simone@rocce:~/dev/cont/buildenv/swig-2.0.4$ ./configure –prefix=$MGL_ROOT
simone@rocce:~/dev/cont/buildenv/swig-2.0.4$ make
simone@rocce:~/dev/cont/buildenv/swig-2.0.4$ make install
simone@rocce:~/dev/cont/buildenv/swig-2.0.4$ cd ..
simone@rocce:~/dev/cont/buildenv$ which swig
/home/simone/dev/cont/pcty/MglToolsLib/bin/swig

2b. CMake

CMake is a build system, required by Trilinos configuration.

First, check if CMake is already present (in this case no further action is required):
simone@rocce:~/dev/cont/buildenv$ which cmake

If not, proceed as follows:

simone@rocce:~/dev/cont/buildenv$ wget http://www.cmake.org/files/v2.8/cmake-2.8.7.tar.gz
simone@rocce:~/dev/cont/buildenv$ tar xvfz cmake-2.8.7.tar.gz
simone@rocce:~/dev/cont/buildenv$ cd cmake-2.8.7
simone@rocce:~/dev/cont/buildenv/cmake-2.8.7$ ./bootstrap –prefix=$MGL_ROOT
simone@rocce:~/dev/cont/buildenv/cmake-2.8.7$ make
simone@rocce:~/dev/cont/buildenv/cmake-2.8.7$ make install
simone@rocce:~/dev/cont/buildenv/cmake-2.8.7$ cd ..
simone@rocce:~/dev/cont/buildenv$ which cmake
/home/simone/dev/cont/pcty/MglToolsLib/bin/cmake

2c. ParMETIS

ParMETIS is a parallel graph partitioner, used (but not required) by Trilinos and other software. Last major version 4.x is not well supported yet, so we install legacy 3.x version.

simone@rocce:~/dev/cont/buildenv$ wget http://glaros.dtc.umn.edu/gkhome/fetch/sw/parmetis/OLD/ParMetis-3.2.0.tar.gz
simone@rocce:~/dev/cont/buildenv$ tar vxfz ParMetis-3.2.0.tar.gz
simone@rocce:~/dev/cont/buildenv$ cd ParMetis-3.2.0

Change line 8 of Makefile.in to OPTFLAGS = -O3 -fPIC. Then

simone@rocce:~/dev/cont/buildenv/ParMetis-3.2.0$ make
simone@rocce:~/dev/cont/buildenv/ParMetis-3.2.0$ cp lib{par,}metis.a $MGL_ROOT/lib
simone@rocce:~/dev/cont/buildenv/ParMetis-3.2.0$ cp {METISLib/metis.h,parmetis.h} $MGL_ROOT/include
simone@rocce:~/dev/cont/buildenv/ParMetis-3.2.0$ cd ..

2d. SuperLU and SuperLU_Dist

SuperLU is a direct solver based on LU decomposition and implemented in C. SuperLU_Dist is the parallel version.

simone@rocce:~/dev/cont/buildenv$ wget http://crd-legacy.lbl.gov/~xiaoye/SuperLU/superlu_4.3.tar.gz
simone@rocce:~/dev/cont/buildenv$ tar xvfz superlu_4.3.tar.gz
simone@rocce:~/dev/cont/buildenv$ cd SuperLU_4.3

Now edit make.inc file as follows:

Line 24: SuperLUroot = $(HOME)/dev/cont/buildenv/SuperLU_4.3
Comment out (place a # in front of it) lines 30 and 31
Uncomment line 34
Line 49: CFLAGS = -DPRNTlevel=0 -O3 -fPIC
Line 50: NOOPTS = -fPIC
Line 51: FORTRAN = ifort
Line 52: FFLAGS = -O2 -fPIC

NOTE for building on rocce:The test scripts that run during the SuperLU make process are in csh, but rocce’s default shell is in bash. You will need to convert the scripts to bash and the Makefiles to call your new bash scripts to get the build to complete successfully. SuperLU Dist does not have this issue and should build fine following the directions specified.

simone@rocce:~/dev/cont/buildenv/SuperLU_4.3$ make
simone@rocce:~/dev/cont/buildenv/SuperLU_4.3$ cp lib/libsuperlu_4.3.a $MGL_ROOT/lib/libsuperlu.a
simone@rocce:~/dev/cont/buildenv/SuperLU_4.3$ mkdir $MGL_ROOT/include/superlu
simone@rocce:~/dev/cont/buildenv/SuperLU_4.3$ cp SRC/*.h $MGL_ROOT/include/superlu
simone@rocce:~/dev/cont/buildenv/SuperLU_4.3$ cd ..

SuperLU_Dist installation is quite similar, there are only few diffences in make.inc file.

simone@rocce:~/dev/cont/buildenv$ wget http://crd-legacy.lbl.gov/~xiaoye/SuperLU/superlu_dist_2.5.tar.gz
simone@rocce:~/dev/cont/buildenv$ tar vxfz superlu_dist_2.5.tar.gz
simone@rocce:~/dev/cont/buildenv$ cd SuperLU_DIST_2.5

As before, edit make.inc file as follow:

Line 24: SuperLUroot = $(HOME)/dev/cont/buildenv/SuperLU_DIST_2.5
Line 28: BLASLIB = -lblas
Line 29: METISLIB = -L$(HOME)/dev/cont/pcty/MglToolsLib/lib -lmetis
Line 30: PARMETISLIB = -L$(HOME)/dev/cont/pcty/MglToolsLib/lib -lparmetis
Line 33: FLIBS =
Line 48: CC = mpicc
Line 50: CFLAGS = -O3 -fPIC -DDEBUGlevel=0 -DPRNTlevel=0 -DPROFlevel=0 \
Line 54: NOOPTS = -O0 -fPIC
Line 57: FORTRAN = mpif90
Line 58: F90FLAGS = -O3 -fPIC
Line 60: LOADER = $(CC)

simone@rocce:~/dev/cont/buildenv/SuperLU_DIST_2.5$ make
simone@rocce:~/dev/cont/buildenv/SuperLU_DIST_2.5$ cp lib/libsuperlu_dist_2.5.a $MGL_ROOT/lib/libsuperlu_dist.a
simone@rocce:~/dev/cont/buildenv/SuperLU_DIST_2.5$ mkdir $MGL_ROOT/include/superlu_dist
simone@rocce:~/dev/cont/buildenv/SuperLU_DIST_2.5$ cp SRC/*.h $MGL_ROOT/include/superlu_dist
simone@rocce:~/dev/cont/buildenv/SuperLU_DIST_2.5$ cd ..

2e. SuiteSparse

SuiteSparse is a set of libraries that implement direct solvers and renumbering of matrix entries. Mostly used packages are UMFPACK, AMD and CHOLMOD.

simone@rocce:~/dev/cont/buildenv$ wget http://www.cise.ufl.edu/research/sparse/SuiteSparse/SuiteSparse-3.7.0.tar.gz
simone@rocce:~/dev/cont/buildenv$ tar vxfz SuiteSparse-3.7.0.tar.gz
simone@rocce:~/dev/cont/buildenv$ cd SuiteSparse

Now edit UFconfig/UFconfig.mk file:

Line 77: INSTALL_LIB = $(MGL_ROOT)/lib
Line 78: INSTALL_INCLUDE = $(MGL_ROOT)/include

simone@rocce:~/dev/cont/buildenv/SuiteSparse$ make
simone@rocce:~/dev/cont/buildenv/SuiteSparse$ make install
simone@rocce:~/dev/cont/buildenv/SuiteSparse$ cd ..

3. Installing Trilinos

Now we should be able to install Trilinos (and PyTrilinos extension) on Continuity.

simone@rocce:~/dev/cont/buildenv$ wget http://trilinos.sandia.gov/download/files/trilinos-10.8.4-Source.tar.gz
simone@rocce:~/dev/cont/buildenv$ tar vxfz trilinos-10.8.4-Source.tar.gz

Now create a new file config-trilinos.sh inside the current directory, and put this:

#!/bin/sh

_src=${HOME}/dev/cont/buildenv/trilinos-10.8.4-Source
_dst=${MGL_ROOT}

cmake \

• -DCMAKE_BUILD_TYPE:STRING=”Release” \
  -DBUILD_SHARED_LIBS:BOOL=ON \
  -DCMAKE_INSTALL_PREFIX:STRING=$_dst \
  -DTPL_ENABLE_MPI:BOOL=ON \
  -DTPL_ENABLE_BLAS:BOOL=ON \
  -DTPL_ENABLE_LAPACK:BOOL=ON \
  -DTPL_ENABLE_ParMETIS:BOOL=ON \
  -DTPL_ENABLE_Zlib:BOOL=ON \
  -DTPL_ENABLE_Pthread:BOOL=ON \
  -DTPL_ENABLE_SuperLU:BOOL=ON \
  -DTPL_ENABLE_SuperLUDist:BOOL=ON \
  -DTPL_ENABLE_UMFPACK:BOOL=ON \
  -DTPL_ENABLE_AMD:BOOL=ON \
  -DTrilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=ON \
  -DTrilinos_ENABLE_PyTrilinos:BOOL=ON \
  -DParMETIS_LIBRARY_DIR:PATH=$_dst/lib \
  -DParMETIS_INCLUDE_DIR:PATH=$_dst/include \
  -DSuperLU_LIBRARY_DIR:PATH=$_dst/lib \
  -DSuperLUDist_LIBRARY_DIR:PATH=$_dst/lib \
  -DSuperLU_INCLUDE_DIRS:PATH=$_dst/include/superlu \
  -DSuperLUDist_INCLUDE_DIRS:PATH=$_dst/include/superlu_dist \
  -DUMFPACK_INCLUDE_DIRS:PATH=$_dst/include \
  -DAMD_INCLUDE_DIRS:PATH=$_dst/include \
  -DUMFPACK_LIBRARY_DIRS:PATH=$_dst/lib \
  -DUMFPACK_LIBRARY_NAMES:STRING=”umfpack;amd;cholmod;colamd” \
  -DAMD_LIBRARY_DIRS:PATH=$_dst/lib \
  -DPYTHON_EXECUTABLE:PATH=”${MGL_ROOT}/bin/python” \
  -DSWIG_DIR:PATH=${MGL_ROOT} \
  $_src
  

Be sure to change _src variable at the top of the file in order to match the correct Trilinos source path.

NOTE for building 10.12.2 I had to use -DTPL_ParMETIS_INCLUDE_DIRS:PATH=$_dst/include and -DTPL_SuperLU_LIBRARIES:PATH=$_dst/lib for my configuration to find all the dependencies.

simone@rocce:~/dev/cont/buildenv$ cd trilinos-10.8.4-Source
simone@rocce:~/dev/cont/buildenv/trilinos-10.8.4-Source$ mkdir build && cd build
simone@rocce:~/dev/cont/buildenv/trilinos-10.8.4-Source/build$ sh ../../config-trilinos.sh
simone@rocce:~/dev/cont/buildenv/trilinos-10.8.4-Source/build$ make
simone@rocce:~/dev/cont/buildenv/trilinos-10.8.4-Source/build$ make install
simone@rocce:~/dev/cont/buildenv/trilinos-10.8.4-Source/build$ cd ../..

Compilation typically takes alot of time, because all available features are enabled.

4. Testing

The final step is to check that everything goes smoothly. To do so, let’s write a simple Python script test.py:

from PyTrilinos import Epetra

comm = Epetra.PyComm ()
ilead = (comm.MyPID() == 0)

if ilead:

• print “Leader!”
  

else:

• print “Slave %d of %d” % (comm.MyPID(), comm.NumProc())
  

Then:[[BR]] simone@rocce:~/dev/cont/buildenv$ python test.py
Leader!
simone@rocce:~/dev/cont/buildenv$ mpirun -np 4 python test.py
Slave 1 of 4
Leader!
Slave 2 of 4
Slave 3 of 4

To get test.py to run with mpirun on rocce, you must submit a job through sge. Below is an example for the last line in the submit script. /opt/openmpi/bin/mpirun -np 4 python ~/libs/Pytrilinos/test.py

Finally, line 152 in Problem.py (after #Trilinos implementation”) must be changed from ‘if False:’ to ‘if True:’.

Have fun!