Building Distributed SuperLU

  1. Download and install Intel Fortran Compiler
  2. Make sure Intel Fortran compiler is available to PATH and it’s libraries and is capable of compiling
    • I added this to my path: 
      source /opt/intel/fc/10.1.015/bin/ifortvars.sh

       

    • I also needed to install libstd5 
      #found this with apt-file search filename
sudo apt-get install libstdc++5

       

  3. Download openmpi
  4. Configure openmpi; make sure it can build fortran code 
    • #http://support.intel.com/support/performancetools/sb/CS-026988.htm
tar -xzvf openmpi-1.2.6.tar.gz
cd openmpi-1.2.6
./configure --prefix=/usr/local F77=/opt/intel/fc/10.1.015/bin/ifort FC=/opt/intel/fc/10.1.015/bin/ifort
sudo make all install

       

  5. Download and extract SuperLU_DIST_2.2

  6. Copy MAKE_INC/<yourplatform.inc> to make.inc. For me it was make.i386_linux.

  7. Edit make.inc
    • Change DSuperLUroot to point to wherever you’ve extracted the SuperLU source code
    • Change LOADER to mpicc. I’m not sure yet if this was a good thing.
    • Make kept failing because it was missing for_write_seq_fmt_xmit. I Added this 
      LOADOPTS        = -lblas -L/opt/intel/fc/10.1.015/lib/ -lifcore

       

  8. Make

  9. Copy <superlu>/lib to /opt/pydusa/pydusa/mympi/src

  10. Change Options.mk so that LLIBS includes -L/opt/pydusa/pydusa/mympi/src/ -lsuperlu_dist_2.0
  11. ???
  12. make
  13. make test

On hexahedral

  1. make.inc

On Lys (32 bit linux Red Hat )

I followed essentially the same directions as above, but did not install libstd5

  1. Installed latest version of OpenMPI
  2. Configure openmpi; make sure it can build fortran code 
    • #http://www.open-mpi.org/software/ompi/v1.4/
tar -xzvf openmpi-1.4.2.tar.gz
cd openmpi-1.4.2
./configure --prefix=/usr/local F77=/opt/intel/fc/10.0.023/bin/ifort FC=/opt/intel/fc/10.0.023/bin/ifort
make all install <-- This needs to be done as root

       

  3. Compile the mpi module within Continuity, no changes where made to any Makefiles there. 
    • cd $CONT_ROOT/src/mpi
make

       

  4. Compile the distributed version of SuperLU (version 2.4 was used here), as above I had to modify the make.inc file to conform to my platform. The main difference from above is I had to add ‘-lg2c’ to the LOADOPTS parameter, and of course change the locations/names of the BLAS, METISLIB, and PARMETISLIB library files. lys_make.inc

    • LOADOPTS        = -lblas -L/opt/intel/fc/10.0.023/lib/ -lifcore -lg2c

       

  5. Compile the python interface to SuperLU_Dist, located at $CONT_ROOT/src/solvers/dist_superlu. I had to make the following changes to the default Makefile, in addition to the location of various libraries. 
    • PINC = -I$(PY_ROOT)/include/python2.5 -I$(PY_ROOT)/lib/python2.5/site-packages/numpy/core/include/numpy -I$(PY_ROOT)/lib/python2.5/site-packages/numpy/core/include -I$(CONT_ROOT)/src/solverLibraries/SuperLU_DIST_2.4/SRC/

LIBS = -lm -lg2c -L/opt/intel/fc/10.0.023/lib/ -lifcore $(SLU_LIB) $(BLAS_LIB) $(PARMETIS_LIB) $(METIS_LIB)

       

  6. Lastly, I was getting a python error when running the src/solvers/dist_superlu/make test: python2.5: relocation error: /usr/local/lib/openmpi/mca_paffinity_linux.so: undefined symbol: mca_base_param_reg_int, so I had to set:

    • export LD_PRELOAD=/usr/local/openmpi/lib/libmpi.so

       

    and the test succeeded after setting LD_PRELOAD.

On oolite (64 bit linux cluster )

The instructions on lys mostly hold for oolite. Not having root priviledges, I had to install things in my local directories.

  1. I downloaded/installed openMPI and configured it as above (i.e., I included Intel Fortran compiler), the openmpi on oolite at /opt/openmpi seemed to have been built using gfortran.

  2. Downloaded and installed Parmetis from http://glaros.dtc.umn.edu/gkhome/metis/parmetis/download, added -fPIC to OPTFLAGS

  3. Compiled $CONT_ROOT/src/mpi
  4. Compiled $CONT_ROOT/src/solverLibraries/SuperLU_DIST_2.4, after modifying the make.inc file, the major change from what was done on lys was: 
    LOADOPTS        = -fPIC -lblas -L/opt/intel/Compiler/11.1/072/lib/intel64/ -lifcore -lirc -lsvml

     

  5. Compiled $CONT_ROOT/src/solvers/dist_superlu, making the same changes to the Makefile as for lys, expect: 
    LIBS = -lm -L/opt/intel/Compiler/11.1/072/lib/intel64 -lifcore -lsvml $(SLU_LIB) $(BLAS_LIB) $(PARMETIS_LIB) $(METIS_LIB)

     

  6. As with lys, I had to do the LD_PRELOAD setting to get the ‘make test’ to work.