...
If the environmental variable is not set every task will write the same gmon.out file.
MPI environment
We offer two options for MPI environment on Marconi100:
> Spectrum-MPI
> OpenMPI
Here you can find some useful details on how to use them on Marconi100.
Spectrum-MPI
It is an IBM implementation of MPI. Together with XL compiler it is the recommended enviroment to be used on Marconi100.
In addition to OpenMPI it adds unique features optimized for IBM systems such as CPU affinity features, dynamic selection of interface libraries, workload manager integrations and better performance.
Spectrum-MPI supports both CUDA-aware and GPUDirect technologies.
The spectrum_mpi module provides the following compiler wrappers:
| compiler | wrapper | usage |
|---|---|---|
| xlc_r | mpixlc | Compile C source files with XL |
| xlC_r | mpixlC | Compile C++ source files with XL |
| xlf_r | mpixlf | Compile FORTRAN source files with XL |
| pgcc | mpipgicc | Compile C source files with PGI |
| pgc++ | mpipgic++ | Compile C++ source files with PGI |
| pgfortran | mpipgifort | Compile FORTRAN source files with PGI |
| gcc | mpicc | Compile C source files with GNU |
| g++ | mpic++ | Compile C++ source files with GNU |
| gfortran | mpif77 mpif90 mpifort oshfort shmemfort | Compile FORTRAN source files with GNU |
Detailed documentation can be found in the original IBM website.
Important flags
In order to enable the GPU awareness you need to run mpirun with the flag -gpu. This is required for applications that pass pointers to GPU buffers to MPI API calls.
OpenMPI
This most common MPI implementation is installed inside the GNU environment.
It is configured to support both CUDA-aware and GPUDirect.
The openmpi module provides the following wrappers:
| compiler | wrapper | Usage |
|---|---|---|
| gcc | mpicc opalcc ortecc oshcc shmemcc | Compile C source files with GNU |
| g++ | mpic++ | Compile C++ source files with GNU |
| gfortran | mpif77 | Compile FORTRAN source files with GNU |
Scientific libraries
ESSL: Engineering and Scientific Subroutine Library by IBM
...