...
This will place the temporary outputs of the nsys code in your TMPDIR folder that by default is /dev/shm/slurm_job.$SLURM_JOB_ID where you have about 250 GB of free space.
This workaround may cause conflicts between multiple jobs running this profiler on a compute node at the same time, so we strongly suggest also to request the compute node exclusively:
#SBATCH --exclusive
MPI environment
We offer two options for MPI environment on LEONARDO:
- For GPU: CUDA aware Open MPI
- For CPU: Intel-OneAPI-MPI
Here you can find some useful details on how to use them on LEONARDO.
OpenMPI
This most common MPI implementation is installed inside the GNU environment.
It is configured to support both CUDA-aware and GPUDirect.
To install MPI applications using Open MPI you have to load openmpi module (use "modmap -m openmpi" command to see the available Open MPI versions) and select the MPI compiler wrapper for Fortran, C or C++ codes.
The openmpi module provides the following wrappers:
Compiler | Wrapper | Usage |
|---|---|---|
| g++ | mpic++ | Compile C++ source files with GNU |
| gcc | mpicc | Compile C source files with GNU |
| gfortran | mpif77 | Compile FORTRAN source files with GNU |
e.g. Compiling C code
> module load openmpi/<version>
> mpicc -o myexec myprog.c (uses the gcc compiler)
You can add all options available for the backend compiler (you can show it by "-show" flag, e.g. "mpicc -show"). In order to list them type the "man" command:
> man mpicc
Intel-OneAPI-MPI
For Leonardo CPUs the availbale MPI implementation is Intel MPI.
To install MPI applications using Intel MPI you have to load intel-oneapi-mpi module (use "modmap -m intel-oneapi-mpi command to see the available versions).
The intel-oneapi-mpi module provides the following wrappers for classic intel compilers and oneapi ("x") compilers:
Compiler | Wrapper | Usage |
|---|---|---|
icpc icpx | mpiicpc mpiicpc -cxx=icpx | Compile C++ source files with classic Intel Compile C++ source files with oneapi |
icc icx | mpiicc mpiicc -cc=iccx | Compile C source files with classic Intel Compile C source files with oneapi |
ifort ifx | mpiifort (Fortran90/77) mpiifort -fc=ifx | Compile FORTRAN source files with classic Intel Compile FORTRAN source files with oneapi |
e.g. Compiling Fortran code
> module load intel-oneapi-mpi/<version>
> mpiifort -o myexec myprog.f90 (uses the ifort compiler)
You can add all options available for the backend compiler (you can show it by "-show" flag, e.g. "mpicc -show"). In order to list them type the "man" command
> man mpiifort
Scientific libraries
...