Link to the new User Guide https://docs.hpc.cineca.it/index.html
Production Environment
Since LEONARDO is a general purpose system and is used by several users at the same time, long production jobs must be submitted using a queuing system (scheduler). The scheduler guarantees that the access to the resources is as fair as possible. The production environment on LEONARDO Data Centric General Purpose (DCGP) partition is based on the SLURM scheduler.
LEONARDO is based on a policy of node sharing among different jobs, i.e. a job can ask for resources and these can also be a part of a node, for example few cores. This means that, at a given time, one physical node can be allocated to multiple jobs of different users. Nevertheless, exclusivity at the level of the single core is guaranteed by low-level mechanisms.
There are two main modes of using compute nodes:
- Batch Mode: This mode is intended for production runs. Users must prepare a shell script with all the operations to be executed once the requested resources are available. The job will then run on the compute nodes. Store all your data, programs, and scripts in the $WORK or $SCRATCH filesystems, as these are best for compute node access. You must have valid active projects to run batch jobs, and be aware of any specific policies regarding project budgets on our systems.
- Interactive Mode: Jobs submitted in this mode are similar to batch mode in that the user must specify the resources to allocate. The job is then managed like any other submitted job. The key difference from batch mode is that once the job is running, the user can interactively execute applications within the limits of the allocated resources. All allocated resources are available for the entire requested walltime (and consequently billed) during the submission process.
Note: interactive Mode under SLURM has a different meaning compared to the common understanding of interactive execution of an application under a Linux shell or prompt. Interactive execution of applications is allowed on compute nodes only via SLURM (see the next sections).
On login nodes, it is permitted to perform tasks such as data movement, archiving, code development, compilations, basic debugging, and very short test runs, provided these tasks do not exceed 10 minutes of CPU time and are free of charge under the current billing policy.
SLURM partitions
A list of partitions defined on the cluster, with access rights and resources definition, can be displayed with the command sinfo:
$ sinfo -o "%10D %20F %P"
The command returns a more readable output which shows, for each partition, the total number of nodes and the number of nodes by state in the format "Allocated/Idle/Other/Total".
In the following table you can find the main features and limits imposed on the partitions of LEONARDO Data Centric.
SLURM partition | Job QOS | # cores/ # GPU per job | max walltime | max n. of nodes/cores/mem per user max n. of nodes per account | priority | Notes |
lrd_all_serial (default) | normal | max = 4 physical cores max mem = 30800 MB | 04:00:00 | 1 node / 4 cores / 30800 MB | 40 | No GPUs Hyperthreading x2 |
dcgp_usr_prod | normal | max = 16 nodes | 24:00:00 | 512 nodes per account | 40 | |
| dcgp_qos_dbg | max = 2 nodes | 00:30:00 | 2 nodes / 224 cores per user 512 nodes per account | 80 | ||
| dcgp_qos_bprod | min = 17 nodes max =128 nodes | 24:00:00 | 128 nodes per user 512 nodes per account | 60 | GrpTRES=1536 node min is 17 FULL nodes | |
| dcgp_qos_lprod | max = 3 nodes | 4-00:00:00 | 3 nodes / 336 cores per user 512 nodes per account | 40 |
Programming environment
LEONARDO Data Centric compute nodes are not provided with GPUs, thus applications running on GPUs can be used only on the Booster partition. The programming environment include a list of compilers and of debugger and profiler tools, suitable for programming on CPUs.
Compilers
You can check the complete list of available compilers on LEONARDO with the command:
$ modmap -c compilers
The native, and recommended, compilers for LEONARDO Data Centric partition are the Intel ones, since the architecture is based on Intel processors and therefore using the Intel compilers may result in a significant improvement in performance and stability of your code. On the other side, since Intel compilers do not support CUDA, they are not recommended when working on GPUs with LEONARDO Booster partition.
For these reason, CUDA-aware compilers, such as GNU, NVIDIA nvhpc, and CUDA compilers, are suitable and recommended for LEONARDO Booster partition, and they are described in the dedicated page.
Intel OneAPI Compilers
Initialize the environment with the module command:
$ module load intel-oneapi-compilers/<VERSION>
The suite contains the new Intel oneAPI nextgen compilers (icx, icpx, ifx) and the classic compilers (icc, icpc, ifort):
| Classic | oneAPI | Notes | |
|---|---|---|---|
| C/C++ compilers | icc/icpc | icx/icpx |
|
| Fortran compilers | ifort | ifx |
|
Note
- ICX is a new compiler. It has functional and behavioural differences compared to ICC. You can expect some porting will be needed for existing applications using ICC. According to Intel, the transition from ICC Classic to ICX is smooth and effortless. However, you must port and tune any existing applications from ICC Classic to ICX. Please refer to the official Intel Porting Guide for ICC Users to DPCPP or ICX
- IFORT is a completely new compiler. According to Intel, although considerable effort is being made to make the transition from ifort to ifx as smooth and as effortless as possible, customers can expect that some effort may be required to tune their application. IFORT will remain Intel’s recommended production compiler until ifx has performance and features superior to ifort. Please refer to the official Intel Porting Guide for ifort Users to ifx
- Please refer to the official Intel C++ Developer Guide and Reference and Fortran Developer Guide and Reference for an exhaustive list of compiler options
After loading the module, the documentation can be obtained with the man command:
$ man ifort $ man icc
Debugger and Profilers
If at runtime your code dies, then there is a problem. In order to solve it, you can decide to analyze the core file (core not available with PGI compilers) or to run your code using the debugger.
Compiler flags
Whatever your decision, in any case, you need to enable compiler runtime checks, by putting specific flags during the compilation phase. In the following we describe those flags for the different Fortran compilers: if you are using the C or C++ compiler, please check before because the flags may differ.
The following flags are generally available for all compilers and are mandatory for an easier debugging session:
-O0 Lower level of optimization
-g Produce debugging information
Other flags are compiler specific and are described in the following.
PORTLAND Group (PGI) Compilers
The following flags are useful (in addition to "-O0 -g") for debugging your code:
-C Add array bounds checking
-Ktrap=ovf,divz,inv Controls the behavior of the processor when exceptions occur:
FP overflow, divide by zero, invalid operands
GNU Fortran compilers
The following flags are useful (in addition to "-O0 -g")for debugging your code:
-Wall Enables warnings pertaining to usage that should be avoided -fbounds-check Checks for array subscripts.
Debuggers availables
GNU: gdb (serial debugger)
GDB is the GNU Project debugger and allows you to see what is going on 'inside' your program while it executes -- or what the program was doing at the moment it crashed.
VALGRIND
Valgrind is a framework for building dynamic analysis tools. There are Valgrind tools that can automatically detect many memory management and threading bugs, and profile your programs in detail. The Valgrind distribution currently includes six production-quality tools: a memory error detector, two thread error detectors, a cache and branch-prediction profiler, a call-graph generating cache profiler, and a heap profiler.
Valgrind is Open Source / Free Software, and is freely available under the GNU General Public License, version 2.
Profilers
In software engineering, profiling is the investigation of a program's behavior using information gathered as the program executes. The usual purpose of this analysis is to determine which sections of a program to optimize - to increase its overall speed, decrease its memory requirement or sometimes both.
A (code) profiler is a performance analysis tool that, most commonly, measures only the frequency and duration of function calls, but there are other specific types of profilers (e.g. memory profilers) in addition to more comprehensive profilers, capable of gathering extensive performance data.
gprof
The GNU profiler gprof is a useful tool for measuring the performance of a program. It records the number of calls to each function and the amount of time spent there, on a per-function basis. Functions which consume a large fraction of the run-time can be identified easily from the output of gprof. Efforts to speed up a program should concentrate first on those functions which dominate the total run-time.
gprof uses data collected by the -pg compiler flag to construct a text display of the functions within your application (call tree and CPU time spent in every subroutine). It also provides quick access to the profiled data, which let you identify the functions that are the most CPU-intensive. The text display also lets you manipulate the display in order to focus on the application's critical areas.
Usage:
$ gfortran -pg -O3 -o myexec myprog.f90 $ ./myexec $ ls -ltr ....... -rw-r--r-- 1 aer0 cineca-staff 506 Apr 6 15:33 gmon.out $ gprof myexec gmon.out It is also possible to profile at code line-level (see "man gprof" for other options). In this case, you must use also the “-g” flag at compilation time: $ gfortran -pg -g -O3 -o myexec myprog.f90 $ ./myexec $ ls -ltr ....... -rw-r--r-- 1 aer0 cineca-staff 506 Apr 6 15:33 gmon.out $ gprof -annotated-source myexec gmon.out
It is possible to profile MPI programs. In this case, the environment variable GMON_OUT_PREFIX must be defined in order to allow to each task to write a different statistical file.
$ export GMON_OUT_PREFIX=<name>
once the run is finished each task will create a file with its process ID (PID) extension:
$ <name>.$PID
If the environmental variable is not set every task will write the same gmon.out file.
MPI environment
The MPI implementation of Intel, i.e. Intel-OneAPI-MPI, is recommended on the LEONARDO Data Centric partition, and it doesn't support CUDA. Here you can find some useful details on how to use it on this partition.
See the page dedicated to LEONARDO Booster partition for a description of OpenMPI, which instead is installed for supporting CUDA.
Compiling
Intel-OneAPI-MPI
To install MPI applications using IntelMPI you have to load intel-oneapi-mpi module (use "modmap -m intel-oneapi-mpi" command to see the available versions).
The intel-oneapi-mpi module provides the following wrappers for classic Intel compilers and OneAPI ("x") compilers.
Compiler | Wrapper | Usage |
|---|---|---|
icpc icpx | mpiicpc mpiicpc -cxx=icpx | Compile C++ source files with classic Intel Compile C++ source files with oneapi |
icc icx | mpiicc mpiicc -cc=iccx | Compile C source files with classic Intel Compile C source files with oneapi |
ifort ifx | mpiifort (Fortran90/77) mpiifort -fc=ifx | Compile FORTRAN source files with classic Intel Compile FORTRAN source files with oneapi |
e.g. Compiling Fortran code:
$ module load intel-oneapi-compilers/<VERSION> $ module load intel-oneapi-mpi/<version> $ mpiifort -o myexec myprog.f90 (uses the ifort compiler)
You can add all options available for the backend compiler (you can show it by "-show" flag, e.g. "mpicc -show"). In order to list them type the "man" command:
$ man mpiifort
Running
To run MPI applications there are two ways:
- using mpirun launcher
- using srun launcher
mpirun launcher
To use mpirun launcher on LEONARDO Data Centric partition, the intel-oneapi-mpi module needs to be loaded:
$ module load intel-onepi-mpi/<VERSION>
After loading the module, MPI applications can be directly launched as:
$ mpirun ./mpi_exec
or via salloc:
$ salloc -N 2 (allocate a job of 2 nodes) $ mpirun ./mpi_exec
or via sbatch:
$ sbatch -N 2 my_batch_script.sh (allocate a job of 2 nodes) $ cat my_batch_script.sh #!/bin/sh mpirun ./mpi_exec
srun launcher
MPI applications can also be launched directly with the SLURM launcher srun:
$ srun -N 2 ./mpi_exec
or via salloc:
$ salloc -N 2 (allocate a job of 2 nodes) $ srun ./mpi_exec
or via sbatch:
$ sbatch -N 2 my_batch_script.sh (allocate a job of 2 nodes) $ vi my_batch_script.sh #!/bin/sh srun -N 2 ./mpi_exec
Scientific libraries
Libraries listed in this section do not support CUDA (see LEONARDO Booster section for GPU-accelerated libraries).
Linear Algebra
- BLAS: openblas, intel-oneapi-mkl
- LAPACK: openblas, intel-oneapi-mkl
- SCALAPACK: netlib-scalapack, intel-oneapi-mkl
- SPARCE MATRICES : PetSc (multi-node), SuperLU-dist (multi-node)
PetSc and SuperLU-dist are GPU-accelerated libraries and are also listed in LEONARDO Booster dedicated page. However, we report them here for the frequent use also in non-accelerated applications.
Fast Fourier Transform
- FFTW (single and multi-node)
Hardware locality
Each compute node in the DCGP partition is equipped with:
- 2 sockets, each containing one multi-core processor.
- 112 cores in total (56 cores per socket).
- 503 GiB of available RAM, divided into 8 NUMA nodes (4 per socket).
The multi-core processors are Intel Xeon Platinum 8480+ (3.80 GHz, Turbo enabled), featuring:
- 56 cores per processor, each with 2 MiB of L2 cache and 80 KiB of L1 cache.
- 105 MiB of L3 cache, shared across all cores.