...
Each KNL node exposes itself to SLURM as having 68 cores (corresponding to the physical cores of the KNL processor). Jobs should request the entire node (hence, #SBATCH -n 68), and the KNL PBS SLURM server is configured so that to assign the KNL nodes in an exclusive way (even if less ncpus are asked). Hyper-threading is enabled, hence you can run up to 272 processes/threads on each assigned node.
...
#!/bin/bash
#SBATCH -N 1
#SBATCH -A <accnt<account_name>
##SBATCH#SBATCH --mem=1740883GB
#SBATCH -p knl_usr_prod
#SBATCH --time 00:05:00
#SBATCH --job-name=KNL_batch_job
#SBATCH --mail-type=ALL
#SBATCH --mail-user=<user_email>
srun ./myexecutable --mpi=pmi2
where <accnt<account_name> is the project account user has to specify to pay for the job, and <user_email> is a valid email address user can specify to receive real-time information about his job status changes.
...
Each SKL node exposes itself to PBS SLURM as having 48 cores (corresponding to the 48 physical cores of the SKL processor). Jobs should request the entire node (hence, ncpus=48).
...
For example, to request a single SKL node in a production queue the following PBS SLURM job script can be used:
#!/bin/bash
#SBATCH -N 1
#SBATCH -A <accnt<account_name>
##SBATCH#SBATCH --mem=cd90GB
#SBATCH -p skl_usr_prod
#SBATCH --time 00:05:00
#SBATCH --job-name=KNL_batch_job
#SBATCH --mail-type=ALL
#SBATCH --mail-user=<user_email>
...
In the following table you can find all the main features and limits imposed on the queues of the shared A1 and A2 partitions. For Marconi-FUSION dedicated queues please refer to the dedicated document.
MARCONI Partition | SLURM partition |
| QOS | # cores per job | max walltime | max running jobs per user/ max n. of cpus per user | max memory per job | priority | HBM/clustering mode | notes | ||
front-end | bdw_all_serial (default partition) | (not applicable) | 1 | 04:00:00 | Max 12 running jobs Max 4 jobs/user | 1 GB | 30 | ||
| A1 |
| bdw_all_rcm | (not applicable) | min = 1 max = 144 |
| 03:00:00 |
| 1/144 | 123 GB/node value suggested: 118 GB/node | 40 |
| runs on 24 nodes shared with the |
debug queue |
| A1 | bdw_usr_dbg | (not applicable) |
min = 1 max = |
144 |
| 02:00:00 |
| 4/ |
| 144 | 123 GB/node value suggested: 118 GB/node |
| 40 | managed by route |
runs on 24 nodes shared with the visualrcm queue |
| A1 |
| bdw_usr_prod | (not applicable) | min = |
1 max = |
2304 | 24:00:00 |
| 20/ |
| 2304 | 123 GB/node value suggested: 118 GB/node |
| 50 |
| A1 | bdw_usr_prod | bdw_qos_bprod |
min = |
2305 max = |
6000 |
| 24:00:00 |
1/6000 | 123 GB/node value suggested: 118 GB/node |
| 60 |
ask superc@cineca.it
#PBS -q special
max 12 jobs on this queue
max 4 jobs per user
#SBATCH -p bdw_usr_prod #SBATCH --qos=bdw_qos_bprod | |||
| A1 | bdw_usr_prod | bdw_qos_special | min = 1 max = |
36 |
| 180:00:00 |
123 GB/node value suggested: 118 GB/node |
| 100 | ask superc@cineca.it
|
runs on 24 nodes shared with
the debug queue#SBATCH -p bdw_usr_prod #SBATCH --qos=bdw_qos_special
| ||||||||
| A2 |
| knl_usr_dbg | (not applicable) | min = 1 max = 136 (2 nodes) | 00:30:00 | 5/340 | 90 GB/node (mcdram=cache) value suggested: 86 GB/node | 40 | mcdram=cache numa=quadrant |
managed by knlroute
runs on 144 dedicated nodes |
| A2 | knl_usr_prod | (not applicable) |
min >136 max = |
13260 ( |
195 nodes) | 24:00:00 | 20/68000 | 90 GB/node (mcdram=cache) value suggested: 86 GB/node | 50 | mcdram=cache numa=quadrant |
|
| A2 | knl_usr_prod | knl_qos_bprod | min > 13260 |
max = |
68000 ( |
1000 nodes) | 24:00:00 |
Max 1 jobs/user Max 2 jobs/account | 90 GB/node (mcdram=cache) value suggested: 86 GB/node 105 GB/node (mcdram=flat) value suggested: 101 GB/node | 30 | mcdram= |
cache numa=quadrant | ask superc@cineca.it
|
#PBS -q knltest
#PBS -W group_list=<account_no>#SBATCH -p knl_usr_prod #SBATCH --qos=knl_qos_bprod
|
Marconi-A3 is restricted to the Eurofusion EUROfusion community, no queues are available for the general production.
PLEASE NOTE: the PBS SLURM characteristics have not completely defined for MARCONI, some changes will be possible.
...
Compiler flags
Whatever your decision, in any case you need to enable compiler runtime checks, by putting specific flags during the compilation phase. In the following we describe those flags for the different Fortran compilers: if you are using the C or C++ compiler, please check before because the flags may differ.
The following flags are generally available for all compilers and are mandatory for an easier debuggin session:
-O0 Lower level of optimisation -g Produce debugging information
Other flags are compiler specific and are described in the following:
INTEL Fortran compiler
The following flags are useful (in addition to "-O0 -g") for debugging your code:
-traceback generate extra information to provide source file traceback at run time -fp-stack-check generate extra code to ensure that the floating-point stack is in the expected state -check bounds enables checking for array subscript expressions -fpe0 allows some control over floating-point exception handling at run-time
PORTLAND Group (PGI) Compilers
The following flags are usefull (in addition to "-O0 -g") for debugging your code:
-C Add array bounds checking
-Ktrap=ovf,divz,inv Controls the behavior of the processor when exceptions occur:
FP overflow, divide by zero, invalid operands
GNU Fortran compilers
The following flags are usefull (in addition to "-O0 -g")for debugging your code:
-Wall Enables warnings pertaining to usage that should be avoided -fbounds-check Checks for array subscripts.
Debuggers available
Totalview
NOTE: if debugging with Intel compiler suite, you may want to add in your jobscript the following line, before the execution of TotalView:
source $INTELMPI_HOME/bin64/mpivars.sh debug
to provide a better population of the message queue.
Scalasca
....
In the following we report information about other ways to debug your codes:
IDB (serial debugger for INTEL compilers)
The Intel Debugger (idb) is a source-level, symbolic debugger that lets you:
- Control the execution of individual source lines in a program.
- Set stops (breakpoints) at specific source lines or under various conditions.
- Change the value of variables in your program.
- Refer to program locations by their symbolic names
- Print the values of variables and set tracepoints
- Perform other functions (examining core files, examining the call stack, displaying registers)
The idb debugger has two modes: dbx (default mode) or gdb (optional mode) For complete information about idb, see the online Intel Debugger (IDB) Manual in:
$INTEL_HOME/Documentation/en_US/idb
To use this debugger, you should specify the ifort command and the debugging command-line options described above, then you run your executable inside the "idb" environment:
> module load intel > ifort -O0 -g -traceback -fp-stack-check -check bounds -fpe0 -o myexec myprog.f90 > idb ./myexec
On MARCONI the debugger runs in GUI mode by default. You can also start the debugger in command line mode on these systems by specifying idbc instead of idb in the command line.
PGI: pgdbg (serial/parallel debugger)
pgdbg is the Portland Group Inc. symbolic source-level debugger for F77, F90, C, C++ and assembly language programs. It is capable of debugging applications that exhibit various levels of parallelism, including:
- Single-thread, serial applications
- Multi-threaded applications
- Distributed MPI applications
- Any combination of the above
There are two forms of the command used to invoke pgdbg. The first is used when debugging non-MPI applications, the second form, using mpirun, is used when debugging MPI applications:
> pgdbg [options] ./myexec [args] > mpirun [options] -dbg=pgdbg ./myexec [args]
More details in the on line documentation, using the "man pgdbg" command after loading the module.
To use this debugger, you should compile your code with one of the pgi compilers and the debugging command-line options described above, then you run your executable inside the "pgdbg" environment:
> module load pgi > pgf90 -O0 -g -C -Ktrap=ovf,divz,inv -o myexec myprog.f90 > pgdbg ./myexec
By default, pgdbg presents a graphical user interface (GUI). A command-line interface is also provided though the "-text" option.
GNU: gdb (serial debugger)
GDB is the GNU Project debugger and allows you to see what is going on 'inside' your program while it executes -- or what the program was doing at the moment it crashed.
GDB can do four main kinds of things (plus other things in support of these) to help you catch bugs in the act:
- Start your program, specifying anything that might affect its behavior.
- Make your program stop on specified conditions.
- Examine what has happened, when your program has stopped.
- Change things in your program, so you can experiment with correcting the effects of one bug and go on to learn about another.
More details in the on line documentation, using the "man gdb" command.
To use this debugger, you should compile your code with one of the gnu compilers and the debugging command-line options described above, then you run your executable inside the "gdb" environment:
> gfortran -O0 -g -Wall -fbounds-check -o myexec myprog.f90 > gdb ./myexec
VALGRIND
Valgrind is a framework for building dynamic analysis tools. There are Valgrind tools that can automatically detect many memory management and threading bugs, and profile your programs in detail. The Valgrind distribution currently includes six production-quality tools: a memory error detector, two thread error detectors, a cache and branch-prediction profiler, a call-graph generating cache profiler, and a heap profiler.
Valgrind is Open Source / Free Software, and is freely available under the GNU General Public License, version 2.
To analyse a serial application:
- Load Valgrind module --> module load valgrind
- Load module for the compiler and compile your code with the compiler you prefer (Use -O0 -g flags)
- Run the executable under Valgrind.
If you normally run your program like this:
myprog arg1 arg2
Use this command line:
valgrind (valgrind-options) myprog arg1 arg2
Memcheck is the default tool. You can add the --leak-check option that turns on the detailed memory leak detector. Your program will run much slower than normal, and use a lot more memory. Memcheck will issue messages about memory errors and leaks that it detects.
- Load Valgrind module --> module load valgrind
- Load modules for compiler and openmpi libraries (at present only available for intel and gnu)
- Compile your code with the "-O0 -g" flags both at compiling and linking time
- Run the executable under Valgrind (both in interactive than in bacth mode)
mpirun -np 4 valgrind (valgrind-options) myprog arg1 arg2
Core file analysis
In order to understand what problem was affecting you code, you can also try a "Core file" analysis. Since core files are usually quite large, be sure to work in the /scratch area.
There are several steps to follow:
- Increase the limit for possible core dumping
> ulimit -c unlimited (bash) > limit coredumpsize unlimited (csh/tcsh)
- If you are using Intel compilers, set to TRUE the decfort_dump_flag environment variable
> export decfort_dump_flag=TRUE (bash) > setenv decfort_dump_flag TRUE (csh/tcsh)
- Compile your code with the debug flags described above.
- Run your code and create the core file.
- Analyze the core file using different tools depending on the original compiler.
INTEL compilers
> module load intel > ifort -O0 -g -traceback -fp-stack-check -check bounds -fpe0 -o myexec prog.f90 > ulimit -c unlimited > export decfort_dump_flag=TRUE > ./myexec > ls -lrt -rwxr-xr-x 1 aer0 cineca-staff 9652 Apr 6 14:34 myexec -rw------- 1 aer0 cineca-staff 319488 Apr 6 14:35 core.25629 > idbc ./myexec core.25629
PGI compilers
> module load pgi > pgf90 -O0 -g -C -Ktrap=ovf,divz,inv -o myexec myprog.f90 > ulimit -c unlimited > ./myexec > ls -lrt -rwxr-xr-x 1 aer0 cineca-staff 9652 Apr 6 14:34 myexec -rw------- 1 aer0 cineca-staff 319488 Apr 6 14:35 core.25666 > pgdbg -text -core core.25666 ./myexec
GNU Compilers
> gfortran -O0 -g -Wall -fbounds-check -o myexec prog.f90 > ulimit -c unlimited > ./myexec > ls -lrt -rwxr-xr-x 1 aer0 cineca-staff 9652 Apr 6 14:34 myexec -rw------- 1 aer0 cineca-staff 319488 Apr 6 14:35 core.25555 > gdb ./myexec core.2555
Profilers (gprof)
In software engineering, profiling is the investigation of a program's behavior using information gathered as the program executes. The usual purpose of this analysis is to determine which sections of a program to optimize - to increase its overall speed, decrease its memory requirement or sometimes both.
A (code) profiler is a performance analysis tool that, most commonly, measures only the frequency and duration of function calls, but there are other specific types of profilers (e.g. memory profilers) in addition to more comprehensive profilers, capable of gathering extensive performance data.
gprof
The GNU profiler gprof is a useful tool for measuring the performance of a program. It records the number of calls to each function and the amount of time spent there, on a per-function basis. Functions which consume a large fraction of the run-time can be identified easily from the output of gprof. Efforts to speed up a program should concentrate first on those functions which dominate the total run-time.
gprof uses data collected by the -pg compiler flag to construct a text display of the functions within your application (call tree and CPU time spent in every subroutine). It also provides quick access to the profiled data, which let you identify the functions that are the most CPU-intensive. The text display also lets you manipulate the display in order to focus on the application's critical areas.
Usage:
> gfortran -pg -O3 -o myexec myprog.f90 > ./myexec > ls -ltr ....... -rw-r--r-- 1 aer0 cineca-staff 506 Apr 6 15:33 gmon.out > gprof myexec gmon.out
It is also possible to profile at code line-level (see "man gprof" for other options). In this case you must use also the “-g” flag at compilation time:
> gfortran -pg -g -O3 -o myexec myprog.f90 > ./myexec > ls -ltr ....... -rw-r--r-- 1 aer0 cineca-staff 506 Apr 6 15:33 gmon.out > gprof -annotated-source myexec gmon.out
It is possilbe to profile MPI programs. In this case the environment variable GMON_OUT_PREFIX must be defined in order to allow to each task to write a different statistical file. Setting
export GMON_OUT_PREFIX=<name>
once the run is finished each task will create a file with its process ID (PID) extension
<name>.$PID
If the environmental variable is not set every task will write the same gmon.out file.
Scientific libraries
MKL
The Intel Math Kernel Library (Intel MKL) enables improving performance of scientific, engineering, and financial software that solves large computational problems. Intel MKL provides a set of linear algebra routines, fast Fourier transforms, as well as vectorized math and random number generation functions, all optimized for the latest Intel processors, including processors with multiple cores.
Intel MKL is thread-safe and extensively threaded using the OpenMP technology
documentation can be found in:
${MKLROOT}/../Documentation/en_US/mkl
To use the MKL in your code you to load the module, then to define includes and libraries at compile and linking time:
> module load mkl > icc -I$MKL_INC -L$MKL_LIB -lmkl_intel_lp64 -lmkl_core -lmkl_sequential
For more inormation please refer to the documentation.
Parallel programming
The parallel programming on MARCONI is based on IntelMPI and OpenMPI versions of MPI. The libraries and special wrappers to compile and link the personal programs are contained in several modules, one for each supported suite of compilers.
The main four parallel-MPI commands for compilation are:
- mpiifort (IntelMPI)/ mpif90 (OpenMPI) (Fortran90)
- mpiifort (IntelMPI) / mpif77 (OpenMPI) (Fortran77)
- mpiicc (IntelMPI) / mpicc (OpenMPI) (C)
- mpiicpc (IntelMPI) / mpicxx (C++)
These command names refers to wrappers around the actual compilers, they behave differently depending on the module you have loaded.
> module avail intelmpi intelmpi/2017--binary(default) intelmpi/5.1--binary > module load intel/pe-xe-2017--binary intelmpi/2017--binary > man mpiifort > mpiifort -o myexec myprof.f90 (uses the ifort compiler)
Please note that the Intel mpif90 wrapper is meant for using the GNU gfortran compiler instead of Intel ifort. If you need to use the IntelMPI libraries with the GNU gfortran compiler you need to redefine the I_MPI_F90 variable:
> module load autoload intelmpi/2017–binary
> export I_MPI_F90=gfortran
> mpif90 -o myexec myprof.f90 (uses the gfortran compiler)
With the above setting the system gfortran (4.8) compiler will be used. You can use a more recent version loading the gnu/6.1.0 module together with the intelmpi/2017–binary.
> module avail openmpi openmpi/1-10.3--gnu--6.1.0
> module load gnu openmpi/1-10.3--gnu--6.1.0 > man mpif90 > mpif90 -o myexec myprof.f90 (uses the gfortran compiler)
The parallel applications have to be executed with the command:
> mpirun ./myexec
There are limitations on running parallel programs in the login shell. You should use the "Interactive PBSSLURM" mode, as described in the "Interactive" section, previously in this page.
-Wall Enables warnings pertaining to usage that should be avoided -fbounds-check Checks for array subscripts.