Page History

...

> ./program

> salloc -N 1 --ntasks-per-node=2 -A <account_name> 

> srun ./myprogram

> sbatch script.x

You can get a list of defined partitions with the command:

> sinfo

>#SBATCH --partition=g100_spc_prod

>#SBATCH --qos=qos_ind

Programming environment 

The programming environment of GALILEO100 consists of a choice of compilers for the main scientific languages (Fortran, C and C++), debuggers to help users in finding bugs and errors in their codes, profilers to help with code optimization. The use of the classic compilers is suggested in this first phase for a smoother migration from other clusters. In principle, binaries generated on Galileo should work, but we strongly recommend you to reinstall all your software applications since on Galileo100 there is a different Operating System (Centos 8.3).

Intel Compilers

The native, and recommended, compilers on Galileo100 are the Intel ones, since the architecture is based on Intel processors and therefore using the Intel compilers may result in a significant improvement in performance and stability of your code. On the cluster is installed the new suite Intel OneAPI. Initialize the environment with the module command:

> module load intel/oneapi-2021–binary

> module list
   Currently Loaded Modulefiles:
   intel/oneapi-2021–binary

The suite contains the new Intel oneAPI nextgen compilers (icx, icpx, ifx) and the classic compilers (icc, icpc, ifort, ...):

	Classic	oneAPI	Notes
C compilers	icc	icx	icx is the Intel nextgen compiler based on Clang/LLVM technology  plus Intel proprietary optimizations and code generation, and it enables OpenMP TARGET offload to Intel GPU targets (irrelevant on Galileo100)
C++ compilers	icpc	icpx
Fortran compilers	ifort	ifx

• icx/icc: C compilers (oneAPI/classic)
• icpx/icpc: C++ compilers (oneAPI/classic)
• ifx/ifort: Fortran compilers (oneAPI/classic)

ICX is Intel nextgen compiler based on Clang /LLVM technology plus Intel proprietary optimizations and code generation.

• You may use ICC for performance on CPU targets.
• ICX enables OpenMP TARGET offload to Intel GPU targets.

NOTE:

• the use of the new Intel® oneAPI C/C++ Compilers (icx, icpx)   is equivalent with respect to the classic C/C++ Compilers (icc, icpc)
• the use of Intel® Fortran Compiler Classic (ifort) is recommended against the Intel® oneAPI Fortran Compiler (Beta) (ifx)

The documentation can be obtained with the man command after loading the relevant module:

> man ifort
> man icx

Some miscellaneous flags are described in the following:

-extend_source    Extend over the 77 column F77's limit 
-free / -fixed    Free/Fixed form for Fortran
-ip               Enables additional interprocedural optimization for single-file compilation
-ipo              Enables interprocedural optimization 

Compilers

The native compiler is the Intel one. On the cluster is installed the new suite Intel OneAPI.

You can check the complete list of available compilers on GALILEO100 with the command:

> module available

checking the "compilers" section.

INTEL Compiler

The native compiler is the Intel one. On the cluster is installed the new suite Intel OneAPI. Using the Intel compilers may result in a significant improvement in performance and stability of your code. Initialize the environment with the module command:

> module load intel

> module list
   Currently Loaded Modulefiles:
   intel/oneapi-2021–binary

The suite contains the new Intel oneAPI compilers (icx, icpx, ifx), and also the classic compilers (icc, icpc, ifort, ...).

The names of the Intel compilers are:

• ifx / ifort: Fortran77 and Fortran90 compiler
• icx / icc: C compiler
• icpx / icpc: C++ compiler

The documentation can be obtained with the man command after loading the relevant module:

> man ifort
> man icc

Some miscellaneous flags are described in the following:

-extend_source    Extend over the 77 column F77's limit
-free / -fixed    Free/Fixed form for Fortran
-ip               Enables interprocedural optimization for single-file compilation
-ipo              Enables interprocedural optimization between files - whole program optimisation

GNU compilers

The gnu compilers are always available but they are not the best optimizing compilers, especially for an IntelOneAPI-based cluster like GALILEO100. The default version is 10.2.0.

NOTE for the migration from Galileo to Galileo100: In principle, binaries generated on Galileo should work, but we strongly recommend you to reinstall all your software applications since on Galileo100 there is a different Operating System (Centos 8.3).

PORTLAND Group (PGI)

Initialize For a more recent version of the compiler, initialize the environment with the module command:

> module load profile/advanced
> module load

gnu

 pgi

The name of the

GNU

PGI compilers are:

• g77pgf77: Fortran77 compiler
• gfortranpgf90: Fortran95 Fortran90 compiler
• gccpgf95: C Fortran95 compiler
• pghpf: High Performance Fortran compiler
• pgcc: C compiler
• pgCCg++: C++ compiler

The documentation can be obtained with the man command after loading the relevant module:

> man gfortanpgf95
> man gccpgcc

Some miscellaneous flags are described in the following:

-

ffixed-line-length-132

Mextend            To extend over the 77 column F77's limit
-

ffree-form

Mfree / -

ffixed-form

Mfixed    Free/Fixed form for Fortran

Debuggers and Profilers

If your code aborts at runtime, there may be a problem with it. In order to solve it, you can decide to analyze the core file (feature not available if the code is compiled with PGI) or to run your code using a debugger.

Compiler flags

In both cases, you need to enable compiler runtime checks, by putting specific flags during the compilation phase. In the following we describe those flags for the different Fortran compilers: if you are using the C or C++ compiler, please keep in mind that the flags may differ.

The following flags are generally available for all compilers and are mandatory for an easier debugging session:

-O0

-fast               Chooses generally optimal flags for the target platform
-fastsse           

Lower

level

Chooses 

of optimisation -g Produce debugging information

Other flags are compiler-specific and are described in the following.

INTEL Fortran compiler

The following flags are useful (in addition to "-O0 -g") for debugging your code:

-traceback        generate extra information to provide source file traceback at run time
-fp-stack-check   generate extra code to ensure that the floating-point stack is in the expected state
-check bounds     enables checking for array subscript expressions
-fpe0             allows some control over floating-point exception handling at run-time

GNU Fortran compilers

The following flags are useful (in addition to "-O0 -g") for debugging your code:

-Wall             Enables warnings pertaining to usage that should be avoided
-fbounds-check    Checks for array subscripts.

In the following we report information about some ways to debug your codes:

GNU: gdb (serial debugger)

GDB is the GNU Project debugger and allows you to see what is going on 'inside' your program while it executes -- or what the program was doing at the moment it crashed.

GDB can do four main kinds of things (plus other things in support of these) to help you catch bugs in the act:

• Start your program, specifying anything that might affect its behavior.
• Make your program stop on specified conditions.
• Examine what has happened, when your program has stopped.
• Change things in your program, so you can experiment with correcting the effects of one bug and go on to learn about another.

More details in the online documentation, using the "man gdb" command.

To use this debugger, you should compile your code with one of the gnu compilers and the debugging command-line options described above, then you run your executable inside the "gdb" environment:

> module load gnu
> gfortran -O0 -g -Wall -fbounds-check -o myexec myprog.f90
> gdb ./myexec

Core file analysis

In order to understand what problem was affecting your code, you can also try a "Core file" analysis. Since core files are usually quite large, be sure to work in the /scratch area.

There are several steps to follow:

1. Increase the limit for possible core dumping

> ulimit -c unlimited (bash)
> limit coredumpsize unlimited (csh/tcsh)

1. If you are using Intel compilers, set to TRUE the decfort_dump_flag environment variable

> export decfort_dump_flag=TRUE  (bash)       
> setenv decfort_dump_flag TRUE  (csh/tcsh)

1. Compile your code with the debug flags described above.
2. Run your code and create the core file.
3. Analyze the core file using different tools depending on the original compiler.

INTEL compilers

> module load intel
> ifort -O0 -g -traceback -fp-stack-check -check bounds -fpe0 -o myexec prog.f90
> ulimit -c unlimited
> export decfort_dump_flag=TRUE
> ./myexec
> ls -lrt
  -rwxr-xr-x 1 aer0 cineca-staff   9652 Apr  6 14:34 myexec
  -rw------- 1 aer0 cineca-staff 319488 Apr  6 14:35 core.25629
> idbc ./myexec core.25629

GNU Compilers

> module load gnu
> gfortran -O0 -g -Wall -fbounds-check -o myexec prog.f90
> ulimit -c unlimited
> ./myexec
> ls -lrt
  -rwxr-xr-x 1 aer0 cineca-staff   9652 Apr  6 14:34 myexec
  -rw------- 1 aer0 cineca-staff 319488 Apr  6 14:35 core.25555
> gdb ./myexec core.2555

VALGRIND

Valgrind is a framework for building dynamic analysis tools. There are Valgrind tools that can automatically detect many memory management and threading bugs, and profile your programs in detail. The Valgrind distribution currently includes six production-quality tools: a memory error detector, two thread error detectors, a cache and branch-prediction profiler, a call-graph generating cache profiler, and a heap profiler.

Valgrind is Open Source / Free Software, and is freely available under the GNU General Public License, version 2.

To analyse a serial application:

1. Load Valgrind module --> module load valgrind
2. Load module for the compiler and compile your code with the compiler you prefer (Use -O0 -g flags)
3. Run the executable under Valgrind.  
   

   If you normally run your program like this:

     myprog arg1 arg2
   

   Use this command line:

     valgrind  (valgrind-options) myprog arg1 arg2 

   Memcheck is the default tool. You can add the --leak-ceck option that turns on the detailed memory leak detector. Your program will run much slower  than normal, and use a lot more memory. Memcheck will issue messages about memory errors and leaks that it detects.

To analyse a parallel application:

1. Load Valgrind module --> module load valgrind
2. Load modules for compiler and openmpi libraries (at present only available for intel and gnu)
3. Compile your code with the "-O0 -g" flags both at compiling and linking time
4. Run the executable under Valgrind (both in interactive than in batch mode)

generally optimal flags for a processor that supports SSE instructions

GNU compilers

The gnu compilers are always available but they are not the best optimizing compilers, especially for an IntelOneAPI-based cluster like GALILEO100. The default version is 10.2.0.

For a more recent version of the compiler, initialize the environment with the module command:

> module load gnu

The name of the GNU compilers are:

• g77: Fortran77 compiler
• gfortran: Fortran95 compiler
• gcc: C compiler
• g++: C++ compiler

The documentation can be obtained with the man command:

> man gfortan
> man gcc

Some miscellaneous flags are described in the following:

-ffixed-line-length-132       To extend over the 77 column F77's limit
-ffree-form / -ffixed-form    Free/Fixed form for Fortran

Debuggers and Profilers

If your code aborts at runtime, there may be a problem with it. In order to solve it, you can decide to analyze the core file (feature not available if the code is compiled with PGI) or to run your code using a debugger.

Compiler flags

In both cases, you need to enable compiler runtime checks, by putting specific flags during the compilation phase. In the following we describe those flags for the different Fortran compilers: if you are using the C or C++ compiler, please keep in mind that the flags may differ.

The following flags are generally available for all compilers and are mandatory for an easier debugging session:

-O0     Lower level of optimisation
-g      Produce debugging information

Other flags are compiler-specific and are described in the following.

INTEL Fortran compiler

The following flags are useful (in addition to "-O0 -g") for debugging your code:

-traceback        generate extra information to provide source file traceback at run time
-fp-stack-check   generate extra code to ensure that the floating-point stack is in the expected state
-check bounds     enables checking for array subscript expressions
-fpe0             allows some control over floating-point exception handling at run-time

PORTLAND Group (PGI) Compilers

The following flags are useful (in addition to "-O0 -g") for debugging your code:

-C                     Add array bounds checking
-Ktrap=ovf,divz,inv    Controls the behavior of the processor when exceptions occur: 
                       FP overflow, divide by zero, invalid operands

GNU Fortran compilers

The following flags are useful (in addition to "-O0 -g") for debugging your code:

-Wall             Enables warnings pertaining to usage that should be avoided
-fbounds-check    Checks for array subscripts.

In the following we report informations about some ways to debug your codes:

PGI: pgdbg (serial/parallel debugger)

pgdbg is the Portland Group Inc. symbolic source-level debugger for F77, F90, C, C++ and assembly language programs. It is capable of debugging applications that exhibit various levels of parallelism, including:

• Single-thread, serial applications
• Multi-threaded applications
• Distributed MPI applications
• Any combination of the above

There are two forms of the command used to invoke pgdbg. The first is used when debugging non-MPI applications, the second form, using mpirun, is used when debugging MPI applications:

> pgdbg [options] ./myexec [args]
> mpirun [options] -dbg=pgdbg ./myexec [args]

More details in the on line documentation, using the "man pgdbg" command after loading the module.

To use this debugger, you should compile your code with one of the pgi compilers and the debugging command-line options described above, then you run your executable inside the "pgdbg" environment:

> module load profile/advanced
> module load pgi
> pgf90 -O0 -g -C -Ktrap=ovf,divz,inv  -o myexec myprog.f90
> pgdbg ./myexec

By default, pgdbg presents a graphical user interface (GUI). A command-line interface is also provided through the "-text" option.

GNU: gdb (serial debugger)

GDB is the GNU Project debugger and allows you to see what is going on 'inside' your program while it executes -- or what the program was doing at the moment it crashed.

GDB can do four main kinds of things (plus other things in support of these) to help you catch bugs in the act:

• Start your program, specifying anything that might affect its behavior.
• Make your program stop on specified conditions.
• Examine what has happened, when your program has stopped.
• Change things in your program, so you can experiment with correcting the effects of one bug and go on to learn about another.

More details in the online documentation, using the "man gdb" command.

To use this debugger, you should compile your code with one of the gnu compilers and the debugging command-line options described above, then you run your executable inside the "gdb" environment:

> module load gnu
> gfortran -O0 -g -Wall -fbounds-check -o myexec myprog.f90
> gdb ./myexec

Core file analysis

In order to understand what problem was affecting your code, you can also try a "Core file" analysis. Since core files are usually quite large, be sure to work in the /scratch area.

There are several steps to follow:

1. Increase the limit for possible core dumping

> ulimit -c unlimited (bash)
> limit coredumpsize unlimited (csh/tcsh)

1. If you are using Intel compilers, set to TRUE the decfort_dump_flag environment variable

> export decfort_dump_flag=TRUE  (bash)       
> setenv decfort_dump_flag TRUE  (csh/tcsh)

1. Compile your code with the debug flags described above.
2. Run your code and create the core file.
3. Analyze the core file using different tools depending on the original compiler.

INTEL compilers

> module load intel
> ifort -O0 -g -traceback -fp-stack-check -check bounds -fpe0 -o myexec prog.f90
> ulimit -c unlimited
> export decfort_dump_flag=TRUE
> ./myexec
> ls -lrt
  -rwxr-xr-x 1 aer0 cineca-staff   9652 Apr  6 14:34 myexec
  -rw------- 1 aer0 cineca-staff 319488 Apr  6 14:35 core.25629
> idbc ./myexec core.25629

PGI compilers

> module load profile/advenced
> module load pgi
> pgf90 -O0 -g -C -Ktrap=ovf,divz,inv  -o myexec myprog.f90
> ulimit -c unlimited
> ./myexec
> ls -lrt
  -rwxr-xr-x 1 aer0 cineca-staff   9652 Apr  6 14:34 myexec
  -rw------- 1 aer0 cineca-staff 319488 Apr  6 14:35 core.25666
> pgdbg -text -core core.25666 ./myexec 

GNU Compilers

> module load gnu
> gfortran -O0 -g -Wall -fbounds-check -o myexec prog.f90
> ulimit -c unlimited
> ./myexec
> ls -lrt
  -rwxr-xr-x 1 aer0 cineca-staff   9652 Apr  6 14:34 myexec
  -rw------- 1 aer0 cineca-staff 319488 Apr  6 14:35 core.25555
> gdb ./myexec core.2555

VALGRIND

Valgrind is a framework for building dynamic analysis tools. There are Valgrind tools that can automatically detect many memory management and threading bugs, and profile your programs in detail. The Valgrind distribution currently includes six production-quality tools: a memory error detector, two thread error detectors, a cache and branch-prediction profiler, a call-graph generating cache profiler, and a heap profiler.

Valgrind is Open Source / Free Software, and is freely available under the GNU General Public License, version 2.

To analyse a serial application:

1. Load Valgrind module --> module load valgrind
2. Load module for the compiler and compile your code with the compiler you prefer (Use -O0 -g flags)
3. Run the executable under Valgrind.  
   

   If you normally run your program like this:

     myprog arg1 arg2
   

   Use this command line:

     valgrind  (valgrind-options) myprog arg1 arg2 

   Memcheck is the default tool. You can add the --leak-ceck option that turns on the detailed memory leak detector. Your program will run much slower  than normal, and use a lot more memory. Memcheck will issue messages about memory errors and leaks that it detects.

To analyse a parallel application:

1. Load Valgrind module --> module load valgrind
2. Load modules for compiler and openmpi libraries (at present only available for intel and gnu)
3. Compile your code with the "-O0 -g" flags both at compiling and linking time
4. Run the executable under Valgrind (both in interactive than in batch mode)

  mpirun -np 4 valgrind (valgrind-options) myprog arg1 arg2

Totalview
Totalview is a parallel debugger with a practical GUI that assist users to debug their parallel code. It has functionalities like stopping and reprising a code mid-run, setting breakpoints, checking the value of variables anytime, browse between the different tasks and threads to see the different behaviours, memory check functions and so on. For information about how to run the debugger (by connecting the compute nodes to your display via RCM), type the command:

> module help totalview

Scalasca
Scalasca is a tool for profiling parallel scientific and engineering applications that make use of MPI and OpenMP.
Details how to use scalasca in
http://www.scalasca.org/software/scalasca-2.x/documentation.html

Profilers (gprof)

In software engineering, profiling is the investigation of a program's behaviour using information gathered as the program executes. The usual purpose of this analysis is to determine which sections of a program to optimize - to increase its overall speed, decrease its memory requirement or sometimes both.

A (code) profiler is a performance analysis tool that, most commonly, measures only the frequency and duration of function calls, but there are other specific types of profilers (e.g. memory profilers) in addition to more comprehensive profilers, capable of gathering extensive performance data.

gprof

The GNU profiler gprof is a useful tool for measuring the performance of a program. It records the number of calls to each function and the amount of time spent there, on a per-function basis. Functions which consume a large fraction of the run-time can be identified easily from the output of gprof. Efforts to speed up a program should concentrate first on those functions which dominate the total run-time.

gprof uses data collected by the -pg compiler flag to construct a text display of the functions within your application (call tree and CPU time spent in every subroutine). It also provides quick access to the profiled data, which let you identify the functions that are the most CPU-intensive. The text display also lets you manipulate the display in order to focus on the application's critical areas.

Usage:

>  gfortran -pg -O3 -o myexec myprog.f90
> ./myexec
> ls -ltr
   .......
   -rw-r--r-- 1 aer0 cineca-staff    506 Apr  6 15:33 gmon.out
> gprof myexec gmon.out

It is also possible to profile at code line-level (see "man gprof" for other options). In this case you must use also the “-g” flag at compilation time:

>  gfortran -pg -g -O3 -o myexec myprog.f90
> ./myexec
> ls -ltr
   .......
   -rw-r--r-- 1 aer0 cineca-staff    506 Apr  6 15:33 gmon.out
> gprof -annotated-source myexec gmon.out

It is possible to profile MPI programs. In this case, the environment variable GMON_OUT_PREFIX must be defined in order to allow to each task to write a different statistical file. Setting

export GMON_OUT_PREFIX=<name>

 once the run is finished each task will create a file with its process ID (PID) extension

<name>.$PID

 If the environmental variable is not set every task will write the same gmon.out file

  mpirun -np 4 valgrind (valgrind-options) myprog arg1 arg2

Totalview - will be soon available
Totalview is a parallel debugger with a practical GUI that assist users to debug their parallel code. It has functionalities like stopping and reprising a code mid-run, setting breakpoints, checking the value of variables anytime, browse between the different tasks and threads to see the different behaviours, memory check functions and so on. For information about how to run the debugger (by connecting the compute nodes to your display via RCM), type the command:

> module help totalview

Scalasca
Scalasca is a tool for profiling parallel scientific and engineering applications that make use of MPI and OpenMP.
Details how to use scalasca in
http://www.scalasca.org/software/scalasca-2.x/documentation.html

Profilers (gprof) - will be updated

In software engineering, profiling is the investigation of a program's behaviour using information gathered as the program executes. The usual purpose of this analysis is to determine which sections of a program to optimize - to increase its overall speed, decrease its memory requirement or sometimes both.

A (code) profiler is a performance analysis tool that, most commonly, measures only the frequency and duration of function calls, but there are other specific types of profilers (e.g. memory profilers) in addition to more comprehensive profilers, capable of gathering extensive performance data.

Scientific libraries (MKL)

MKL

The Intel Math Kernel Library (Intel MKL) enables improving performance of scientific, engineering, and financial software that solves large computational problems. Intel MKL provides a set of linear algebra routines, fast Fourier transforms, as well as vectorized math and random number generation functions, all optimized for the latest Intel processors, including processors with multiple cores.

Intel MKL is thread-safe and extensively threaded using the OpenMP technology.

documentation can be found by loading the mkl module and searching in the directory:

${MKLROOT}/../Documentation/en_US/mkl

To use the MKL in your code you to load the module, then to define includes and libraries at compile and linking time:

> module load mkl
> icc -I$MKL_INC -L$MKL_LIB  -lmkl_intel_lp64 -lmkl_core -lmkl_sequential

For more informations please refer to the documentation.

...

The parallel programming on GALILEO100 Galileo is based on IntelMPI and OpenMPI versions of MPI. The libraries and special wrappers to compile and link the personal programs are contained in several modules, one for each supported suite of compilers.

These command names refer to wrappers around the actual compilers, they behave differently depending on the module you have loaded. 

IntelMPI

IntelMPI in GALILEO100 GALILEO is recommended since the architecture is based on Intel processors.

To load IntelMPI on GALILEO100GALILEO, check the module versions available with the "module avail" command, then load the relevant module. After you have loaded the Intelmpi module

...

• mpiifort (Fortran90/77)
• mpiicc (C)
• mpiicpc (C++)

 

OpenMPI

On GALILEO100GALILEO, "gnu" versions of OpenMPI are available. After  you have load the openmpi module (check the available version with the "module avail" command):

...