...
Intel Compilers
The native, and recommended, compilers on GALILEO100 are the Intel ones, since the architecture is based on Intel processors and therefore using the Intel compilers may result in a significant improvement in performance and stability of your code. On the cluster is installed the new suite Intel OneAPI. Initialize the environment with the module command:
> module load intel/oneapi-2021–binary
> module list
Currently Loaded Modulefiles:
intel/oneapi-2021–binary
The suite contains the new Intel oneAPI nextgen compilers (icx, icpx, ifx) and the classic compilers (icc, icpc, ifort):
| Classic | oneAPI | Notes | |
|---|---|---|---|
| C/C++ compilers | icc/icpc | icx/icpx |
|
| Fortran compilers | ifort | ifx |
|
NOTE:
- ICX is a new compiler. It has functional and behavioral differences compared to ICC. You can expect some porting will be needed for existing applications using ICC. According to Intel, the transition from ICC Classic to ICX is smooth and effortless. However, you must port and tune any existing applications from ICC Classic to ICX. Please refer to the official Intel Porting Guide for ICC Users to DPCPP or ICX
- IFORT is a completely new compiler. According to Intel, although considerable effort is being made to make the transition from ifort to ifx as smooth and as effortless as possible, customers can expect that some effort may be required to tune their application. IFORT will remain Intel’s recommended production compiler until ifx has performance and features superior to ifort. Please refer to the official Intel Porting Guide for ifort Users to ifx
- Please refer to the official Intel C++ Developer Guide and Reference and Fortran Developer Guide and Reference for an exhaustive list of compiler options
The documentation can be obtained with the man command after loading the relevant module:
> man ifort > man icxicc
Some miscellaneous flags are described in the following:
-extend_source Extend over the 77 column F77's limit -free / -fixed Free/Fixed form for Fortran -ip Enables additional interprocedural optimization for single-file compilation -ipo Enables interprocedural optimization between files - whole program optimisation
-qopenmp enables the parallelizer to generate multi-threaded code based on OpenMP directives
NOTE for the migration from Galileo to Galileo100: In principle, binaries generated on Galileo should work, but we strongly recommend you to reinstall all your software applications since on Galileo100 there is a different Operating System (Centos 8.3).
GNU compilers
The gnu compilers are always available but they are not the best optimizing compilers, especially for an IntelOneAPI-based cluster like GALILEO100.
For a more recent version of the compiler, initialize the environment with the module command:
> module load gnu
The name of the GNU compilers are:
- g77: Fortran77 compiler
- gfortran: Fortran compiler with "gnu" standard
- gcc: C compiler
- g++: C++ compiler
The "gnu" standard is the default value for the -std option. It specifies a superset of the latest Fortran standard that includes all of the extensions supported by GNU Fortran, although warnings will be given for obsolete extensions not recommended for use in new code. To change the standard to which the program is expected to conform, set the -std option to one of the possible values (f95, f2003, f2008, f2018, gnu, or legacy).
The documentation can be obtained with the man command:
> man gfortan > man gcc
Some miscellaneous flags are described in the following:
-ffixed-line-length-132 To extend over the 77 column F77's limit -ffree-form / -ffixed-form Free/Fixed form for Fortran
-fopenmp Enable handling of OpenMP directives "#pragma omp" in C/C++ and "!$omp" in Fortran.
When -fopenmp is specified, the compiler generates parallel code according to the
OpenMP Application Program Interface v4.5. This option implies -pthread and -fopenmp-simd
Debuggers and Profilers
If your code aborts at runtime, there may be a problem with it. In order to solve it, you can decide to analyze the core file or to run your code using a debugger.
Compiler flags
In both cases, you need to enable compiler runtime checks, by putting specific flags during the compilation phase. In the following we describe those flags for the different Fortran compilers: if you are using the C or C++ compiler, please keep in mind that the flags may differ.
The following flags are generally available for all compilers and are mandatory for an easier debugging session:
-O0 Lower level of optimisation
-g Produce debugging information
Other flags are compiler-specific and are described in the following.
INTEL Fortran compiler
The following flags are useful (in addition to "-O0 -g") for debugging your code:
-traceback generate extra information to provide source file traceback at run time -fp-stack-check generate extra code to ensure that the floating-point stack is in the expected state
-fvar-tracking Generates enhanced debug information useful in finding scalar local variables
-fvar-tracking-assignments Generates enhanced debug information useful for breakpoints and stepping. It
tells the debugger to stop only at machine instructions that achieve the final
effect of a source statement. -check bounds
Debuggers and Profilers
If your code aborts at runtime, there may be a problem with it. In order to solve it, you can decide to analyze the core file or to run your code using a debugger.
In both cases, you need to enable compiler runtime checks, by putting specific flags during the compilation phase. In the following we describe those flags for the different Fortran compilers: if you are using the C or C++ compiler, please keep in mind that the flags may differ.
The following flags are generally available for all compilers and are mandatory for an easier debugging session:
-O0 Lower level of optimisation
-g Produce debugging information
Other flags are compiler-specific and are described in the following.
INTEL Fortran compiler
The following flags are useful (in addition to "-O0 -g") for debugging your code:
-traceback generate extra information to provide source file traceback at run time -fp-stack-check generate extra code to ensure that the floating-point stack is in the expected state -check bounds enables checking for array subscript expressions -fpe0 allows some control over floating-point exception handling at run-time
Please also refer to the debug C++ compiler option and the debug Fortran compiler option.
GNU Fortran compilers
The following flags are useful (in addition to "-O0 -g") for debugging your code:
-Wall Enables warnings pertaining to usage that should be avoided
-fbounds-check Checks for array subscripts.
GNU: gdb (serial debugger)
GDB is the GNU Project debugger and allows you to see what is going on 'inside' your program while it executes -- or what the program was doing at the moment it crashed.
GDB can do four main kinds of things (plus other things in support of these) to help you catch bugs in the act:
- Start your program, specifying anything that might affect its behavior.
- Make your program stop on specified conditions.
- Examine what has happened, when your program has stopped.
- Change things in your program, so you can experiment with correcting the effects of one bug and go on to learn about another.
More details in the online documentation, using the "man gdb" command.
To use this debugger, you should compile your code with one of the gnu compilers and the debugging command-line options described above, then you run your executable inside the "gdb" environment:
> module load gnu
> gfortran -O0 -g -Wall -fbounds-check -o myexec myprog.f90
> gdb ./myexec
Core file analysis
In order to understand what problem was affecting your code, you can also try a "Core file" analysis. Since core files are usually quite large, be sure to work in the scratch area.
There are several steps to follow:
- Increase the limit for possible core dumping
> ulimit -c unlimited (bash) > limit coredumpsize unlimited (csh/tcsh)
- If you are using Intel Fortran compilers, set to TRUE the the for_dump_core_file (or decfort_dump_flag environment ) environment variable
> export decfortfor_dump_core_flagfile=TRUE (bash) > setenv decfortfor_dump_core_flagfile TRUE (csh/tcsh)
- Compile your code with the debug flags described above.
- Run your code and create the core file.
- Analyze the core file using different tools depending on the original compiler.
INTEL compilers
> module load intel > ifort -O0 -g -traceback -fp-stack-check -check bounds -fpe0 -o myexec prog.f90 > ulimit -c unlimited > export decfortfor_dump_core_flagfile=TRUE > ./myexec > ls -lrt -rwxr-xr-x 1 aer0 cineca-staff 9652 Apr 6 14:34 myexec -rw------- 1 aer0 cineca-staff 319488 Apr 6 14:35 core.25629 > idbc ./myexec35 core.25629
GNU Compilers
> module load gnu
> gfortran -O0 -g -Wall -fbounds-check -o myexec prog.f90 > ulimit -c unlimited > ./myexec > ls -lrt -rwxr-xr-x 1 aer0 cineca-staff 9652 Apr 6 14:34 myexec -rw------- 1 aer0 cineca-staff 319488 Apr 6 14:35 core.25555 > gdb ./myexec core.2555
VALGRIND - will be soon available
Valgrind is a framework for building dynamic analysis tools. There are Valgrind tools that can automatically detect many memory management and threading bugs, and profile your programs in detail. The Valgrind distribution currently includes six production-quality tools: a memory error detector, two thread error detectors, a cache and branch-prediction profiler, a call-graph generating cache profiler, and a heap profiler.
Valgrind is Open Source / Free Software, and is freely available under the GNU General Public License
Totalview - will be soon available
Totalview is a parallel debugger with a practical GUI that assist users to debug their parallel code. It has functionalities like stopping and reprising a code mid-run, setting breakpoints, checking the value of variables anytime, browse between the different tasks and threads to see the different behaviours, memory check functions and so on. For information about how to run the debugger (by connecting the compute nodes to your display via RCM), type the command:
> module help totalview
Scalasca
Scalasca is a tool for profiling parallel scientific and engineering applications that make use of MPI and OpenMP.
Details how to use scalasca in
http://www.scalasca.org/software/scalasca-2.x/documentation.html
Profilers - it will be updated
In software engineering, profiling is the investigation of a program's behaviour using information gathered as the program executes. The usual purpose of this analysis is to determine which sections of a program to optimize - to increase its overall speed, decrease its memory requirement or sometimes both.
A (code) profiler is a performance analysis tool that, most commonly, measures only the frequency and duration of function calls, but there are other specific types of profilers (e.g. memory profilers) in addition to more comprehensive profilers, capable of gathering extensive performance data.
Scientific libraries (MKL)
MKL
The Intel Math Kernel Library (Intel MKL) enables improving performance of scientific, engineering, and financial software that solves large computational problems. Intel MKL provides a set of linear algebra routines, fast Fourier transforms, as well as vectorized math and random number generation functions, all optimized for the latest Intel processors, including processors with multiple cores.
Intel MKL is thread-safe and extensively threaded using the OpenMP technology.
documentation Examples can be found by loading the mkl module and searching in the directory:
${MKLROOT}/../Documentation/en_US/mkl
To use the MKL in your code you to load the module, then to define includes and libraries at compile and linking time:
> module load mkl
> icc -I$MKL_INC -L$MKL_LIB -lmkl_intel_lp64 -lmkl_core -lmkl_sequential
}/examples
To use the MKL in your code you to load the module, then to define includes and libraries at compile and linking time, please refer to the Intel oneAPI Math Kernel Library Link Line Advisor:
https://software.intel.com/content/www/us/en/develop/tools/oneapi/components/onemkl/link-line-advisor.htmlFor more informations please refer to the documentation.
Parallel programming
The parallel programming on GALILEO100 is based on IntelMPI and OpenMPI versions of MPI. The libraries and special wrappers to compile and link the personal programs are contained in several modules, one for each supported suite of compilers.
...