Page History

System Architecture

Compute Nodes: There are 360 currently 1022 36-core compute nodes. Each one contains 2 18-cores Intel Xeon E5-2697 v4 (Broadwell) at 2.30 GHz. All the compute nodes have 128 GB of memory. 15 of Of these compute nodes, 60 are equipped with 2  nVidia K80 GPU and two with nVidia V100.

Login and Service nodes: There are 8  Login Login & Viz node are available, equipped with 2 nVidia K40 GPU each. 8  service nodes available (5 are academic nodes and 3 are for industrial users). There are 8 service nodes for I/O and cluster management.

All the nodes are interconnected through a an Infiniband network, with OFED v1.5.3OPA v10.6, capable of a maximum bandwidth of 40Gbit100Gbit/s between each pair of nodes.

Accounting

For more information about accounting information , please consult our dedicated section.

Budget Linearization policy

On GALILEO a linearization policy for the usage of project budgets has been defined and implemented. For each account, a monthly quota is defined as:

monthTotal = (total_budget / total_no_of_months)

Starting from the first day of each month, the collaborators of any account are allowed to use the quota at full priority. As long as the budget is consumed, the jobs submitted from the account will gradually lose priority, until the monthly budget (monthTotal) is fully consumed. At that moment, their jobs will still be considered for execution, but with a lower priority than the jobs from accounts that still have some monthly quota left.

This policy is similar to those already applied by other important HPC centers in Europe and worldwide. The goal is to improve the response time, giving users the opportunity of using the cpu hours assigned to their project in relation of to their actual size (total amount of core-hours). 

Disks and Filesystems

The storage organisation conforms to the CINECA infrastructure (see Section "Data storage and Filesystems") . In addition to the home directory ($HOME), for each user is defined a scratch area $CINECA_SCRATCH, a large disk for storing run time data and files. $WORK is defined for each active project on the system, reserved for all the collaborators of the project. This is a safe storage area to keep run time data for the whole life of the project.

 It is also available a temporary storage local on compute nodes generated when the job starts and accessible via environment variable $TMPDIR. For more details please see the dedicated section of UG2.5: Data storage and FileSystems. On Galileo the $TMPDIR local area has 49 GB of available space.

$DRES points to the shared repository where Data RESources are maintained. This is a data archive area availble available only on-request, shared with all CINECA HPC systems and among different projects.

$DRES is not mounted on the compute nodes. This means that you can't access it within a batch job: all data needed during the batch execution has to be moved on $WORK or $CINECA_SCRATCH before the run starts.

Since all the filesystems are based on gpfs (General Parallel FIle System), the usual unix command "quota" is not working. Use the local command "cindata" to query for disk usage and quota ("cindata -h" for help):

  > cindata

Modules environment

As usual, the The software modules are collected in different profiles and organized by functional category (compilers, libraries, tools, applications,..).

On GALILEO a new feature has been added to the module environment: the profiles are of two types,  “domain” “domain” type (chem, phys, lifesc,..) for the production activity and “programming” type (base and advanced)  for for compilation, debugging and profiling activities and that they can be loaded together.

"Base" profile is the default. It is automatically loaded after login and it contains basic modules for the programming activities (intel e gnu compilers, math libraries, profiling and debugging tools,..).

If you want to use a module placed under others other profiles, for example an application module, you will have to load preventively the corresponding profile:

>module load profile/<profile name>

>module load autoload <module name>

For listing all profiles you have loaded you can use the following command:

>module list

In order to detect all profiles, categories and modules available on GALILEO the command “modmap” is available:

>modmap

We allow you to load old-Galileo environment modules.

To switch from new environmengt (the default one) to the old and vice-versa, you can load the superc module

>module load superc

and run one of the following commands:

FOR BASH USERS

. $SUPERC_SWITCH/switch_to_old.sh

to switch from new ro old environment, or

. $SUPERC_SWITCH/switch_to_new.sh

to switch from old to new environment.

FOR CSH/TCSH USERS:

source $SUPERC_SWITCH/switch_to_old.csh

to switch from new ro old environment, or

source $SUPERC_SWITCH/switch_to_new.csh

to switch from old to new environment.

Production environment

Since GALILEO is a general purpose system and it is used by several users at the same time, long production jobs must be submitted using a queuing system. This guarantees that the access to the resources is as fair as possible.

Roughly speaking, there are two different modes to use an HPC system: Interactive and Batch. For a general discussion see the section "Production Environment and Tools".

Interactive

A serial program can be executed in the standard UNIX way:

> ./program

This is allowed only for very short runs, since the interactive environment has a 10 minutes time limit: for longer runs please use the "batch" mode.

A parallel program can be executed interactively only within an "Interactive" SLURM batch job, using the "srun" command: the job is queued and scheduled as any other job, but when executed, the standard input, output, and error streams are connected to the terminal session from which srun was launched.

For example, to start an interactive session with the MPI program myprogram, using one node, two processors, launch the command:

SLURM will then schedule your job to start, and your shell will be unresponsive until free resources are allocated for you.

When the shell come back with the prompt, you can execute your program by typing:

> mpirun ./myprogram

or

> srun --mpi=pmi2 ./myprogram

Pay attention in putting the flag --mpi=pmi2 immediatly after srun.

SLURM automatically exports the environment variables you defined in the source shell, so that if you need to run your program myprogram in a controlled environment (i.e. specific library paths or options), you can prepare the environment in the origin shell being sure to find it in the interactive shell.

> sbatch script.x

> sinfo -a

#SBATCH -p gll_all_serial

Graphic session

If a graphic session is desired we recommend to use the RCM tool or the EnginFrame environment. Both of them are under construction, we will report here all required information asap.

Submitting parallel Batch jobs

To run parallel batsch jobs on GALILEO you need to specify the partition gll_usr_prod, or any other partition we invited you to use.

If you do not specify the partition, your jobs will try to run on the bdw_all_serial partition, eventually failing if specific partition limits (maximum one core for maximum walltime of 4 hours) are violated.

The minimum number of cores to require is 1. The maximum number of cores that you can request is the 2304 (about 167 nodes) with a maximum walltime of 24 hours:

• If you do not specify the walltime (by means of the #SBATCH --time directive), a default value of 30 minutes will be assumed.
• If you do not specify the number of cores (by means of the "SBATCH -n" directive) a default value of 36 will be assumed.
• If you do not specify the amount of memory (as the value of the "SBATCH --mem" DIRECTIVE), a default value of 3000MB will be assumed.
• The maximum memory per node is 118000MB.

The special QOS (bdw_qos_special) is designed for not-ordinary types of jobs, and users need to be enabled in order to use it. Please write to superc@cineca.it in case you think you need to use it.

Summary

In the following table you can find all the main features and limits imposed on the queues of the shared A1 and A2 partitions. For Marconi-FUSION dedicated queues please refer to the dedicated document.

SLURM

partition

max running jobs per user/

max n. of cpus/nodes per user

max memory per node

(MB)

gll_all_serial

(default partition)

Max 12 running jobs

Max 4 jobs/user

min = 1

max = 2304

118000

PLEASE NOTE: the SLURM characteristics have not completely defined for GALILEO, some changes will be possible.

Users with reserved resources

Users of projects that require reserved resources will be associated to a QOS.

Using the specific QOS (i.e. specifying the QOS in the submission script) , an specifying the partition gll_spc_prod partition, users associated to the allowed project will run their jobs on reserved nodes in the gll_spc_prod partitioN

>#SBATCH --partition="gll_spc_prod"

>#SBATCH --qos=<specific_qos>

With modmap you can see if the desired module is available and which profile you have to load to use it.

>modmap -m <module name>

Production environment

Since GALILEO is a general purpose system and it is used by several users at the same time, long production jobs must be submitted using a queuing system. This guarantees that access to the resources is as fair as possible.

Roughly speaking, there are two different modes to use an HPC system: Interactive and Batch. For a general discussion see the section "Production Environment".

Interactive

A serial program can be executed in the standard UNIX way:

> ./program

This is allowed only for very short runs, since the interactive environment set on the login nodes has a 10 minutes time limit: for longer runs please use the "batch" mode.

A parallel program can be executed interactively only by submitting an "Interactive" SLURM batch job, using the "srun" command: the job is queued and scheduled as any other job, but when executed, the standard input, output, and error streams are connected to the terminal session from which srun was launched.

For example, to start an interactive session with the MPI program "myprogram", using one node and two processors, you can launch the command:

> srun -N 1 --ntasks-per-node=2 -A <account_name> --pty /bin/bash

SLURM will then schedule your job to start, and your shell will be unresponsive until free resources are allocated for you. If not specified, the default time limit for this kind of jobs is one hour.

When the shell returns a prompt inside the compute node, you can execute your program by typing:

> srun ./myprogram

or

> mpirun ./myprogram

SLURM automatically exports the environment variables you defined in the source shell, so that if you need to run your program "myprogram" in a controlled environment (i.e. with specific library paths or options), you can prepare the environment in the login shell and be sure to find it again in the interactive shell o the compute node.

Batch

As usual, on systems using SLURM, you can submit a script script.x using the command:

> sbatch script.x

You can get a list of defined partitions with the command:

> sinfo

For more information and examples of job scripts, see section Batch Scheduler SLURM.

Submitting serial Batch jobs

The gll_all_serial partition is available with one core and a maximum walltime of 4 hours. It runs on two nodes and it is designed for pre/post-processing serial analysis, and for moving your data (via rsync, scp etc.) in case more than 10 minutes are required to complete the data transfer. In order to use this partition you have to specify the SLURM flag "-p":

#SBATCH -p gll_all_serial

The gll_all_serial partition has a limit of 1 task per job and 4GB of memory per job. If you wish to ask for more than a core on a single job, remember to add on your jobscript the specific about the memory limit, since the default per core is 3.5GB and therefore your job won't enter because the required memory exceeds the partition limit.

IMPORTANT FOR USERS OF THE PRE-UPGRADED VERSION OF GALILEO:  the old filesystems (such as $CINECA_SCRATCH) haven't been migrated to the upgraded cluster, but will remain available for a certain period of time (until the end of 2019) and visible only on specific nodes. You can access such nodes for transferring your data from the old environment to the new, by using the partition:

#SBATCH -p gll_all_transfer 

with the limitations of one core and 24 hours per job. Inside the gll_all_transfer nodes, you will find your old scratch area at the mount point /gpfs/scratch_old .

Graphic session

If a graphic session is desired we recommend to use the tool "RCM". See the corresponding session to know more about how to download and use RCM.

A complete reconfiguration of the RCM environment is in progress. This guide will be completed as soon as a final configuration will be implemented.

Submitting parallel Batch jobs

To run parallel batch jobs on GALILEO you need to specify the partition gll_usr_prod, or any other partition described in this user guide.

Users who need to run on GPU-equipped nodes need to specify the partition gll_usr_gpuprod.

If you do not specify the partition, your jobs will try to run on the default partition bdw_all_serial, meant for serial jobs, eventually failing if specific partition limits (maximum four tasks per job and maximum walltime of 4 hours) are violated.

The minimum number of cores you can request for a batch job is 1. The maximum number of cores that you can request is 2304 (64 nodes). It is also possible to request a maximum walltime of 24 hours. Defaults are as follows:

• If you do not specify the walltime (by means of the #SBATCH --time directive), a default value of 30 minutes will be assumed.

• If you do not specify the number of cores (by means of the "SBATCH -n" directive) a default value of 36 will be assumed.

• If you do not specify the amount of memory (as the value of the "SBATCH --mem" DIRECTIVE), a default value of 3000MB will be assumed.

• The maximum memory per node is 118000MB

A special QOS (qos_special) is also available for not-ordinary types of jobs, such as a walltime larger than 24 hours. Since it violates our standard policy, there are restrictions in its usage, and users who want to use it need to be enabled by the User Support staff. Please write to superc@cineca.it in case you think you need to use it. Your request will be evaluated and, if approved, you will be allowed to use the special QOS for a limited period of time.

Use of GPUs on GALILEO

The gll_usr_gpuprod partition is defined on 42 Broadwell nodes (18-cores Intel Xeon E5-2697 v4 @ 2.40GHz), each equipped with 1 nVidia K80 GPUs (seen as two K40 gpus). All users using an account with available budget can submit jobs on this partition and use GPU nodes on GALILEO.  

The maximum number of nodes that can be required on gll_usr_gpuprod is 4 for a maximum walltime of 08:00:00 hours. The maximum memory is 118000 MB. 

In regards of writing a SLURM jobscript, you need to request the GPU as "gres":

#SBATCH --partition=gll_usr_gpuprod

#SBATCH --gres=gpu:kepler:N     (N=1,2)

GALILEO is also equipped with two nodes with one nVIDIA Volta (V100) GPUs each, accessible for tests for a limited period of time. Please write to superc@cineca.it if you are interested to test the Volta GPUs, with a brief motivation for your request. Once your request is approved and you are enabled to use these resources (via the association to a special QOS "gll_qos_gpudev"), you can submit jobs to the Volta node with the following options:

#SBATCH --partition=gll_usr_gpudev

#SBATCH --qos=gll_qos_gpudev

#SBATCH --gres=gpu:volta:1

Users with reserved resources

Users of projects that require reserved resources (such as industrial users or users associated to an agreement that involves dedicated resources) will be associated to a QOS.

Using the specific QOS (i.e. specifying the QOS in the submission script) , and specifying the partition gll_spc_prod, users associated to the allowed project will run their jobs on reserved nodes in the gll_spc_prod partition:

>#SBATCH --partition=gll_spc_prod

>#SBATCH --qos=<specific_qos>

Summary

In the following table, you can find all the main features and limits imposed on the SLURM partitions and QOS.

Compilers

You can check the complete list of available compilers on GALILEO with the command:

> module available

checking the "compilers" section.

In general, the available compilers are:

• INTEL (ifort, icc, icpc) : ► module load intel
• PGI - Portland Group (pgf77,pgf90,pgf95,pghpf, pgcc, pgCC): ► module load pgi (profile/advanced)
• GNU (gcc, g77, g95): ► module load gnu

After loading the appropriate module, use the "man" command to get the complete list of the flags supported by the compiler, for example:

> module load intel
> man ifort

There are some flags that are common for all these compilers. Others are more specific. The most common are reported later for each compiler.

1. If you want to use a specific library or a particular include file, you have to give their paths, using the following options

-I/path_include_files        specify the path of the include files
-L/path_lib_files -l<xxx>    specify a library lib<xxx>.a in /path_lib_files

1. If you want to debug your code you have to turn off optimisation and turn on run time checkings: these flags are described in the following section.
2. If you want to compile your code for normal production you have to turn on optimization by choosing a higher optimization level

-O2 or -O3      Higher optimisation levels

Other flags are available for specific compilers and are reported later.

INTEL Compiler

Intel family compiler suite is recommended on GALILEO, since the architecture is based on Intel processors and therefore using the Intel compilers may result in a significant improvement in performance and stability of your code. Initialize the environment with the module command:

> module load intel

The names of the Intel compilers are:

• ifort: Fortran77 and Fortran90 compiler
• icc: C compiler
• icpc: C++ compiler

The documentation can be obtained with the man command after loading the relevant module:

> man ifort
> man icc

Some miscellaneous flags are described in the following:

-extend_source    Extend over the 77 column F77's limit
-free / -fixed    Free/Fixed form for Fortran
-ip  

Compilers

You can check the complete list of available compiler on GALILEO with the command:

> module available

and checking the "compilers" section.

In general the available compilers are:

• INTEL (ifort, icc, icpc) : ► module load intel
• PGI - Portland Group (pgf77,pgf90,pgf95,pghpf, pgcc, pgCC): ► module load pgi
• GNU (gcc, g77, g95): ► module load gnu

After loading the appropriate module, use the "man" command to get the complete list of the flags supported by the compiler, for example:

> module load intel
> man ifort

There are some flags that are common for all these compilers. Others are more specifics. The most common are reported later for each compiler.

1. If you want to use a specific library or a particular include file, you have to give their paths, using the following options

-I/path_include_files        specify the path of the include files
-L/path_lib_files -l<xxx>    specify a library lib<xxx>.a in /path_lib_files

1. If you want to debug your code you have to turn off optimisation and turn on run time checkings: these flags are described in the following section.
2. If you want to compile your code for normal production you have to turn on optimisation by choosing a higher optimisation level

-O2 or -O3      Higher optimisation levels

Other flags are available for specific compilers and are reported later.

INTEL Compiler

Initialize the environment with the module command:

> module load intel

The names of the Intel compilers are:

• ifort: Fortran77 and Fortran90 compiler
• icc: C compiler
• icpc: C++ compiler

The documentation can be obtained with the man command after loading the relevant module:

> man ifort
> man icc

Some miscellanous flags are described in the following:

-extend_source    Extend over the 77 column F77's limit
-free / -fixed    Free/Fixed form for Fortran
-ip               Enables interprocedural optimization for single-file compilation
-ipo              Enables interprocedural optimization between files - whole program optimisation

PORTLAND Group (PGI)

Initialize the environment with the module command:

> module load profile/advanced
> module load pgi

The name of the PGI compilers are:

• pgf77: Fortran77 compiler
• pgf90: Fortran90 compiler
• pgf95: Fortran95 compiler
• pghpf: High Performance Fortran compiler
• pgcc: C compiler
• pgCC: C++ compiler

The documentation can be obtained with the man command after loading the relevant module:

> man pgf95
> man pgcc

Some miscellanous flags are described in the following:

-Mextend            To extend over the 77 column F77's limit
-Mfree / -Mfixed    Free/Fixed form for Fortran
-fast               Chooses generally optimal flags for the target platform
-fastsse            Chooses generallyEnables optimalinterprocedural flagsoptimization for a processor that supports SSE instructions

GNU compilers

The gnu compilers are always available but they are not the best optimizing compilers. The default version is 4.8.2, you do not need to load the module for using them.

For a more recent version of the compiler, initialize the environment with the module command:

> module load gnu

The name of the GNU compilers are:

• g77: Fortran77 compiler
• gfortran: Fortran95 compiler
• gcc: C compiler
• g++: C++ compiler

The documentation can be obtained with the man command:

> man gfortan
> man gcc

Some miscellanous flags are described in the following:

-ffixed-line-length-132       To extend over the 77 column F77's limit
-ffree-form / -ffixed-form    Free/Fixed form for Fortran

Hybrid programming (Intel PHI)

Some of the nodes of the system are equipped with 2 accelerators per node. They can be addressed within C or Fortran programs by means of the Intel compilers suite.

You can find a brief guide on the production and programming environment for the Intel Xeon Phi (MIC) nodes in a separate document here.

Debuggers and Profilers

If at runtime your code dies, then there is a problem. In order to solve it, you can decide to analyze the core file (core not available with PGI compilers) or to run your code using the debugger.

Compiler flags

Whatever your decision, in any case you need enable compiler runtime checks, by putting specific flags during the compilation phase. In the following we describe those flags for the different Fortran compilers: if you are using the C or C++ compiler, please check before because the flags may differ.

The following flags are generally available for all compilers and are mandatory for an easier debuggin session:

-O0     Lower level of optimisation
-g      Produce debugging information

Other flags are compiler specific and are described in the following.

INTEL Fortran compiler

The following flags are usefull (in addition to "-O0 -g")for debugging your code:

 single-file compilation
-ipo              Enables interprocedural optimization between files - whole program optimisation

PORTLAND Group (PGI)

Initialize the environment with the module command:

> module load profile/advanced
> module load pgi

The name of the PGI compilers are:

• pgf77: Fortran77 compiler
• pgf90: Fortran90 compiler
• pgf95: Fortran95 compiler
• pghpf: High Performance Fortran compiler
• pgcc: C compiler
• pgCC: C++ compiler

The documentation can be obtained with the man command after loading the relevant module:

> man pgf95
> man pgcc

Some miscellaneous flags are described in the following:

-Mextend            To extend over the 77 column F77's limit
-Mfree / -Mfixed    Free/Fixed form for Fortran
-fast

Chooses 

PORTLAND Group (PGI) Compilers

The following flags are usefull (in addition to "-O0 -g")for debugging your code:

generally optimal flags for the target platform
-fastsse            Chooses generally optimal flags for a processor 

that 

supports 

GNU Fortran compilers

SSE instructions

GNU compilers

The gnu compilers are always available but they are not the best optimizing compilers, especially for an Intel-based cluster like GALILEO. The default version is 4.8.5, you do not need to load the module for using it.

For a more recent version of the compiler, initialize the environment with the module command:

> module load gnu

The name of the GNU compilers are:

• g77: Fortran77 compiler
• gfortran: Fortran95 compiler
• gcc: C compiler
• g++: C++ compiler

The documentation can be obtained with the man command:

> man gfortan
> man gcc

Some miscellaneous flags are described in the following:

-ffixed-line-length-132

The following flags are usefull (in addition to "-O0 -g")for debugging your code:

       To extend over 

$INTEL_HOME/Documentation/en_US/idb

> module load intel
> ifort -O0 -g -traceback -fp-stack-check -check bounds -fpe0 -o myexec myprog.f90
> idb ./myexec

> pgdbg [options] ./myexec [args]
> mpirun [options] -dbg=pgdbg ./myexec [args]

> module load profile/advanced
> module load pgi
> pgf90 -O0 -g -C -Ktrap=ovf,divz,inv  -o myexec myprog.f90
> pgdbg ./myexec

> module load gnu
> gfortran -O0 -g -Wall -fbounds-check -o myexec myprog.f90
> gdb ./myexec

  mpirun -np 4 valgrind (valgrind-options) myprog arg1 arg2

Core file analysis

In order to understand what problem was affecting you code, you can also try a "Core file" analysis. Since core files are usually quite large, be sure to work in the /scratch area.

There are several steps to follow:

1. Increase the limit for possible core dumping

> ulimit -c unlimited (bash)
> limit coredumpsize unlimited (csh/tcsh)

1. If you are using Intel compilers, set to TRUE the decfort_dump_flag environment variable

> export decfort_dump_flag=TRUE  (bash)       
> setenv decfort_dump_flag TRUE  (csh/tcsh)

1. Compile your code with the debug flags described above.
2. Run your code and create the core file.
3. Analyze the core file using different tools depending on the original compiler.

INTEL compilers

the 77 column F77's limit
-ffree-form / -ffixed-form    Free/Fixed form for Fortran

Debuggers and Profilers

If your code aborts at runtime, there may be a problem with it. In order to solve it, you can decide to analyze the core file (feature not available if the code is compiled with PGI) or to run your code using a debugger.

-O0     Lower level of optimisation
-g      Produce debugging information

-traceback        generate extra information to provide source file traceback at run time
-fp-stack-check   generate extra code to ensure that the floating-point stack is in the expected state
-check bounds     enables checking for array subscript expressions
-fpe0             allows some control over floating-point exception handling at run-time

-C                     Add array bounds checking
-Ktrap=ovf,divz,inv    Controls the behavior of the processor when exceptions occur: 
                       FP overflow, divide by zero, invalid operands

-Wall             Enables warnings pertaining to usage that should be avoided
-fbounds-check    Checks for array subscripts.

> pgdbg [options] ./myexec [args]
> mpirun [options] -dbg=pgdbg ./myexec [args]

> module load profile/advanced
> module load pgi
> pgf90 -O0 -g -C -Ktrap=ovf,divz,inv  -o myexec myprog.f90
> pgdbg ./myexec

> module load gnu
> gfortran -O0 -g -Wall -fbounds-check -o myexec myprog.f90
> gdb ./myexec

> ulimit -c unlimited (bash)
> limit coredumpsize unlimited (csh/tcsh)

> export decfort_dump_flag=TRUE  (bash)       
> setenv decfort_dump_flag TRUE  (csh/tcsh)

> module load intel
> ifort -O0 -g -traceback -fp-stack-check -check bounds -fpe0 -o myexec prog.f90
> ulimit -c unlimited
> export decfort_dump_flag=TRUE
> ./myexec
> ls -lrt
  -rwxr-xr-x 1 aer0 cineca-staff   9652 Apr  6 14:34 myexec
  -rw------- 1 aer0 cineca-staff 319488 Apr  6 14:35 core.25629
> idbc ./myexec core.25629

> module load profile/advenced
> module load pgi
> pgf90 -O0 -g -C -Ktrap=ovf,divz,inv  -o myexec myprog.f90
> ulimit -c unlimited

> ./myexec
> ls -lrt
  -rwxr-xr-x 1 aer0 cineca-staff   9652 Apr  6 14:34 myexec
  -rw------- 1 aer0 cineca-staff 319488 Apr  6 14:35 core.

25666
>

 pgdbg -text -core core.25666 ./myexec 

> module load 

gnu
> 

gfortran -O0 -g -

Wall -

fbounds-check -o myexec 

prog.f90
> ulimit -c unlimited
> ./myexec
> ls -lrt
  -rwxr-xr-x 1 aer0 cineca-staff   9652 Apr  6 14:34 myexec
  -rw------- 1 aer0 cineca-staff 319488 Apr  6 14:35 core.25555
> gdb ./myexec core.2555

  mpirun -np 4 valgrind (valgrind-options) myprog arg1 arg2

Totalview
Totalview is a parallel debugger with a practical GUI that assist users to debug their parallel code. It has functionalities like stopping and reprising a code mid-run, setting breakpoints, checking the value of variables anytime, browse between the different tasks and threads to see the different behaviours, memory check functions and so on. For information about how to run the debugger (by connecting the compute nodes to your display via RCM), type the command:

> module help totalview

Scalasca
Scalasca is a tool for profiling parallel scientific and engineering applications that make use of MPI and OpenMP.
Details how to use scalasca in
http://www.scalasca.org/software/scalasca-2.x/documentation.html

GNU Compilers

Profilers (gprof)

In software engineering, profiling is the investigation of a program's behavior behaviour using information gathered as the program executes. The usual purpose of this analysis is to determine which sections of a program to optimize - to increase its overall speed, decrease its memory requirement or sometimes both.

A (code) profiler is a performance analysis tool that, most commonly, measures only the frequency and duration of function calls, but there are other specific types of profilers (e.g. memory profilers) in addition to more comprehensive profilers, capable of gathering extensive performance data.

gprof

The GNU profiler gprof is a useful tool for measuring the performance of a program. It records the number of calls to each function and the amount of time spent there, on a per-function basis. Functions which consume a large fraction of the run-time can be identified easily from the output of gprof. Efforts to speed up a program should concentrate first on those functions which dominate the total run-time.

gprof uses data collected by the -pg compiler flag to construct a text display of the functions within your application (call tree and CPU time spent in every subroutine). It also provides quick access to the profiled data, which let you identify the functions that are the most CPU-intensive. The text display also lets you manipulate the display in order to focus on the application's critical areas.

Usage:

>  gfortran -pg -O3 -o myexec myprog.f90
> ./myexec
> ls -ltr
   .......
   -rw-r--r-- 1 aer0 cineca-staff    506 Apr  6 15:33 gmon.out
> gprof myexec gmon.out

It is also possible to profile at code line-level (see "man gprof" for other options). In this case you must use also the “-g” flag at compilation time:

>  gfortran -pg -g -O3 -o myexec myprog.f90
> ./myexec
> ls -ltr
   .......
   -rw-r--r-- 1 aer0 cineca-staff    506 Apr  6 15:33 gmon.out
> gprof -annotated-source myexec gmon.out

It is possilbe possible to profile MPI programs. In this case, the environment variable GMON_OUT_PREFIX must be defined in order to allow to each task to write a different statistical file. Setting

export GMON_OUT_PREFIX=<name>

 once the run is finished each task will create a file with its process ID (PID) extension

<name>.$PID

 If the environmental variable is not set every task will write the same gmon.out file.

Scientific libraries (MKL)

MKL

The Intel Math Kernel Library (Intel MKL) enables improving performance of scientific, engineering, and financial software that solves large computational problems. Intel MKL provides a set of linear algebra routines, fast Fourier transforms, as well as vectorized math and random number generation functions, all optimized for the latest Intel processors, including processors with multiple cores.

Intel MKL is thread-safe and extensively threaded using the OpenMP technology.

documentation can be found by loading the mkl module and searching in the directory:

${MKLROOT}/../Documentation/en_US/mkl

To use the MKL in your code you to load the module, then to define includes and libraries at compile and linking time:

> module load mkl
> icc -I$MKL_INC -L$MKL_LIB  -lmkl_intel_lp64 -lmkl_core -lmkl_sequential

For more inormation informations please refer to the documentation.