Page History

Production environment

Since GALILEO100 is a general purpose system and it is used by several users at the same time, long production jobs must be submitted using a queuing system. This guarantees that access to the resources is as fair as possible.

Roughly speaking, there are two different modes to use an HPC system: Interactive and Batch. For a general discussion see the section "Production Environment".

Interactive

A serial program can be executed in the standard UNIX way:

> ./program

This is allowed only for very short runs, since the interactive environment set on the login nodes has a 10 minutes time limit: for longer runs please use the "batch" mode.

A parallel program can be executed interactively only by submitting an "Interactive" SLURM batch job, using the "srun" command: the job is queued and scheduled as any other job, but when executed, the standard input, output, and error streams are connected to the terminal session from which srun was launched.

For example, to start an interactive session with the MPI program "myprogram", using one node and two processors, you can launch the command:

> salloc -N 1 --ntasks-per-node=2 -A <account_name> 

SLURM will then schedule your job to start, and your shell will be unresponsive until free resources are allocated for you. If not specified, the default time limit for this kind of jobs is one hour.

When the shell returns a prompt inside the compute node, you can execute your program by typing:

> srun ./myprogram

(srun is recommended with respect to mpirun for this environment)

SLURM automatically exports the environment variables you defined in the source shell, so that if you need to run your program "myprogram" in a controlled environment (i.e. with specific library paths or options), you can prepare the environment in the login shell and be sure to find it again in the interactive shell o the compute node.

On systems using SLURM, you can submit a script script.x using the command:

> sbatch script.x

You can get a list of defined partitions with the command:

> sinfo

For more information and examples of job scripts, see section Batch Scheduler SLURM.

Submitting serial Batch Jobs

The  partition will be  available in the full production.

Graphic session

If a graphic session is desired we recommend to use the tool "RCM". Please install the latest version of RCM. See the corresponding paragraph to know more about how to download and use RCM.

Submitting parallel Batch Jobs

To run parallel batch jobs on GALILEO100 you need to specify the partition  and the qos that are described in this user guide.

If you do not specify the partition, your jobs will try to run on the default partition  g100_usr_prod.

The minimum number of cores you can request for a batch job is 1. The maximum number of cores that you can request is  (16 nodes). It is also possible to request a maximum walltime of 24 hours. Defaults are as follows:

• If you do not specify the walltime (by means of the #SBATCH --time directive), a default value of 30 minutes will be assumed.

• If you do not specify the number of cores (by means of the "SBATCH -n" directive) a default value of 1 core will be assumed.

• If you do not specify the amount of memory (as the value of the "SBATCH --mem" DIRECTIVE), a default value of 7800 MBper core will be assumed.

The maximum memory per node is 375300MB (366.5GB) for thin and viz nodes, about 3TB for fat nodes.

Processor affinity

Processor affinity, or CPU pinning, enables the binding of processes and threads to a CPU (or group of CPUs). It is crucial to ensure the correct affinity so to avoid the CPUs overallocation, with a significant reduction of performances. It becomes a critical matter when you ask for a full node but, for your specific reasons (memory needs etc.) you don't use all the cores.  

The following indications work when running your executables with srun, which is the recommended option against mpirun. We refer to a hybrid MPI/OpenMP case compiled with the Intel oneAPI suite.

Given your optimal value of OMP_NUM_THREADS and number of processes, to obtain the full node ask for a number of task such that  ( --ntasks-per-node * --cpus-per-task )= 48.

• To avoid the processes over allocation of cores rely on the --cpu-bind=cores option of srun  (you can skip it if you use all the requested cores)
• To enforce the threads affinity use the Intel parameter KMP_AFFINITY, or the OpenMP parameter OMP_PLACES
• To distribute the MPI tasks consecutively inside the sockets, use the -m block:block option of srun (or the equivalent sbatch directive #SBATCH -m block:block)

#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=12
#SBATCH --cpus-per-task=4
#SBATCH --account=<your_account>

module load autoload intelmpi/oneapi-2021–binary

export OMP_NUM_THREADS=4

export KMP_AFFINITY=compact    # or OMP_PLACES=cores

srun --cpu-bind=cores -m block:block <your_exe>

Use of GPUs on Galileo100

to be soon defined

Users with reserved resources

Users of projects that require reserved resources (such as industrial users or users associated to an agreement that involves dedicated resources) will be associated to a QOS qos_ind.

Using the  qos_ind (i.e. specifying the QOS in the submission script) , and specifying the partition g100_spc_prod, users associated to the allowed project will run their jobs on reserved nodes in the g100_spc_prod partition with the features and limits imposed for the particular account.

>#SBATCH --partition=g100_spc_prod

>#SBATCH --qos=qos_ind

Summary

In the following table, you can find all the main features and limits imposed on the SLURM partitions and QOS.

Intel Compilers

The native, and recommended, compilers on GALILEO100 are the Intel ones, since the architecture is based on Intel processors and therefore using the Intel compilers may result in a significant improvement in performance and stability of your code. On the cluster is installed the new suite Intel OneAPI. Initialize the environment with the module command:

> module load intel/oneapi-2021–binary

> module list
   Currently Loaded Modulefiles:
   intel/oneapi-2021–binary

The suite contains the new Intel oneAPI nextgen compilers (icx, icpx, ifx) and the classic compilers (icc, icpc, ifort):

NOTE:

• ICX is a new compiler. It has functional and behavioral differences compared to ICC. You can expect some porting will be needed for existing applications using ICC. According to Intel, the transition from ICC Classic to ICX is smooth and effortless. However, you must port and tune any existing applications from ICC Classic to ICX. Please refer to the official Intel Porting Guide for ICC Users to DPCPP or ICX
• IFORT is a completely new compiler. According to Intel, although considerable effort is being made to make the transition from ifort to ifx as smooth and as effortless as possible, customers can expect that some effort may be required to tune their application. IFORT will remain Intel’s recommended production compiler until ifx has performance and features superior to ifort. Please refer to the official Intel Porting Guide for ifort Users to ifx

The documentation can be obtained with the man command after loading the relevant module:

> man ifort
> man icx

Some miscellaneous flags are described in the following:

-extend_source    Extend over the 77 column F77's limit 
-free / -fixed    Free/Fixed form for Fortran
-ip               Enables additional interprocedural optimization for single-file compilation
-ipo              Enables interprocedural optimization between files - whole program optimisation

NOTE for the migration from Galileo to Galileo100: In principle, binaries generated on Galileo should work, but we strongly recommend you to reinstall all your software applications since on Galileo100 there is a different Operating System (Centos 8.3).

GNU compilers

The gnu compilers are always available but they are not the best optimizing compilers, especially for an IntelOneAPI-based cluster like GALILEO100.

For a more recent version of the compiler, initialize the environment with the module command:

> module load gnu

The name of the GNU compilers are:

• g77: Fortran77 compiler
• gfortran: Fortran compiler with "gnu" standard
• gcc: C compiler
• g++: C++ compiler

The "gnu" standard is the default value for the -std option. It specifies a superset of the latest Fortran standard that includes all of the extensions supported by GNU Fortran, although warnings will be given for obsolete extensions not recommended for use in new code. To change the standard to which the program is expected to conform, set the -std option to one of the possible values (f95, f2003, f2008, f2018, gnu, or legacy).

The documentation can be obtained with the man command:

> man gfortan
> man gcc

Some miscellaneous flags are described in the following:

-ffixed-line-length-132       To extend over the 77 column F77's limit
-ffree-form / -ffixed-form    Free/Fixed form for Fortran