Page History

System Architecture

Compute Nodes: There are currently 1022 36-core compute nodes. Each one contains 2 18-cores Intel Xeon E5-2697 v4 (Broadwell) at 2.30 GHz. All the compute nodes have 128 GB of memory. Of these compute nodes, 60 are equipped with nVidia K80 GPU and two with nVidia V100.

Login and Service nodes: There are 8 Login & Viz nodes available (5 are academic nodes and 3 are for industrial users). There are 8 service nodes for I/O and cluster management.

All the nodes are interconnected through an Infiniband network, with OPA v10.76, capable of a maximum bandwidth of 100Gbit/s between each pair of nodes.

Accounting

For more informations about accounting, please consult our dedicated section.

Budget Linearization policy

On GALILEO a linearization policy for the usage of project budgets has been defined and implemented. For each account, a monthly quota is defined as:

monthTotal = (total_budget / total_no_of_months)

Starting from the first day of each month, the collaborators of any account are allowed to use the quota at full priority. As long as the budget is consumed, the jobs submitted from the account will gradually lose priority, until the monthly budget (monthTotal) is fully consumed. At that moment, their jobs will still be considered for execution, but with a lower priority than the jobs from accounts that still have some monthly quota left.

This policy is similar to those already applied by other important HPC centers in Europe and worldwide. The goal is to improve the response time, giving users the opportunity of using the cpu hours assigned to their project in relation of their actual size (total amount of core-hours).

Production environment

Since GALILEO is a general purpose system and it is used by several users at the same time, long production jobs must be submitted using a queuing system. This guarantees that the access to the resources is as fair as possible.

Roughly speaking, there are two different modes to use an HPC system: Interactive and Batch. For a general discussion see the section "Production Environment and Tools".

Interactive

A serial program can be executed in the standard UNIX way:

> ./program

This is allowed only for very short runs, since the interactive environment set on the login nodes has a 10 minutes time limit: for longer runs please use the "batch" mode.

A parallel program can be executed interactively only by submitting an "Interactive" SLURM batch job, using the "srun" command: the job is queued and scheduled as any other job, but when executed, the standard input, output, and error streams are connected to the terminal session from which srun was launched.

For example, to start an interactive session with the MPI program "myprogram", using one node and two processors, you can launch the command:

> srun -N 1 --ntasks-per-node=2 -A <account_name> --pty /bin/bash

SLURM will then schedule your job to start, and your shell will be unresponsive until free resources are allocated for you. If not specified, the default time limit for this kind of jobs is one hour.

When the shell returns a prompt inside the compute node, you can execute your program by typing:

> srun ./myprogram

or

> mpirun ./myprogram

SLURM automatically exports the environment variables you defined in the source shell, so that if you need to run your program "myprogram" in a controlled environment (i.e. with specific library paths or options), you can prepare the environment in the login shell and be sure to find it again in the interactive shell o the compute node.

Batch

As usual, on systems using SLURM, you can submit a script script.x using the command:

> sbatch script.x

You can get a list of defined partitions with the command:

> sinfo

For more information and examples of job scripts, see section Batch Scheduler SLURM.

Submitting serial Batch jobs

The gll_all_serial partition is available with one core and a maximum walltime of 4 hours. It runs on two nodes and it is designed for pre/post-processing serial analysis, and for moving your data (via rsync, scp etc.) in case more than 10 minutes are required to complete the data transfer. In order to use this partition you have to specify the SLURM flag "-p":

#SBATCH -p gll_all_serial

The gll_all_serial partition has a limit of 1 task per job and 4GB of memory per job. If you wish to ask for more than a core on a single job, remember to add on your jobscript the specific about the memory limit, since the default per core is 3.5GB and therefore your job won't enter because the required memory exceeds the partition limit.

IMPORTANT FOR USERS OF THE PRE-UPGRADED VERSION OF GALILEO:  the old filesystems (such as $CINECA_SCRATCH) haven't been migrated to the upgraded cluster, but will remain available for a certain period of time and visible only on specific nodes. You can access such nodes for transfering your data from the old environment to the new, by using the partition:

#SBATCH -p gll_all_transfer 

with the limitations of one core and 24 hours per job. Inside the gll_all_transfer nodes, you will find your old scratch area at the mount point /gpfs/scratch_old .

Graphic session

If a graphic session is desired we recommend to use the tool "RCM". See the corresponding session to know more about how to download and use RCM.

A complete reconfiguration of the RCM environment is in progress. This guide will be completed as soon as a final configuration will be implmented.

Submitting parallel Batch jobs

To run parallel batch jobs on GALILEO you need to specify the partition gll_usr_prod, or any other partition described in this userguide.

Users who need to run on GPU-equipped nodes need to specify the partition gll_usr_gpuprod.

If you do not specify the partition, your jobs will try to run on the default partition bdw_all_serial, meant for serial jobs, eventually failing if specific partition limits (maximum four tasks per job and maximum walltime of 4 hours) are violated.

The minimum number of cores you can request for a batch job is 1. The maximum number of cores that you can request is 2304 (64 nodes). It is also possible to request a maximum walltime of 24 hours. Defaults are as follows:

• If you do not specify the walltime (by means of the #SBATCH --time directive), a default value of 30 minutes will be assumed.

• If you do not specify the number of cores (by means of the "SBATCH -n" directive) a default value of 36 will be assumed.

• If you do not specify the amount of memory (as the value of the "SBATCH --mem" DIRECTIVE), a default value of 3000MB will be assumed.

• The maximum memory per node is 118000MB

A special QOS (qos_special) is also available for not-ordinary types of jobs, such as a walltime larger than 24 hours. Since it violates our standard policy, there are restrictions in its usage, and users who want to use it need to be enabled by the User Support staff. Please write to superc@cineca.it in case you think you need to use it. You request will be evaluated and, if approved, you will be allowed to use the special QOS for a limited period of time.

Use of GPUs on GALILEO

The gll_usr_gpuprod partition is defined on 42 Broadwell nodes (18-cores Intel Xeon E5-2697 v4 @ 2.40GHz), each equipped with + 1 nVidia K80 GPUs (seen as two K40 gpus). All users using an account with available budget can submit jobs on this partition and use GPU nodes on GALILEO.  

The maximum number of nodes that can be required on gll_usr_gpuprod is 2 4 for a maximum walltime of 08:00:00 hours. The maximum memory is 117000 118000 MB. 

In regards of writing a SLURM jobscript, you need to request the GPU as "gres":

#SBATCH --partition=gll_usr_gpuprod

#SBATCH --gres=gpu:kepler:N     (N=1,2)

GALILEO is also equipped with two nodes with one nVIDIA Volta (V100) GPUs each, accessible for tests for a limited period of time. Please write to superc@cineca.it if you are interested to test the Volta GPUs, with a brief motivation for your request. Once your request is approved and you are enabled to use these resources (via the association to a special QOS "gll_qos_gpudev"), you can submit jobs to the Volta node with the following options:

#SBATCH --partition=gll_usr_gpudev

#SBATCH --qos=gll_qos_gpudev

#SBATCH --gres=gpu:volta:1

Users with reserved resources

Users of projects that require reserved resources (such as industrial users or users associated to an agreement that involves dedicated resources) will be associated to a QOS.

Using the specific QOS (i.e. specifying the QOS in the submission script) , and specifying the partition gll_spc_prod, users associated to the allowed project will run their jobs on reserved nodes in the gll_spc_prod partition:

>#SBATCH --partition=gll_spc_prod

>#SBATCH --qos=<specific_qos>

Summary

In the following table, you can find all the main features and limits imposed on the SLURM partitions and QOS.

PLEASE NOTE: the SLURM characteristics have not been completely defined for GALILEO, some changes will be possible. In such a case, the above table and the rest of the documentation will be updated accordingly.