...
Starting from January 2018 GALILEO has been reconfigured with Intel Xeon E5-2697 v4 (Broadwell) nodes, inherited from MARCONI system.
Starting from March 2018 GALILEO is again in production, available for italian research communityAugust 2019 the new reconfiguration phase has begun leading to the new, upgraded environment, equipped with Intel Omni-Path internal network with 1024 Intel Xeon E5-2697 v4 (Broadwell) nodes (at the end of August).
System Architecture
Compute Nodes: There are 360 352 36-core compute nodes. Each one contains 2 18-cores Intel Xeon E5-2697 v4 (Broadwell) at 2.30 GHz. All the compute nodes have 128 GB of memory. 15 36 of these compute nodes are equipped with 2 with nVidia K80 GPU and two with nVidia V100.
Login and Service nodes: 8 Login & Viz node are available (5 academic and 3 industrials), equipped with 2 nVidia K40 GPU each. 8 service nodes for I/O and cluster management.
All the nodes are interconnected through a Infiniband network, with OFED v3OPA v1.185-34, capable of a maximum bandwidth of 40Gbit100Gbit/s between each pair of nodes.
Accounting
For accounting information please consult our dedicated section.
Budget Linearization policy
On GALILEO a linearization policy for the usage of project budgets has been defined and implemented. For each account, a monthly quota is defined as:
monthTotal = (total_budget / total_no_of_months)
Starting from the first day of each month, the collaborators of any account are allowed to use the quota at full priority. As long as the budget is consumed, the jobs submitted from the account will gradually lose priority, until the monthly budget (monthTotal) is fully consumed. At that moment, their jobs will still be considered for execution, but with a lower priority than the jobs from accounts that still have some monthly quota left.
This policy is similar to those already applied by other important HPC centers in Europe and worldwide. The goal is to improve the response time, giving users the opportunity of using the cpu hours assigned to their project in relation of their actual size (total amount of core-hours).
...
Production environment
Since GALILEO is a general purpose system and it is used by several users at the same time, long production jobs must be submitted using a queuing system. This guarantees that the access to the resources is as fair as possible.
Roughly speaking, there are two different modes to use an HPC system: Interactive and Batch. For a general discussion see the section "Production Environment and Tools".
Interactive
A serial program can be executed in the standard UNIX way:
> ./program
This is allowed only for very short runs, since the interactive environment has a 10 minutes time limit: for longer runs please use the "batch" mode.
A parallel program can be executed interactively only within an "Interactive" SLURM batch job, using the "srun" command: the job is queued and scheduled as any other job, but when executed, the standard input, output, and error streams are connected to the terminal session from which srun was launched.
For example, to start an interactive session with the MPI program myprogram, using one node, two processors, launch the command:
> srun -N1 -n2 --ntasks-per-node=2 -A <account_name> --pty /bin/bash
SLURM will then schedule your job to start, and your shell will be unresponsive until free resources are allocated for you.
When the shell come back with the prompt, you can execute your program by typing:
> mpirun ./myprogram
or
> srun ./myprogram
The default SLURM MPI type has been set equal to PMI2.
SLURM automatically exports the environment variables you defined in the source shell, so that if you need to run your program myprogram in a controlled environment (i.e. specific library paths or options), you can prepare the environment in the origin shell being sure to find it in the interactive shell.
Batch
As usual on systems using SLURM, you can submit a script script.x using the command:
> sbatch script.x
You can get a list of defined partitions with the command:
> sinfo -a
For more information and examples of job scripts, see section Batch Scheduler SLURM.
Submitting serial Batch jobs
The gll_all_serial partition is available with one core and a maximum walltime of 4 hours. It runs on the login nodes and it is designed for pre/post-processing serial analysis, and for moving your data (via rsync, scp etc.) in case more than 10 minutes are required to complete the data transfer. In order to use this partition you have to specify the SLURM flag "-P":
#SBATCH -p gll_all_serial
The gll_all_serial partition has a limit of 4 tasks per job and 4GB of memory per job. If you wish to ask for more than a core on a single job, remember to add on your jobscript the specific about the memory limit, since the default per core is 3.5GB and therefore your job won't enter because the required memory exceeds the partition limit.
Graphic session
If a graphic session is desired we recommend to use the RCM tool or the EnginFrame environment. See the corresponding session to know more how to use RCM or EnginFrame.
During the first few weeks on the new system the RCM service will not be available.
Submitting parallel Batch jobs
To run parallel batch jobs on GALILEO you need to specify the partition gll_usr_prod, or any other partition we invited you to use.
Users who need to run on GPU-equipped nodes need to specify che partition gll_usr_gpuprod.
If you do not specify the partition, your jobs will try to run on the bdw_all_serial partition, eventually failing if specific partition limits (maximum one core for maximum walltime of 4 hours) are violated.
The minimum number of cores to require is 1. The maximum number of cores that you can request is the 2304 (64 nodes) with a maximum walltime of 24 hours:
- If you do not specify the walltime (by means of the #SBATCH --time directive), a default value of 30 minutes will be assumed.
- If you do not specify the number of cores (by means of the "SBATCH -n" directive) a default value of 36 will be assumed.
- If you do not specify the amount of memory (as the value of the "SBATCH --mem" DIRECTIVE), a default value of 3000MB will be assumed.
- The maximum memory per node is 118000MB (117000 MB for gpu nodes)
Example of batch script to submit a batch job
The special QOS (gll_qos_special) is designed for not-ordinary types of jobs, and users need to be enabled in order to use it. Please write to superc@cineca.it in case you think you need to use it.
Summary
In the following table, you can find all the main features and limits imposed on the SLURM partitions and QOS.
SLURM partition | QOS | # cores per job | max walltime | max running jobs per user/ max n. of cpus/nodes per user | max memory per node (MB) | priority | HBM/clustering mode | notes |
gll_all_serial (default partition) | gll_all_serial | 1 | 04:00:00 | Max 12 running jobs Max 4 jobs/user | 3000 | |||
| gll_usr_prod | noQOS | min = 1 max = 2304 | 24:00:00 | 20/2304 | 118000 | |||
| gll_usr_gpuprod | noQOS | min = 1 max = 64 | 08:00:00 | 4 | 117000 | --gres=gpu:kepler:N (N=1,4) | ||
| gll_spc_prod | Every account needs to have a valid QOS to access this partition | Depending on kind of users | 24:00:00 | / | 118000 | Partition dedicated to specific kind of users. | ||
| gll_meteo_prod | Partition reserved to meteo services, NOT opened to production |
PLEASE NOTE: the SLURM characteristics have not completely defined for GALILEO, some changes will be possible.
Use of GPUs on GALILEO
gll_usr_gpuprod partiton is defined on 14 Haswell nodes (2*8-cores Intel Xeon E5-2630 v3 @ 2.40GHz), each equipped with + 2 nVidia K80 GPUs. All users using an account with positive budget can launch jobs on this partition.
The maximum number of nodes that can be require on gll_usr_gpuprod is 4, for a maximum walltime of 08:00:00 hours. The maximum memory is 117000 MB.
You need to request the GPU as gres:
#SBATCH --partition=gll_usr_gpuprod
#SBATCH --gres=gpu:kepler:N (N=1,4)
Galileo is also equipped of a node with two nVIDIA Volta (V100) GPUs, accessible for tests for a limited period of time. Please write to superc@cineca.it if you are interested to test the Volta GPUs, with a brief motivation for your request. Once enabled to use these resources (via the association to the QOS gll_qos_gpudev) you can submit jobs to the node with the following options:
#SBATCH --partition=gll_usr_gpudev
#SBATCH --qos=gll_qos_gpudev
#SBATCH --gres=gpu:volta:N (N=1,2)
Users with reserved resources
Users of projects that require reserved resources will be associated to a QOS.
Using the specific QOS (i.e. specifying the QOS in the submission script) , an specifying the partition gll_spc_prod partition, users associated to the allowed project will run their jobs on reserved nodes in the gll_spc_prod partitioN
>#SBATCH --partition="gll_spc_prod"
>#SBATCH --qos=<specific_qos>
...