...
- on Marconi-A3 (SkyLake) SKL, consisting in 2912 dedicated nodes;
- on Marconi100 with 80 dedicated nodes starting from April 2020 increased to 99 from April 2021.on Leonardo (currently) with 99 dedicated 72 nodes starting from May August 2023.
Previous dedicated partitions:
- From Jul 2016 up to Aug 2017 EUROfusion community had 806 dedicated nodes of the Marconi-A1 partition (from Jul 2016 up to Aug 2017). Starting from October 15th, 2017 the Eurofusion EUROfusion activity on Marconi-A1 has been transferred to Marconi-A3.
- From beginning of 2017 up to the end of 2019 the community could use 449 dedicated nodes of Marconi-A2 (Knights landing) KNL partition. From January 2020 up to May 2020 the number of dedicated nodes has been 288. Starting from May 2020 the activity on Marconi-A2 has been stopped.
- From January 2019 up to February 2020 the community had a dedicated partition on D.A.V.I.D.E. consisting in 40 nodes. On February 2020 the activity on D.A.V.I.D.E. has been stopped.
- From April 2020 the community had 80 dedicated nodes on Marconi100, increased to 99 from April 2021. The Marconi100 production has been stopped on 24 of July of 2023.
The general environment defined on our clusters for the EUROfusion cmmunity is the same as the one defined for all the users of the cluster. The general environment refers to:
...
In the following you find an extraction of the basic knowledge needed to properly take advantage of our clusters
Data storage and Filesystems
The storage organization conforms to the CINECA infrastructure (see Section Data Storage and Filesystems). In addition to the home directory $HOME, for each user is defined a scratch area $CINECA_SCRATCH, a large disk for the storage of run time data and files. A $WORK area is defined for each active project on the system, reserved for all the collaborators of the project. This is a safe storage area to keep run time data for the whole life of the project.
Storage Production Areas: $WORK and $CINECA_SCRATCH
These two areas share the same physical device, have the same block size and they also have the same performance in terms of data throughput. $WORK and $CINECA_SCRATCH are conceived as working directories for large files used and produced by batch jobs. Also, the blocking features make these areas more suitable for large binary files.
* $WORK: permanent, project specific, local *
There is one $WORK area for each active project on the machine that all users belonging to can use for production runs and storage of their output data. The owner of the main directory is the PI (Principal Investigator), but all collaborators are allowed to read/write in there. Collaborators are advised to create a personal directory in $WORK for storing their personal files. By default the personal directory will be protected (only the owner can read/write), but protection can be easily modified, for example by allowing write permission to project collaborators through "chmod" command. The default quota for a project $WORK area is 1TB, but it is possible to consider a quota extension if needed (please mailto: superc@cineca.it). File retention in the $WORK area is related to the life of the project. Files in this area will be conserved up to 6 months after the project expiring date, and then they will be cancelled. Please note that there is no back-up on this area.
...
To check for the occupancy of this area please use the "cindata" command, that will list all filesets containing any file owned by your username.
* $CINECA_SCRATCH: temporary, user specific, local *
The main difference of this area with respect to $WORK is that it is user specific (not project specific) and that it can be used for sharing data with people outside your project. There is one $CINECA_SCRATCH area for each username on the machine. By default, file access is open to everyone, in case you need more restrictive protections, you can set them with "chmod" command. On this area a periodic cleaning procedure is applied, with a normal retention time of 40 days: files are cancelled on a daily basis by an automatic procedure if not accessed for more than 40 days. Please take in mind that this time interval of 40 days may be reduced in case of critical usage ratio of the area. In this case, users will be notified via HPC-News. When files are deleted, a file listing al deleted files for a given day will be created: CLEAN_<yyyymmdd>.log, where <yyyymmdd> = date when files were cancelled.
$CINECA_SCRATCH does not have any disk quota. However, it is strongly recommended to maintain a low occupancy of this area in order to prevent very dangerous filling condition. Please, be aware that on Galileo and Marconi clusters, in order to prevent a very dangerous filling condition, a 20TB disk quota will be temporarily imposed to all users when the global quota area will reach the 88% of occupancy; this disk quota will be removed when the global occupancy lowers back to normal. To check for the occupancy of this area please use the "cindata" command, that will list all filesets containing any file owned by your username.
* $TMPDIR: temporary, user specific, local *
Each compute node is equipped with a local storage which dimension differs depending on the cluster (please look at the specific page of the cluster for more details).
When a job starts, a temporary area is defined on the storage local to each compute node:
...
On Marconi100 the $TMPDIR area has 1 TB of available space, while on Marconi the available space is about 49 GB.
Repository area for collaborative work among different projects and across platforms: $DRES
This is a data archive area available only on-request (please mailto: superc@cineca.it), shared with all CINECA HPC systems and among different projects. $DRES is not mounted on the compute nodes. This means that you cannot access it within a batch job: all data needed during the batch execution has to be moved to $WORK or $CINECA_SCRATCH before the run starts.
MARCONI-A3
This partition, made of 2410 nodes (SkyLake, 48 cores, 192000MB) is in production since August 2017 (initially with 1512 nodes, enlarged to 2410 nodes since November 2018) and is reserved to the EUROfusion community.
...
For more information and examples of job scripts, see section Batch Scheduler SLURM.
Euterpe
It has been noticed a problem with EUTERPE. The job may go in hang using the command mpiexec. The mpiexec command launches only the execution of the application on the selected cores without initialising them, therefore in some cases, the run could crash without exiting.
In order to avoid this issue, we strongly suggest users to use SRUN when launching jobs of EUTERPE.
MARCONI100
On Marconi100, EUROfusion community has access to 80 dedicated nodes each one containing 4 NVIDIA Volta V100 GPU.
Similarly to Marconi production jobs must be submitted using a queuing system. Batch jobs are managed by the SLURM batch scheduler, described in section Batch Scheduler SLURM.
In the following table you can find all the main features and limits (e.g. maximum number of cores per job or maximum walltime) imposed on the SLURM partitions available to EUROfusion community.
For up-to-date information, use the "sinfo" and "scontrol show partition <partition_name>" commands on the system itself.
LEONARDO
A presentation of Leonardo dedicated to the EUROfusion community was held on June 6th, 2023, and the slides and recording are available here (you should log in through the button Log in as a guest).
The mandatory access to Leonardo is the two-factor authentication (2FA). Please refer to this link of the User Guide to activate and connect via 2FA and to the slides of the 2FA presentation (07/06/2023) dedicated to the EUROfusion community.
All the login nodes have an identical environment and can be reached with SSH (Secure Shell) protocol using the "collective" hostname:
> login.leonardo.cineca.it
On Leonardo, EUROfusion community has (currently) access to 72 nodes, each one containing 4 NVIDIA A100 GPUs.
Similarly to other systems jobs must be submitted using a queuing system. Batch jobs are managed by the SLURM batch scheduler, described in section Batch Scheduler SLURM.
In the following table you can find all the main features and limits (e.g. maximum number of cores per job or maximum walltime) imposed on the SLURM partitions available to EUROfusion community.
For up-to-date information, use the "sinfo" and "scontrol show partition <partition_name>" commands on the system itself.
It is not possible to occupy more than 32 nodes for a single user on the partition boost_fua_prod.
SLURM partition | Job QOS | # cores/# GPU per job | max walltime | max running jobs per user/ max n. of cores/nodes/GPUs per Grp | priority | notes |
boost |
SLURM
partition
per job
max running jobs per user/
max n. of cpus/nodes/GPUs per user
m100_all_serial
(def. partition)
max = 1 core, 1 GPU
max mem= 7600MB
4 cpus/1 GPU
m100_fua_prod
runs on 12 nodes
_fua_prod | normal | max = 16 nodes | 24:00:00 | 40 |
runs on 68 nodes
| boost_qos_fuabprod | min = 17 nodes max =32 nodes | 24:00:00 |
>24:00:00
| 49 nodes / 1568 cores / 196 GPUs | 60 | runs on 49 nodes, min is 17 FULL nodes | ||||
boost_fua_dbg | normal | max = 2 nodes | 00:10:00 | 2 nodes / 64 cores / 8 GPUs | 40 | runs on 2 nodes |
| qos_fualowprio | max = 16 nodes | 08:00:00 | 0 |
|
|
For information on how to submit and manage jobs for M100 partition, please refer to the Marconi100 UserGuide.
You will find also important details on compilers, profilers and how to fully exploit the GPU compute capacity.
As usual on systems with the SLURM scheduler, you submit a batch job script with the command:
> sbatch <options> scriptFor more information and examples of job scripts, see section Batch Scheduler SLURM and other specific examples for Marconi100.
LEONARDO
The mandatory access to Leonardo is the two-factor authentication (2FA). Please refer to this link of the User Guide to activate and connect via 2FA and to the slides of the 2FA presentation (07/06/2023) dedicated to the Eurofusion community.
All the login nodes have an identical environment and can be reached with SSH (Secure Shell) protocol using the "collective" hostname:
...
On Leonardo, EUROfusion community has (currently) access to 99 dedicated nodes each one containing 4 NVIDIA A100 GPU.
Similarly to other systems jobs must be submitted using a queuing system. Batch jobs are managed by the SLURM batch scheduler, described in section Batch Scheduler SLURM.
In the following table you can find all the main features and limits (e.g. maximum number of cores per job or maximum walltime) imposed on the SLURM partitions available to EUROfusion community.
For up-to-date information, use the "sinfo" and "scontrol show partition <partition_name>" commands on the system itself.
...
SLURM
partition
...
max running jobs per user/
max n. of cores/nodes/GPUs per Grp
...
lrd_all_serial
(default)
not yet available
...
min = 17 nodes
max =32 nodes
...
|
Low-priority jobs
1) If you consume all the budget assigned to your projects, you can keep running on Leonardo boost_fua_prod partition at low priority by requesting in your submission script the qos_fualowprio QOS:
#SBATCH --qos=qos_fualowprio
The QOS is automatically added to your account upon budget exhaustion.
2) You can also request to run low priority jobs, without consuming your active budget, by association to the FUAL7_LOWPRIO account (write a mail to superc@cineca.it). You always need to specify the qos_fualowprio QOS in your submission script as above together with this LOWPRIO account.
2FA Access
The mandatory access to Leonardo and Marconi (starting from June 13 2023) is the two-factor authentication (2FA). Please refer to this link of the User Guide to activate and connect via 2FA and to the slides of the 2FA presentation (07/06/2023) dedicated to the EUROfusion community.
Other remarks
PLEASE NOTE THE FOLLOWING IMPORTANT REMARKS:
...
#SBATCH --account=FUAC6_LOWPRIO
#SBATCH --partition=m100_fua_prod
#SBATCH --qos=qos_fualowprio
8) On Marconi-A3, Eurofusion EUROfusion community users are suggested to use some particular settings for the Intel I_MPI_ADJUST family environment variables. This choice should guarantee optimal performances controlling the corresponding collective algorithm selection. In particular:
...