...
Please note that we are still optimizing Leonardo software stack, and more installations may be added/replaced. Always check with "module av" (the hash in the module name can change).
Beta-production environment
The production environment is based on the slurm scheduler, already in place on the cluster but in a very preliminary configuration.
- The only available partition is "prod" (#SBATCH --partition=prod). Please refer to the general online guide to slurm and on task/thread bindings, and please pay attention to the setting of the SRUN_CPUS_PER_TASK for hybrid applications dispatched with "srun". In this preliminary configuration, please explicit the request of the correct pmix plugin when launching your parallel applications with "srun": srun --mpi=pmix_v3 <options> <exe>No mpii <exe>. No mpi settings are needed if you launch with "mpirun".
- The GPUs are not yet defined as G(eneral)res(ources) (Gres), and all the 4 GPUs of a node will be available in a job. Do not ask for gres=gpu:X (or analogous --gpus-per-node) in your script. Take the node in exclusive with the #SBATCH --exclusive directive
- The $SBATCH --exclusive directive is also recommended to avoid annoying drawbacks on the $TMPDIR of job
Pre-production environment
Storage:
- the scratch areas are now available ($CINECA_SCRATCH or $SCRATCH)
- home filesystem:
BETA-0 users: your old home is still available at the path /leonardo/smcx_home/home/<userinternal>. A quota of 100 GB has been enforced, and most likely you will not be able to add more data. We already started the sync of the /home/<username> contents to the corresponding user scratch in the <username> sub-directory. We'll inform you when the copy is finished (a stop will be required for the last rsync). After a check from your side that everything was successfully copied, we'll proceed with the data removal in /xxxx/home. You can however reprise your activity
- BETA-1 users: you started already with the home at the correct path, /leonardo/home/userexternal/<username>. The 100 GB quota is enforced as well. Please copy the contents not supposed to be in your $HOME to your $SCRATCH (we will NOT move the contents of the new homes), and remove from $HOME the transferred data. You can reprise your activity now on your scratch area, just copy there the needed input files and scripts.
- work filesystem: the $WORK areas are not available yet. Until they will be configured and put in place the automatic cleaning of the scratch area will NOT be active.
Slurm:
- use the "boost_usr_prod" partition in the place of "prod". You can use at most 32 nodes on this partition (MaxTime=24:00:00). Please request the boost_qos_bprod QOS to go up to 512 nodes (MaxTime=10:00:00) This limit will be in place until May 25, when it will be reduced to 256 nodes with MaxTime=24:00:00 (production environment) before May 25.
- you have to request the gpus with --gres=gpu:X or --gpus-per-node=X
- the --mpi=pmix_v3 is not required anymore when launching with srun