Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Please note that we are still optimizing Leonardo software stack, and more installations may be added/replaced. Always check with "module av" (the hash in the module name can change).

Beta-production environment

The production environment is based on the slurm scheduler, already in place on the cluster but in a very preliminary configuration.

  • The only available partition is "prod" (#SBATCH --partition=prod).  Please refer to the general online guide to slurm and on task/thread bindings, and please pay attention to the setting of the SRUN_CPUS_PER_TASK for hybrid applications dispatched with "srun". In this preliminary configuration, please explicit the request of the correct pmix plugin when launching your parallel applications with "srun": srun --mpi=pmix_v3 <options>  <exe>No mpii <exe>. No mpi settings are needed if you launch with "mpirun".
  • The GPUs are not yet defined as G(eneral)res(ources) (Gres), and all the 4 GPUs of a node will be available in a job. Do not ask for gres=gpu:X (or analogous --gpus-per-node)  in your script. Take the node in exclusive with the #SBATCH --exclusive directive
  • The $SBATCH --exclusive directive is also recommended to avoid annoying drawbacks on the $TMPDIR of job

Pre-production environment

Storage:

  • the scratch areas are now available ($CINECA_SCRATCH or $SCRATCH)
  • home filesystem:
    • BETA-0 users: your old home is still available at the path /leonardo/smcx_home/home/<userinternal>.  A quota of 100 GB has been enforced, and most likely you will not be able to add more data. We already started the sync of the /home/<username> contents to the corresponding user scratch in the <username> sub-directory. We'll inform you when the copy is finished (a stop will be required for the last rsync). After a check from your side that everything was successfully copied, we'll proceed with the data removal in /xxxx/home. You can however reprise your activity

    • BETA-1 users: you started already with the home at the correct path, /leonardo/home/userexternal/<username>. The 100 GB quota is enforced as well. Please copy the contents not supposed to be in your $HOME to your $SCRATCH (we will NOT move the contents of the new homes), and remove from $HOME the transferred data. You can reprise your activity now on your scratch area, just copy there the needed input files and scripts.
  • work filesystem: the $WORK areas are not available yet. Until they will be configured and put in place the automatic cleaning of the scratch area will NOT be active.

Slurm:

  • use the "boost_usr_prod" partition in the place of "prod". You can use at most 32 nodes on this partition (MaxTime=24:00:00). Please request the boost_qos_bprod QOS to go up to 512 nodes (MaxTime=10:00:00) This limit will be in place until May 25, when it will be reduced to 256 nodes with MaxTime=24:00:00 (production environment) before May 25.
  • you have to request the gpus with --gres=gpu:X or --gpus-per-node=X
  • the --mpi=pmix_v3 is not required anymore when launching with srun