UG3.2: LEONARDO UserGuide

Sections Production environment, Programming environment are specific for the two partitions:

• LEONARDO Booster UserGuide
• LEONARDO Data Centric UserGuide

hostname: login.leonardo.cineca.it

                       login01-ext.leonardo.cineca.it

                       login02-ext.leonardo.cineca.it

                       login05-ext.leonardo.cineca.it

                       login07-ext.leonardo.cineca.it

early availability:      March, 2023

start of pre-production: June, 2023 (Booster) 

                                  January 2024 (Data Centric)

start of production: August 2023 (Booster)

This HPC system is the new pre-exascale Tier-0 EuroHPC Joint Undertaking supercomputer hosted by CINECA and currently built in the Bologna Technopole, Italy. It is supplied by EVIDEN ATOS, and it is based on two new specifically-designed compute blades, which are available through two distinct SLURM partitions on the cluster:

• X2135 GPU blade based on NVIDIA Ampere A100-64 accelerators - LEONARDO Booster partition
• X2140 CPU-only blade based on Intel Sapphire Rapids processors - LEONARDO Data Centric partition

The overall system architecture also uses NVIDIA Mellanox InfiniBand High Data Rate (HDR) connectivity, with smart in-network computing acceleration engines that enable extremely low latency and high data throughput to provide the highest AI and HPC application performance and scalability.

The system also includes a Capacity Tier and a Fast Tier storage, based on DDN Exascaler.

The Operating System is RedHat Enterprise Linux 8.6.

System Architecture

Architecture: Atos BullSequana X2135 "Da Vinci" blade - Booster - Atos BullSequana X2140 compute blade - Data-centric (will be available in the first quarter of the 2024)
Internal Network: Nvidia Mellanox HDR DragonFly+ 200 Gb/s
Storage: 106 PB (raw) Large capacity storage, 620 GB/s
                   High Performance Storage 5.4 PB, 1.4 TB/s Based on 31 x DDN Exascaler ES400NVX2

Login nodes: 4 nodes, icelake no-gpu

Peak performance details

Access

All the login nodes have an identical environment and can be reached with SSH (Secure Shell) protocol using the "collective" hostname:

$ login.leonardo.cineca.it

which establishes a connection to one of the available login nodes.  To connect to Leonardo you can also indicate explicitly  the login nodes:

$ login01-ext.leonardo.cineca.it
$ login02-ext.leonardo.cineca.it 
$ login05-ext.leonardo.cineca.it
$ login07-ext.leonardo.cineca.it

The mandatory access to Leonardo is the two-factor authentication (2FA). Please refer to this link of the User Guide to activate and connect via 2FA. For information about data transfer from other computers please follow the instructions and caveats on the dedicated section Data storage or the document  Data Management.

Accounting

The accounting (consumed budget) is active from the start of the production phase. For accounting information please consult our dedicated section.

The account_name (or project) is important for batch executions. You need to indicate an account_no to be accounted for in the scheduler, using the flag "-A"

#SBATCH -A <account_name>

With the "saldo -b" command you can list all the account_name associated with your username. 

> saldo -b   (reports projects defined on LEONARDO )

Please note that the accounting is in terms of consumed core hours, but it strongly depends also on the requested memory and number of GPUs, please refer to the dedicated section.

Budget Linearization policy

On LEONARDO, as on the other HPC clusters in Cineca, a linearization policy for the usage of project budgets has been defined and implemented. The goal is to improve the response time, giving users the opportunity of using the cpu hours assigned to their project in relation to their actual size (total amount of core-hours).

Disks and Filesystems

The storage organization conforms to the CINECA infrastructure (see Section Data Storage and Filesystems). 

In addition to the home directory $HOME, for each user is defined a scratch area $CINECA_SCRATCH, a large disk for the storage of run time data and files.

A $WORK area is defined for each active project on the system, reserved for all the collaborators of the project.

• The automatic cleaning of the scratch area is NOT active yet, but it will soon be enforced.

It is also available a temporary storage local to nodes on login and compute nodes (on the latter it is generated when the job starts) and accessible via environment variable $TMPDIR. For more details please see the dedicated section of UG2.5: Data storage and FileSystems.

Since all the filesystems are based on Lustre, the usual unix command "quota" is not working. Use the local command cindata to query for disk usage and quota ("cindata -h" for help) that will be available soon.

$ cindata

Software environment

Module environment

The software modules are collected in different profiles and organized by functional categories (compilers, libraries, tools, applications,...). The profiles are of two types: “programming” type (base and advanced) for compilation, debugging and profiling activities, and  “domain” type (chem-phys, lifesc,..) for the production activity. They can be loaded together.

"Base" profile is the default. It is automatically loaded after login and it contains basic modules for the programming activities (ibm, gnu, pgi, cuda compilers, math libraries, profiling and debugging tools,..).

If you want to use a module placed under other profiles, for example an application module, you will have to previously load the corresponding profile:

$ module load profile/<profile name>
$ module load <module name>

Almost all the softwares on Leonardo were installed with Spack manager, which loads automatically the possible dependencies, so autoload command is unnecessary.

For listing all profiles you have loaded you can use the following command:

$ module list

In order to detect all profiles, categories and modules available on LEONARDO, the command “modmap” is available as for the other clusters. With modmap you can see if the desired module is available and which profile you have to load to use it.

$ modmap -m <module_name> 

 Spack environment

In case you don't find a software you are interested in, you can install it by yourself. 
In this case, on Leonardo  we also offer the possibility to use the “spack” environment by loading the corresponding module. Please refer to the dedicated section in UG2.6: Production Environment

Please note that we are still optimizing Leonardo software stack, and more installations may be added/replaced. Always check with "module av" (the hash in the module name can change).

Graphic session

It will be available soon. 

You can proceed with the sections related to Software environment, Production environment and Programming environment in the specific pages for the two partitions:

• LEONARDO Booster UserGuide
• LEONARDO Data Centric UserGuide