Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

System Architecture


Compute Nodes:

  • 636 computing nodes each 2 x CPU Intel CascadeLake 8260,  with 24 cores each, 2.4 GHz, 384GB RAM, subdivided in:
      • 422 standard nodes ("thin nodes") 480 GB SSD
      • 180  data processing nodes ("fat nodes") 2TB SSD, 3TB Intel Optane
      •   34  GPU nodes  (visualization "viz" ) with 2x NVIDIA GPU V100 with 100Gbs Infiniband interconnection and 2TB SSD.
  • 77 computing servers OpenStack for cloud computing (ADA CLOUD), 2x CPU 8260 Intel CascadeLake, 24 cores, 2.4 GHz, 768 GB RAM, with 100Gbs Ethernet interconnection.
  • 20 PB of active storage accessible from both cloud and HPC nodes.
  • 1 PB Ceph storage for Cloud (full NVMe/SSD)
  • 720 TB fast storage (IME DDN solution)

Login and Service nodes:

10 login nodes and 5 service nodes. All the nodes are interconnected through an Infiniband network, with OPA v10.6, capable of a maximum bandwidth of 100Gbit/s between each pair of nodes.


Access


All the login nodes have an identical environment and can be reached with SSH (Secure Shell) protocol using the "collective" hostname:

...

For information about data transfer from other computers please follow the instructions and caveats on  the dedicated section section Data storage or the document  Data Management.

...

Disks and Filesystems

The storage organization conforms to the CINECA infrastructure (see Section "Data storage and Filesystems") . In addition to the home directory ($HOME), for each user is defined a scratch area $CINECA_SCRATCH, a large disk for storing run time data and files. The new variable $SCRATCH is also available, resolving in the same path of $CINECA_SCRATCH. $WORK is defined for each active project on the system, reserved for all the collaborators of the project. This is a safe storage area to keep run time data for the whole life of the project.

The filesystem organization is based on LUSTRE open source parallel file system.


Total Dimension (TB)Quota (GB)Notes
$HOME100T50 GB quota per user
  • permanent/backed up, user specific, local

$SCRATCH (on G100)

1PBno quota
  • temporary, user specific, local
  • automatic cleaning procedure of data older than 50 40 days (time interval can be reduced in case of critical usage ratio of the area. In this case, users will be notified via HPC-News)
$WORK2PB1TB quota per project
  • permanent, project specific, local
  • extensions can be considered if needed (mailto: superc@cineca.it)

It is also available a temporary storage local on compute nodes generated when the job starts and accessible via environment variable $TMPDIR. For more details please see the dedicated section of UG2.5: Data storage and FileSystems. On Galileo100 the $TMPDIR local area has 293 GB of available space.

$DRES points to the shared repository where Data RESources are maintained. This is a data archive area available only on-request, shared with all CINECA HPC systems and among different projects.

$DRES is not mounted on the compute nodes. This means that you can't access it within a batch job: all data needed during the batch execution has to be moved on $WORK or $CINECA_SCRATCH before the run starts.

Use the local command "cindata" to query for disk usage and quota ("cindata -h" for help):

  > cindata

or the tool "cinQuota" available in the module cintools

  > cinQuota

For more details about both these commands, please consult the section dedicated to how to monitor the occupancy

Dedicated node for Data transfer and download

A time limit of 10 cpu-minutes for processes running on login nodes has been set.
For Data transfer or download that may require more time, we set up a dedicated "data" VM accessible with a dedicated alias.
Login via ssh to this VM is not allowed. Environment variables as $HOME or $WORK are not defined, so you always have to make the complete path to the files you need to copy explicit.
For example to copy data to Galileo100 using rsync you can run the following command:

rsync -PravzHS </data_path_from/file> <your_username>@data.g100.cineca.it:<complete_data_path_to>

You can also use the "data" VM onto login nodes to move data from Galileo100 to another location with public IP:

ssh -xt data.g100.cineca.it rsync -PravzHS <complete_data_path_from/file> </data_path_to> 

this command will open a session on the VM that will not be closed until the rsync command is completed.

In similar ways you can use also scp and sftp commands if you prefer them.

The data VM also offers wget and curl commands. If you need to wget a large amount of data for a public site, you can run the following command:

ssh -xt data.g100.cineca.it wget <url/file> -P </data_path_to> 

this command will open a session on the VM that will not be closed until the wget command is completed.

Modules environment

The software modules are collected in different profiles and organized by functional category (compilers, libraries, tools, applications,..).

On GALILEO100 the profiles are of two types, “domain” type (bioinf, chem-phys, lifesc,..) for the production activity and “programming” type (base and advanced) for compilation, debugging and profiling activities and that they can be loaded together.

"Base" profile is the default. It is automatically loaded after login and it contains basic modules for the programming activities (intel e gnu compilers, math libraries, profiling and debugging tools,..).

If you want to use a module placed under other profiles, for example an application module, you will have to load preventively the corresponding profile:

>module load profile/<profile name>
>module load autoload <module name>

For listing all profiles you have loaded you can use the following command:

>module list

In order to detect all profiles, categories and modules available on GALILEO100 the command “modmap” is available:

>modmap

With modmap you can see if the desired module is available and which profile you have to load to use it.

>modmap -m <module name>

Spack

environment - will be available soon

environment 

In case you don't find a software you are interested in, you can install it by yourself. 
In this case, on GALILEO100 we also offer the possibility to use the “spack” environment by loading the corresponding module. Please refer to the dedicated section in UG2.6: Production Environment.


Production environment

Since GALILEO100 is a general purpose system and it is used by several users at the same time, long production jobs must be submitted using a queuing system. This guarantees that access to the resources is as fair as possible.

Roughly speaking, there are two different modes to use an HPC system: Interactive and Batch. For a general discussion see the section "Production Environment".

Interactive

serial program can be executed in the standard UNIX way:

> ./program

This is allowed only for very short runs, since the interactive environment set on the login nodes has a 10 minutes time limit: for longer runs please use the "batch" mode.

A parallel program can be executed interactively only by submitting an "Interactive" SLURM batch job, using the "srun" command: the job is queued and scheduled as any other job, but when executed, the standard input, output, and error streams are connected to the terminal session from which srun was launched.

For example, to start an interactive session with the MPI program "myprogram", using one node and two processors, you can launch the command:

> salloc -N 1 --ntasks-per-node=2 -A <account_name> 

SLURM will then schedule your job to start, and your shell will be unresponsive until free resources are allocated for you. If not specified, the default time limit for this kind of jobs is one hour.

When the shell returns a prompt inside the compute node, you can execute your program by typing:

> srun ./myprogram

(srun is recommended with respect to mpirun for this environment)


SLURM automatically exports the environment variables you defined in the source shell, so that if you need to run your program "myprogram" in a controlled environment (i.e. with specific library paths or options), you can prepare the environment in the login shell and be sure to find it again in the interactive shell o the compute node.

On systems using SLURM, you can submit a script script.x using the command:

> sbatch script.x

You can get a list of defined partitions with the command:
> sinfo

For more information and examples of job scripts, see section Batch Scheduler SLURM.

Submitting serial Batch Jobs

The partition will be available in the full production.


Graphic session

If a graphic session is desired we recommend to use the tool "RCM". Please install the latest version of RCM. See the corresponding paragraph to know more about how to download and use RCM.

Submitting parallel Batch Jobs

To run parallel batch jobs on GALILEO100 you need to specify the partition and the qos that are described in this user guide.

If you do not specify the partition, your jobs will try to run on the default partition g100_all_serial.

The minimum number of cores you can request for a batch job is 1. The maximum number of cores that you can request is 3072 (64 nodes). It is also possible to request a maximum walltime of 24 hours. Defaults are as follows:

  • If you do not specify the walltime (by means of the #SBATCH --time directive), a default value of 30 minutes will be assumed.

  • If you do not specify the number of cores (by means of the "SBATCH -n" directive) a default value of 1 core will be assumed.

  • If you do not specify the amount of memory (as the value of the "SBATCH --mem" DIRECTIVE), a default value of 7800 MB per core will be assumed on g100_usr_prod and g100_usr_smem partitions. While on the g100_usr_bmem partition, the default value of memory assumed is  63200 MB per core.

The maximum memory per node is 375300MB (366.5GB) for thin and viz nodes, about 3TB for fat nodes. A study on the performances of the thin and fat nodes is ongoing.

On GALILEO100 there are four partitions dedicated to the production:

  • g100_usr_prod: this partition collects all the thin and persistent memory nodes (memory per node 375300 MB).
  • g100_usr_smem: this partition is composed only of thin nodes (memory per node 375300 MB).
  • g100_usr_pmem: this partition is composed only of Persistent memory nodes - more info in a following update (memory per node 375300 MB).
  • g100_usr_bmem: this partition is composed only of fat nodes (memory per node 3 TB). It is reserved for jobs that need more than 375300 MB of memory per node and is out of the g100_usr_prod partition.

We encourage you to use the partition more suitable for your job. For more information on the partitions look to Summary.


Processor affinity

Processor affinity, or CPU pinning, enables the binding of processes and threads to a CPU (or group of CPUs). It is crucial to ensure the correct affinity so to avoid the CPUs overallocation, with a significant reduction of performances. It becomes a critical matter when you ask for a full node but, for your specific reasons (memory needs etc.) you don't use all the cores.  

The following indications work when running your executables with srun, which is the recommended option against mpirun. We refer to a hybrid MPI/OpenMP case compiled with the Intel oneAPI suite.

Given your optimal value of OMP_NUM_THREADS and number of processes, to obtain the full node ask for a number of task such that  ( --ntasks-per-node * --cpus-per-task )= 48.

  • To avoid the processes over allocation of cores rely on the --cpu-bind=cores option of srun  (you can skip it if you use all the requested cores)
  • To enforce the threads affinity use the Intel parameter KMP_AFFINITY, or the OpenMP parameter OMP_PLACES
  • To distribute the MPI tasks consecutively inside the sockets, use the -m block:block option of srun (or the equivalent sbatch directive #SBATCH -m block:block)
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=12
#SBATCH --cpus-per-task=4
#SBATCH --account=<your_account>
module load autoload intelmpi/oneapi-2021--binary
export OMP_NUM_THREADS=4
export KMP_AFFINITY=compact    # or OMP_PLACES=cores
srun --cpu-bind=cores -m block:block <your_exe>

Use of GPUs on Galileo100

to be soon defined

Users with reserved resources

Users of projects that require reserved resources (such as industrial users or users associated to an agreement that involves dedicated resources) will be associated to a QOS qos_ind.

Using the  qos_ind (i.e. specifying the QOS in the submission script) , and specifying the partition g100_spc_prod, users associated to the allowed project will run their jobs on reserved nodes in the g100_spc_prod partition with the features and limits imposed for the particular account.

>#SBATCH --partition=g100_spc_prod
>#SBATCH --qos=qos_ind


Summary

In the following table, you can find all the main features and limits imposed on the SLURM partitions and QOS. For all partitions, if the memory is not explicitly requested, a DefMemPerCPU = 7800 will be assigned for each requested core. If the walltime is not explicitly requested, a DefaultTime=00:30:00 will be assigned to the job. If the partition is not explicitly requested, the default g100_all_serial partition will be assigned.

SLURM
partition

QOS

# cores per job

max walltime

max 

...

jobs

...

/resources

...

per user

max memory
per node
(MB)

priority

notes


g100_all_serial
(DEFAULT)

noQOSmax = 4

04:00:00

4 cores
120 submitted jobs

31,200 (30 GB)40

on two login nodes
budget-free

qos_installmax = 1604:00:00

...

16 cores
1 running job

...


100 GB40request to superc@cineca.it
g100_usr_dbgnoQOS
max = 2 nodes

01:00:00


375,300 (366 GB)40


qos_ind
Depending on the specific agreement
90

Partition dedicated to specific kinds of users.




g100_usr_prod

g100_usr_smem

g100_usr_pmem

noQOS

min = 1
max =  32 nodes

24:00:00

...


100 running jobs
120 submitted jobs






375,300 (366 GB)
40



runs on thin and persistent memory nodes

runs only on thin nodes

runs only on persistent memory

...

g100_qos_dbg

...

nodes

...

...

g100_qos_bprod

min = 1537 (33 nodes)
max = 3072 (64 nodes)

24:00:00

60
g100_qos_lprod

min = 1
max = 2 nodes

4-00:00:00

...

2 nodes

...


100 running jobs
120 submitted jobs
40
qos_special> 32 nodes> 24:00:00
40request to superc@cineca.it

g100_usr_bmem


noQOS25 nodes

24:00:00



100 running jobs
120 submitted jobs

3,036,000  (3 TB)

40runs on fat nodes

g100_usr_interactive


noQOS

...

max = 0.5 node

8:00:00

375,300 (366 GB)

40on nodes with GPUs
--gres=gpu:

...

g100_spc_prod

...

qos_ind

...

Depending on the specific agreement

...

24:00:00

...

375,300 (366 GB)

...

Partition dedicated to specific kinds of users.

...

N (N=1)

g100_meteo_prod

qos_meteo


24:00:00
375,300 (366 GB)
40

Partition reserved to meteo services,
NOT opened to production.

Runs on thin nodes

...

Programming environment 

The programming environment of the GALILEO cluster consists of a choice of compilers for the main scientific languages (Fortran, C and C++), debuggers to help users in finding bugs and errors in the codes, profilers to help in code optimisation.

...

You can check the complete list of available compilers on GALILEO100 LEONARDO with the command:

> module availablemodmap -c compilers

and checking the "compilers" section. The available compilers are:

  • Intel OneAPI (native and recommended)
  • GNU
  • CUDA
  • Compilers Collection (GCC)
  • NVIDIA nvhpc HPC-SDK (ex PGI) (soon available)
  • CUDA


Intel Compilers

The native, and recommended, compilers on GALILEO100 are the Intel ones, since the architecture is based on Intel processors and therefore using the Intel compilers may result in a significant improvement in performance and stability of your code. On the cluster is installed the new suite Intel OneAPI. Initialize the environment with the module command:

> module load intel/oneapi-2021--binary> module list
   Currently Loaded Modulefiles:
  intel/oneapi-2021--binary<VERSION>

The suite contains the new Intel oneAPI nextgen compilers (icx, icpx, ifx) and the classic compilers (icc, icpc, ifort):


ClassiconeAPINotes
C/C++ compilersicc/icpcicx/icpx
  • ICX is the Intel nextgen compiler based on Clang/LLVM technology  plus Intel proprietary optimizations and code generation,
  • ICX enables OpenMP TARGET offload to Intel GPU targets (irrelevant on Galileo100) 
  • ICX and ICC Classic use different compiler drivers. The ICC Classic drivers are icc, icpc, and icl. The ICX drivers are icx and icpx. Use icx to compile and link C programs, and icpx for C++ programs. 
  • Unlike the icc driver, icx does not use the file extension to determine whether to compile as C or C+. Users must invoke icpx to compile C+ files. In addition to providing a core C++ Compiler, ICX is the base compiler for the Intel® oneAPI Data Parallel C++ Compiler and its new driver, dpcpp.
  • Intel still recommends ICC/ICPC for standard C/C++ applications
Fortran compilersifortifx
  • The Intel® Fortran Compiler (Beta) IFX is s a new compiler based on the Intel® Fortran Compiler Classic (ifort) frontend and runtime libraries using LLVM backend technology.  ifx is released as a Beta version for users interested in trying offloading to supported Intel GPUs using OpenMP* TARGET directives which ifort does not support (irrelevant on Galileo100)

  • Intel recommends IFORT for standard Fortran applications


NOTE:

  • ICX is a new compiler. It has functional and behavioral differences compared to ICC. You can expect some porting will be needed for existing applications using ICC. According to Intel, the transition from ICC Classic to ICX is smooth and effortless. However, you must port and tune any existing applications from ICC Classic to ICX. Please refer to the official Intel Porting Guide for ICC Users to DPCPP or ICX
  • IFORT is a completely new compiler. According to Intel, although considerable effort is being made to make the transition from ifort to ifx as smooth and as effortless as possible, customers can expect that some effort may be required to tune their application. IFORT will remain Intel’s recommended production compiler until ifx has performance and features superior to ifort. Please refer to the official Intel Porting Guide for ifort Users to ifx
  • Please refer to the official Intel C++ Developer Guide and Reference and Fortran Developer Guide and Reference for an exhaustive list of compiler options

The documentation can be obtained with the man command after loading the relevant module:

> man ifort
> man icc

Some miscellaneous flags are described in the following:

-extend_-source    Extend over the 77 column F77's limit 
-free / -fixed    Free/Fixed form for Fortran
-ip               Enables additional interprocedural optimization for single-file compilation
-ipo              Enables interprocedural optimization between files - whole program optimisation
-qopenmp enables the parallelizer to generate multi-threaded code based on OpenMP directives


NOTE for the migration from Galileo to Galileo100: In principle, binaries generated on Galileo should work, but we strongly recommend you to reinstall all your software applications since on Galileo100 there is a different Operating System (Centos 8.3).


GNU compilers

The gnu compilers are always available but they are not the best optimizing compilers, especially for an Intel OneAPI-based cluster like GALILEO100.

For a more recent version of the compiler, initialize the environment with the module command:

> module load gnu

The name of the GNU compilers are:

  • g77: Fortran77 compiler
  • gfortran: Fortran compiler with "gnu" standard
  • gcc: C compiler
  • g++: C++ compiler

The "gnu" standard is the default value for the -std option. It specifies a superset of the latest Fortran standard that includes all of the extensions supported by GNU Fortran, although warnings will be given for obsolete extensions not recommended for use in new code. To change the standard to which the program is expected to conform, set the -std option to one of the possible values (f95, f2003, f2008, f2018, gnu, or legacy).

The documentation can be obtained with the man command:

> man gfortan
> man gcc

Some miscellaneous flags are described in the following:

-ffixed-line-length-132       To extend over the 77 column F77's limit
-ffree-form / -ffixed-form    Free/Fixed form for Fortran
-fopenmp Enable handling of OpenMP directives "#pragma omp" in C/C++ and "!$omp" in Fortran. 
When -fopenmp is specified, the compiler generates parallel code according to the
OpenMP Application Program Interface v4.5. This option implies -pthread and -fopenmp-simd


...