...
start of production:
Model: DUal-Socket Dell PowerEdge Architecture: Cores:48 cores/node + |  |
|---|
...
System Architecture
Compute Nodes:
- 528 computing nodes each 2 x CPU Intel CascadeLake 8260, with 24 cores each, 2.4 GHz, 384GB RAM, subdivided in:
- 348 standard nodes ("thin nodes") 480 GB SSD
- 180 180 data processing nodes ("fat nodes") 2TB SSD, 3TB Intel Optain
- 36 (viz nodes) GPU nodes with 2x NVIDIA GPU V100 with 100Gbs Infiniband interconnection and 2TB SSD.
- 77 computing server OpenStack for cloud computing, 2x CPU 8260 Intel CascadeLake, 24 cores, 2.4 GHz, 768 GB RAM, with 100Gbs Ethernet interconnection.
- 20 PB of active storage accessible from both cloud and HPC nodes.
- 5 PB of fast storage for HPC system.
- 1 PB Ceph storage for Cloud (full NVMe/SSD)
- 720 TB fast storage (IME DDN solution)
Login and Service nodes:
10 login nodes and 5 service nodes. All the nodes are interconnected through an Infiniband network, with OPA v10.6, capable of a maximum bandwidth of 100Gbit/s between each pair of nodes.
Accounting
For more information about accounting, please consult our dedicated section.
Budget Linearization policy
On GALILEO100 a linearization policy for the usage of project budgets has been defined and implemented. For each account, a monthly quota is defined as:
monthTotal = (total_budget / total_no_of_months)
Starting from the first day of each month, the collaborators of any account are allowed to use the quota at full priority. As long as the budget is consumed, the jobs submitted from the account will gradually lose priority, until the monthly budget (monthTotal) is fully consumed. At that moment, their jobs will still be considered for execution, but with a lower priority than the jobs from accounts that still have some monthly quota left.
This policy is similar to those already applied by other important HPC centers in Europe and worldwide. The goal is to improve the response time, giving users the opportunity of using the cpu hours assigned to their project in relation to their actual size (total amount of core-hours).
Disks and Filesystems
The storage organisation conforms to the CINECA infrastructure (see Section "Data storage and Filesystems") . In addition to the home directory ($HOME), for each user is defined a scratch area $CINECA_SCRATCH, a large disk for storing run time data and files. $WORK is defined for each active project on the system, reserved for all the collaborators of the project. This is a safe storage area to keep run time data for the whole life of the project.
The filesystem organisation is based on LUSTREm an ioen source parallel file system.
| Total Dimension (TB) | Quota (GB) | Notes | |
|---|---|---|---|
| $HOME | 50 GB quota per user |
| |
| $CINECA_SCRATCH | no quota |
| |
| $WORK | 1TB quota per project |
|
It is also available a temporary storage local on compute nodes generated when the job starts and accessible via environment variable $TMPDIR. For more details please see the dedicated section of UG2.5: Data storage and FileSystems. On Galileo Galileo100 the $TMPDIR local area has 49 ## GB of available space.
$DRES points to the shared repository where Data RESources are maintained. This is a data archive area available only on-request, shared with all CINECA HPC systems and among different projects.
$DRES is not mounted on the compute nodes. This means that you can't access it within a batch job: all data needed during the batch execution has to be moved on $WORK or $CINECA_SCRATCH before the run starts.
Since all the filesystems are based on gpfs (General Parallel FIle System), the usual unix command "quota" is not working. Use Use the local command "cindata" to query for disk usage and quota ("cindata -h" for help):
> cindata
Modules environment
The software modules are collected in different profiles and organized by functional category (compilers, libraries, tools, applications,..).
On GALILEO100 the profiles are of two types, “domain” type (bioinf, chem-phys, lifesc,..) for the production activity and “programming” type (base and advanced) for compilation, debugging and profiling activities and that they can be loaded together.
"Base" profile is the default. It is automatically loaded after login and it contains basic modules for the programming activities (intel e gnu compilers, math libraries, profiling and debugging tools,..).
If you want to use a module placed under other profiles, for example an application module, you will have to load preventively the corresponding profile:
>module load profile/<profile name>
>module load autoload <module name>
For listing all profiles you have loaded you can use the following command:
>module list
In order to detect all profiles, categories and modules available on GALILEO100 the command “modmap” is available:
>modmap
With modmap you can see if the desired module is available and which profile you have to load to use it.
>modmap -m <module name>
...