...
This supercomputer takes advantage of the new Intel® Omni-Path Architecture, which provides the high performance interconnectivity required to efficiently scale out the system’s thousands of servers.
...
Login nodes: 8 Login (3 available for regular users). Each one contains 2 x Intel Xeon Processor E5-2697 v4 with a clock of 2.30GHz and 128 GB of memory. Login nodes are shared between three partitions: A1 (BDW), A2 (KNL) and A3 (SKL). The three partitions are served by three different PBS servers that must be selected in order to address the required resourcea single SLURM server.
System A1 (Broadwell)
Model: Lenovo NeXtScale Racks: 21
|
|
|---|
...
KNL is the evolution of Knights Corner (KNC), part of the Xeon Phi family and currently available on CINECA's system GALILEO GALILEO until January 2018. The main differences between KNC and KNL are a better power in performance, a faster internal network, and the fact that :
- KNLs are
...
- standalone, self-bootable processors, unlike KNCs, which are treated as accelerators;
- a better power in performance;
- a faster internal network.
Applications compiled for KNL are also binary compatible with regular computing nodes.
KNL supports Intel AVX-512 instruction set extensions. The same three login nodes serve the Marconi-Broadwell (Marconi-A1) and the Marconi-KNL (Marconi-A2) partitions and queueing systems. Also the storage
Storage devices are in common between the two partitions.
...
> saldo -b --knl (reports projects defined on Marconi-KNL)
Budget Linearization policy
On MARCONI a linearization policy for the usage of project budgets has been defined and implemented. For each account, a monthly quota is defined as:
monthTotal = (total_budget / total_no_of_months)Starting from the first day of each month, the collaborators of any account are allowed to use the quota at full priority. As long as the budget is consumed, the jobs submitted from the account will gradually lose priority, until the monthly budget (monthTotal) is fully consumed. At that moment, their jobs will still be considered for execution, but with a lower priority than the jobs from accounts that still have some monthly quota left.
This policy is similar to those already applied by other important HPC centers in Europe and worldwide. The goal is to improve the response time, giving users the opportunity of using the cpu hours assigned to their project in relation of their actual size (total amount of core-hours).
Disks and Filesystems
The storage organization conforms to the CINECA infrastructure (see Section Data Storage and Filesystems).
...
| Total Dimension (TB) | Quota (GB) | Notes | |
|---|---|---|---|
| $HOME | 200 | 50 |
|
| $CINECA_SCRATCH | 2.500 | no quota |
|
| $WORK | 7.100 | 1.000 |
|
$DRES points environment variable points to the shared repository where Data RESources are maintained. This is a data archive area availble only on-request, shared with all CINECA HPC systems and among different projects. $DRES is not mounted on the compute nodes. This means that you cannot access it within a batch job: all data needed during the batch execution has to be moved to $WORK or $CINECA_SCRATCH before the run starts.
...
Applications available on MARCONI are compiled for being executable on both A1-A3 and A2 partitions. At present we did not recompile all softwares to have a KNL optimized version, but the binaries built for the Broadwell nodes can run on the KNL as well. We will inform you via newsletter when the optimized version of all softwares will be available.
"Base" profile is the default. It is automatically loaded after login and it contains basic modules for the programming activities (intel e gnu compilers, math libraries, profiling and debugging tools,..).
...
When the shell come back with the prompt, you can execute your program by typing:
> mpirun ./myprogramor
> srun ./myprogram --mpi=pmi2
SLURM automatically exports the environment variables you defined in the source shell, so that if you need to run your program myprogram in a controlled environment (i.e. specific library paths or options), you can prepare the environment in the origin shell being sure to find it in the interactive shell SLURruns for you on allocated resources.
Batch
The info reported here refer to the general MARCONI partition. The production environment of MARCONI_Fusion is discussed in a separate document.
...
You can get a list of defined partitions with the command:
> sinfo -ad
For more information and examples of job scripts, see section Batch Scheduler SLURM.
...
On MARCONI it is possible to submit jobs of different types using a "routing" partition: just declare how many resources you need and your job will be directed into the right production partition (bdw_usr_dbg, bdw_usr_prod) with a correct priority. Furthermore, there are two additional partitions not managed by the default routing, devoted to specific categories of jobs: the serial partition and the special partition.
The minimum number of cores to requere
The maximum number of cores that you can request is 6000 (about 167 nodes) with a maximum walltime of 24 hours:
...