...
Disks and Filesystems
The storage organisation conforms to the CINECA infrastructure (see Section "Data storage and Filesystems" ). In addition to the home directory ($HOME), for each user is defined a scratch area $CINECA_SCRATCH, a large disk for storing run time data and files. $WORK is defined for each active project on the system, reserved for all the collaborators of the project. This is a safe storage area to keep run time data for the whole life of the project.
| Total Dimension (TB) | Quota (GB) | Notes | |
|---|---|---|---|
| $HOME | 200 | 50 |
|
| $CINECA_SCRATCH | 2.500 | no quota |
|
| $WORK | 7.100 | 1.000 |
|
$DRES points to the shared repository where Data RESources are maintained. This is a data archive area availble only on-request, shared with all CINECA HPC systems and among different projects.
$DRES is not mounted on the compute nodes. This means that you can't access it within a batch job: all data needed during the batch execution has to be moved on $WORK or $CINECA_SCRATCH before the run starts.
Since all the filesystems are based on gpfs (General Parallel FIle System), the usual unix command "quota" is not working. Use the local command "cindata" to query for disk usage and quota ("cindata -h" for help):
> cindata
Modules environment
As usual, the software modules are collected in different profiles and organized by functional category (compilers, libraries, tools, applications,..).
On GALILEO a new feature has been added to the module environment: the profiles are of two types, “domain” type (chem, phys, lifesc,..) for the production activity and “programming” type (base and advanced) for compilation, debugging and profiling activities and that they can be loaded together.
"Base" profile is the default. It is automatically loaded after login and it contains basic modules for the programming activities (intel e gnu compilers, math libraries, profiling and debugging tools,..).
If you want to use a module placed under others profiles, for example an application module, you will have to load preventively the corresponding profile:
>module load profile/<profile name>
>module load autoload <module name>
For listing all profiles you have loaded use the following command:
>module list
In order to detect all profiles, categories and modules available on GALILEO the command “modmap” is available:
>modmap
We allow you to load old-Galileo environment modules.
To switch from new environmengt (the default one) to the old and vice-versa, you can load the superc module
>module load supercand run one of the following commands:
FOR BASH USERS
. $SUPERC_SWITCH/switch_to_old.shto switch from new ro old environment, or
. $SUPERC_SWITCH/switch_to_new.shto switch from old to new environment.
FOR CHS/TCSH USERS:
source $SUPERC_SWITCH/switch_to_old.cshto switch from new ro old environment, or
source $SUPERC_SWITCH/switch_to_new.cshto switch from old to new environment.
Production environment
Since GALILEO is a general purpose system and it is used by several users at the same time, long production jobs must be submitted using a queuing system. This guarantees that the access to the resources is as fair as possible.
Roughly speaking, there are two different modes to use an HPC system: Interactive and Batch. For a general discussion see the section "Production Environment and Tools".
Interactive
A serial program can be executed in the standard UNIX way:
> ./program
This is allowed only for very short runs, since the interactive environment has a 10 minutes time limit: for longer runs please use the "batch" mode.
A parallel program can be executed interactively only within an "Interactive" SLURM batch job, using the "srun" command: the job is queued and scheduled as any other job, but when executed, the standard input, output, and error streams are connected to the terminal session from which srun was launched.
For example, to start an interactive session with the MPI program myprogram, using one node, two processors, launch the command:
> srun -N1 -n2 --ntasks-per-node=2 -A <account_name> --pty /bin/bash
SLURM will then schedule your job to start, and your shell will be unresponsive until free resources are allocated for you.
When the shell come back with the prompt, you can execute your program by typing:
> mpirun ./myprogram
or
> srun --mpi=pmi2 ./myprogram
Pay attention in putting the flag --mpi=pmi2 immediatly after srun.
SLURM automatically exports the environment variables you defined in the source shell, so that if you need to run your program myprogram in a controlled environment (i.e. specific library paths or options), you can prepare the environment in the origin shell being sure to find it in the interactive shell.
Batch
As usual on systems using SLURM, you can submit a script script.x using the command:
> sbatch script.x
You can get a list of defined partitions with the command:
> sinfo -d
For more information and examples of job scripts, see section Batch Scheduler SLURM.
Submitting serial Batch jobs
The gll_all_serial partition is available with one core and a maximum walltime of 4 hours. It runs on the login nodes and it is designed for pre/post-processing serial analysis, and for moving your data (via rsync, scp etc.) in case more than 10 minutes are required to complete the data transfer. In order to use this partition you have to specify the SLURM flag "-P":
#SBATCH -p gll_all_serial
The gll_all_serial partition has a limit of 4 tasks per job and 4GB of memory per job. If you wish to ask for more than a core on a single job, remember to add on your jobscript the specific about the memory limit, since the default per core is 3.5GB and therefore your job won't enter because the required memory exceeds the partition limit.
Graphic session
If a graphic session is desired we recommend to use the RCM tool or the EnginFrame environment. Both of them are under construction, we will report here all required information asap.
Submitting parallel Batch jobs
To run parallel batsch jobs on GALILEO you need to specify the partition gll_usr_prod, or any other partition we invited you to use.
If you do not specify the partition, your jobs will try to run on the bdw_all_serial partition, eventually failing if specific partition limits (maximum one core for maximum walltime of 4 hours) are violated.
The minimum number of cores to require is 1. The maximum number of cores that you can request is the 2304 (about 167 nodes) with a maximum walltime of 24 hours:
- If you do not specify the walltime (by means of the #SBATCH --time directive), a default value of 30 minutes will be assumed.
- If you do not specify the number of cores (by means of the "SBATCH -n" directive) a default value of 36 will be assumed.
- If you do not specify the amount of memory (as the value of the "SBATCH --mem" DIRECTIVE), a default value of 3000MB will be assumed.
- The maximum memory per node is 118000MB.
The special QOS (bdw_qos_special) is designed for not-ordinary types of jobs, and users need to be enabled in order to use it. Please write to superc@cineca.it in case you think you need to use it.
Summary
In the following table you can find all the main features and limits imposed on the queues of the shared A1 and A2 partitions. For Marconi-FUSION dedicated queues please refer to the dedicated document.
SLURM partition | QOS | # cores per job | max walltime | max running jobs per user/ max n. of cpus/nodes per user | max memory per node (MB) | priority | HBM/clustering mode | notes |
bdw_all_serial (default partition) | gll_all_serial | 1 | 04:00:00 | Max 12 running jobs Max 4 jobs/user | 4096 | |||
| bdw_usr_prod | noQOS | min = 1 max = 2304 | 24:00:00 | 20/2304 | 118000 |
PLEASE NOTE: the SLURM characteristics have not completely defined for GALILEO, some changes will be possible.
...