Page History

The User Environment

There are a number of environment variables provided to the SLURM job. Some of them are taken from the user's environment and carried with the job. Others are created by SLURM.

All SLURM-provided environment variable names start with the characters SLURM_.

Below are listed some of the more useful variables, and some typical values taken as an example:

SLURM_JOB_NAME=job
SLURM_NNODES (or SLURM_JOB_NUM_NODES)=2
SLURM_JOBID (or SLURM_JOB_ID)=453919
SLURM_JOB_NODELIST=node1,node2,...
SLURM_SUBMIT_DIR=/marconi_scratch/userexternal/username
SLURM_SUBMIT_HOST=node1
SLURM_CLUSTERNAME=cluster1
SLURM_JOB_PARTITION=partition1

There are a number of ways that you can use these environment variables to make more efficient use of SLURM. For example, SLURM_JOB_NAME can be used to retrieve the SLURM jobname. Another commonly used variable is SLURM_SUBMIT_DIR which contains the name of the directory from which the user submitted the SLURM job.

WARNING: $SLURM_JOB_NODELIST will display the node names in contracted forms, meaning that for consecutive nodes you will get their range instead of the full list. You will see in square brackets the ID of the first and the last node of the chunk, meaning that all the nodes between them are also part of the actual node list.

Job TMPDIR:

When a job starts, a temporary area is defined on the storage local to each compute node:

TMPDIR=/scratch_local/slurm_job.$SLURM_JOB_ID

which can be used exclusively by the job's owner. During your jobs, you can access the area with the (local) variable $TMPDIR. The directory is removed at the end of the job, hence remember to save the data stored in such area to a permanent directory. Please note that the area is located on local disks, so it can be accessed only by the processes running on the node. For multinode jobs, if you need all the processes to access some data, please use the shared filesystems $HOME, $WORK, $CINECA_SCRATCH.

SLURM Resources

A job requests resources through the SLURM syntax; SLURM matches requested resources with the available ones, according to the rules defined by the administrator. When the resources are allocated to the job, the job can be executed.

There are different types of resources, i.e. server level resources, like walltime, chunk resources, like number of cpus or nodes, and generic resource (GRES) like GPUs on the systems that have them.

Other resources may be added to manage access to software resources, for example when resources are limited and the lack of availability leads to jobs abort when they are scheduled for execution. More details may be found in the module help of the application you are trying to execute.

The syntax of the request depends on the type of resource:

#SBATCH --<resource>=<value>          (server level resources, e.g. walltime)
#SBATCH --<chunk_resource>=<value>    (chunk resources, e.g. cpus, nodes,...)
#SBATCH --gres=gpu:<value>            (generic resources, e.g. gpus)

For example:

#SBATCH --time=10:00:00
#SBATCH --ntasks-per-node=1
#SBATCH --gres=gpu:2

Resources can be required either:

1) using SLURM directives in the job script

2) using options of the sbatch/salloc command

• --nodes=NN                        number of nodes 
• --ntasks-per-node=CC       number of tasks/processes per node 
• --cpus-per-task=TT            number of threads/cpus per task

Please note that there are specific limitations on the maximum walltime on a system, also depending on the partition. Check the system specific guide for more information.

SLURM directives: memory allocation

The default memory depends on the partition/queue you are working with. Usually we set it as the Total Memory of the Node divided by the total number of cores in a single node that we call here memory-per-core. So if you request 3 cores by default you would get the equivalent of 3 times the memory-per-core. Alternatively, you can specify the requested memory with the --mem=<value> directive up to maximum memory available on the nodes.

#SBATCH --mem=10000

The default measurement unit for memory requests is the Megabyte (in the example above, we are requesting for 10000MB per node). It is possible to ask for an amount of memory expressed in GB, like this:

#SBATCH --mem=10GB

However, the default request method in MB is preferable, since the memory limits defined for any partition are expressed in these terms. For example, Marconi SkyLake partition has 182000MB as a limit, corresponding to approx. 177GB.

Please note: if you are requiring a larger memory with respect to the "main amount" for the system, the number of "effective cores" and the cost of your job could increase. For more information check the accounting section.

SLURM directives: MPI tasks/OpenMP threads affinity

You may have to modify the default affinity, in order to ensure optimal performances on A3 Marconi.

The slurm directives that concern the processes binding are the following:

--cpu-bind=<cores|threads>

--cpus-per-task=<physical or logical cpus number to allocate for single task>

In order to modify them correctly, we suggest to follow our guidelines.

Other SLURM directives

#SBATCH --account=<account_no>          --> name of the project to be accounted to ("saldo -b" for a list of projects)
#SBATCH --job-name=<name>               --> job name
#SBATCH --partition=<destination>       --> partition/queue destination. For a list and description of available partitions, please refer to the specific cluster description of the guide.
#SBATCH --qos=<qos_name>                --> quality of service. Please refer to the specific cluster description of the guide.
#SBATCH --output=<out_file>             --> redirects output file (default, if missing, is slurm-<Pid> containing merged output and error file)
#SBATCH --error=<err_file>              --> redirects error file (as above)
#SBATCH --mail-type=<mail_events>       --> specify email notification (NONE, BEGIN, END, FAIL, REQUEUE, ALL)
#SBATCH --mail-user=<user_list>         --> set email destination (email address)

Directives in contracted form

Some SLURM directives can be written with a contracted syntax. Here are all the possibilities:

#SBATCH -N <NN>                         --> #SBATCH --nodes=<NN>
#SBATCH -c <TT>                         --> #SBATCH --cpus-per-task=<TT>
#SBATCH -t <value>                      --> #SBATCH --time=<value>
#SBATCH -A <account_no>                 --> #SBATCH --account=<account_no>
#SBATCH -J <name>                       --> #SBATCH --job-name=<name>
#SBATCH -p <destination>                --> #SBATCH --partition=<destination>
#SBATCH -q <qos_name>                   --> #SBATCH --qos=<qos_name>
#SBATCH -o <out_file>                   --> #SBATCH --output=<out_file>
#SBATCH -e <err_file>                   --> #SBATCH --error=<err_file>

Note: the directives --mem, --mail-type, --mail-user and --ntasks-per-node can't be contracted. About the latter, it exists a SLURM directive "-n" for the number of tasks, but it can be misleading since it is used to indicate the TOTAL number of tasks and not the number of tasks per node. Therefore, it is not recommended since it can lead to confusion and unexpected behaviour. Use of the uncontracted --ntasks-per-node is recommended instead.

Using sbatch attributes to assign job attributes and resource request

It is also possible to assign the job attributes using the sbatch command options:

> sbatch [--job-name=<name>]  [--partition=<queue/partition>]  [--out=<out_file>] [--err=<err_file>] [--mail-type=<mail_events>] [--mail-user=<user_list>] <name of script>

And the resources can also be requested using the sbatch command options:

> sbatch  [--time=<value>] [-ntasks=<value>] [--account=<account_no>] <name of script>

The sbatch command options override script directives if present.