Page History

...

By default SLURM uses the auto-affinity module to bind MPI tasks to cpus. In some cases, you may want to modify this default, in order to ensure optimal performances on Marconi-A2 (KNL) and Marconi-A3 (SkyLake).

You can find below the guidelines to do it.

1) If  On "exclusive" nodes if you require for a the single KNL or SKL node (exlusive nodes) to use less tasks node fewer tasks than the available physical cpus (< 48 cpus on SKL nodes, < 68 cpus on KNL nodes), you have to specify the following slurm option as a parameter of the "srun" command line:

--cpu-bind=cores

...

Alternatively, you can request more cpus for the single task so that you will use all the cpus of the node:
--cpus-per-task=<n° cpus per node / n° tasks per node>
...

If not specified, the default of --cpus-per-task is 1.
 

For SKL nodesIf the “n° tasks per node” is not a divisor of “n° cpus per node“, then “n° cpus per task” should be equal to:
--cpus-per-task=<48 <floor (n° cpus per node/ n° tasks per node>
 
For KNL nodes:
node)>

You you have to remember that if hyperthreading is enabled, thus the “ n° cpus per node” refers to logical cpus rather than physical . So there are in total 272 maximum logical cpus for a single node (68 cpus (logical cpus = physical cpus * 4 n° hyper threads):
--cpus-per-task=<272<n° logical cpus per node / n° tasks per node>
 
...
--cpus-per-task=
...
floor (n° physical cpus per node/ n° tasks per node
...
 
...
)
...
 * n° 
...
hyper threads

In case the number of tasks is not a divisor of the number of cpus per node, covering all the tasks is not an alternative, and you should add the srun directive "--cpu-bind=cores" anyway. 

2)
...
 On "hyperthreading nodes
...
" if you require for a single 
...
node more tasks of available physical cpus
...
  you have to specify the following SLURM option:
--cpu-bind=threads
in order to ensure the binding between mpi tasks and logical cpus and to avoid the overload of physical cpus.
 
Alternatively,
...
 you can request more cpus for the single task until you use all logical cpus of the node 
...
 :
--cpus-per-task=
...
<n° logical cpus per node / n° tasks per node>
 

3) If  If you require for a single node a a number of tasks that is are equal to a number of physical cpus (48 on SKL node, 68 on KNL nodes) or  or number of logical cpus (272 on KNL nodes) there sthere is no need for adding --cpu-bind or --cpus-per-task slurm options. Each task is assigned a CPU in sequential order.
...
--cpus-per-task=<n° cpus>
 
On 
...
nodes without hyperthreading the cpu 
...
concept coincides with physical cpu (core) and consequently the n° of cpus for single task (--cpus-per-task) can be up the maximum number of physical cpus of the node. For example, on BDW and  SKL nodes the n° of cpus for single task can be up to 36/ 48 cpus respectively.
...

On  hyperthreading nodes, it coincides 
...
with logical cpu (thread) and consequently, the n° of cpus for a single task can be up 
...
 
...
to the maximum number of logical cpus of the node.
In order to define if the OpenMP threads have to bind 
...
physical ( core) or logical cpus (thread) you can use the following variable:
export OMP_PLACES= <cores|threads>
...


For example, if you request 68 logical cpus for a single task:
--cpus-per-task=68
 you are allocating 17 physical cpus (68/4) and than if you want the OpenMP threads bind physical cpus you have to set the variable to “cores”:
export OMP_PLACES=cores
and you can specify up to 17 OpenMP threads for single task:
export OMP_NUM_THREADS=<up to 17>
 
have to run on a node with 32 cores and 8 hyper threads for single core, you can require up to 256 logical cpus for single task. If you require 256 openmp threads for a single task, each one binds necessary a logical cpus.  If you require 32 openmp threads they can bind logical or physical cpus.
In order to bind physical cpus: 1 task, 32 openmp threads for a single task, 32 physical cpus (=256 logical cpus) for a single thread
--cpus-per-task=256
export OMP_NUM_THREADS=32If you select more of 17 OpenMP threads you can bind them only to logical cpus because the physical cpus allocated for single task are 17 and then you have to specify "threads" for "OMP_PLACES variable":
export OMP_PLACES=
...
 <cores>
In order to bind logical cpus: 1 task, 32 openmp threads for single task, 32 logical cpus for single thread
--cpus-per-task=256
...
export OMP_NUM_THREADS=<up to 68>
 
=32
export OMP_PLACES= <threads>
 
If you are using intel you can set KMP_AFFINITY variable  By default for intelmpi module on Marconi the kmp_affinity environment variable is set to "compact" value in order to bind the threads to available cpus consecutively:
export KMP_AFFINITY=compact 
It binds threads to available cpus consecutively.
You can modify this default in “scatter” way :
export KMP_AFFINITY=scatter

Alternatively, you can use non intel options:
export OMP_PROC_BIND=<close|spread|true>
...
You can find, at the following web page, some MPI and MPI/OpenMP jobs scripts examplest:
UG2.65.1:  How to submit the job -  Batch Scheduler SLURM