Page History

PBS directives: num. of processors

The number of cpus required for a serial or parallel MPI/OpenMP/mixed job must be required with the "select" directive:

#PBS -l select=NN:ncpus=CC:mpiprocs=TT:ngpus=GG:nmics=MM[+N1....]

where:

• NN: number of nodes (max depending on the queue)
• ncpus=CC:  number of physical cores per node 
• mpiprocs=TT:  number of MPI tasks per node
• ngpus=GG:  number of physical gpus per node (if available on the system)
• nmics=MM:  number of physical Intel MICs per node (if available on the system)

for example:

#PBS -l select=1:ncpus=1             --> serial job
#PBS -l select=2:ncpus=8:mpiprocs=8  --> MPI job (2 nodes and 8 procs per node)
#PBS -l select=2:ncpus=8:mpiprocs=1  --> mixed job (2 MPI tasks and 8 threads/task)
#PBS -l select=1:ncpus=16:mpiprocs=16:ngpus=2     
                                     --> MPI job with cuda (16 MPI tasks and 2 GPUs)
#PBS -l select=2:ncpus=8:mpiprocs=8+1:ncpus=5:mpiprocs=5 
                                     --> 21 MPI tasks on three nodes: 8+8+5

Please note that hyper-threading is disabled, so it is warmly recommended to set a value for mpiprocs less or equal to ncpus. If you specify a higher number of mpiprocs you will overload the physical cores and slow down your own execution.

On different systems there could be different values of cores/node and accelerator devices. Check the system page of the Guide for more information.

PBS directives: processing time

Resources as the computing time, must be requested in this form:

#PBS -l walltime=<value>

where <value> express the actual elapsed time (wall-clock) in the format hh:mm:ss

for example:

#PBS -l walltime=1:00:00 (one hour)

Please note that there are specific limitations on the maximum walltime on a system. Check the system page of the Guide for more information.

PBS directives: memory allocation

The default memory depends on the system you are working with. You can specify the requested memory (up to to maximum memory available on the nodes:  on Galileo it is 120 GB, on Pico 122GB, on Marconi 123GB118GB) using the "mem" directive:

#PBS -l select=NN:ncpus=CC :mpiprocs=TT:mem=24GB

Please note: if you are requiring a larger memory with respect to the "main amount" on the system, the number of "effective cores" and the cost of your job could increase. For more information refers to the "accounting" section.

Other PBS directives

#PBS -A <account_no> --> name of the project to be accounted to ("saldo -b" for a list of prjs)
#PBS -N name        -->  job name
#PBS -q destination --> queue destination. For a list and description of available 
                                     queues, please refer to the specific cluster description of the guide
#PBS -o path        --> redirects output file
#PBS -e path        --> redirects error file
#PBS -j eo            --> merge std-err and std-out
#PBS -m mail_events --> specify email notification (a=aborted,b=begin,e=end,n=no_mail)
#PBS -M user_list       --> set email destination (email addr)

Please note that Galileo and Marconi are served by a routing queue (directing all jobs to the "shared" queue), which is also the default queue. Hence there is no need to specify the #PBS -q directive. The explicit request of the shared queue (on Galileo) and of the debug, prod, bigprod queues (on Marconi) as the destination will be rejected by PBS. You need to use that directive only if you need to submit jobs to the archive queue.

qsub attributes

The attributes  can also be set using the qsub command options:

> qsub [-N name]  [-q destination]  [-o path] [-e path] [-j eo]
            [-m mail_events] [-M user_list]                                       <name of script>

The resources can also be required using the qsub command options.

> qsub  [-l walltime=<value>] [-select=<value>] [-A <account_no>] <name of script>

The qsub command options override directives.