EURORA Batch Scheduler PBS

With the exception of the p_devel queue, you do not need to specify the job type: according to the resources (number of CPUs/GPUs/MICs) and the walltime requested, the scheduler will automatically route your job to the appropriate queue. On the contrary, since the p_devel queue is defined on specific nodes with a different configuration (tuned for the optimal use of profiling and performance analysis tools), you need to explicitly specify its use via the flags -q of qsub or the directive #PBS -q <queue_name>.

PBS commands

The main user's commands of PBS are reported in the table below: Please consult the man pages for more information.

Submit a job:

> qsub [opts ] job_script
> qsub -I [opts] -- /bin/bash  (interactive job)

The second command is related to a so-called "Interactive job": sets job's interactive attribute to TRUE. The job is queued and scheduled as any PBS batch job, but when executed, the standard input, output, and error streams of the job are connected to the terminal session in which qsub is running. When the job begins execution, all input to the job is taken from the terminal session. Use CNTR-D to close the session.

Displaying Job Status:

> qstat             (only jobs submitted by the user, same as qstat -u $USER)
> qstat    <job_id> (only the specified job)
> qstat -f <job_id> (full display of the specified job)

Displaying Queue Status:

> qstat -Q 
> qstat -Qf  <queuename>     (Long format of the specified  queue)

Delete a job:

> qdel  <jobID>.node351

More information about these commands are available with the man command

The User Environment

There are a number of environment variables provided to the PBS job. Some of them are taken from the user's environment and carried with the job. Others are created by PBS.

All PBS-provided environment variable names start with the characters PBS_. Some are then followed by a capital O (PBS_O_) indicating that the variable is from the job's originating environment (i.e. the user's).

The following short example lists some of the more useful variables, and typical values:

PBS_JOBNAME=jobb
PBS_ENVIRONMENT=PBS_BATCH
PBS_JOBID=6207.node129
PBS_QUEUE=parallel

PBS_O_WORKDIR=/gpfs/scratch/usercin/aer0
PBS_O_HOME=/eurora/home/usercin/aer0
PBS_O_QUEUE=parallel
PBS_O_LOGNAME=aer0
PBS_O_SHELL=/bin/bash
PBS_O_HOST=node129
PBS_O_MAIL=/var/spool/mail/aer0
PBS_O_PATH=/cineca/bin:/cineca/sysprod/pbs/default/bin: ...

There are a number of ways that you can use these environment variables to make more efficient use of PBS. For example PBS_ENVIRONMENT can be used to test if we were running under PBS. Another commonly used variable is PBS_O_WORKDIR which contains the name of the directory from which the user submitted the PBS job

(NOTE: PBS does execute job script in the $HOME directory by default!)

PBS Resources

A job requests resources through the PBS sintax; PBS matches requested resources with available resources, according to rules defined by the administrator; when resources are allocated to the job, the job can be executed.

There are different types of resources, i.e. server level resources like walltime, and chunked reources like number of cpus or nodes. Other resources may be added to manage access to software resources for example, particularly when resources are limited and lack of their availability leads to aborting the jobs when they are scheduled for execution. Details may be found in documentation (module help) of the interested applications.

The sintax of the request depends on which type is concerned:

#PBS -l <resourse>=<value>        (server level resources, e.g. walltime)
#PBS -l select=[N:]chunk[+[N:]chunk ...] (chunk resources, e.g. cpus, gpus,mpiprocs)

For example:

#PBS -l walltime=10:00
#PBS -l select=1:ncpus=1

Moreover, resources can be required in one of two possible ways:

1) using PBS directives in the job script

2) using options of the qsub command

PBS directives: number and type of processors

The number and the type (at 2 GHz or 3GHz) of cpus required for a serial or parallel MPI/OpenMP/mixed job must be required with the "select" directive:

#PBS -l select=NN:ncpus=CC:cpuspeed=HHGHz:ngpus=GG:mpiprocs=TT[+N1....]

where:

• NN = number of nodes (max depending on the queue)
• CC = number of physical cpus per node (max 16)
• HH = CPU clock rate
• GG = number of physical gpus per node (max 2)
• TT = number of MPI tasks per node

for example:

#PBS -l select=1:ncpus=1             --> serial job
#PBS -l select=2:ncpus=8:mpiprocs=8  --> MPI job (2 nodes and 8 proc per node)
#PBS -l select=2:ncpus=8:mpiprocs=1  --> mixed job (2 MPI tasks and 8 threads/task)
#PBS -l select=1:ncpus=16:mpiprocs=16:ngpus=2     
                                     --> MPI job with cuda (16 MPI tasks and 2 GPUs)
#PBS -l select=2:ncpus=16:mpiprocs=16+1:ncpus=5:mpiprocs=5 
                                     --> 37 MPI tasks on three nodes: 16+16+5

Please note that hyper-threadind is disabled, so it is warmly recommended to set a value for mpiprocs less or equal to ncpus. If you specify a higher number of mpiprocs you will overload the physical cores and slow down your own execution.

PBS directives: processing time

Resources as the computing time, must be requested in this form:

#PBS -l walltime=<value>

where <value> express the actual elapsed time (wall-clock) in the format hh:mm:ss

for example:

#PBS -l walltime=1:00:00 (one hour)

PBS directives: memory allocation

The default memory is 1 GB per node (for the classes debug, p_devel, parallel and np_longpar).

The user can specify the requested memory up to 14 GB, on 59 nodes:

#PBS -l select=NN:ncpus=CC :mpiprocs=TT:mem=14GB

There are 5 nodes with 32 GB, that you can require by specifying a value of mem larger than 16 GB up to 30 GB. 

Other PBS directives

#PBS -N name        --> specify job name
#PBS -q destination --> specify the queue destination. For a list a description of available 
                        queues, please refer to the specific cluster description of the guide
#PBS -o path        --> redirects output file
#PBS -e path        --> redirects error file
#PBS -j eo          --> merge std-err and std-out
#PBS -m mail_events --> specify email notification (a=aborted,b=begin,e=end,n=no_mail)
#PBS -M user_list   --> set email destination (email addr)

qsub attributes

The attributes  can also be set using the qsub command options:

> qsub [-N name]  [-q destination]  [-o path] [-e path] [-j eo]
       [-m mail_events] [-M user_list]                          <name of script>

The resources can also be required using the qsub command options.

> qsub  [-l walltime=<value>] [-select=<value>] [-A <account_no>] <name of script>

The qsub command options override directives.