UG3.4.x: PLX UserGuide

Interactive

A serial program can be executed in the standard UNIX way:

> ./program

This is allowed only for very short runs, since the interactive environment has a 10 minutes time limit: for longer runs please use the "batch" mode.

A parallel program can be executed interactively only within an "Interactive" PBS batch job, using the "-I" (capital i) option: the job is queued and scheduled as any PBS batch job, but when executed, the standard input, output, and error streams are connected to the terminal session from which qsub was submitted.

For example, for starting an interactive session with the MPI program myprogram, using 2 processors, enter the commands::

> qsub -A <account_no> -I -l select=1:ncpus=2:mpiprocs=2 -q debug -- /bin/bash
    qsub: waiting for job 4611.node351.plx.cineca.it to start
    qsub: job 4611.node351.plx.cineca.it ready
> mpirun ./myprogram
> ^D

If you want to export variables to the interactive session, use the -v option. For example if myprograms is not compiled statically, you have to define and export the LD_LIBRARY_PATH variables:

> export LD_LIBRARY_PATH= ...
> qsub -I -v LD_LIBRARY_PATH ...

Batch

Batch jobs are managed by the PBS batch scheduler, that is described in section "Batch Scheduler PBS".

On PLX it is possible to submit jobs on different queues, each identified by different resource allocation. You submit a script with the command:

> qsub script

You can get a list of defined queues with the command:

> qstat -Q

At present the following queues are defined for general scientific users (please note that the parallel queue is a routing queue: parallel jobs will be served by the longpar queue together with the longpar jobs):

job type  Max nodes  Max CPUs/GPUs    max wall time  
                         
------------------------------------------------------------------------------
debug         2          24/ 4           0:30:00        
parallel     22         264/44          24:00:00     
longpar      22         264/44          24:00:00      
archive       1           1             04:00:00           
big1          1          32/1           72:00:00            
big2          1          40/0           72:00:00            

The archive queue is not meant for computational jobs; it is a service queue for accessing data on $CINECA_DATA and archiving data on the cartridge medium (see "Batch Scheduler PBS" - archive your data: cart).

The big1 and big2 queues are reserved to use the big memory nodes.

Moreover, several "private" queues are available. They are dedicated to industrial users of the machine.

For more information and examples of job scripts, see section "Batch Scheduler PBS".

Graphic session

If a graphic session is desired we recommend to establish a VNC client/server connection with PLX. To do so, the following procedure has to be followed:

• 1 launch the VNC server on PLX
• 2 establish a ssh tunnelling from your local workstation (i.e. "client" side)
• 3 launch a VNC viewer on your local workstation (i.e. VNC client)

To launch the VNC server on the PLX machine, you can use a specific script:

> module load tightvnc
> vncserver_wrapper

please, make sure you carefully read what is printed to screen by this script! The following steps depend on the nature of your local workstation. In case it is Linux, it should be enough to carefully follow the instruction printed on screen by the vncserver_wrapper after its execution. In case you are working on a Windows workstation, please refer to the relevant document in the "Other documents" folder of this portal.

Compiler flags

Whatever your decision, in any case you need enable compiler runtime checks, by putting specific flags during the compilation phase. In the following we describe those flags for the different Fortran compilers: if you are using the C or C++ compiler, please check before because the flags may differ.

The following flags are generally available for all compilers and are mandatory for an easier debuggin session:

-O0     Lower level of optimisation
-g      Produce debugging information

Other flags are compiler specific and are described in the following.

INTEL Fortran compiler

The following flags are usefull (in addition to "-O0 -g")for debugging your code:

-traceback        generate extra information to provide source file traceback at run time
-fp-stack-check   generate extra code to ensure that the floating-point stack is in the expected state
-check bounds     enables checking for array subscript expressions
-fpe0             allows some control over floating-point exception handling at run-time

PORTLAND Group (PGI) Compilers

The following flags are usefull (in addition to "-O0 -g")for debugging your code:

-C                     Add array bounds checking
-Ktrap=ovf,divz,inv    Controls the behavior of the processor when exceptions occur: 
                       FP overflow, divide by zero, invalid operands

GNU Fortran compilers

The following flags are usefull (in addition to "-O0 -g")for debugging your code:

-Wall             Enables warnings pertaining to usage that should be avoided
-fbounds-check    Checks for array subscripts.

Debuggers available on PLX (ddt, idb, pgdbg, gdb, valgrind)

DDT (Distributed Debugging Tool - Allinea)

DDT is the most advanced debugging tool available for scalar, multi-threaded and large-scale parallel applications. C, C++, Fortran and Fortran 90/95/2003 are all supported by DDT, along with a large number of platforms, compilers and all known MPI libraries. You may also debug GPU programs by enabling CUDA support.

A short tutorial is available on the hpc portal: DDT (Distributed Debugging Tool - Allinea) tutorial

IDB (serial debugger for INTEL compilers)

The Intel Debugger (idb) is a source-level, symbolic debugger that lets you:

• Control the execution of individual source lines in a program.
• Set stops (breakpoints) at specific source lines or under various conditions.
• Change the value of variables in your program.
• Refer to program locations by their symbolic names
• Print the values of variables and set tracepoints
• Perform other functions (examining core files, examining the call stack, displaying registers)

The idb debugger has two modes: dbx (default mode) or gdb (optional mode) For complete information about idb, see the online Intel Debugger (IDB) Manual in:

$INTEL_HOME/Documentation/en_US/idb

To use this debugger, you should specify the ifort command and the debugging command-line options described above, then you run your executable inside the "idb" environment:

> module load intel
> ifort -O0 -g -traceback -fp-stack-check -check bounds -fpe0 -o myexec myprog.f90
> idb ./myexec

On PLX the debugger runs in GUI mode by default. You can also start the debugger in command line mode on these systems by specifying idbc instead of idb in the command line.

PGI: pgdbg (serial/parallel debugger)

pgdbg is the Portland Group Inc. symbolic source-level debugger for F77, F90, C, C++ and assembly language programs. It is capable of debugging applications that exhibit various levels of parallelism, including:

• Single-thread, serial applications
• Multi-threaded applications
• Distributed MPI applications
• Any combination of the above

There are two forms of the command used to invoke pgdbg. The first is used when debugging non-MPI applications, the second form, using mpirun, is used when debugging MPI applications:

> pgdbg [options] ./myexec [args]
> mpirun [options] -dbg=pgdbg ./myexec [args]

More details in the on line documentation, using the "man pgdbg" command after loading the module.

To use this debugger, you should compile your code with one of the pgi compilers and the debugging command-line options described above, then you run your executable inside the "pgdbg" environment:

> module load pgi
> pgf90 -O0 -g -C -Ktrap=ovf,divz,inv  -o myexec myprog.f90
> pgdbg ./myexec

By default, pgdbg presents a graphical user interface (GUI). A command-line interface is also provided though the "-text" option.

GNU: gdb (serial debugger)

GDB is the GNU Project debugger and allows you to see what is going on 'inside' your program while it executes -- or what the program was doing at the moment it crashed.

GDB can do four main kinds of things (plus other things in support of these) to help you catch bugs in the act:

• Start your program, specifying anything that might affect its behavior.
• Make your program stop on specified conditions.
• Examine what has happened, when your program has stopped.
• Change things in your program, so you can experiment with correcting the effects of one bug and go on to learn about another.

More details in the on line documentation, using the "man gdb" command.

To use this debugger, you should compile your code with one of the gnu compilers and the debugging command-line options described above, then you run your executable inside the "gdb" environment:

> gfortran -O0 -g -Wall -fbounds-check -o myexec myprog.f90
> gdb ./myexec

VALGRIND

Valgrind is a framework for building dynamic analysis tools. There are Valgrind tools that can automatically detect many memory management and threading bugs, and profile your programs in detail. The Valgrind distribution currently includes six production-quality tools: a memory error detector, two thread error detectors, a cache and branch-prediction profiler, a call-graph generating cache profiler, and a heap profiler.

Valgrind is Open Source / Free Software, and is freely available under the GNU General Public License, version 2.

To analyse a serial application:

1. Load Valgrind module --> module load valgrind
2. Load module for the compiler and compile your code with the compiler you prefer (Use -O0 -g flags)
3. Run the executable under Valgrind

> valgrind ./test  (or, more detailed, ...)
> valgrind --leak-check=full --show-reachable=yes --leak-resolution=high ./test

To analyse a parallel application, you need a script (mpirun-dbg) that is contained in the "cineca-tools" module :

1. Load Valgrind and cineca-tools modules

> module load valgrind
> module load cineca-tools

1. Load modules for compiler and openmpi libraries (at present only available for intel and gnu)
2. Compile your code with the "-O0 -g" flags both at compiling and linking time
3. Run the executable under Valgrind using an Interactive qsub shell

> qsub -q debug -l select=1:mpiprocs=4 -I -v LD_LIBRARY_PATH mpirun-dbg -openmpi-ib -tool valg -valgrind-args ./prova

The same flags used for the serial code must be added after the arguments "-tool valg -valgrind-args".

By default the output is written on a number of files equal to the number of requested processors.

A typical output name file is Valgrind.report.123445 where 123445 is the PID of the pbs job. For that use the flag:

--log-file=Valgrind.report

To get more details for the parallel execution of valgrind:

> module load cineca-tools
> mprun-dbg --help

Core file analysis

In order to understand what problem was affecting you code, you can also try a "Core file" analysis. Since core files are usually quite large, be sure to work in the /scratch area.

There are several steps to follow:

1. Increase the limit for possible core dumping

> ulimit -c unlimited (bash)
> limit coredumpsize unlimited (csh/tcsh)

1. If you are using Intel compilers, set to TRUE the decfort_dump_flag environment variable

> export decfort_dump_flag=TRUE  (bash)       
> setenv decfort_dump_flag TRUE  (csh/tcsh)

1. Compile your code with the debug flags described above.
2. Run your code and create the core file.
3. Analyze the core file using different tools depending on the original compiler.

INTEL compilers

> module load intel
> ifort -O0 -g -traceback -fp-stack-check -check bounds -fpe0 -o myexec prog.f90
> ulimit -c unlimited
> export decfort_dump_flag=TRUE
> ./myexec
> ls -lrt
  -rwxr-xr-x 1 aer0 cineca-staff   9652 Apr  6 14:34 myexec
  -rw------- 1 aer0 cineca-staff 319488 Apr  6 14:35 core.25629
> idbc ./myexec core.25629

PGI compilers

> module load pgi
> pgf90 -O0 -g -C -Ktrap=ovf,divz,inv  -o myexec myprog.f90
> ulimit -c unlimited
> ./myexec
> ls -lrt
  -rwxr-xr-x 1 aer0 cineca-staff   9652 Apr  6 14:34 myexec
  -rw------- 1 aer0 cineca-staff 319488 Apr  6 14:35 core.25666
> pgdbg -text -core core.25666 ./myexec 

GNU Compilers

> gfortran -O0 -g -Wall -fbounds-check -o myexec prog.f90
> ulimit -c unlimited
> ./myexec
> ls -lrt
  -rwxr-xr-x 1 aer0 cineca-staff   9652 Apr  6 14:34 myexec
  -rw------- 1 aer0 cineca-staff 319488 Apr  6 14:35 core.25555
> gdb ./myexec core.2555