...
Debuggers and Profilers
If your code aborts at runtime, there may be a problem with it. In order to solve it, you can decide to analyze the core file (feature not available if the code is compiled with PGI) or to run your code using a debugger.
Compiler flags
In both cases, you need to enable compiler runtime checks, by putting specific flags during the compilation phase. In the following we describe those flags for the different Fortran compilers: if you are using the C or C++ compiler, please keep in mind that the flags may differ.
The following flags are generally available for all compilers and are mandatory for an easier debugging session:
-O0 Lower level of optimisation -g Produce debugging information
Other flags are compiler-specific and are described in the following.
INTEL Fortran compiler
The following flags are useful (in addition to "-O0 -g") for debugging your code:
-traceback generate extra information to provide source file traceback at run time -fp-stack-check generate extra code to ensure that the floating-point stack is in the expected state -check bounds enables checking for array subscript expressions -fpe0 allows some control over floating-point exception handling at run-time
GNU Fortran compilers
The following flags are useful (in addition to "-O0 -g") for debugging your code:
-Wall Enables warnings pertaining to usage that should be avoided -fbounds-check Checks for array subscripts.
PGI: pgdbg (serial/parallel debugger)
pgdbg is the Portland Group Inc. symbolic source-level debugger for F77, F90, C, C++ and assembly language programs. It is capable of debugging applications that exhibit various levels of parallelism, including:- Single-thread, serial applications
- Multi-threaded applications
- Distributed MPI applications
- Any combination of the above
There are two forms of the command used to invoke pgdbg. The first is used when debugging non-MPI applications, the second form, using mpirun, is used when debugging MPI applications:
> pgdbg [options] ./myexec [args]
> mpirun [options] -dbg=pgdbg ./myexec [args]
More details in the on line documentation, using the "man pgdbg" command after loading the module.
To use this debugger, you should compile your code with one of the pgi compilers and the debugging command-line options described above, then you run your executable inside the "pgdbg" environment:
> module load profile/advanced
> module load pgi > pgf90 -O0 -g -C -Ktrap=ovf,divz,inv -o myexec myprog.f90 > pgdbg ./myexec
By default, pgdbg presents a graphical user interface (GUI). A command-line interface is also provided through the "-text" option.GNU: gdb (serial debugger)
GDB is the GNU Project debugger and allows you to see what is going on 'inside' your program while it executes -- or what the program was doing at the moment it crashed.
GDB can do four main kinds of things (plus other things in support of these) to help you catch bugs in the act:
- Start your program, specifying anything that might affect its behavior.
- Make your program stop on specified conditions.
- Examine what has happened, when your program has stopped.
- Change things in your program, so you can experiment with correcting the effects of one bug and go on to learn about another.
More details in the online documentation, using the "man gdb" command.
To use this debugger, you should compile your code with one of the gnu compilers and the debugging command-line options described above, then you run your executable inside the "gdb" environment:
> module load gnu
> gfortran -O0 -g -Wall -fbounds-check -o myexec myprog.f90
> gdb ./myexec
Core file analysis
In order to understand what problem was affecting your code, you can also try a "Core file" analysis. Since core files are usually quite large, be sure to work in the /scratch area.
There are several steps to follow:
- Increase the limit for possible core dumping
> ulimit -c unlimited (bash) > limit coredumpsize unlimited (csh/tcsh)
- If you are using Intel compilers, set to TRUE the decfort_dump_flag environment variable
> export decfort_dump_flag=TRUE (bash) > setenv decfort_dump_flag TRUE (csh/tcsh)
- Compile your code with the debug flags described above.
- Run your code and create the core file.
- Analyze the core file using different tools depending on the original compiler.
INTEL compilers
> module load intel > ifort -O0 -g -traceback -fp-stack-check -check bounds -fpe0 -o myexec prog.f90 > ulimit -c unlimited > export decfort_dump_flag=TRUE > ./myexec > ls -lrt -rwxr-xr-x 1 aer0 cineca-staff 9652 Apr 6 14:34 myexec -rw------- 1 aer0 cineca-staff 319488 Apr 6 14:35 core.25629 > idbc ./myexec core.25629
GNU Compilers
> module load gnu
> gfortran -O0 -g -Wall -fbounds-check -o myexec prog.f90 > ulimit -c unlimited > ./myexec > ls -lrt -rwxr-xr-x 1 aer0 cineca-staff 9652 Apr 6 14:34 myexec -rw------- 1 aer0 cineca-staff 319488 Apr 6 14:35 core.25555 > gdb ./myexec core.2555
VALGRIND
Valgrind is a framework for building dynamic analysis tools. There are Valgrind tools that can automatically detect many memory management and threading bugs, and profile your programs in detail. The Valgrind distribution currently includes six production-quality tools: a memory error detector, two thread error detectors, a cache and branch-prediction profiler, a call-graph generating cache profiler, and a heap profiler.
Valgrind is Open Source / Free Software, and is freely available under the GNU General Public License, version 2.
To analyse a serial application:
- Load Valgrind module --> module load valgrind
- Load module for the compiler and compile your code with the compiler you prefer (Use -O0 -g flags)
- Run the executable under Valgrind.
If you normally run your program like this:
myprog arg1 arg2
Use this command line:
valgrind (valgrind-options) myprog arg1 arg2
Memcheck is the default tool. You can add the --leak-ceck option that turns on the detailed memory leak detector. Your program will run much slower than normal, and use a lot more memory. Memcheck will issue messages about memory errors and leaks that it detects.
- Load Valgrind module --> module load valgrind
- Load modules for compiler and openmpi libraries (at present only available for intel and gnu)
- Compile your code with the "-O0 -g" flags both at compiling and linking time
- Run the executable under Valgrind (both in interactive than in batch mode)
mpirun -np 4 valgrind (valgrind-options) myprog arg1 arg2
...