...
Disks and Filesystems
The storage organisation conforms to the CINECA infrastructure (see Section "Data storage and Filesystems") . In addition to the home directory ($HOME), for each user is defined a scratch area $CINECA_SCRATCH, a large disk for storing run time data and files. The new variable $SCRATCH is also available, resolving in the same path of $CINECA_SCRATCH. $WORK is defined for each active project on the system, reserved for all the collaborators of the project. This is a safe storage area to keep run time data for the whole life of the project.
The filesystem organisation is based on LUSTRE an open source parallel file system.
| Total Dimension (TB) | Quota (GB) | Notes | |
|---|---|---|---|
| $HOME | 100T | 50 GB quota per user |
|
$CINECA_SCRATCH $SCRATCH | 1PB | no quota |
|
| $WORK | 2PB | 1TB quota per project |
|
It is also available a temporary storage local on compute nodes generated when the job starts and accessible via environment variable $TMPDIR. For more details please see the dedicated section of UG2.5: Data storage and FileSystems. On Galileo100 the $TMPDIR local area has 293 GB of available space.
$DRES points to the shared repository where Data RESources are maintained. This is a data archive area available only on-request, shared with all CINECA HPC systems and among different projects.
$DRES is not mounted on the compute nodes. This means that you can't access it within a batch job: all data needed during the batch execution has to be moved on $WORK or $CINECA_SCRATCH before the run starts.
Use the local command "cindata" to query for disk usage and quota ("cindata -h" for help):
> cindata
Modules environment
The software modules are collected in different profiles and organized by functional category (compilers, libraries, tools, applications,..).
On GALILEO100 the profiles are of two types, “domain” type (bioinf, chem-phys, lifesc,..) for the production activity and “programming” type (base and advanced) for compilation, debugging and profiling activities and that they can be loaded together.
"Base" profile is the default. It is automatically loaded after login and it contains basic modules for the programming activities (intel e gnu compilers, math libraries, profiling and debugging tools,..).
If you want to use a module placed under other profiles, for example an application module, you will have to load preventively the corresponding profile:
>module load profile/<profile name>
>module load autoload <module name>
For listing all profiles you have loaded you can use the following command:
>module list
In order to detect all profiles, categories and modules available on GALILEO100 the command “modmap” is available:
>modmap
With modmap you can see if the desired module is available and which profile you have to load to use it.
>modmap -m <module name>
Spack environment - will be available soon
In case you don't find a software you are interested in, you can install it by yourself.
In this case, on GALILEO100 we also offer the possibility to use the “spack” environment by loading the corresponding module. Please refer to the dedicated section in UG2.6: Production Environment.
...
Debuggers and Profilers
If your code aborts at runtime, there may be a problem with it. In order to solve it, you can decide to analyze the core file or to run your code using a debugger.
Compiler flags
In both cases, you need to enable compiler runtime checks, by putting specific flags during the compilation phase. In the following we describe those flags for the different Fortran compilers: if you are using the C or C++ compiler, please keep in mind that the flags may differ.
The following flags are generally available for all compilers and are mandatory for an easier debugging session:
-O0 Lower level of optimisation -g Produce debugging information
Other flags are compiler-specific and are described in the following.
INTEL Fortran compiler
The following flags are useful (in addition to "-O0 -g") for debugging your code:
-traceback generate extra information to provide source file traceback at run time -fp-stack-check generate extra code to ensure that the floating-point stack is in the expected state -check bounds enables checking for array subscript expressions -fpe0 allows some control over floating-point exception handling at run-time
GNU Fortran compilers
The following flags are useful (in addition to "-O0 -g") for debugging your code:
-Wall Enables warnings pertaining to usage that should be avoided -fbounds-check Checks for array subscripts.
GNU: gdb (serial debugger)
GDB is the GNU Project debugger and allows you to see what is going on 'inside' your program while it executes -- or what the program was doing at the moment it crashed.
GDB can do four main kinds of things (plus other things in support of these) to help you catch bugs in the act:
- Start your program, specifying anything that might affect its behavior.
- Make your program stop on specified conditions.
- Examine what has happened, when your program has stopped.
- Change things in your program, so you can experiment with correcting the effects of one bug and go on to learn about another.
More details in the online documentation, using the "man gdb" command.
To use this debugger, you should compile your code with one of the gnu compilers and the debugging command-line options described above, then you run your executable inside the "gdb" environment:
> module load gnu
> gfortran -O0 -g -Wall -fbounds-check -o myexec myprog.f90
> gdb ./myexec
Core file analysis
In order to understand what problem was affecting your code, you can also try a "Core file" analysis. Since core files are usually quite large, be sure to work in the / scratch area.
There are several steps to follow:
- Increase the limit for possible core dumping
> ulimit -c unlimited (bash) > limit coredumpsize unlimited (csh/tcsh)
- If you are using Intel compilers, set to TRUE the decfort_dump_flag environment variable
> export decfort_dump_flag=TRUE (bash) > setenv decfort_dump_flag TRUE (csh/tcsh)
- Compile your code with the debug flags described above.
- Run your code and create the core file.
- Analyze the core file using different tools depending on the original compiler.
INTEL compilers
> module load intel > ifort -O0 -g -traceback -fp-stack-check -check bounds -fpe0 -o myexec prog.f90 > ulimit -c unlimited > export decfort_dump_flag=TRUE > ./myexec > ls -lrt -rwxr-xr-x 1 aer0 cineca-staff 9652 Apr 6 14:34 myexec -rw------- 1 aer0 cineca-staff 319488 Apr 6 14:35 core.25629 > idbc ./myexec core.25629
GNU Compilers
> module load gnu
> gfortran -O0 -g -Wall -fbounds-check -o myexec prog.f90 > ulimit -c unlimited > ./myexec > ls -lrt -rwxr-xr-x 1 aer0 cineca-staff 9652 Apr 6 14:34 myexec -rw------- 1 aer0 cineca-staff 319488 Apr 6 14:35 core.25555 > gdb ./myexec core.2555
VALGRIND - will be soon available
Valgrind is a framework for building dynamic analysis tools. There are Valgrind tools that can automatically detect many memory management and threading bugs, and profile your programs in detail. The Valgrind distribution currently includes six production-quality tools: a memory error detector, two thread error detectors, a cache and branch-prediction profiler, a call-graph generating cache profiler, and a heap profiler.
Valgrind is Open Source / Free Software, and is freely available under the GNU General Public License
...