In this page:

Data Storage architecture

All HPC systems share the same logical disk structure and file systems definition.

The available storage areas can be

• temporary (data are cancelled after a given period);
• permanent (data are never cancelled or cancelled only few months after the "end" of the project);

they can also be

• user specific (each username has a different data area);
• project specific (accessible by all users within the same project).

Finally the storage areas can be:

• Local (specific for each system);
• Shared (the same area can be accessed by all HPC systems)

The available data areas are defined through predefined "environment variables": $HOME, $CINECA_SCRATCH, $WORK.

Important: It is the user's responsibility to backup your important data. We only guarantee a daily backup of data in the $HOME area.

This is a scratch area for collaborative work within a given project. File retention is related to the life of the project. Files in $WORK will be conserved up to 6 months after the project end, then they will be cancelled. Please note that there is no back-up on this area.

This area is conceived for hosting large working data files, since it is characterized by high bandwidth of a parallel file system. It behaves very well when I/O is performed accessing large blocks of data, while it is not well suited for frequent and small I/O operations. This is the main area for maintaining scratch files resulting from batch processing.

There is one $WORK area for each active project on the machine. Default quota is 1 TB per project, but extensions can be considered by Help Desk (mailto: superc@cineca.it) if motivated. Owner of the main directory is the PI (Principal Investigator) of the project, all collaborators are allowed to read/write in there. Collaborators are advised to create a personal directory in $WORK for storing their personal files. By default the personal directory will be protected (only the owner can read/write), but protection can be easily modified, for example by allowing write permission to project collaborators through chmod command. This second approach does not affect global files security.

The (chprj - change project) command makes it easier to manage the different WORK areas for different projects, see here for details.

$CINECA_SCRATCH: temporary , user specific, local

This is a local temporary storage, like $WORK, conceived for temporary files from batch applications. There are important differences with respect to $WORK area. It is user specific (not project specific) and it can be used for sharing data with people outside your project. By default, file access is open to everyone, in case you need more restrictive protections, you can set them with chmod command.

On this area, a periodic cleaning procedure could be applied, with a normal  retention time of 40 days: files are daily cancelled by an automatic procedure if not accessed for more than 40 days. In each user's home directory ($HOME) a file lists all deleted files for a given day.

CLEAN_<yyyymmdd>.log
      <yyyymmdd> = date when files were cancelled

There is one $CINECA_SCRATCH area for each username on the machine.

$CINECA_SCRATCH does not have any disk quota. However, it is strongly recommended  to maintain a low occupancy of this area in order to prevent the very dangerous filling condition. To verify if and how the cleaning procedure is active on a given cluster, check the Mott-of-the-Day.

$DRES: permanent, shared (among platforms and projects)

This is a repository area for collaborative work among different projects and across platforms. You need to explicitly ask for this kind of resource: it does not come as part of a project (mailto: superc@cineca.it).

File retention is related to life of DRES itself. Files in DRES will be conserved up to 6 months after DRES completion, then they will be cancelled. Several types of DRES are available, at present:

• FS: normal filesystem access on high throughput disks, shared among all HPC platforms (mounted only on login nodes);
• ARCH: magnetic tape archiving with a disk-like interface via LTFS;
• REPO: smart repository based on iRODS.

This area is conceived for hosting data files to be used by several projects, in particular if you need to use them among different platforms. For example, you would need to post-process data produced on Marconi, taking advantage of the visualization environment of Galileo; or you would require a place for your data from experiments to be processed by several related projects. This filesystem is mounted only on login nodes and on the nodes of the "serial" partitions of all HPC clusters in Cineca (e.g. bdw_all_serial on Marconi, gll_all_serial on Galileo) - please recur to batch jobs on the serial partitions for the rsync transfers of great amounts of data, or to gridftp clients. As a consequence you have to move data from $DRES to $CINECA_SCRATCH or $WORK area in order them to be seen by the compute nodes.

WARNING: DRES of type ARCH have a limit in the number of inodes available: 2000 inodes for each TB of quota. This means that no more than 2000 files can be stored in 1 TB of disk space. It is then recommended that you compress your files for storage purposes, and that the dimension of each file stored should be in an average quota of 500MB. DRES of types FS and REPO do not have this limitation.

Files stored in DRES of type FS or ARCH will be moved automatically to tape storage, when specific conditions are met:

- for ARCH: files are older than 3 months and bigger than 50 MB
- for FS: files are older than 3 months and bigger than 100 MB

Data movement is transparent for the user. Only physical support changes, while logical environment will not be affected (this means that you can reach data stored in tape using same path you used for storing it in first place)

$TAPE: permanent, user specific, shared

This is an archive area conceived for saving personal data on magnetic media. The list of file is maintained on disks, the file content is moved automatically to tape using the LTFS technology. This archive space is not created by default for all users: you have to ask for it, by specifying the maximum space required (mailto: superc@cineca.it).

This filesystem is mounted on login nodes of all HPC clusters in Cineca. Files retention is related to the life of the username: data are preserved until the username remains active.

$HOME, $WORK and $CINECA_SCRATCH and $DRES environment variables are defined on all HPC Systems, and you can access these areas simply using those names:

cd $HOME
cd $CINECA_SCRATCH
cd $WORK
cd $DRES

 You are strongly encouraged to use these environment variables instead of full paths to refer to data in your scripts and codes.

 SInce storage areas are based on GPFS, you cannot use usual "quota" project to show quotas and occupancies. A command is available on all HPC systems to check the quota and the occupancy of common data areas visible from the cluster:

cindata

Option "-h" shows the help for this command.

If you are a DRES user, output of cindata command will contain similar lines:

USER       AREADESCR                          AREAID                                   FRESH    USED      QTA    USED%    aUSED     aQTA   aUSED%

myuser00   /gss/gss_work/DRES_myAcc           work_OFFLINE-DRES_myAcc-FS               9hou     2.9G       --      --%      11T      15T    73.3%

myuser00   /gss/gss_work/DRES_myAcc           work_ONLINE-DRES_myAcc-FS                9hou     1.2T       --      --%     2.8T       4T    70.0%

This may be tricky to interpret. OFFLINE area is the DRES data that has been stored on tape, after 3 months of storage (see DRES description above). ONLINE area is the DRES data still in FS or ARCH area. The total quota of storage capability assigned to your DRES is indicated in the "aQTA" parameter of the line "OFFLINE". When DRES is empty, the correspondent value of the line "ONLINE" will be the same as "OFFLINE",  but when files begin to be moved it will decrease, while the "aUSED" parameter of OFFLINE will increase accordingly. It means that you have less space that you can use to store your data, since some of the already used space has been moved to tape. Similarly, deleting offline data will decrease the aUSED parameter in OFFLINE, and increase the aQTA parameter in ONLINE of the same amount. The formula to remember is:

TOTAL DRES STORAGE = aQTA-OFF = aQTA-ON + aUSED-OFF

Backup policies

The $HOME filesystem is guaranteed by daily backups. In the following the standard backup policies are reported; in particular cases, a different agreement is possible: contact the HPC support (superc@cineca.it) for further details.

The backup procedure runs daily and we preserve a maximum of three different copies of the same file. Older versions are kept for 1 month. The last version of deleted files is kept for 2 months, then definitely removed from the backup archive.

Endianness

Endianness is the attribute of a system that indicates whether integers are represented from left to right or right to left. At present all CLusters in CIneca are "little-endian".

When moving and/or transferring your data from a big-endian system to a little-endian one, a special care must be taken: your data may need tobe converted beforehand. To this purpose, we recommend you to follow different strategies, depending on your own special needs.

• Convert to/from ASCII text files: before transferring, data files can be converted in ASCII format, that does not have an endianness issue. Once copied they can be converted back to binary to be used as inputs for jobs and/or processing.
• change/modify your code: use this approach if you have a code of your own. For example, HDF5 library can be used to manage your data to take into account endianness in I/O operations. For details see: https://www.hdfgroup.org/training/HDFtraining/UsersGuide/Fundmtls.fm3.html
• If your code is written in fortran you can follow this procedure in order to manage endianness in I/O operations on Marconi: http://www.lrz.de/services/compute/troubleshoot/#TOC7
• Some software in the domain module profiles are able to read data files generated on Fermi and therefore no need to convert them when used to start simulations from checkpoints or restart files.

Managing your data

A comprehensive discussion on how to manage your data is presented in a specific document.