...
In this page:
| Table of Contents |
|---|
...
Data Storage architecture
All HPC systems share the same logical disk structure and file systems definition.
...
The available data areas are defined through predefined "environment variables": $HOME, $CINECA_SCRATCH, $WORK.
$HOME: permanent/backed up, user specific, local
This is a local area where you are placed after the login procedure. It is where system and user applications store their dot-files and dot-directories (.nwchemrc, .ssh, ...) and where users keep initialization files specific for the systems (.cshrc, .profile, ...). There is a $HOME area for each username on the machine.
This area is conceived to store programs and small personal data. Files are never deleted from this area, moreover they are guaranteed by daily backups: if you delete or accidentally overwrite a file, you can ask our Help Desk (superc@cineca.it) to restore it. File retention is related to the life of the username, data are preserved until the username remains active.
$WORK: permanent, project specific, local
This is a scratch area for collaborative work within a given project. File retention is related to the life of the project. Files in $WORK will be conserved up to 6 months after the project end, then they will be cancelled. Please note that there is no back-up on this area.
...
A newly defined command (chprj - change project) makes it easier to manage the different WORK areas for different projects, see here for details.
$CINECA_SCRATCH: temporary , user specific, local
This is a local temporary storage, like $WORK, conceived for temporary files from batch applications. Main difference with respect to $WORK is that it is user specific (not project specific) and that it can be used for sharing data with people outside your project. By default, file access is open to everyone, in case you need more restrictive protections, you can set them with chmod command.
...
$CINECA_SCRATCH does not have any disk quota. However, it is strongly recommended to maintain a low occupancy of this area in order to prevent the very dangerous filling condition. To verify if and how the cleaning procedure is active on a given cluster, check the Mott-of-the-Day.
$TAPE: permanent, user specific, shared
This is an archive area conceived for saving personal data on magnetic media. The list of file is maintained on disks, the file content is moved automatically to tape using the LTFS technology. This archive space is not created by default for all users: you have to ask for it, by specifying the maximum space required (mailto: superc@cineca.it).
This filesystem is mounted on login nodes of all HPC clusters in Cineca. Files retention is related to the life of the username: data are preserved until the username remains active.
$DRES: permanent, shared (among platforms and projects)
This is a repository area for collaborative work among different projects and across platforms. You need to explicitly ask for this kind of resource: it does not come as part of a project (mailto: superc@cineca.it).
...
This policy may be subject to change: updates will be notified as soon as possible.
...
...
$HOME, $WORK and $CINECA_SCRATCH and $DRES environment variables are defined on all HPC Systems, and you can access these areas simply using those names:
| UI Text Box |
|---|
cd $HOME |
...
You are strongly encouraged to use these environment variables instead of full paths to refer to data in your scripts and codes.
...
TOTAL DRES STORAGE = aQTA-OFF = aQTA-ON + aUSED-OFF
Endianness
Endianness is the attribute of a system that indicates whether integers are represented from left to right or right to left. At present all CLusters in CIneca are "little-endian".
...
- Convert to/from ASCII text files: before transferring, data files can be converted in ASCII format, that does not have an endianness issue. Once copied they can be converted back to binary to be used as inputs for jobs and/or processing.
- change/modify your code: use this approach if you have a code of your own. For example, HDF5 library can be used to manage your data to take into account endianness in I/O operations. For details see: https://www.hdfgroup.org/training/HDFtraining/UsersGuide/Fundmtls.fm3.html
- If your code is written in fortran you can follow this procedure in order to manage endianness in I/O operations on Marconi: http://www.lrz.de/services/compute/troubleshoot/#TOC7
- Some software in the domain module profiles are able to read data files generated on Fermi and therefore no need to convert them when used to start simulations from checkpoints or restart files.
Managing your data
A comprehensive discussion on how to manage your data is presented in a specific document.
...
...
Outgoing links: