In this page:
| Table of Contents |
|---|
...
Data Storage architecture
All HPC systems share the same logical disk structure and file systems definition.
...
Important: It is the user's responsibility to backup your important data. We only guarantee a daily backup of data in the $HOME area.
$HOME: permanent/backed up, user specific, local
$HOME is a local area where you are placed after the login procedure. It is where system, and user applications store their dot-files and dot-directories (.nwchemrc, .ssh, ...) and where users keep initialization files specific for the systems (.cshrc, .profile, ...). There is a $HOME area for each username on the machine.
This area is conceived to store programs and small personal data. It has a quota of 50 GB. Files are never deleted from this area. Moreover, they are guaranteed by daily backups: if you delete or accidentally overwrite a file, you can ask our Help Desk (superc@cineca.it) to restore it. A maximum of 3 versions of each file is stored as a backup. The last version of the deleted file is kept for two months, then definitely removed from the backup archive. File retention is related to the life of the username; data are preserved until the username remains active.
$WORK: permanent, project specific, local
$WORK is a scratch area for collaborative work within a given project. File retention is related to the life of the project. Files in $WORK will be conserved up to 6 months after the project end, and then they will be cancelled. Please note that there is no back-up in this area.
...
The (chprj - change project) command makes it easier to manage the different WORK areas for different projects, see here for details.
$CINECA_SCRATCH: temporary , user specific, local
This is local temporary storage, like $WORK, conceived for temporary files from batch applications. There are important differences with respect to $WORK area. It is user specific (not project specific), and it can be used for sharing data with people outside your project. By default, file access is open to everyone, in case you need more restrictive protections, you can set them with chmod command.
...
To verify if and how the cleaning procedure is active on a given cluster, check the Mott-of-the-Day.
$TMPDIR: temporary, user specific, local
Each compute node is equipped with a local storage whose dimension differs depending on the cluster (please look at the specific page of the cluster for more details).
When a job starts, a temporary area is defined on the storage local to each compute node:
...
However, the directory is removed at the job's end; hence always remember to save the data stored in such area to a permanent directory in your sbatch script at the end of the run. Please note that the area is located on local disks, so it can be accessed only by the processes running on the specific node. For multinode jobs, if you need all the processes to access some data, please use the shared filesystems $HOME, $WORK, $CINECA_SCRATCH.
$DRES: permanent, shared (among platforms and projects)
This is a repository area for collaborative work among different projects and across platforms. You need to explicitly ask for this kind of resource: it does not come as part of a project (mailto: superc@cineca.it).
...
The Data movement is transparent for the user. Only physical support changes, while the logical environment will not be affected (this means that you can reach data stored in tape using the same path you used for storing it in the first place)
...
Backup policies
Daily backups guarantee the $HOME filesystem. In particular cases, a different agreement is possible: contact the HPC support (superc@cineca.it) for further details.
The backup procedure runs daily, and we preserve a maximum of three different copies of the same file. Older versions are kept for 1 month. The last version of deleted files is kept for 2 months, then definitely removed from the backup archive.
Environment variables
$HOME, $WORK,$CINECA_SCRATCH ($SCRATCH on G100) and $DRES environment variables are defined on all HPC Systems, and you can access these areas simply using those names:
...
You are strongly encouraged to use these environment variables instead of full paths to refer to your scripts and codes data.
Summary
| $CINECA_SCRATCH | $WORK | $DRES |
|---|---|---|
Created when username has granted access. | Created when a project is opened. | Created on request. |
A clean-up procedure is active. | Data are preserved up to few months after the end of the project. | Data are preserved up few months after the expiring date No backup. |
No quota. | Default quota of 1 TB. Motivated requests for quota increase will be taken into account. | Quota based on the needs. |
| By default files are public (read only). The user can change the permission (chmod) and make files private. It is not possible to restrict access to the group (all usernames share the same mail unix group). | By default files are private. The user can change the permission (chmod) and make files visible (R o R/W) to project collaborators. | Same as $WORK area. |
What to use when...
| Data are critical, not so large, I want to be sure to preserve them. | $HOME is the right place. Pay attention to the 50 GB quota limit for each user on this space. |
| Large data to be shared with collaborators of my project. | $WORK is the right place. Here each collaborator can have his own directory. He can open it for reading or even writing and be sure, at the same time, that data are not public. People not included in the project will not be able to access the data. Moreover, data are preserved up to a few months after the project’s end. |
| Data to be shared with other users, not necessarily participating in common projects | $CINECA_SCRATCH is the right place. |
| Data to be maintained even beyond the end of the project. I'll use the data on different CINECA hosts | $DRES is a better solution. |
| Data to be shared among different platforms | $DRES allows this. |
Monitoring the occupancy
A command is available on all HPC systems to check the status of the occupancy of all the areas accessible by the user and eventually the presence of a quota limit.
...
The tool is available in the module cintools, which is automatically loaded in your environment. However, the module can be unloaded as all the other modules (Modules).
File permissions
$WORK and $DRES are environmental variables automatically set in the user environment.
...
Since the $WORK/$DRES fileset is closed to non collaborators, the data sharing is active only among the project collaborators.
Pointing $WORK to a different project: the chprj command
The user can modify the project pointed to by the variable $WORK using the "change project" command.
...
More details are in the help page of the command:
chprj -h
chprj --help
Endianness
Endianness is the attribute of a system that indicates whether integers are represented from left to right or right to left. At present, all CLusters in Cineca are "little-endian".
Managing your data
A comprehensive discussion on how to manage your data can be found in a specific document.
...