Versions Compared

Old Version 1

changes.mady.by.user Neva Besker

Saved on

compared with

New Version Current

changes.mady.by.user Neva Besker

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


Updated: 

Table of Contentschildren

Data storage resources

A complete description about of HPC data storage resources is available at this web page.

In general, CINECA storage is "user oriented": each user has its own space in the $HOME filesystem (with back-up) and some space in the $CINECA_SCRATCH filesystem (without back-up and for a short time) where to store data for all the projects he is involved in. Moreover, on demand, a $TAPE filesystem is available for saving personal data on magnetic media.

However, two "project oriented" filesystem have been introduced: WORK and DRES.

For each project (or account_no) a WORK directory is created on default and it's accessible to all the project's collaborators. For example, if you have three projects running at CINECA you'll have three separated storage area in the $WORK file system to write in.

A DRES directory can be created on request of an user. It's only-storage resource, based on GSS technology. It's characterized by:

  • an Owner (a user who owns that resource and is allowed to manage it), 
  • some possible Collaborators (users who can access the resource but not manage it)
  • a validity time, an extension and a storage type
  • some possible computational Projects (all collaborators of the project can access the resource)

Actually, three main types of DRES are available:

  • FS: a storage area consisting in a normal unix FileSystem object. 
  • REPO: a more sophysticated data repository for long-time archiving, where data are described by metadata and different security levels are available. The Data Repository (REPO) is based on iRODS technology, please refer to the link REPO for more informations. 
  • ARCHIVE: a storage area for long-time archiving that is actually maintained mainly on magnetic tape via LTFS technology.

You can ask for a DRES if you are interested in:

  • persistence of your data beyond a give project duration
  • sharing of the data among different platforms 
  • sharing of the data among different projects 

by sending an email to superc@cineca.it where you specify the type, quota and validity of the DRES you are interested in. The "owner" of a DRES can manage it, defining collaborators and participating projects, in his personal area on the UserDB. This resource will expire when running out of the defined validity, following the usual policies.

A CINECA user can ask also for a DRES (Data RESource) not depending on active projects. This kind of resource will not depend on a specific project, owner and duration will be DRES specific.

A DRES can connect more than one project to it, all projects being able to use the same data

.

Other generic information about data storage in CINECA, are available in the User Guide at the link Data storage and filesystem.

Data maintenance 

Sensible data can be kept in the $HOME area: this area is limited by a quota of some GB (enlargements can be required to the UserSupport). Data are back-upped and maintained up to 6 months after the username expiration and then archived and preserved for another year.

The results of the simulations, in particular big amount of data, should be stored in the $CINECA_SCRATCH area or in $WORK. These areas:

  • are very large: the actual size could be defined according to your project requests (check with the command "cindata")
  • has NO back-up
  • On CINECA_SCRATCH an automatic clean-up procedure could be in place: all files older then 30 days are automatically removed.
  • On WORK data are preserved up to the end of the project (removed 6 months after the project's end).

Long term data can be archived in the REPO dres-type (or Data Repository) or ARCHIVE dres-type or TAPE user space.

Use Cases

data are critical, not so large, I want to be sure ...$HOME is the right place. The only limitation is the quota limit on this area, usually several GB, you can ask to enlarge up to 50GB.
large data to be shared with collaborators of my project$WORK is the right place. Here each collaborator can have his own directory. He can open it for reading or even writing and be sure, at the same time, that data are not public. People not included in the project will not be able to access the data. Moreover, even if this filesystem is not back-upped, data are deleted only months after the project completion, so you can consider this area to be of medium security.
data to be shared with other users, not necessarily partecipating to common projects$CINECA_SCRATCH is the right place. 
data to be maintained even beyond the project. I'll use the data on CINECA hosts

$DRES repo or archive or $TAPE are the possible solutions.

data to be shared among different platforms$DRES file system

 

 

 

outgoing links:


© Copyright 2