Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Children Display


When dealing with data produced or used by HPC applications, you should keep in mind two separated separate issues: where to put them and how to move them. This means "Data Storage" and "Data transfer".

Where: The management of a large amount of data (BigData) is not a trivial task: you can use a normal FileSystem (FS) based area or you should consider to describe your data  with with metadata, in order to better organize them and share among your colleagues (REPO).

How: The transfer of big data sets from or to an HPC cluster with classical tools like scp is not straightforward because various constraints could limit bandwidth and execution time.



...