Page tree
Skip to end of metadata
Go to start of metadata

(updated: Mar 21, 2016 10:55)


When dealing with data produced by HPC applications, you should keep in mind two separated issues: where to put them and how to move them. This means "Data Storage" and "Data transfer".

Where: The management of large amount of data (BigData) is not a trivial task: you can use a normal FileSystem (FS) based area or you should consider to describe your data  with metadata, in order to better organize them and share among your colleagues (REPO).

How: The transfer of big data sets from or to an HPC cluster with classical tools like scp is not straightforward because various constraints could limit bandwidth and execution time.



 

  • No labels