...
- You need to upload or download data FROM/TO your local machine TO/FROM a CINECA HPC cluster
$ sftp <username>@data.<cluster_name>.cineca.it:/absolute/remote/path/to/
sftp> put relative/local/path/to/file
Uploading /absolute/local/path/to/file to /absolute/remote/path/to/file
file 100% 414 365.7KB/s 00:00
sftp> get relative/remote/path/to/file
Fetching /absolute/remote/path/to/file to file
file 100% 1455KB 19.0MB/s 00:00
sftp> - You need to transfer files between 2 CINECA HPC clusters
$ ssh -xt <username>data.<cluster_name_1>.cineca.it sftp <username>@data.<cluster_name_2>.cineca.it:/absolute/path/to/
It is also possible to use the flag -b and execute sftp in batch mode (TOBE TESTED).
wget
Sometimes, the 10-minute CPU time limit or the 4-hour wall time limit on the serial queue are not enough to download a large dataset for ML. In this case, you can use wget from the datamover. Here you can find a simple example
...
$ globus-url-copy -vb sshftp://<username>@gftp.<cluster_name_1>.cineca.it:22/absolute/path/from/directory sshftp://<username>@gftp.<cluster_name_2>.cineca.it:22/absolute/path/to/
$ globus-url-copy -vb gsiftp://<username>@gftp.<cluster_name_1>.cineca.it:2811/absolute/path/from/directory gsiftp://<username>@gftp.<cluster_name_2>.cineca.it:2811/absolute/path/to/
CAVEAT: at present the gsiftp mode is not available on Leonardo.
Transfer data FROM/TO local machine TO/FROM a CINECA HPC cluster
...