(Updated: )
The following examples show how to use rsync via command line and batch script to transfer from, to and between CINECA HPC machines. The rsync parameters are tuned for CINECA internal network.
For very large data set (>~ 500GB), CINECA'staff strongly suggests to use globus on line via GridFTP protocol
rsync via command line
You can lunch rsync via command line in the following way:
------CINECA /space1/ <-> CINECA /space2/--------------
rsync -ravzHS --progress </data_path_from/dir> <data_path_to>
rsync -avzHS --progress --sparse </data_path_from/file> <data_path_to>
------CINECA -> LOCAL/HPC machine----------------------
rsync -ravzHS --progress username@login.<hostname>.cineca.it:</data_path_from/dir> <data_path_to>
rsync -avzHS --progress --sparse username@login.<hostname>.cineca.it:</data_path_from/file> <data_path_to>
------LOCAL/HPC machine -> CINECA----------------------
rsync -ravzHS --progress <data_path_from/dir> username@login.<hostname>.cineca.it:<data_path_to>
rsync -avzHS --progress --sparse </data_path_from/file> username@login.<hostname>.cineca.it:<data_path_to>
Please note that, on CINECA's cluster, the maximum cpu time available via command line is 10 min. If your rsync connection will be killed after this time (i.e for big file >10 GB) and your transfer has not been completed ri-execute rsync command line in this way:
rsync -ravzHS --progress </data_path_from/dir> <data_path_to>
rsync -avzH --append --sparse --progress </data_path_from/file> <data_path_to>
Rsync will re-start the transfer by updating the chunck file on the destination (--append option). If necessary, repeat this rsync command line untill the data transfer will be completed. Alternatevely, you can use rsync via batch job.
rsync via batch job
Single job
If your data copy requires up to 4 hours you can lunch rsync via batch file by submitting a job to a dedicated queue.
This way allows you to have up to 4 hours of time limit for the data copy without consuming your budget. In fact, the job will run on the serial queue (login nodes).
Example:
################ serial queue 1 cpu ##########
#!/bin/bash
#SBATCH --out=job.out
#SBATCH --time=04:00:00
#SBATCH --nodes=1 --tasks-per-node=1 --cpus-per-task=1 --mem=4096
#SBATCH --account=<account name> ##you can find the name of the account by using "saldo -b" command
#SBATCH --partition=<serial queue name> ##you can find the name of the dedicated queue by using "sinfo|grep serial" command
#
cd <data_path_to>
#In order to transfer from Galileo to Marconi log into Galileo, create the public key and move it to Marconi(*)
rsync -ravzHS --progress </data_path_from/dir> username@login.marconi.cineca.it:<data_path_to>
##########################################
(*)Copy of -How to connect by a public key
On Marconi system you can use up to 4 cpus for single job and execute on each cpu an rsync command for different data.
Example:
################ serial queue 2 cpu ##########
#!/bin/bash
#SBATCH --out=%j.job.out
#SBATCH --time=04:00:00
#SBATCH --nodes=1 --tasks-per-node=1 --cpus-per-task=2 --mem=4096
#SBATCH --account=<account name> ##you can find the name of the account by using "saldo -b" command
#SBATCH --partition=<serial queue name> ##you can find the name of the dedicated queue by using "sinfo|grep serial" command
#
cd <data_path_to>
rsync -ravzHS --progress </data_path_from/dir1> . &
rsync -ravzHS --progress </data_path_from/dir2> . &
wait
########################################
Chaining multiple jobs
If your data copy requires more than 4 hours you can run concatenated jobs. Each concatenated job has up to 4 hours of time limit in order to copy the data starting from the file where the previous job was interrupted.
> sbatch job1.cmd
submitted batch job 100
> sbatch --dependency=afternotok:100 job2.cmd
submitted batch job 102
The available options for -d or --dependency are:
afterany:job_id[:jobid...], afternotok:job_id[:jobid...], afterok:job_id[:jobid...], ... etc..
See the sbatch man page for more detail.