Globus-url-copy client

myproxy-destroy -s grid.hpc.cineca.it -l <username>

NB. The proxy will destroy itself 7 days from its "init". So after this time you have to create again the proxy for a new transfer. 

Install the standard client

Once you have the valid proxy certificate, the globus-url-copy client has to be installed on the machine with your source data. On all the CINECA HPC Clusters (MARCONI, PICO, GALILEO and MARCONI) it is already available. On your local PC, if you are in a UNIX environment you can a) install the system package (RPMs and Debian package available) or b) build it from source. The former solution is the most suitable for most users and you can follow the instructions being available here http://www.ige-project.eu/downloads/software/releases/downloads. See also http://www.ige-project.eu/downloads/documents/guide/component-installation-guide for more details.

Use the standard client

Now that you have a valid proxy on the machine with the source data, you can start to transfer your data. 

To transfer file from CINECA HPC Custer to yout Local PC: 

$ globus-url-copy gsiftp://[username@]<ENDPOINT-CINECA HPC Cluster>/<remote_path/to/yourfile>   file:///home/user/<local_path/to/yourfile>

To transfer file from your local PC to CINECA HPC Custer 

$ globus-url-copy /path/to/your/local/file gsiftp://[username@]<ENDPOINT-CINECA HPC Cluster>/remote/path/ 

where the ENDPOINT-CINECA HPC Clusters are:

iRODS repository --> gftp.repo.cineca.it:2811

MARCONI machine --> gftp.marconi.cineca.it:2811

MARCONI machine for PRACE users --> gftp-prace.marconi.cineca.it:2811

GALILEO machine --> gftp.galileo.cineca.it:2811

MARCONI-100 machine --> gftp.m100.cineca.it:2811

Examples

1-  for synching recursively a directory and its subdirectories to MARCONI (like with rsync) 

$ globus-url-copy -cd -r -sync /path/to/your/dir/ gsiftp://[username@]gftp.marconi.cineca.it:2811/~/remote/dir/

2-  for moving a big chunk of data from MARCONI to PICO, the parallel option can be used

$ globus-url-copy -p 4 gsiftp://[username@]gftp.marconi.cineca.it:2811/~/path/to/file gsiftp://gftp.pico.cineca.it:2811/~/path/ 

3-  for listing the file in your directory on iRODS repository. 

$ globus-url-copy -list gsiftp://gftp.repo.cineca.it:2811/CINECA01/home/your-remote-dir/

Gridftp transfer via batch script

To perform a gridftp transfer longer than 10 cpu minutes, it is suggested to submit a job batch on the serial queue. In what follows the list of needed steps

1. Create a proxy certificate in a location available from all the nodes of the cluster (e.g. your $HOME directory)

   $ grid-proxy-init -out $HOME/proxy.cert

2. Below an example of job script:

   #!/bin/bash
   #SBATCH --err=slurm_%J.err

   #SBATCH --out=slurm_%J.out
   #SBATCH --time=04:00:00 #max time 4h
   #SBATCH --nodes=1 --ntasks-per-node=1 --cpus-per-task=1
   #SBATCH --partition=bdw_all_serial #on GALILEO the partition is gll_all_serial, on DAVIDE the partition is dvd_all_serial
   
   globus-url-copy -cred $HOME/proxy.cert <file in local position> gsiftp://<gridftp endpoint>/remote/path/