Globus-url-copy client

myproxy-destroy -s grid.hpc.cineca.it -l <username>

NB. The proxy will destroy itself 7 days from its "init". So after this time you have to create again the proxy for a new transfer. 

Use the standard client

Now that you have a valid proxy on the machine with the source data, you can start to transfer your data bby using the standard client globus-url-copy. Please note that the client is already available on CINECA HPC clusters.

To transfer file from CINECA HPC Custer to yout Local PC: 

$ globus-url-copy gsiftp://[username@]<ENDPOINT-CINECA HPC Cluster>/<remote_path/to/yourfile>   file:///home/user/<local_path/to/yourfile>

To transfer file from your local PC to CINECA HPC Custer 

$ globus-url-copy /path/to/your/local/file gsiftp://[username@]<ENDPOINT-CINECA HPC Cluster>/remote/path/ 

where the ENDPOINT-CINECA HPC Clusters are:

iRODS repository --> gftp.repo.cineca.it:2811

MARCONI machine --> gftp.marconi.cineca.it:2811

MARCONI machine for PRACE users --> gftp-prace.marconi.cineca.it:2811

GALILEO-100 machine --> gftp.g100.cineca.it:2811

MARCONI-100 machine --> gftp.m100.cineca.it:2811

Examples

1-  for synching recursively a directory and its subdirectories to MARCONI (like with rsync) 

$ globus-url-copy -cd -r -sync /path/to/your/dir/ gsiftp://[username@]gftp.marconi.cineca.it:2811/~/remote/dir/

2-  for moving a big chunk of data from MARCONI to PICO, the parallel option can be used

$ globus-url-copy -p 4 gsiftp://[username@]gftp.marconi.cineca.it:2811/~/path/to/file gsiftp://gftp.pico.cineca.it:2811/~/path/ 

3-  for listing the file in your directory on iRODS repository. 

$ globus-url-copy -list gsiftp://gftp.repo.cineca.it:2811/CINECA01/home/your-remote-dir/

Gridftp transfer via batch script

To perform a gridftp transfer longer than 10 cpu minutes, it is suggested to submit a job batch on the serial queue. In what follows the list of needed steps

1. Create a proxy certificate in a location available from all the nodes of the cluster (e.g. your $HOME directory)

   $ grid-proxy-init -out $HOME/proxy.cert

2. Below an example of job script:

   #!/bin/bash
   #SBATCH --err=slurm_%J.err

   #SBATCH --out=slurm_%J.out
   #SBATCH --time=04:00:00 #max time 4h
   #SBATCH --nodes=1 --ntasks-per-node=1 --cpus-per-task=1
   #SBATCH --partition=bdw_all_serial #on GALILEO the partition is gll_all_serial, on DAVIDE the partition is dvd_all_serial
   
   globus-url-copy -cred $HOME/proxy.cert <file in local position> gsiftp://<gridftp endpoint>/remote/path/

GridFTP TCP Port Range configuration

Please note that GridFTP servers on our clusters are configured to use the port range 20000 - 25000 for the incoming and outgoing connections.