Galileo will stop its production phase on November 20th 2017.
All projects and usernames active on GALILEO with their deadline beyond November 30th will be moved to MARCONI (BDW partition) on the basis of a scheduled time frames
that will be communicated in details later on this fall, during which all users should migrate to MARCONI.
All other projects, namely those with deadline earlier than November 30th 2017, will not be moved to Marconi A1 but their data and files will be accessible to the users till 6 months after the deadline of the project.
New Projects on MARCONI
All projects active on GALILEO with end date after final shutdown day will be duplicated on MARCONI.
The project budget on MARCONI will be set to the residual budget of the original project on GALILEO at July the 4th, with 1-to-1 ratio (one core-h on GALILEO = one core-h on MARCONI).
The project name and residual budget on MARCONI can be visualized with the command:
saldo -b
The users will have the possibility to run their jobs on both clusters up to final shutdown day.
How to move data from GALILEO to MARCONI
GALILEO storage is not shared with MARCONI. The user should move the data of interest by himself/herself. In this page we will post suggestions on how to move data as soon as possible. GALILEO interactive logins (login.marconi.cineca.it) and storage areas $HOME, $CINECA_SCRATCH and $WORK will remain accessible to users up to December 31st, 2017.
There are different ways to move data from Galileo cluster to Marconi cluster.
Using ftp (file transfer protocol) is not permitted on CINECA systems for security reasons, but other tools are available.
SCP secure copy protocol
scp is useful to move small amount of data. This program allows to move data, using ssh encryption, between two servers. sintax:
scp -r * myusername@login.marconi.cineca.it:/my_directory/.
In this way all files are copied from the local system current directory to "/my_directory/" of the user "myusername" on Marconi.
For further info type
man scp
RSYNC remote sync
By this utility it is possible to synchronize your local directory with a remote one.
Respect to scp, before copying it is possible to do a check of the files in the remote directory. If the file to be copied already exists with the same size, the copy is not performed.
The following examples show how to use rsync via command line and batch script to transfer from Galileo to Marconi.
Rsync requires the specification of a source and of a destination. Generic syntax:
rsync [OPTION] SRC myusername@HOST:DEST
Rsync [OPTION] myusername@HOST:SRC [DEST]
where SRC is the file or directory (or a list of multiple files and directories) to copy from, DEST is the file or directory to copy to, and square brackets indicate optional parameters.
To see the possible options please consult this documentation or type
man rsync.
Please note that, on CINECA's cluster, the maximum cpu time available via command line, is 10 min. If your rsync connection will be killed after this time (i.e for big file >10 GB) and your transfer has not been completed it is possible to ri-execute rsync command line in following example:
rsync -r -vzu -e ssh *.bin myusername@login.marconi.cineca.it:/my_directory/.
Rsync will re-start the transfer by updating the chunk file on the destination (--append option). If necessary, repeat this rsync command line until the data transfer will be completed.
For data transfers that require more than 10 minutes a good way is to launch rsync command via batch job.
Rsync via batch job
- Single step
If your data copy requires up to 4 hours you can lunch rsync via batch file by setting a single step. This way allows you to have up to 4 hours of time limit for the data copy without consuming your budget. In fact, the job will run on the archive queue (login nodes).
Example:
--------------CINECA-GALILEO <-> CINECA-MARCONI---------------
#!/bin/bash
#PBS -l walltime=4:00:00
#PBS -l select=1:mpiprocs=1
## PBS -N myjob
#PBS -o rsync$job_id.out
#PBS -e rsync$job_id.err
#PBS -q <archive || serial>
#### Load Modules
. /cineca/prod/environment/module/3.1.6/none/init/bash
module purge
## Move to directory from which you have submitted the PBS, i.e. the working dir
cd $PBS_O_WORKDIR
## Define the source (in the example Galileo scratch) and destination folder (In the example Marconi)
source=/gpfs/scratch/mydirectory/........ ## do not put the / here
dest=username@login.marconi.cineca.it:/marconi/mydirectory ## put the / here
### Launch rsync
rsync -avHS -r $source $dest > logrsync.out
rsync --timeout=600 -r -avzHS --bwlimit=80000 --block-size=1048576 --progress
$source $dest
The "-z" flag compress/decompress file to save bandwidth. For further info type
man rsync
or consult this documentation
- Multiple steps
If your data copy requires more than 4 hours you can run a multisteps job. Each step of this job has up to 4 hours of time limit and will copy the data starting from the file where the previous step was interrupted.
Example:
> qsub -W depend=afterok:JOBID.io01 job.sh (Galileo)
where JOBID is the job id (e.g. 204375) of the job you want to concatenate. There are multiple choices for the dependency (afterok, afternotok, afterany), please refer to the PBS manual (PBS Professional User Guide 13.0) .
For very large data set (>~ 500GB), CINECA'staff strongly suggests to use globus on line via GridFTP protocol
How to get familiar with MARCONI
MARCONI is an HPC cluster, made of three different partitions: A1 (based on Broadwell nodes), A2 (based on Knightslanding nodes) and A3 (based on Skylake nodes, to be installed this summer). The HPC Users' Guide for MARCONI can be found at the following link:
https://wiki.u-gov.it/confluence/display/SCAIUS/UG3.1%3A+MARCONI+UserGuide
Users migrating from GALILEO will get access to A1 partition only.
On MARCONI the latest releases of Intel and Gnu compilers and related libraries are available. Users will need to re-compile their own codes before using them on MARCONI compute nodes, in order to exploit the advanced features of the new hardware.
MARCONI environment is similar to the one of GALILEO. The number of cores/node is different: 16 cores on Galileo and 36 on Marconi. The internal network is different too (Omnipath network on Marconi), but the performance should be comparable. For more info please check our on-line documentation.
Pay attention to the different "module" environment on MARCONI. On Marconi a new feature has been added to the module environment: the profiles are of two types, “domain” type (chem, phys, lifesc,..) for the production activity and “programming” type (base and advanced) for compilation, debugging and profiling activities, and they can be loaded together. Please, see the on-line guide: MARCONI UserGuide. All the applications available on GALILEO should be present also on MARCONI.
Sometimes on MARCONI are present newer releases than on GALILEO and we do not plan to install older versions.
If the application you are interested in is not present on Marconi, please feel free to ask superc@cineca.it.
The present document is upgraded continuously. Come back here often!