In this page:

This is a base guide to learn using UNICORE Services and tools. In case of troubles please report any problem to superc@cineca.it

Introduction

UNICORE (Uniform Interface to Computing Resources) offers a ready-to-run Job submission system including client and server software. UNICORE makes distributed computing and data resources available in a seamless and secure way in intranets and the internet (https://www.unicore.eu/).

UCC (Unicore Command Line Client)

The UNICORE Commandline Client (UCC) is a full-featured client for accessing UNICORE services. It requires a Java runtime environment, version 11 or later. We recommend using the OpenJDK.

UCC offers:

  • Job submission and management
  • Data movement (upload, download, server-to-server copy, etc) using the UNICORE storage management functions and the available data transfer protocols
  • Storage functions (ls, mkdir, ...) including creation of storage instances via storage factories
  • UNICORE Workflow service support and much more.

Installation and Startup

The following instructions were taken from the UNICORE Command Line Client User guide, for more info please refer to the UNICORE official documentation.

Prerequisites

  • Operating Systems: currently Linux and Microsoft Windows are supported;
  • To run UCC, you need a Java runtime version 11 or later (OpenJDK preferred).

Installation

  1. Download the installation archive
  2. Untar the tar.gz file and copy it in your home directory or in the folder you prefer.
  3. Download the CA certificates by clicking on "Package with individual certs from CAcerts file in gz format" at this page. Copy in the folder ~/.unicore/certificates/ and untar it.
  4. Download the following certificate file unicore.hpc.cineca.it.CA.pem and add it to the certificate folder (not mandatory, but avoids "Add exception" requests at each ucc command launch).
  5. Login to UserDB using OpenID credentials and copy your OIDC token bearer visualized at this page.
  6. Configure your UCC preferences file (~/.ucc/preferences) in which you include the location of the certificate folder and insert your personal OIDC/OAuth2 token bearer.
    When finished, the file should appear as follows:



# User preferences for UCC
authentication-method=bearer-token
token=eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwi.....FkyV3pVSUFzc0RwaVprRURZV1U4Ukh3In0.e.....xLTQzMGQtOTBjZC05ZDY4OTc4ODY4MTciLCJleHAiOjE2Njc5MzY0MTEsIm5iZiI6MCw.......cLXA

# configure truststore containing trusted CA certs
truststore.type=directory

# if type is 'directory'
truststore.directoryLocations.1=/home/<user>/.unicore/certificates/*.pem
truststore.directoryEncoding=PEM

# do not check the SSL cert contains correct hostname
client.serverHostnameChecking=NONE

# The address(es) of the registries to contact
# (space separated list)

registry=https://unicore.hpc.cineca.it:9111/PRACE/services/Registry?res=default_registry
# default directory for output
output=.
protocols=UFTP BFT


After these simple steps you can use the UCC service.


Useful basic commands

  • Connect to the UCC servic

$ ./bin/ucc connect -v
[ucc connect] UCC 9.0.0, https://www.unicore.eu
[ucc connect] Reading properties file </home/michele/.ucc/preferences>
[ucc connect] Current directory is </home/michele/UNICORE/ucc/unicore-ucc-9.0.0>
[ucc connect] Output goes to <.>
[ucc connect] Registry = https://unicore.hpc.cineca.it:9111/PRACE/services/Registry?res=default_registry
[ucc connect] Using converted Registry URL https://unicore.hpc.cineca.it:9111/PRACE/rest/registries/default_registry
[ucc connect] Checking registry connection.
[ucc connect] Registry connection status: OK
[ucc connect] Current directory is </home/michele/UNICORE/ucc/unicore-ucc-9.0.0>
[ucc connect] Output goes to <.>
[ucc connect] Using site default for TSS lifetime.
[ucc connect] Connecting to https://unicore.hpc.cineca.it:9111/CINECA-G100/rest/core/factories/default_target_system_factory
[ucc connect] TSS at address https://unicore.hpc.cineca.it:9111/CINECA-G100/rest/core/sites/eb3d2edc-2912-469a-889a-cdcc5642d39e
[ucc connect] Connecting to https://unicore.hpc.cineca.it:9111/CINECA-M100/rest/core/factories/default_target_system_factory
[ucc connect] TSS at address https://unicore.hpc.cineca.it:9111/CINECA-M100/rest/core/sites/d96ec76d-215d-45b4-958b-be19333d6c73
[ucc connect] Connecting to https://unicore.hpc.cineca.it:9111/CINECA-MARCONI/rest/core/factories/default_target_system_factory


  • Lists all available clusters

$ ./bin/ucc list-sites -v
[ucc list-sites] UCC 9.0.0, https://www.unicore.eu
[ucc list-sites] Reading properties file </home/michele/.ucc/preferences>
[ucc list-sites] Current directory is </home/michele/UNICORE/ucc/unicore-ucc-9.0.0>
[ucc list-sites] Output goes to <.>
[ucc list-sites] Registry = https://unicore.hpc.cineca.it:9111/PRACE/services/Registry?res=default_registry
[ucc list-sites] Using converted Registry URL https://unicore.hpc.cineca.it:9111/PRACE/rest/registries/default_registry
[ucc list-sites] Checking registry connection.
[ucc list-sites] Registry connection status: OK
[ucc list-sites] Current directory is </home/michele/UNICORE/ucc/unicore-ucc-9.0.0>
[ucc list-sites] Output goes to <.>
[ucc list-sites] Filtering = false
[ucc list-sites] Detailed listing = false
CINECA-G100 https://unicore.hpc.cineca.it:9111/CINECA-G100/rest/core/sites/eb3d2edc-2912-469a-889a-cdcc5642d39e 
CINECA-M100 https://unicore.hpc.cineca.it:9111/CINECA-M100/rest/core/sites/d96ec76d-215d-45b4-958b-be19333d6c73 


  • List the available storage areas and their UNICORE paths

$ ./bin/ucc list-storages

https://unicore.hpc.cineca.it:9111/CINECA-M100/rest/core/storages/HOME

https://unicore.hpc.cineca.it:9111/CINECA-G100/rest/core/storages/HOME

https://unicore.hpc.cineca.it:9111/CINECA-M100/rest/core/storages/ROOT

https://unicore.hpc.cineca.it:9111/CINECA-G100/rest/core/storages/ROOT

https://unicore.hpc.cineca.it:9111/CINECA-M100/rest/core/storages/426538bf-72c2-4cd3-ac32-3907b80b94d1-uspace

https://unicore.hpc.cineca.it:9111/CINECA-G100/rest/core/storages/7befdf20-fba9-45a8-a5fe-5a92a9789ebd-uspace

https://unicore.hpc.cineca.it:9111/CINECA-M100/rest/core/storages/8ad3b031-7144-4368-8d7d-41df51fcd59f-uspace

https://unicore.hpc.cineca.it:9111/CINECA-M100/rest/core/storages/395b7fc5-ec8f-4884-87d0-75a3c4552885-uspace

https://unicore.hpc.cineca.it:9111/CINECA-M100/rest/core/storages/5c643e6d-5d47-4669-98f2-c8cf461f0559-uspace

https://unicore.hpc.cineca.it:9111/CINECA-M100/rest/core/storages/bbc372b3-3377-4a9b-995b-02aa81d2ae97-uspace

https://unicore.hpc.cineca.it:9111/CINECA-M100/rest/core/storages/41b59bab-f310-4bad-b49b-65a1328dca6e-uspace

https://unicore.hpc.cineca.it:9111/CINECA-M100/rest/core/storages/171bb395-7246-44a6-9b07-f7849d191a01-uspace


  • List files in an area

$ ./bin/ucc ls -v https://unicore.hpc.cineca.it:9111/CINECA-M100/rest/core/storages/HOME
[ucc ls] UCC 9.0.0, https://www.unicore.eu
[ucc ls] Reading properties file </home/diego/.ucc/preferences>
[ucc ls] Current directory is </home/diego/Documenti/Accesso/UNICORE/unicore-ucc-9.0.0>
[ucc ls] Output goes to <.>
[ucc ls] Registry = https://unicore.hpc.cineca.it:9111/PRACE/services/Registry?res=default_registry
[ucc ls] Using converted Registry URL https://unicore.hpc.cineca.it:9111/PRACE/rest/registries/default_registry
[ucc ls] Checking registry connection.
[ucc ls] Registry connection status: OK
[ucc ls] Current directory is </home/diego/Documenti/Accesso/UNICORE/unicore-ucc-9.0.0/bin>
[ucc ls] Output goes to <.>
/consuma_lowprio2.sh
/sbatch_example/
/venv_mpi4py/
/prova_jobs.txt
/.fltk/
/.spack/
...


  • Copy a local file ON the cluster

$ ./bin/ucc cp samples/povray.u -v https://unicore.hpc.cineca.it:9111/CINECA-M100/rest/core/storages/HOME
[ucc cp] UCC 9.0.0, https://www.unicore.eu
[ucc cp] Reading properties file </home/diego/.ucc/preferences>
[ucc cp] Current directory is </home/diego/Documenti/Accesso/UNICORE/unicore-ucc-9.0.0>
[ucc cp] Output goes to <.>
[ucc cp] Registry = https://unicore.hpc.cineca.it:9111/PRACE/services/Registry?res=default_registry
[ucc cp] Using converted Registry URL https://unicore.hpc.cineca.it:9111/PRACE/rest/registries/default_registry
[ucc cp] Checking registry connection.
[ucc cp] Registry connection status: OK
[ucc cp] Current directory is </home/diego/Documenti/Accesso/UNICORE/unicore-ucc-9.0.0/bin>
[ucc cp] Output goes to <.>
[ucc cp] Uploading local file '/home/diego/Documenti/Accesso/UNICORE/unicore-ucc-9.0.0/bin/./../samples/povray.u' -> 'https://unicore.hpc.cineca.it:9111/CINECA-M100/rest/core/storages/HOME/files//povray.u'
[ucc cp] File transfer URL : https://unicore.hpc.cineca.it:9111/CINECA-M100/rest/core/client-server-transfers/47e3c4be-e6d0-4c59-9ccf-bbbca91cded3
povray.u   
                                                            100%  336 1,8KB/s


  • copy a file locally FROM the cluster (notice the "files" folder to be added to the path)

$ ./bin/ucc cp https://unicore.hpc.cineca.it:9111/CINECA-M100/rest/core/storages/HOME/files/lista_nodi.txt -v .
[ucc cp] UCC 9.0.0, https://www.unicore.eu
[ucc cp] Reading properties file </home/diego/.ucc/preferences>
[ucc cp] Current directory is </home/diego/Documenti/Accesso/UNICORE/unicore-ucc-9.0.0>
[ucc cp] Output goes to <.>
[ucc cp] Registry = https://unicore.hpc.cineca.it:9111/PRACE/services/Registry?res=default_registry
[ucc cp] Using converted Registry URL https://unicore.hpc.cineca.it:9111/PRACE/rest/registries/default_registry
[ucc cp] Checking registry connection.
[ucc cp] Registry connection status: OK
[ucc cp] Current directory is </home/diego/Documenti/Accesso/UNICORE/unicore-ucc-9.0.0/bin>
[ucc cp] Output goes to <.>
[ucc cp] Downloading remote file 'https://unicore.hpc.cineca.it:9111/CINECA-M100/rest/core/storages/HOME/files//lista_nodi.txt' -> /home/diego/Documenti/Accesso/UNICORE/unicore-ucc-9.0.0/./lista_nodi.txt
[ucc cp] File transfer URL : https://unicore.hpc.cineca.it:9111/CINECA-M100/rest/core/client-server-transfers/806d55f9-570e-49d0-ba9a-6a1edabd0592
lista_nodi.txt                                                                                                           100%  152 279,4B/s



  • Start remote jobs via SLURM on the HPC infrastructure (need to specify the HPC cluster with -s <cluster>)

$ ./bin/ucc run -v samples/date.u -s CINECA-G100
[ucc run] UCC 9.0.0, https://www.unicore.eu
[ucc run] Reading properties file </home/michele/.ucc/preferences>
[ucc run] Current directory is </home/michele/UNICORE/ucc/unicore-ucc-9.0.0>
[ucc run] Output goes to <.>
[ucc run] Registry = https://unicore.hpc.cineca.it:9111/PRACE/services/Registry?res=default_registry
[ucc run] Using converted Registry URL https://unicore.hpc.cineca.it:9111/PRACE/rest/registries/default_registry
[ucc run] Checking registry connection.
[ucc run] Registry connection status: OK
[ucc run] Current directory is </home/michele/UNICORE/ucc/unicore-ucc-9.0.0>
[ucc run] Output goes to <.>
[ucc run] Synchronous processing = true
[ucc run] Adding job id to output file names = true
[ucc run] Dry run = false
[ucc run] Quiet mode = false
[ucc run] Read job from <samples/date.u>
[ucc run] Using broker LOCAL
[ucc run] Job is NEW
[ucc run] Checking https://unicore.hpc.cineca.it:9111/CINECA-G100/rest/core/sites/eb3d2edc-2912-469a-889a-cdcc5642d39e
[ucc run] Check requirement: Application: Date
[ucc run] Have <1> candidate resource(s)
[ucc run]   https://unicore.hpc.cineca.it:9111/CINECA-G100/rest/core/sites/eb3d2edc-2912-469a-889a-cdcc5642d39e
[ucc run] Selected TSS at https://unicore.hpc.cineca.it:9111/CINECA-G100/rest/core/sites/eb3d2edc-2912-469a-889a-cdcc5642d39e


More features will be added soon and fully documented here.

FAQ

The service fails when try to connect

It may happen that when trying to connect the command "ucc connect -v" gives the following Error

[ucc connect] Connecting to https://unicore.hpc.cineca.it:9111/CINECA-G100/rest/core/factories/default_target_system_factory
Can't create target system.
The root error was: eu.unicore.services.rest.client.RESTException: n/a [HTTP 403 Forbidden]


It is possible that you OIDC token is expired.
Please login via OpenID on UserDB and download the new OIDC token. Replace it in ~/.ucc/preferences and try again.


  • No labels