Loading the spack module available on the cluster
When you connect to a Cineca cluster "preprod" and "prod" spack modules are available. The pre-production module should only be used when any production module of spack is not available.
$ module av spack
$ module load spack/<version>
By loading the spack module, the setup-env.sh file is sourced.
Then $SPACK_ROOT is initialised, the spack command is added to your PATH and some nice command line integration tools too.
The folder $USER/spack-<version> is created into your "default" $WORK space ("chprj -l" command to see the default work) and it contains some subfolders created and used by spack during the phase of the packages installation:
- sources cache: $WORK/$USER/spack-<version>/cache
- software installation root: $WORK/$USER/spack-<version>/install
- modulefiles location: $WORK/$USER/spack-<version>/modulefiles
- user scope: $WORK/$USER/spack-<version>/user_cache
Remember that $WORK space will be removed at the end of the corresponding project. If you want to define different paths for cache, installation, modules and user scope directories refer to spack manual.
Listing the software packages available to be installed
Check if the software package you want install is available via spack.
$ spack list <pack name> or
$ spack list|grep <pattername>
Providers
Multiple specific packages (e.g. intel-oneapi-mpi, openmpi,...) are able to provide a generic package called “virtual package” (e.g. mpi)
# List the available virtual packages
$ spack providers
# List the real packages that are able to provide a specific virtual package (e.g. intel-oneapi-mpi and openmpi packages provide “mpi” virtual package , netlib-lapack and openblas provide “lapack” virtual package)
$ spack providers <virtual package>
$ spack providers mpi
Variants an dependencies
If the package of your interest is available to be installed via spack, look at its build “variants” yon can switch in/off or customize and the “dependencies” that will be used for the building, linking and running phases and you are able to personalize.
$ spack info <package name>
Listing the software packages installed from Cineca staff by the spack module
Cineca staff has already installed through the spack module that you loaded a suite of compilers, libraries, tools and applications and you can use them to install additional software (it is strongly recommended).
# List the packages already installed
$ spack find
# Check if a specific package is already installed or what packages have been already installed to provide for example mpi virtual package
$ spack find <package name>
$ spack find <virtual package name> (e.g mpi)
# List the packages already installed and display the used variants (-v) and depedencies (-d), without and with the hash (-l)(*)
$ spack find -v <package>
$ spack find -lv <package>
$ spack find -ld <package>
# List the packages already installed with their installation path
$ spack find -lp
# List the packages already installed with a specific variant
$ spack find -l +<variant>
e.g. spack find -l +cuda
# List the packages already installed and that depend on a specific package (e.g openmpi) or a generic virtual package (e.g. mpi)
$ spack find -l ^<package name>
e.g. spack find -l ^openmpi
e.g. spack find -l ^mpi
# List the compilers already installed and ready to be used and the packages installed with a specific compiler
$ spack compilers
$ spack find %<compiler>
(*) What is the hash is explained in the next paragraph.
Installing a new package
Spec command
In order to install a package with the spack module you have to select for it a version, a compiler, the dependencies and the building variants (we’ll see how to do it in the next paragraphes) .
The combination of all these parameters is the spec with which the package will be installed.
If you don’t select any combination a default spec is selected from spack and you can know it by “spack spec” command to be launched before executing “spack install”.
# Show the default spec with which a package is installed
$ spack spec <pack name>
Each spec is identified by a unique number (e.g. lunjxl7), named “hash” and displayed by “-l” (long) option that distinguishes the several installations of a same package
# Display the hash number for a spec
$ spack spec -l <pack name>
# Example: see the different hash for openmpi%intel and openmpi%gcc
$ spack spec -l openmpi %intel
aouyzha openmpi@4.1.1%intel@2021.4.0~atomics+cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~java+legacylaunchers~lustre~memchecker~pmi+pmix~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=ucx schedulers=slurm arch=linux-centos8-cascadelake
$ spack spec -l openmpi %gcc
2ovg6ub openmpi@4.1.1%gcc@10.2.0~atomics+cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~java+legacylaunchers~lustre~memchecker~pmi+pmix~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=ucx schedulers=slurm arch=linux-centos8-cascadelake
The "spec" command also shows the installation status of the requested package and its dependencies if you use the -I option.
# Display the hash and install status of the selected packages and its dependencies
$ spack spec -Il <pack name>
The symbol “-” (not installed) or “+/^” (installed/installed from an other user), displayed by “-I” option (Install) of the spec command precedes the hash of a spec and indicates if that spec is already installed.
If the dependency spec (e.g. cuda) of the selected package (e.g openmpi) is not installed (-), it will be by default from spack during the installation of the selected package. The dependency spec installed by default from spack to meet the building requirement of the selected pack is called “implicit” installation, the selected pack “explicit” installation.
# Example: see hash and installing status of openmpi and its cuda dependency
$ spack spec -Il openmpi
- aouyzha openmpi@4.1.1%intel@2021.4.0~atomics+cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~java+legacylaunchers~lustre~memchecker~pmi+pmix~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=ucx schedulers=slurm arch=linux-centos8-cascadelake
[-] rdd7huh ^cuda@11.5.0%intel@2021.4.0~dev arch=linux-centos8-cascadelake
On Cineca clusters it’s recommended to execute always “spec” command before installing a package to make sure its dependencies are fullfilled by the cineca installations (^) if they are available. The cineca installations are optimised and tested for the architecture of the specific cluster.
If a dependency is not fullfilled by the cineca installation even if it is available, it shows “-” symbol and it will be installed with a different hash because it's a different spec.
In the following example openmpi dependency is not fullfilled by the cineca installation (-) even if it is already available (spack find openmpi %gcc@10.2.0) and it will be installed again from spack with a different combination of building values (hash xxxxxxx).
# Example: parmetis will be installed with a spec of openmpi that shows a new different hash (xxxxxxx) and has not been installed from Cineca staff (-)
$ spack spec -Il parmetis %gcc@10.2.0 ^openmpi
- oytpej7 parmetis@4.0.3%gcc@10.2.0~gdb~int64~ipo+shared build_type=RelWithDebInfo patches=4f892531eb0a807eb1b82e683a416d3e35154a455274cf9b162fb02054d11a5b,50ed2081bc939269689789942067c58b3e522c269269a430d5d34c00edbc5870,704b84f7c7444d4372cb59cca6e1209df4ef3b033bc4ee3cf50f369bce972a9d arch=linux-centos8-cascadelake
[-] xxxxxxx ^openmpi@4.1.1%gcc@10.2.0~atomics+cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~java+legacylaunchers~lustre~memchecker~pmi+pmix~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=ucx schedulers=slurm arch=linux-centos8-cascadelake
To use the cineca openmpi installation you would have to provide its spec to spack, for example through the hash that you can get by the spack command “spack find -l openmpi %gcc@10.2.0”.
# Example: parmetis will be installed with the spec installed from Cineca staff defined by the hash ov3ei7j as the command "spack find -l openmpi %gcc@10.2.0" shows
$ spack spec -Il parmetis %gcc@10.2.0 ^/ov3ei7j
- oefbg6x parmetis@4.0.3%gcc@10.2.0~gdb~int64~ipo+shared build_type=RelWithDebInfo patches=4f892531eb0a807eb1b82e683a416d3e35154a455274cf9b162fb02054d11a5b,50ed2081bc939269689789942067c58b3e522c269269a430d5d34c00edbc5870,704b84f7c7444d4372cb59cca6e1209df4ef3b033bc4ee3cf50f369bce972a9d arch=linux-centos8-cascadelake
[^] ov3ei7j ^openmpi@4.1.1%gcc@10.2.0~atomics+cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~java+legacylaunchers~lustre~memchecker~pmi+pmix~singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=ucx schedulers=slurm arch=linux-centos8-cascadelake
Install command
For a specific package you can install the default spec or a custom spec by selecting a specific version (through @ symbol), and/or a specific compiler (through % symbol), the building variants (through +,-,= symbols) and specific dependencies (through ^ symbol)
# default package installation
$ spack install pack@vers
# custom package installation
$ spack install pack@vers +/-/variant variant=<value> %compiler@version ^dependency@version
or
$ spack install pack@vers +/-var variant=<value> %comp@vers ^/<hash>
Remember to install the new package after executing “spack spec” command and use the cineca installations for the dependencies if they are available.
If you installed a new compiler, in order to use it remember to add it to your “compilers.yaml” file in the following way.
# to install a new compiler and make it available
$ spack install <compiler_package>
$ spack load <compiler_package>
$ spack compiler add
$ spack compilers
Module command
After installing a package you can use it as module.
# To create the modulefile of your installation
$ spack module tcl refresh --upstream-modules <pack name> or
$ spack module tcl refresh --upstream-modules /<spec hash>
# To use the created modulefile
$ module load spack
$ module av <package module>