You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Next »

The bulk of the "cineca-ai" package, provided by the deeplrn profile, is based on the Open Cognitive Environment (Open-CE) tool, which includes (for example) Tensorflow, Pytorch, XGBoost, and other related packages and dependencies

This cognitive environment has been personalised by CINECA AI experts and its installations have been performaed through the "spack" tool. You can find several versions of the cineca-ai module in profile/deeplrn, differing in the versions of their main components  (pytorch, tensorflow etc.). The module help reports the versioning for these components. For a complete list load the module and launch the "pip list" command. 

The CINECA AI project can be used in several ways, depending on the method more suited to your needs and on the availability of conda/pip packages.

1. Loading cineca-ai module

The way to use the installations of  cineca-ai environment goes through the loading of the module:

module load profile/deeplrn
module load autoload cineca-ai/<version>

# see all available python installations
python -m pip list

# enjoy the environment
python -c "import PACKAGE"

If you need to use a package not included in the list of those provided by the cineca-ai modules, you can always rely on the cineca-ai environment for the dependencies and install what you need within a personal virtual environment and/or a conda environment.

1.1 install additional packages within a virtual environment

If your package is available to pip:

# create the virtual env loading cineca-ai module
module load profile/deeplrn
module load cineca-ai/<version>
python -m venv <myvenv> --system-site-packages

# activate the virtual env just created and install your python packages
source <myvenv>/bin/activate
pip list
pip install PACKAGE
deactivate

# access your packages AND those of the cineca-ai environment at any time activating the created virtual env. No need to load the cineca-ai module
source <myvenv>/bin/activate
pip list
deactivate

NOTES:

  • <myvenv>: choose an arbitrary, up-to-you name for your personal virtual env
  • the --system-site-packages flag gives the virtual environment access to the system site-packages directory (otherwise you cannot access the cineca-ai environment)
  • it is advised to create your personal envs in your $WORK area, since the $HOME disk quota is limited to 50 GB
  • to test the installation launch: python -c "import PACKAGE"
  • to use the installed PACKAGE, just source your env (source <myvenv>/bin/activate): you will access your packages AND those of the cineca-ai environment, no need to load the cineca-ai module
  • if PACKAGE is not available to pip, see Section 2.2 and 2.3

Example: torch-scatter

It requires torch, which will be taken from the cineca-ai env (i.e., no need to install it):

$ module load profile/deeplrn
$ module load autoload cineca-ai/2.1.0
$ python -m venv torch_venv  --system-site-packages
$ source torch_venv/bin/activate
$ pip install torch-scatter

2. Loading spack module

An alternative way to use the installations of  cineca-ai environment to install additional packages goes through the loading of the spack module that CINECA staff used to performe them

# find 
module load spack
spack find <specific CINECA-AI PACKAGE>
e.g. spack find py-torch

spack install <your PACKAGE>
  • No labels