The bulk of the cineca-ai package, provided by the deeplrn profile, is based on the Open Cognitive Environment (Open-CE) tool, which includes (for example) Tensorflow, Pytorch, XGBoost, and other related packages and dependencies.
This cognitive environment has been personalised by CINECA AI experts and will be published in a public channel. You can find several versions of the cineca-ai module in profile/deeplrn, differing in the versions of their main components (pytorch, tensorflow etc.). The module help reports the versioning for these components. For a complete list load the module and launch the "pip list" command.
The CINECA AI project can be used in several ways, depending on the method more suited to your needs and on the availability of conda/pip packages.
The first way to use the cineca-ai environment goes through the loading of the module:
module load profile/deeplrn module load autoload cineca-ai/<version> # see all available packages pip list # enjoy the environment |
If you prefer working in a conda environment, you can activate the cineca-ai env via "conda activate $CINECA_AI_ENV", and explore the environment with "conda list".
If you need to use a package not included in the list of those provided by the cineca-ai modules, you can always rely on the cineca-ai environment for the dependencies and install what you need within a personal virtual environment and/or a conda environment.
If your package is available to pip:
module load profile/deeplrn module load autoload cineca-ai/<version> python -m venv <myvenv> --system-site-packages source <myvenv>/bin/activate pip install PACKAGE |
NOTES:
Example: torch-scatter
It requires torch, which will be taken from the cineca-ai env (i.e., no need to install it):
$ module load profile/deeplrn
$ module load autoload cineca-ai/2.1.0
$ python -m venv torch_venv --system-site-packages
$ source torch_venv/bin/activate
$ pip install torch-scatter
module load autoload cineca-ai/<version> conda create --prefix <mycondaenv> -y conda activate <mycondaenv> # see all packages in the channel conda search -c $CINECA_AI_CHANNEL --override-channels # install a package from the channel conda install -c $CINECA_AI_CHANNEL PACKAGE |
# after activating <mycondaenv> and installed the needed prerequisites of PACKAGE from CINECA channel python -m venv <myvenv> --system-site-packages source <myvenv>/bin/activate pip install PACKAGE |
# after activating <mycondaenv> and installed the needed prerequisites of PACKAGE from CINECA channel, install the additional packages conda install PACKAGE1 conda install -c conda-forge PACKAGE2 |
NOTES:
# after activating <mycondaenv> and installed the needed prerequisites of PACKAGE from CINECA channel, install the additional conda packages conda install PACKAGE # create a virtual env and install the additional pip packages python -m venv <myvenv> --system-site-packages source <myvenv>/bin/activate pip install PACKAGE |
NOTES:
Example: netket
It requires some of the conda packages provided by the cineca-ai environment, and netket is only available via pip.
$ module load autoload cineca-ai/2.1.0
$ conda create --prefix <mycondaenv> -y
$ conda activate <mycondaenv>
$ conda install -c $CINECA_AI_CHANNEL mpi4py jaxlib jax dm-tree mpi4jax
$ conda install cmake h5py -y
$ conda install -c conda-forge llvmlite=0.38.0 orjson -y
$ python -m venv <myvenv> --system-site-packages
$ source <myvenv>/bin/activate
$ pip install 'git+https://github.com/netket/netket.git#egg=netket[all]'
conda create --prefix <mycondaenv> --clone /cineca/prod/opt/libraries/cineca-ai/<version>/none/cineca-ai-conda-env-py3.8-cuda-openmpi-11.0 -y conda activate mycondaenv # see all packages in the channel conda search -c <mycondaenv> --override-channels # install a package from your personal channel conda install -c <mycondaenv> PACKAGE |