...
The CINECA AI project can be used in several ways, depending on the method more suited to your needs and on the availability of conda/pip packages.
NOTE: at present the cineca-ai tool is only available in MARCONI100 environment.
1. Loading cineca-ai module
The first way to use the cineca-ai environment goes through the loading of the module:
...
If you need to use a package not included in the list of those provided by the cineca-ai modules, you can always rely on the cineca-ai environment for the dependencies and install what you need within a personal virtual environment and/or a conda environment.
1.1 install additional packages within a virtual environment
If your package is available to pip:
...
$ module load profile/deeplrn
$ module load autoload cineca-ai/2.1.0
$ python -m venv torch_venv --system-site-packages
$ source torch_venv/bin/activate
$ pip install torch-scatter
2. create a personal conda environment with selected packages from CINECA channel
Code Block |
---|
module load autoload cineca-ai/<version> conda create --prefix <mycondaenv> -y conda activate <mycondaenv> # see all packages in the channel conda search -c $CINECA_AI_CHANNEL --override-channels # install a package from the channel conda install -c $CINECA_AI_CHANNEL PACKAGE |
2.1 install additional packages within a virtual environment
Code Block |
---|
# after activating <mycondaenv> and installed the needed prerequisites of PACKAGE from CINECA channel python -m venv <myvenv> --system-site-packages source <myvenv>/bin/activate pip install PACKAGE |
2.2 install additional packages within a conda environment
Code Block |
---|
# after activating <mycondaenv> and installed the needed prerequisites of PACKAGE from CINECA channel, install the additional packages conda install PACKAGE1 conda install -c conda-forge PACKAGE2 |
...
- for some packages you need to explicit the external channel to use, e.g. conda-forge
2.3 install additional packages within a conda and a virtual environment
Code Block |
---|
# after activating <mycondaenv> and installed the needed prerequisites of PACKAGE from CINECA channel, install the additional conda packages conda install PACKAGE # create a virtual env and install the additional pip packages python -m venv <myvenv> --system-site-packages source <myvenv>/bin/activate pip install PACKAGE |
...
- <mycondaenv>: choose an arbitrary, up-to-you name for your personal conda env
- install the selected packages from the $CINECA_AI_CHANNEL
- install the additional packages (specifying specific external channels if needed, e.g. conda-forge)
- <myvenv>: choose an arbitrary, up-to-you name for your personale conda env
- the --system-site-packages flag gives the virtual environment access to the system site-packages directory (otherwise you cannot access the cineca-ai environment)
- it is advised to create both personal envs in your $WORK area, since the $HOME disk quota is limited to 50 GB
- to test the installation launch: python -c "import PACKAGE"
- to use the installed PACKAGE, just source your env (source <myvenv>/bin/activate): you will access your packages AND those of the cineca-ai installed in your conda environment, no need to load the cineca-ai module and to activate <mycondaenv>
...
$ module load autoload cineca-ai/2.1.0
$ conda create --prefix <mycondaenv> -y
$ conda activate <mycondaenv>
$ conda install -c $CINECA_AI_CHANNEL mpi4py jaxlib jax dm-tree mpi4jax
$ conda install cmake h5py -y
$ conda install -c conda-forge llvmlite>llvmlite=0.38.0 orjson -y
$ python -m venv <myvenv> --system-site-packages
$ source <myvenv>/bin/activate
$ pip install 'git+https://github.com/netket/netket.git#egg=netket[all]'
3. clone the CINECA channel in your personal channel
Code Block |
---|
conda create --prefix <mycondaenv> --clone /cineca/prod/opt/libraries/cineca-ai/<version>/none/cineca-ai-conda-env-py3.8-cuda-openmpi-11.0 -y conda activate mycondaenv # see all packages in the channel conda search -c <mycondaenv> --override-channels # install a package from your personal channel conda install -c <mycondaenv> PACKAGE |
...