Conda¶

Conda is a package and environment manager popular especially for Python and R. It lets you create isolated environments with exactly the package versions your project needs, without root access and without affecting other users.

Everything described on this page is done in your own home directory, or in a project directory if you have one. You do not need any help from the cluster administrators.

We recommend using Miniforge, a minimal Conda distribution that uses the community conda-forge channel by default and includes the faster mamba solver.

Warning

The default channels of Anaconda and Miniconda are subject to the Anaconda Terms of Service, which require a paid license for use in larger organizations. Use Miniforge with the free conda-forge and bioconda channels instead.

Tip

Before creating your own environment, check whether the software is already available as a module using module spider <name>. Modules installed with EasyBuild are optimized for our cluster and do not take space in your home directory.

Note

The cluster also provides Anaconda3 and Miniconda3 modules (module spider Anaconda3, module spider Miniconda3). They use the Anaconda default channels, so if you decide to use them, follow the Configuration section below and remove the default channel with conda config --remove channels defaults. Your own Miniforge installation is still recommended, because you fully control its version and packages.

Installing Miniforge¶

Install Miniforge once into your home directory on the login node.

wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh
bash Miniforge3-Linux-x86_64.sh -b -p ${HOME}/miniforge3      # Install without prompts
rm Miniforge3-Linux-x86_64.sh

To make the conda command available in your current shell, run:

source ${HOME}/miniforge3/etc/profile.d/conda.sh

Configuration¶

Conda reads its settings from the ~/.condarc file in your home directory. We recommend the following settings, so packages are installed from the free community channels in a consistent order.

conda config --add channels bioconda           # Bioinformatics packages
conda config --add channels conda-forge        # Community packages (highest priority)
conda config --set channel_priority strict     # Prefer packages from higher-priority channels
conda config --set auto_activate_base false    # Do not activate base environment automatically

You can check your current configuration with conda config --show-sources.

Tip

If your home directory is running out of space, you can store environments and the package cache in your personal directory on /project, if you have one assigned:

conda config --add envs_dirs /project/${USER}/conda/envs
conda config --add pkgs_dirs /project/${USER}/conda/pkgs

Note

We do not recommend running conda init, which automatically activates the base environment in every shell. The activated environment then also applies to your jobs and can conflict with modules loaded by Lmod. If you have already run it, undo it with conda init --reverse.

Creating Environment¶

Create a new environment with the packages you need. Specifying the Python version and the packages at once lets Conda resolve all dependencies together.

conda create -n myenv python=3.12 numpy pandas scikit-learn    # Create environment
conda activate myenv                                           # Activate environment
python --version                                               # Use the environment
conda deactivate                                               # Deactivate environment

Bioinformatics tools are available in the bioconda channel, which is already enabled by the configuration above:

conda create -n bioenv samtools bwa fastqc

Tip

mamba is a faster drop-in replacement for conda included in Miniforge. If resolving dependencies takes too long, use mamba create ... or mamba install ... with the same arguments.

Environment File¶

For reproducible research, describe your environment in an environment.yml file and keep it together with your code.

Example content of environment.yml file:

name: myenv
channels:
  - conda-forge
  - bioconda
dependencies:
  - python=3.12
  - numpy
  - pandas
  - samtools=1.21
  - pip
  - pip:
      - my-pip-only-package

conda env create -f environment.yml                 # Create environment from file
conda env update -f environment.yml --prune         # Update environment after changing the file
conda env export --from-history > environment.yml   # Export packages you installed explicitly

Managing Environments¶

conda env list                          # List your environments
conda list -n myenv                     # List packages in environment
conda install -n myenv matplotlib       # Install a package into environment
conda remove -n myenv --all             # Delete environment
conda clean --all                       # Remove cached package files

Note

Conda environments contain tens of thousands of small files and the package cache grows quickly. Run conda clean --all regularly and remove environments you no longer need.

Shared Project Environment¶

To share one environment with all members of a project, create it in the project directory using --prefix instead of a name.

conda create --prefix /project/my_project/envs/myenv python=3.12 numpy
conda activate /project/my_project/envs/myenv

Using Environment In Job¶

Create and update environments on the login node, then only activate them in your jobs. Because conda init is not used, load Conda explicitly at the beginning of the job script.

Example¶

#!/bin/bash
#PBS -N conda_job                                   # Job name
#PBS -l select=1:ncpus=8:mem=16gb                   # One node, 8 cores, 16 GB RAM
#PBS -l walltime=02:00:00                           # Max run time (hh:mm:ss)

cd ${PBS_O_WORKDIR}                                 # Enter submission directory

module reset                                        # Reset modules to avoid conflicts
source ${HOME}/miniforge3/etc/profile.d/conda.sh    # Make conda command available
conda activate myenv                                # Activate environment

python my_script.py                                 # Run your application

This example activates the myenv environment on the compute node and runs my_script.py with the Python interpreter and packages installed in that environment.

Warning

Many Python libraries such as NumPy or PyTorch start as many threads as there are cores on the node, not as many as your job requested. Limit them to the requested cores, e.g. by adding export OMP_NUM_THREADS=${NCPUS} and export MKL_NUM_THREADS=${NCPUS} to your job script.

Interactive Work¶

For testing and interactive data analysis, activate the environment inside an Interactive Job instead of on the login node.

qsub -I -l select=1:ncpus=4:mem=8gb -l walltime=02:00:00
source ${HOME}/miniforge3/etc/profile.d/conda.sh
conda activate myenv
python

Conda In Containers¶

If your environment is very large or you need to run it on another system as well, consider packing it into a Singularity container. A container is a single file, which is faster to load than thousands of small files and is fully portable.

Example content of conda_container.def file:

Bootstrap: docker
From: condaforge/miniforge3:latest

%files
    environment.yml /opt/environment.yml

%post
    conda env update -n base -f /opt/environment.yml
    conda clean --all --yes

%runscript
    exec python "$@"

singularity build conda_container.sif conda_container.def
singularity run conda_container.sif my_script.py