Python

Python is the most widely used language for data analysis, machine learning and scientific scripting. On our cluster, you can use Python from a module or set up your own Python with Conda. Packages are installed into a virtual environment in your home or project directory, without any help from the cluster administrators.

Approach	Best for
Python module + virtual environment	Pure Python packages installed with pip.
Conda	Packages with compiled dependencies, specific Python versions, mixing Python and R.
Singularity	Complex software stacks, e.g. deep learning frameworks, or ready-made images.

Loading Python

List the available Python versions and load the one you need.

module spider Python            # List available Python versions
module load Python/<version>    # Load a specific version
python --version                # Check loaded version

Tip

Always load the same Python version when creating a virtual environment and when using it in your jobs. A virtual environment created with one Python version does not work with another.

Virtual Environment

A virtual environment is an isolated directory with its own Python packages. Create one environment per project, so packages of different projects do not conflict.

module load Python/<version>                        # Load Python
python -m venv ${HOME}/venvs/myproject              # Create virtual environment
source ${HOME}/venvs/myproject/bin/activate         # Activate it
pip install --upgrade pip                           # Upgrade pip in the environment
pip install numpy pandas scikit-learn               # Install packages
deactivate                                          # Deactivate environment

Warning

Do not use pip install --user. Packages installed into ~/.local are visible to every Python with the same version, including Conda environments and containers, and often cause hard-to-find conflicts. Always install packages into a virtual environment.

Requirements File

For reproducible research, record the packages of your environment in a requirements.txt file and keep it together with your code.

pip freeze > requirements.txt                       # Save installed packages and versions
pip install -r requirements.txt                     # Install packages from file

Using Environment In Job

Create the environment and install packages on the login node, then only activate it in your jobs.

Example

#!/bin/bash
#PBS -N python_job                                  # Job name
#PBS -l select=1:ncpus=8:mem=16gb                   # One node, 8 CPUs, 16 GB RAM
#PBS -l walltime=02:00:00                           # Max run time (hh:mm:ss)

cd ${PBS_O_WORKDIR}                                 # Enter submission directory

module reset                                        # Reset environment
module load Python/<version>                        # Load the same Python version
source ${HOME}/venvs/myproject/bin/activate         # Activate virtual environment

export OMP_NUM_THREADS=${NCPUS}                     # Limit threads of NumPy, SciPy, ...
export OPENBLAS_NUM_THREADS=${NCPUS}
export MKL_NUM_THREADS=${NCPUS}

python my_script.py                                 # Run your script

This example activates the virtual environment on the compute node and runs my_script.py with up to 8 threads.

Note

Python output is buffered, so print() messages may appear in the output file late or only after the job finishes. Use python -u my_script.py or set export PYTHONUNBUFFERED=1 to see the output immediately.

Parallel Python

Python runs code in a single process by default. To use multiple CPUs for independent tasks, use the built-in multiprocessing or concurrent.futures modules. Read the number of allocated CPUs from the NCPUS environment variable, because os.cpu_count() returns all CPUs of the node.

import os
from concurrent.futures import ProcessPoolExecutor

def process(path):
    with open(path) as f:
        return sum(1 for _ in f)

ncpus = int(os.environ.get("NCPUS", 1))            # CPUs allocated by PBS
files = [f"mydata/sample_{i:02d}.txt" for i in range(1, 101)]

with ProcessPoolExecutor(max_workers=ncpus) as pool:
    results = list(pool.map(process, files))

print(results)

Tip

For many independent runs of a whole script, Job Arrays or Capacity Computing are often simpler than parallelizing the code itself.

Jupyter

To work interactively with notebooks on a compute node, see Jupyter.