Skip to content

Jupyter

JupyterLab lets you work with notebooks in your web browser while the code runs on a cluster compute node with the resources of your job. JupyterLab runs inside a regular PBS job, and you connect to it through an SSH tunnel via the login node.

Warning

Do not run JupyterLab on the login node. Notebooks often use a lot of CPU and memory, which slows down the login node for all users.

Installing JupyterLab

Install JupyterLab into your Python virtual environment or Conda environment on the login node.

module load Python/<version>
source ${HOME}/venvs/myproject/bin/activate
pip install jupyterlab

source ${HOME}/miniforge3/etc/profile.d/conda.sh
conda activate myenv
conda install jupyterlab

Tip

To use R in notebooks, install the R kernel into your R library: install.packages("IRkernel") and then IRkernel::installspec().

Starting JupyterLab

Submit JupyterLab as a batch job, so it keeps running even if you close your terminal.

Example content of jupyter.pbs file:

#!/bin/bash
#PBS -N jupyter                                     # Job name
#PBS -l select=1:ncpus=4:mem=16gb                   # One node, 4 CPUs, 16 GB RAM
#PBS -l walltime=08:00:00                           # Max run time (hh:mm:ss)

cd ${PBS_O_WORKDIR}                                 # Enter submission directory

module reset                                        # Reset environment
module load Python/<version>                        # Load Python
source ${HOME}/venvs/myproject/bin/activate         # Activate virtual environment

export OMP_NUM_THREADS=${NCPUS}                     # Limit threads to allocated CPUs

PORT=$(shuf -i 20000-40000 -n 1)                    # Random port to avoid conflicts
NODE=$(hostname -s)

echo "ssh -N -L ${PORT}:${NODE}:${PORT} ${USER}@login.hpc.imtm.cz" > jupyter-${PBS_JOBID}.txt

jupyter lab --no-browser --ip=${NODE} --port=${PORT} > jupyter-${PBS_JOBID}.log 2>&1

qsub jupyter.pbs

Connecting

  1. Wait until the job is running (qstat -u $USER shows state R).
  2. Print the SSH tunnel command prepared by the job: 
    cat jupyter-<job_id>.txt
  3. Find the address with the access token in the JupyterLab log: 
    grep "http://" jupyter-<job_id>.log
  4. On your own computer, run the SSH tunnel command from step 2 and keep it running.
  5. Open the address from step 3 in your browser, replacing the node name with localhost, e.g. http://localhost:23456/lab?token=....

Tip

On Windows, run the SSH tunnel command in PowerShell. In PuTTY, set up the tunnel under Connection → SSH → Tunnels with the same source port and destination <node>:<port>.

Stopping JupyterLab

When you are finished, stop JupyterLab with File → Shut Down in the browser, or delete the job with qdel <job_id>. This releases the allocated resources for other users.

Note

The job runs until its walltime expires, even if you close your browser. Always shut it down when you stop working, otherwise the resources stay reserved and count towards your fair-share usage.