Troubleshooting¶

This page lists the most common problems users run into on the cluster, what usually causes them and how you can solve them yourself. If none of the solutions help, see Getting Help at the end of this page.

Tip

Most problems with jobs can be diagnosed from three sources: the job error output file <job_name>.e<id>, the exit status from qstat -xf <job_id> | grep Exit_status and the scheduler comment from qstat -xf <job_id> | grep comment. See Managing Jobs for details.

Connection¶

Permission denied (publickey)¶

The server did not accept your SSH key.

Check that you are using the correct username and the private key that belongs to the public key you sent with your access request: ssh -i ~/.ssh/id_ed25519 your_user_name@login.hpc.imtm.cz.
SSH refuses keys with too open permissions. Fix them on your computer:
```
chmod 700 ~/.ssh
chmod 600 ~/.ssh/id_ed25519
```
Run ssh -v ... to see which keys are offered to the server.

Remote host identification has changed¶

WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!

The key of the login node differs from the one your computer remembered. This happens after a reinstallation of the login node, but it may also indicate an attack. Verify with the cluster administrators that the key has changed, then remove the old key with ssh-keygen -R login.hpc.imtm.cz and connect again.

Connection drops after a while of inactivity¶

Network devices close idle connections. Add the following to the ~/.ssh/config file on your computer, so SSH sends keep-alive messages:

Host login.hpc.imtm.cz
    User your_user_name
    IdentityFile ~/.ssh/id_ed25519
    ServerAliveInterval 60

With this configuration, you can also connect simply with ssh login.hpc.imtm.cz. To keep long-running commands alive when the connection drops, use Tmux.

Jobs¶

Job stays in the queue¶

Check why the scheduler has not started your job:

qstat -s <job_id>

Comment contains	What to do
`Insufficient amount of resource: ncpus` / `mem`	The cluster is busy. Wait, or request fewer CPUs or less memory.
`Insufficient amount of resource: scratch_*`	Not enough free scratch. Request less scratch or use a different scratch type.
`Job would conflict with reservation or top job`	Your job does not fit before a higher-priority job. A shorter `walltime` may help it start sooner through backfilling.
`Not Running: Job is held` (state H)	The job waits for a dependency or was held with `qhold`. Release it with `qrls <job_id>`.

Warning

A single chunk must fit on one node. A chunk requesting more CPUs or memory than any node provides, see Node Resources, will never start. Split such a request into multiple chunks.

Job was rejected by qsub¶

qsub: Job exceeds queue and/or server resource limits

The job requests more resources than allowed, e.g. a too long walltime or too many CPUs. Lower the request and submit again.

bad interpreter: No such file or directory¶

/bin/bash^M: bad interpreter: No such file or directory

The job script was written on Windows and contains Windows line endings. Convert it on the cluster:

sed -i 's/\r$//' job.pbs

Job was killed for exceeding walltime¶

The error output file contains a message similar to:

=>> PBS: job killed: walltime 7230 exceeded limit 7200

The exit status of the job is typically -29 or 271, see Exit Status. Your job ran longer than its requested walltime. Request more time, or split the work into smaller jobs, see Job Dependencies and Capacity Computing.

Job was killed for exceeding memory¶

The job ended abruptly, the error output contains Killed, Out of memory or oom-kill, and the exit status is typically 265 (killed by signal SIGKILL), -26 (killed by PBS for exceeding mem) or 137 for the program itself, see Exit Status.

Your job tried to use more memory than requested in mem. Check the memory usage of the job with qstat -xf <job_id> | grep resources_used.mem and request more memory next time, e.g. with 20–30 % reserve.

Job ended immediately or without results¶

Read the error output file <job_name>.e<id>, which usually contains the reason.
Jobs start in your home directory. Use cd ${PBS_O_WORKDIR} at the beginning of the script, otherwise relative paths to input files will not be found.
Check that the first line of the script is #!/bin/bash and that all #PBS directives come before the first command. Directives after a command are ignored.

Output file is missing or empty while the job is running¶

Output files <job_name>.o<id> and <job_name>.e<id> are copied to the submission directory only after the job finishes. To follow a running job, redirect the output of your program into a file, e.g. ./my_program > progress.log 2>&1. For Python, also use python -u to disable output buffering.

Job runs slower than expected¶

Too many threads. Many libraries start one thread per CPU of the whole node, not per allocated CPU. Set export OMP_NUM_THREADS=${NCPUS} and read the number of CPUs from the NCPUS variable in your code, see Parallel Jobs.
Too few threads. If resources_used.cpupercent is around 100 although you requested more CPUs, your program uses only one CPU. Enable multi-threading in your program (e.g. --threads, -t, -@) or request just one CPU.
Slow storage. Working with many files directly on project storage is slow. Copy your data to Scratch Space at the beginning of the job.

No space left on device in scratch¶

Your job wrote more data into scratch than it requested. The scratch size is a hard limit, so request more scratch (e.g. scratch_local=50gb), or clean up temporary files during the run.

$SCRATCH is empty¶

The SCRATCH variable is set only when the job requests scratch, e.g. #PBS -l select=1:ncpus=4:mem=8gb:scratch_local=10gb. See Scratch Space.

Software¶

module: command not found¶

The job script is not run by Bash. Make sure the very first line of the script is #!/bin/bash.

Module cannot be loaded¶

Lmod has detected the following error: These module(s) or extension(s) exist but cannot be loaded as requested

The module depends on other modules that must be loaded first. Show how to load it with:

module spider <name>/<version>

If two loaded modules conflict, start from a clean environment with module reset and load only the modules you need.

The environment of a job is not the same as your interactive shell. Load all modules and activate all environments inside the job script:

module reset
module load Python/<version>
source ${HOME}/venvs/myproject/bin/activate
which python                    # Check which Python is used

conda: command not found in job¶

Conda is not initialized in jobs. Load it at the beginning of the job script, see Conda:

source ${HOME}/miniforge3/etc/profile.d/conda.sh
conda activate myenv

Python package is installed but cannot be imported¶

Check that the right environment is active with which python and pip list.
Packages installed with pip install --user into ~/.local can override packages in your environment. Move them away with mv ~/.local/lib ~/.local/lib.bak and install the packages into a virtual environment instead.

GLIBC_2.xx not found¶

version `GLIBC_2.38' not found

The program was compiled on a newer system than the cluster. Compile it on the cluster, see Compiling Software, install it with Conda, or run it in a Singularity container.

Illegal instruction¶

The program was compiled for a different processor, e.g. with -march=native on another computer. Recompile it on a compute node, see Compiling Software.

Storage¶

Disk quota exceeded in home directory¶

Find out what takes the most space in your home directory:

du -sh ~/* ~/.[!.]* 2>/dev/null | sort -h | tail -20

The most common culprits are caches, which you can safely clean:

conda clean --all                   # Conda package cache
pip cache purge                     # Pip cache
singularity cache clean             # Singularity image cache

Move large datasets to your personal directory /project/<username> or to your project directory. If you do not have space on /project yet, request it. Also remove Conda environments and virtual environments you no longer need.

Permission denied in project directory¶

Files created by other project members may not be writable for you. The owner of the files can allow writing for the whole group:

chmod -R g+rwX /project/my_project/shared_data

Getting Help¶

If you cannot solve the problem yourself, contact the cluster administrator at jan.lostak@upol.cz. To get help quickly, include:

your username,
the job ID of the affected job,
the job script or the exact command you ran,
the error message or the content of the error output file,
when the problem occurred and whether it happens repeatedly.