Long Jobs

A job that runs for many days is risky. A failure of the node, a planned maintenance or a too short walltime means that all its work is lost. It is safer to split a long computation into several shorter jobs, each continuing where the previous one stopped. This page shows how to do that.

Tip

Shorter jobs also start sooner thanks to backfilling.

Checkpointing

A checkpoint is a file in which a program saves its state, so it can later continue from that point instead of starting over. Many scientific applications support checkpoints directly:

Application	Save checkpoints	Continue from checkpoint
GROMACS	gmx mdrun -cpt 15 (every 15 min)	gmx mdrun -cpi state.cpt
PyTorch	torch.save(state, "ckpt.pt")	torch.load("ckpt.pt")
Own Python code	Save results with pickle or numpy.save regularly	Load them at start if the file exists

The job script then only needs to decide whether to start from the beginning or continue:

#!/bin/bash
#PBS -N long_run
#PBS -l select=1:ncpus=32:mem=32gb
#PBS -l walltime=24:00:00
#PBS -j oe

cd ${PBS_O_WORKDIR}

if [ -f checkpoint.dat ]; then
    ./my_program --threads ${NCPUS} --restart checkpoint.dat     # Continue
else
    ./my_program --threads ${NCPUS}                              # First run
fi

Many applications can also stop by themselves before the walltime runs out, e.g. GROMACS with gmx mdrun -maxh 23.5, so the last checkpoint is always complete.

Stopping Before the Walltime

If your program cannot stop by itself, use the timeout command to stop it a while before the walltime, so the script has time to save the results:

#PBS -l walltime=24:00:00

cd ${SCRATCH}
timeout 23h ./my_program                        # Stop the program after 23 hours
cp -r ${SCRATCH}/results ${PBS_O_WORKDIR}/      # One hour is left to copy the results

Warning

When a job reaches its walltime, PBS kills it immediately and its scratch directory is deleted. Results that were not copied to permanent storage before are lost.

Chaining Jobs

Submit a chain of jobs at once with dependencies. Each job starts after the previous one finishes and continues from its checkpoint:

JOB=$(qsub long.pbs)
for i in $(seq 2 5); do
    JOB=$(qsub -W depend=afterany:${JOB} long.pbs)
done

This submits five jobs, which run one after another. afterany is used because a job stopped by the walltime does not end with exit status 0.

Tip

Add a check at the beginning of the job script that ends it immediately when the computation is already complete, e.g. [ -f md.gro ] && exit 0. The remaining jobs of the chain then finish in a few seconds instead of running again.