Job Dependencies

Job dependencies let you build workflows from several jobs, where one job starts only after another one has finished. Typical use is a pipeline such as preprocessing → computation → postprocessing, where each step needs results of the previous one.

Dependencies are set with the -W depend= option when submitting a job.

Dependency	The job starts after the listed job(s)
afterok:jobid	finished successfully (exit status 0)
afternotok:jobid	finished with an error (non-zero exit status)
afterany:jobid	finished, regardless of the exit status
after:jobid	started running

Multiple jobs can be listed separated by colons, e.g. afterok:1234:1235.

Note

If a job depends on afterok and the parent job fails, the dependent job will never be able to run. PBS deletes it automatically, so check the exit status of the parent job if a dependent job disappears from the queue.

Example

The qsub command prints the ID of the submitted job, so you can store it in a variable and use it for the next submission.

JOB1=$(qsub preprocess.pbs)                                 # Submit first step
JOB2=$(qsub -W depend=afterok:${JOB1} compute.pbs)          # Runs after JOB1 succeeds
JOB3=$(qsub -W depend=afterok:${JOB2} postprocess.pbs)      # Runs after JOB2 succeeds

echo "Submitted pipeline: ${JOB1} -> ${JOB2} -> ${JOB3}"

This example submits a three-step pipeline at once. The second and third jobs wait in the queue in the H (held) state until their parent job finishes successfully.

Cleanup After Failure

You can combine dependencies to run a different job depending on the result of the previous one, for example to send results or clean up data when the computation fails.

JOB1=$(qsub compute.pbs)                                    # Main computation
qsub -W depend=afterok:${JOB1} postprocess.pbs              # Runs only on success
qsub -W depend=afternotok:${JOB1} cleanup.pbs               # Runs only on failure

Waiting For Multiple Jobs

A single job can wait for several jobs, for example to merge results of independent computations.

JOB_A=$(qsub sample_a.pbs)                                  # Independent job A
JOB_B=$(qsub sample_b.pbs)                                  # Independent job B
qsub -W depend=afterok:${JOB_A}:${JOB_B} merge.pbs          # Runs after A and B succeed

Tip

Instead of submitting many similar independent jobs, consider using Job Arrays or Capacity Computing.