Capacity Computing

Many workloads consist of a large number of small, independent tasks, for example processing thousands of samples or input files. Submitting each task as a separate job puts a heavy load on the scheduler, slows down commands such as qstat for all users and adds scheduling overhead to every task.

Warning

Please do not flood the queue with thousands of individual jobs. Use one of the approaches below to group many small tasks into a smaller number of jobs.

Approach	Best for
Job Arrays	Hundreds of tasks, each running for at least several minutes.
Task packing	Thousands of short tasks, each using one or a few CPU cores.
Job arrays with task packing	Very large numbers of tasks spread over several nodes.

Job Arrays

A job array is a single submission that the scheduler handles as one job with many elements. It is the simplest way to run the same script on many inputs, as long as each element runs long enough (minutes rather than seconds) to make its scheduling overhead negligible.

Task Packing

Task packing runs many small tasks inside one job, in parallel on all allocated cores. The job requests the cores once, and a tool such as xargs keeps them busy by starting a new task whenever the previous one finishes.

Example

#!/bin/bash
#PBS -N packed_tasks                        # Job name
#PBS -l select=1:ncpus=32:mem=64gb          # One node, 32 cores, 64 GB RAM
#PBS -l walltime=12:00:00                   # Max run time (hh:mm:ss)

cd ${PBS_O_WORKDIR}                         # Enter submission directory

module load mysoftware                      # Load any necessary modules

# Run ./my_program on every file listed in tasks.txt,
# keeping 32 tasks (one per core) running at the same time.
xargs -a tasks.txt -P ${NCPUS} -I {} \
      sh -c './my_program {} > results/$(basename {}).out 2>&1'

This example processes every input file listed in tasks.txt (one file per line) using all 32 CPU cores of one node. Results of each task are written into a separate file in the results directory.

You can create the task list for example with the find command:

mkdir -p results
find ${PBS_O_WORKDIR}/mydata -name "*.dat" > tasks.txt

Tip

For more advanced task packing with logging, retries of failed tasks and resuming interrupted runs, use GNU Parallel, which is installed on the cluster. The example above can be written as:

parallel -j ${NCPUS} --joblog tasks.log --resume \
         './my_program {} > results/{/}.out 2>&1' :::: tasks.txt

The --joblog and --resume options record finished tasks, so if the job runs out of walltime, you can submit it again and only unfinished tasks will run.

Job Arrays With Task Packing

To process a very large number of tasks across several nodes, combine both approaches. Split the task list into parts, and let each element of a job array process one part with task packing.

Example

split -n l/10 -d -a 2 tasks.txt part_      # Split tasks.txt into part_00 … part_09

#!/bin/bash
#PBS -N packed_array                        # Job name
#PBS -J 0-9                                 # 10 array elements
#PBS -l select=1:ncpus=32:mem=64gb          # Each element: 32 cores, 64 GB RAM
#PBS -l walltime=12:00:00                   # Max run time (hh:mm:ss)

cd ${PBS_O_WORKDIR}                         # Enter submission directory

module load mysoftware                      # Load any necessary modules

PART=$(printf "part_%02d" ${PBS_ARRAY_INDEX})
xargs -a ${PART} -P ${NCPUS} -I {} \
      sh -c './my_program {} > results/$(basename {}).out 2>&1'

This example splits the task list into 10 parts and processes them as a job array with 10 elements, each running 32 tasks in parallel, so up to 320 tasks run at the same time.