Parallel Jobs¶

Most scientific applications can use more than one CPU core. Depending on how the application is parallelized, you request resources differently so the scheduler can place your job correctly.

Model	Typical use	Runs on	Key resources
Multi-threaded	OpenMP, threads, `-t` / `--threads` options	Single node	`ncpus`, `ompthreads`
MPI	Distributed-memory applications	One or more nodes	`select`, `mpiprocs`
Hybrid	MPI processes each running several threads	One or more nodes	`mpiprocs`, `ompthreads`

Note

The ncpus resource counts logical CPUs (hardware threads), two per physical core, see Node Resources. For example, ncpus=48 gives your job 24 physical cores with 2 threads each.

Tip

If you are not sure which model your application supports, check its documentation for options such as --threads, -t, -@ (multi-threaded) or for instructions mentioning mpirun (MPI).

Multi-threaded Jobs¶

A multi-threaded application runs as a single process that uses several CPU cores on one node. Request a single chunk with the number of cores your application will use.

PBS automatically sets the OMP_NUM_THREADS environment variable to the value of ompthreads, or to ncpus if ompthreads is not given.

Example¶

#!/bin/bash
#PBS -N openmp_job                          # Job name
#PBS -l select=1:ncpus=16:mem=32gb          # One node, 16 cores, 32 GB RAM
#PBS -l walltime=04:00:00                   # Max run time (hh:mm:ss)

cd ${PBS_O_WORKDIR}                         # Enter submission directory

module load mysoftware                      # Load any necessary modules
./my_program --threads ${OMP_NUM_THREADS}   # Run with 16 threads

This example requests one node with 16 CPU cores and 32 GB of memory for up to 4 hours and runs the application with 16 threads.

Warning

Always match the number of threads to the number of requested cores. Starting more threads than requested ncpus does not make the job faster, because the job is limited to the allocated cores by cgroups.

MPI Jobs¶

MPI applications run as many independent processes that communicate over the network, so they can span multiple nodes. Each chunk in the select statement describes the resources for one group of MPI processes.

Resource	Description
`select`	Number of chunks
`ncpus`	Number of CPU cores per chunk
`mpiprocs`	Number of MPI processes per chunk
`mem`	Amount of memory per chunk

PBS writes one line per MPI process into the file referenced by PBS_NODEFILE, which mpirun uses to start processes on the allocated nodes.

Example¶

#!/bin/bash
#PBS -N mpi_job                                     # Job name
#PBS -l select=2:ncpus=48:mpiprocs=48:mem=96gb      # 2 chunks, 48 MPI processes each
#PBS -l place=scatter                               # Each chunk on a different node
#PBS -l walltime=08:00:00                           # Max run time (hh:mm:ss)

cd ${PBS_O_WORKDIR}                                 # Enter submission directory

module load OpenMPI                                 # Load MPI library
module load mysoftware                              # Load any necessary modules

NP=$(wc -l < ${PBS_NODEFILE})                       # Total number of MPI processes
mpirun -np ${NP} -hostfile ${PBS_NODEFILE} ./my_mpi_program

This example requests two nodes with 48 CPU cores and 96 GB of memory each and starts 96 MPI processes in total.

Hybrid MPI + OpenMP Jobs¶

Hybrid applications start fewer MPI processes and let each of them run several threads. Use mpiprocs for the number of MPI processes and ompthreads for the number of threads per process, so that mpiprocs × ompthreads = ncpus.

Example¶

#!/bin/bash
#PBS -N hybrid_job                                              # Job name
#PBS -l select=2:ncpus=48:mpiprocs=4:ompthreads=12:mem=96gb     # 4 MPI × 12 threads per node
#PBS -l place=scatter                                           # Each chunk on a different node
#PBS -l walltime=08:00:00                                       # Max run time (hh:mm:ss)

cd ${PBS_O_WORKDIR}                                             # Enter submission directory

module load OpenMPI                                             # Load MPI library
module load mysoftware                                          # Load any necessary modules

NP=$(wc -l < ${PBS_NODEFILE})                                   # Total number of MPI processes
mpirun -np ${NP} -hostfile ${PBS_NODEFILE} \
       -x OMP_NUM_THREADS ./my_hybrid_program

This example runs 8 MPI processes on two nodes, each process using 12 threads.

Chunk Placement¶

The place statement controls how the scheduler distributes chunks across nodes.

Directive	Behavior
`-l place=free`	Chunks are placed wherever resources are available (default).
`-l place=pack`	All chunks are placed on a single node.
`-l place=scatter`	Each chunk is placed on a different node.
`-l place=scatter:excl`	Each chunk on a different node, nodes are not shared with other jobs.

Note

Exclusive placement (excl) reserves whole nodes for your job even if you request only part of their resources. Use it only when your application really needs it, for example for performance benchmarks.