PBS Cheat Sheet¶

A one-page overview of the most used commands and directives. Each section links to a page with details.

Submitting Jobs¶

Command	Action
`qsub job.pbs`	Submit a job script
`qsub -I -l select=1:ncpus=4:mem=8gb`	Start an interactive job
`qsub -J 1-100 job.pbs`	Submit a job array
`qsub -W depend=afterok:<job_id> job.pbs`	Start after another job succeeds
`qsub -h job.pbs`	Submit a job in the held state
`qsub -v INPUT=file.txt job.pbs`	Pass a variable to the job

Checking Jobs¶

Command	Action
`qstat -u $USER`	Your queued and running jobs
`qstat -an1 -u $USER`	Your jobs with allocated nodes
`qstat -s <job_id>`	Why the job is waiting
`qstat -f <job_id>`	Full details of a job
`qstat -x -u $USER`	Including finished jobs
`qstat -xf <job_id> \| grep Exit_status`	Exit status of a finished job
`qstat -xf <job_id> \| grep resources_used`	Resources used by a job
`qstat -Q`	Queues and their jobs
`pbsnodes -aSj`	Nodes and their free resources

Changing Jobs¶

Command	Action
`qhold <job_id>` / `qrls <job_id>`	Hold / release a queued job
`qalter -l walltime=04:00:00 <job_id>`	Change a queued job
`qdel <job_id>`	Delete a job
`qdel '<job_id>[]'`	Delete a whole job array
`qdel $(qselect -u $USER -s Q)`	Delete all your queued jobs

Directives¶

#!/bin/bash
#PBS -N my_job                              # Job name
#PBS -l select=1:ncpus=8:mem=16gb           # Resources of one chunk
#PBS -l select=1:ncpus=8:mem=16gb:scratch_local=50gb   # With local scratch
#PBS -l select=2:ncpus=64:mem=128gb:mpiprocs=64        # Two chunks for MPI
#PBS -l place=scatter                       # Each chunk on a different node
#PBS -l walltime=24:00:00                   # Maximum runtime
#PBS -j oe                                  # Merge output and errors
#PBS -o output.log                          # Output file
#PBS -m ae                                  # E-mail when aborted or ended
#PBS -M your.name@upol.cz                   # E-mail address
#PBS -J 1-100                               # Job array

Environment Variables¶

Variable	Content
`PBS_JOBID`	ID of the job
`PBS_O_WORKDIR`	Directory from which the job was submitted
`PBS_ARRAY_INDEX`	Index of the subjob in a job array
`PBS_NODEFILE`	File with the list of allocated nodes
`NCPUS`	Number of CPUs on this node
`SCRATCH`	Scratch directory of the job

Template Job Script¶

#!/bin/bash
#PBS -N my_job
#PBS -l select=1:ncpus=8:mem=16gb:scratch_local=50gb
#PBS -l walltime=04:00:00
#PBS -j oe

set -e                                              # Stop at the first error

module reset
module load mysoftware

cp -r ${PBS_O_WORKDIR}/input ${SCRATCH}/            # Stage input to scratch
cd ${SCRATCH}

./my_program --threads ${NCPUS} input/ results/     # Compute

cp -r results ${PBS_O_WORKDIR}/results_${PBS_JOBID} # Save results