Managing Jobs¶

After you submit a job, you can check its state, change its parameters while it is waiting in the queue, hold or release it, and investigate why it failed.

Job States¶

The qstat command shows the state of each job in the S column.

State	Meaning
`Q`	Queued, waiting for resources to become available.
`R`	Running.
`H`	Held, waiting for a dependency or held by the user or administrator.
`E`	Exiting, the job is finishing and copying output files.
`F`	Finished.
`B`	Job array has started (at least one element is running).
`X`	Job array element has finished.
`W`	Waiting for its requested start time.

Checking Jobs¶

qstat -u $USER          # Your queued and running jobs
qstat -an1 -u $USER     # Including allocated nodes, one line per job
qstat -s -u $USER       # Including scheduler comment (why the job is waiting)
qstat -f <job_id>       # Full details of a job
qstat -x -u $USER       # Including finished jobs

Note

Information about finished jobs is kept for one year, so you can check the exit status and resource usage of your past jobs with qstat -x.

Tip

If your job stays in the Q state for a long time, check the scheduler comment with qstat -s <job_id> or in the comment field of qstat -f <job_id>. It usually explains which resource is not currently available.

Holding and Releasing Jobs¶

A queued job can be put on hold, so it does not start until you release it. This is useful when you need to update input data before the job starts.

qhold <job_id>          # Put a queued job on hold
qrls <job_id>           # Release the job back to the queue
qsub -h job.pbs         # Submit a job directly in the held state

Modifying Jobs¶

The qalter command changes attributes of a job that is still waiting in the queue, so you do not need to delete and resubmit it.

qalter -l walltime=04:00:00 <job_id>                # Change maximum runtime
qalter -l select=1:ncpus=8:mem=16gb <job_id>        # Change requested resources
qalter -N new_name <job_id>                         # Change job name

Note

Resources of a running job can not be increased. If your job needs more time or memory, delete it and submit it again with a corrected resource request.

Deleting Jobs¶

qdel <job_id>                                       # Delete a single job
qdel '1234[]'                                       # Delete a whole job array
qdel $(qselect -u $USER)                            # Delete all your jobs
qdel $(qselect -u $USER -s Q)                       # Delete all your queued jobs

Warning

Deleting a running job immediately terminates it and its scratch directory is removed. Copy any data you need before deleting the job.

Job Output¶

Standard output and error of the job are written into files in the directory from which the job was submitted, unless set otherwise with -o and -e directives.

File	Content
`<job_name>.o<id>`	Standard output (stdout)
`<job_name>.e<id>`	Standard error (stderr)

Note

Output files are copied to the destination only after the job finishes. To follow the progress of a running job, redirect output of your program to a file in your home or project directory, e.g. ./my_program > progress.log 2>&1.

Exit Status¶

The exit status of a finished job shows whether it ended successfully.

qstat -xf <job_id> | grep Exit_status

Exit status	Meaning
`0`	The job finished successfully.
`1` – `255`	The job script failed, the value is the exit code of the last command.
`> 256`	The job was killed by a signal, the signal number is the value minus 256 (e.g. `271` = `SIGTERM`).
`< 0`	PBS could not start the job, or killed it for exceeding a resource limit.

Tip

The exit status of the job is the exit status of the last command in the script. If an earlier command fails and the last one succeeds, the job reports 0. Add set -e at the beginning of the script to stop the job at the first failing command.

Common Exit Codes of the Job Script¶

These values are set by your program or by the shell running the job script.

Exit status	Meaning	What to do
`1`	General error of the program.	Read the error output file `<job_name>.e<id>`.
`2`	Wrong usage of a command, e.g. bad arguments.	Check the command line in the job script.
`126`	Command found but cannot be executed.	Make the file executable with `chmod +x`, or run it with `bash`/`python`.
`127`	Command not found.	Load the right module or activate the environment inside the job script.
`134`	Program aborted (`SIGABRT`), e.g. a failed assertion.	Read the error output, the program detected an internal error.
`137`	Program killed (`SIGKILL`), most often out of memory.	Request more `mem`, see Troubleshooting.
`139`	Segmentation fault (`SIGSEGV`).	Bug in the program, wrong input, or too small stack (`ulimit -s unlimited`).
`143`	Program terminated (`SIGTERM`).	The job was deleted or killed by PBS, check the end of the error output file.

Values 128 + n mean that a command inside the script was killed by signal n, e.g. 137 = 128 + 9 (SIGKILL).

Exit Codes Set by PBS¶

These values are set by PBS itself, when the whole job was killed by a signal or could not be run.

Exit status	Meaning
`265`	Job killed by `SIGKILL` (`256 + 9`), e.g. after `qdel` or exceeding memory.
`271`	Job terminated by `SIGTERM` (`256 + 15`), e.g. after `qdel`.
`-1`, `-2`	Job could not be started on the compute node, e.g. the output directory is not accessible.
`-3`	Temporary failure when starting the job, PBS will try to run it again.
`-4`	Initialization of the job failed.
`-10`	Invalid user or group of the job.
`-11`	The job was requeued and will run again.
`-26`	Job exceeded its `mem` limit.
`-28`	Job exceeded its `cput` limit.
`-29`	Job exceeded its `walltime` limit.

Note

If your job ends with a negative exit status other than a resource limit, it is usually not caused by your script. Contact the cluster administrator with the job ID, see Getting Help.

Resource Usage¶

To estimate resources for your next jobs, check how much CPU time, memory and walltime a finished job actually used.

qstat -xf <job_id> | grep resources_used

resources_used.cpupercent = 395
resources_used.cput = 07:52:14
resources_used.mem = 6291456kb
resources_used.ncpus = 4
resources_used.walltime = 02:00:41

This example shows a job that used all 4 CPU cores for about two hours and roughly 6 GB of memory, so a request of ncpus=4:mem=8gb was appropriate.

Tip

If resources_used.mem is close to the requested mem, request more memory next time. If cpupercent is much lower than ncpus × 100, your application does not use all requested cores and you can request fewer.