Monitoring Jobs¶

While your job is running, you can check how many CPUs and how much memory it really uses. This helps you find out whether the job runs as expected and how many resources to request next time.

Resource Usage From PBS¶

PBS regularly updates the resources used by every running job:

qstat -f <job_id> | grep resources_used

resources_used.cpupercent = 790
resources_used.cput = 03:12:40
resources_used.mem = 12582912kb
resources_used.ncpus = 8
resources_used.walltime = 00:25:03

Value	What it tells you
`cpupercent`	CPU usage, `100` means one fully used CPU. Compare it with `ncpus × 100`.
`cput`	Total CPU time used by all processes of the job.
`mem`	Current memory usage. Compare it with the `mem` you requested.
`walltime`	How long the job has been running.

Tip

A job with ncpus=8 and cpupercent around 100 uses only one CPU. Check that your program is set to use multiple threads, see Parallel Jobs, or request fewer CPUs.

Looking at the Compute Node¶

To see the individual processes of your job, find out on which node it runs and log in there:

qstat -n1 <job_id>                  # The node is listed at the end of the line
ssh pbs-mom5                        # Log in to the node from the login node
top -u $USER                        # Your processes, press q to quit

In top, the %CPU column shows the CPU usage of each process (800 = 8 CPUs) and RES its memory.

Warning

Use the login to the compute node only to look at your job. Processes you start there run outside your job and share a small limit with all other users on the node, see Running Jobs. Never start computations this way.

Following the Output¶

Output files of a job are copied to the submission directory only after it finishes. To follow the progress of a running job, write the output of your program into a file and watch it from the login node:

./my_program > progress.log 2>&1                # In the job script

tail -f progress.log                            # On the login node, stop with Ctrl+C

Getting an E-mail¶

PBS can send you an e-mail when your job starts, finishes or is aborted:

#PBS -m abe
#PBS -M your.name@upol.cz

Letter	E-mail is sent when the job
`a`	is aborted
`b`	begins
`e`	ends

Note

For job arrays, use only -m ae or none at all, otherwise you get an e-mail for every subjob.