Job Scheduling¶
When resources are not immediately available, submitted jobs wait in the queue. The scheduler decides which job starts next based on its priority and on the resources it requests.
Job Priority¶
Job priority is influenced mainly by the following factors.
| Factor | Description |
|---|---|
| Fair-share | Users who have recently used fewer resources get higher priority. Recent usage gradually decays over time. |
| Waiting time | The longer a job waits in the queue, the higher its priority becomes. |
| Requested resources | A job can only start when all resources it requests are free at the same time. |
Note
Fair-share ensures that the cluster is shared evenly among all users. If you have run many jobs recently, your new jobs may wait longer while other users' jobs start first. Your priority recovers over time.
Job Backfilling¶
To keep the cluster fully utilized, the scheduler uses backfilling. While the highest-priority job is waiting for enough resources to become free, smaller jobs with lower priority can run in the gaps, as long as they do not delay the start of the highest-priority job.
The scheduler can only do this when it knows how long each job will run, which is why an accurate walltime matters.
Getting Your Jobs Started Sooner¶
Tip
- Request a realistic
walltime. Shorter, well-estimated jobs fit into backfilling gaps and often start before jobs with higher priority. - Request only the resources you need. Smaller requests for CPU cores and memory are easier to satisfy. Check the actual usage of your finished jobs as described in Managing Jobs.
- Split large work into smaller jobs. Several shorter jobs usually start sooner than one very long job, and a failure costs you less work. See Job Dependencies and Capacity Computing.
Why Is My Job Waiting?¶
The scheduler writes the reason why a job has not started yet into its comment.
qstat -s <job_id> # Show job with scheduler comment
qstat -f <job_id> | grep comment # Show only the comment
This example shows a job waiting because there are currently not enough free CPU cores. The job will start automatically once resources become available.
To see how many resources are currently free on each node, use: