Notes on Torque

Overview

The CSML uses Torque with Maui for most of its job scheduling. Please see the Cambridge Dept. of Chemistry's user guide for information on submitting and checking jobs, and the associated admin guide for more job scheduling tools.

Special usage notes

• qb is a succinct queue monitoring script that tallies each user's running and queued jobs, total cores used and free, and down nodes (if any). The script was originally written by Daniel Sinkovits and should be available to copy from another user's $HOME/bin folder to your own. Maintenance includes keeping the username list up-to-date and adjusting the variable node_total (currently 37) if a node goes down permanently.
• If qb indicates a node is down, check Ganglia on Minotaur or Hydra to find which node it is (down nodes are in red at the top of the list). Grep the output of qstat to find the jobs running on any down nodes with

  qstat -f | grep -B 3 [node_ID]
  

  where [node_ID] is the name of the down node, e.g., h036 on Hydra. Make sure the owners of all jobs on the node have a chance to take note of which of their jobs went down and then restart the down nodes using Microway control.

• Change the total cpu time allotted to a job via

  qalter -l cput=[new_cput] [job_id]
  

  where [new_cput] is the desired amount of cpu time and [job_id] is the job's ID number. Multiple jobs can be changed simultaneously by using `seq [min_job_id] [max_job_id]` in the place of job_id. This will change all jobs [min_job_id] through [max_job_id]. Lowering the cpu time requirement of a job can decrease its wait time in the queue, as the scheduler is more likely to be able to use it for backfilling. On the other hand, increasing the cpu time requirement can be used to ensure that a job is able to finish properly (and can be done even while the job is running), but requires root permission.

• Move a queued (i.e., waiting) job to a different queue via

  qmove [destination] [job_id]
  

  where [destination] is the new queue (either 'fast' or 'default' for our system) and [job_id] is the job ID.

• Delete a sequence of jobs via

  qdel `seq [min_job_id] [max_job_id]`
  

  where [min_job_id] is the first job ID of the sequence of jobs you want to delete and [max_job_id] is the last one.