Notes on Torque: Difference between revisions

From csml-wiki.northwestern.edu
Jump to navigation Jump to search
No edit summary
Line 1: Line 1:
=== Overview ===
=== Overview ===


The CSML uses Torque with Maui for most of its job scheduling. Please see the Cambridge Dept. of Chemistry's [http://www.ch.cam.ac.uk/computing/maui-and-torque-introduction user guide] for information on submitting and checking jobs, and the associated [http://www.ch.cam.ac.uk/computing/maui-administration admin guide] for more job scheduling tools.



=== General usage ===
=== General usage ===

Revision as of 15:47, 25 June 2015

Overview

The CSML uses Torque with Maui for most of its job scheduling. Please see the Cambridge Dept. of Chemistry's user guide for information on submitting and checking jobs, and the associated admin guide for more job scheduling tools.

General usage

Special usage notes

  • If qb indicates a node is down, check the ganglia of Minotaur or Hydra to find which node it is (down nodes are in red at the top of the list). Grep the output of qstat to find the jobs running on any down nodes with
    qstat -f | grep -B 3 [node_ID]
    

    where [node_ID] is the name of the down node, e.g., h036 on Hydra. Make sure the owners of all jobs on the node have a chance to take note of which of their jobs went down and then restart the down nodes using Microway control.

  • Change the total cpu time allotted to a job via
    qalter -l cput=[new_cput] [job_id]
    

    where [new_cput] is the desired amount of cpu time and [job_id] is the job's ID number. Multiple jobs can be changed simultaneously by using `seq [min_job_id] [max_job_id]` in the place of job_id. This will change all jobs [min_job_id] through [max_job_id]. Lowering the cpu time requirement of a job can decrease its wait time in the queue, as the scheduler is more likely to be able to use it for backfilling. On the other hand, increasing the cpu time requirement can be used to ensure that a job is able to finish properly (and can be done even while the job is running), but requires root permission.

  • Move a queued (i.e., waiting) job to a different queue via
    qmove [destination] [job_id]
    

    where [destination] is the new queue (either 'fast' or 'default' for our system) and [job_id] is the job ID.

  • Delete sequence of jobs via
    qdel `seq [min_job_id] [max_job_id]`
    

    where [min_job_id] is the first job ID of the sequence of jobs you want to delete and [max_job_id] is the last one.