General Usage of Hydra

2014-06-09T22:54:44Z

Wzhwei:

<ul>
<li>Login
<pre>
ssh hydra
</pre>
</li>
<li>Example of job.pbs file
<pre>
### AUTOMATICALLY GENERATED BATCH FILE

# ### name of job
#PBS -N [name_of_job]

# ### mail for begin/end/abort
#PBS -m ea
#PBS -M [email_address]

# ### maximum wall time
#PBS -l walltime=[dd:hh:mm:ss]

# ### number of nodes and processors per node
#PBS -l nodes=1:ppn=1

# ### queue
#PBS -q [queue_name]

# ### indicates that job should not rerun if it fails
#PBS -r n

# ### stdout and stderr merged as stderr
#PBS -j eo

# ### write stderr to file
#PBS -e log.err

# ### the shell that interprets the job script
#PBS -S /bin/bash

cd /home/[job_location]
time /opt/lammps/lmp2013 < input.dat

if [ $? -eq 0 ] ; then
touch COMPLETED
fi
</pre>
<tt>[name_of_job]</tt>
This is the name of the job that will be showed in the queue.

<tt>[email_address]</tt>
This is the email address you used to receive the system notice when job begins, aborted or ended.

<tt>[dd:hh:mm:ss]</tt>
This is the maximum allowed running time for your job. dd: days; hh: hours; mm: minutes; ss: seconds.

<tt>[queue_name]</tt>
There are two options for queue name: fast or default.

<tt>[job_location]</tt>
This is the address of the folder where your input file is located.

</li>
<li>Submit jobs
<pre>
qsub job.pbs
</pre>
<li>Check job status
<pre>
qstat
</pre>
show status of all jobs.
<pre>
qstat -u [username]
</pre>
where <tt>[username]</tt> is your own username. This command will show the status of your own jobs.
</ul>

General Usage of Hydra

Notes on Torque

2014-06-09T21:36:38Z

Wzhwei:

General Usage

2014-06-09T21:33:40Z

Wzhwei: Created page with "</li> <li>Login <pre> ssh [netid]@quest.it.northwestern.edu </pre> where <tt>[netid]</tt> is your NETID. The first time you login, it will ask you to enter file in which t..."

</li>
<li>Login
<pre>
ssh [netid]@quest.it.northwestern.edu
</pre>
where <tt>[netid]</tt> is your NETID. The first time you login, it will ask you to enter file in which to save the key and then to enter passphrase twice. Just press "enter" for these three questions and you should be able to login successfully.

Our group folder is located in /projects/b1011/luijten-group
</ul>

Hardware

2014-06-09T21:22:34Z

Wzhwei: /* Quest */

=== Desktop machines ===

All desktop machines run [http://www.opensuse.org OpenSuSE]. [[Installation instructions for OpenSuSE 13.1]].

=== Clusters ===

==== Minotaur ====
* 38 nodes, each containing two 4-core processors (304 cores total). 8 GB memory per node. Processor type: Intel Xeon E5472, 3.0 GHz.
* Jobs are scheduled via [http://www.adaptivecomputing.com/products/open-source/torque/ Torque]/Maui. [[Notes on Torque]].

==== Hydra ====
* 60 nodes, each containing two 6-core processors (720 cores total). 12 GB memory per node. 8 nodes (queue "fast", nodes h001-h008) have Intel Xeon X5690 3.47 GHz processors. 52 nodes (queue "default", nodes h009-h060) have Intel Xeon E5645 2.40 GHz processors.
* Jobs are scheduled via [http://www.adaptivecomputing.com/products/open-source/torque/ Torque]/Maui. [[Notes on Torque]].

==== Quest ====
* Jobs are scheduled via [http://www.adaptivecomputing.com/products/open-source/torque/ Torque]/Maui. [[Notes on Torque]].
* [[General Usage]]

=== Disk space, backups, and RAID storage ===

==== Disk space allocations and nightly backups ====

Each user has a home directory located on ''ariadne''. This home directory is exported to all desktop machines, so that you see the same home filesystem on each machine. The drive is protected against hardware failure via a [[http://en.wikipedia.org/wiki/RAID_1#RAID_1 RAID-1]] setup. Furthermore, each night all new or modified files on /home are written to tape (located in ariadne). This makes it important not to store temporary data in your home folder, as it would quickly fill up the tape. Since users tend to forget this, a quota system has been enabled on ariadne, restricting each user to 15 GB. To check how much space you are using log on to ariadne and issue the command
<pre>
quota -s
</pre>

In addition, each user has significant additional storage on the scratch partitions. These drives are located in the different desktop machines and protected via RAID-1, but backups are your own responsibility. Note that these partitions are generally only mounted on the desktop machine that contains the corresponding drives. If you need a partition to be exported to a different machine, please ask.

==== Changing the nightly backup tape ====

# Press eject button on tape drive in ariadne.
# Take the tape cartridge out of the drive and put it in its box (should be on top of ariadne). Label the box.
# Insert cleaning tape (on top of ariadne). It will work for less than a minute and then eject automatically.
# Put cleaning tape back in box on top of ariadne.
# Insert new DDS tape. Leave empty box on top of ariadne.
# Update settings in /usr/local/lib/backup, namely ''position'' and ''tapenumber''; update logfile.

==== Recovering data from the nightly backup tape ====

Log files of all nightly backup tapes are located on ariadne, in /usr/local/lib/backup. For privacy reasons, these logfiles are only accessible to root. Once the proper file to be recovered has been identified, insert the corresponding tape into the drive on ariadne and follow these steps (all to be executed as root):

# <tt>cd /</tt> (if you change to a different directory, the recovered file will be placed relative to this directory)
# <tt>/usr/local/bin/tape-rewind</tt> (or <tt>mtst -f /dev/nst0 rewind</tt>)
# <tt>mtst -f /dev/nst0 fsf <position></tt> (see the contents file in /usr/local/lib/backup for the position number)
# <tt>tar xzvf /dev/nst0 <full_file_name_without_leading_slash></tt> This step won't work unless you omit the leading slash; also note that you can specify multiple files, separated by spaces. The 'z' option is necessary because all nightly backups are compressed. For wildcards, use --wildcards and escape '*' and '?'. For example: <tt>tar -x --wildcards -zvf /dev/nst0 \*datafiles\*</tt>
# <tt>/usr/local/bin/tape-rewoffl</tt> (or <tt>mtst -f /dev/nst0 rewoffl</tt>)

==== Archiving data using the LTO tape drive ====

==== Checking RAID status ====

<ul>

<li>Hydra
<ul>
<li>OS is on software RAID (which spans /dev/sda and /dev/sdb). An overview is obtained via
<pre>cat /proc/mdstat</pre>
Detailed information via
<pre>
mdadm --detail /dev/mdX
</pre>
where X = 1, 5, 6, 7.</li>
<li>/home</li>
<li>/archive</li>
</ul>
</li>

<li>Minotaur Web interface. Log in as root to head node and use ''opera''.</li>

<li>Ariadne RAID-5 controller with 4 drives. Status can be checked by interrogating the controller:
<pre>
/opt/MegaRAID/MegaCli/MegaCli64 -AdpAllInfo -aALL | less
</pre>
In the 'Device Present' section, it is reported if any drives are critical or have failed, and what the state of the RAID is. More detailed information can also be found via
<pre>
/opt/MegaRAID/MegaCli/MegaCli64 -LDPDInfo -aAll | less
</pre>
Directly at the beginning (under 'Adapter #0') it should report 'State: Optimal'</li>

<li>Desktop machines, except pelops Hardware RAID-1. The RAID status is reported upon reboot of a machine. Press Ctrl-C (when prompted) to enter the configuration utility. From within Linux, use (as root):
<pre>
mpt-status -i 0
mpt-status -i 2
</pre>
The second command only applies to machines with a second set of hard drives (achilles, agamemnon, nestor, poseidon) 
To allow regular users to verify the RAID status, the <tt>mpt-status</tt> has been added to <tt>sudo</tt>:
<pre>
sudo mpt-status -i 0
sudo mpt-status -i 2
</pre>
</li>

<li>Pelops: Software RAID (for OS and scratch partitions). See [[#hydra|Hydra]].</li>

</ul>

=== Printers ===

=== Scanner ===