csml-wiki.northwestern.edu - User contributions [en]

Gnuplot

2016-04-29T19:30:58Z

Bernie: /* Formatting labels */

=== Overview ===

Gnuplot is a program for plotting scientific data. The resulting figures can be of very high quality, provided that attention is paid to detail. Impressive examples can be found on the [http://www.gnuplot.info/ gnuplot homepage]. '''This page does not provide a detailed manual for gnuplot''', but instead focuses on specific aspects typically encountered in the CSML when creating figures for manuscripts.

=== PDF terminal ===

The version of gnuplot installed on all [[hardware|local machines]] has been compiled with [http://www.pdflib.com/ PDFlib], which makes it possible to directly produce PDF images. Such figures offer the important advantage that they are fully scalable and yet have a small file size. Choose this terminal via

set term pdf enh [additional options]
set output "filename.pdf"

Sometimes, fine-tuning of the resulting PDF figure is most conveniently done in Adobe Illustrator: each component of the figure will appear as an object that you can move, resize, etc.

=== Encapsulated PostScript terminal ===

Often when the output of a gnupot script requires further formatting it is easiest to do so using the [http://en.wikipedia.org/wiki/Encapsulated_PostScript Encapsulated PostScript] (EPS) terminal. This is essentially the same as the PostScript terminal except that it contains some extra lines that allow it to interface with other programs. Invoke the EPS terminal via

set term post eps enh color [additional options]
set output "filename.eps"

For possible options see the instructions for the PostScript terminal on [http://www.gnuplotting.org/output-terminals/ this page]. One advantage of .eps files is that they can be edited directly in a text editor. See [[#eps-edit|below]] for an example.
Once the eps file has been formatted properly, you can convert it to PDF format using [http://www.ctan.org/pkg/epstopdf epstopdf]. Executing the command

epstopdf filename.eps

will create the PDF file <code>filename.pdf</code>.

=== Formatting labels ===

When choosing for instance an axis label via

set xlabel "textlabel"

it is straightforward to use simple text, however often special formatting is necessary. A brief set of rules follows. For italic script place

{/Helvetica-Oblique [text]}

inside the quotes, where <code>[text]</code> is what you wish to have in italics. All variables should be in italics. For greek letters use

{/Symbol [text]}

where <code>[text]</code> is any character off of the list linked [http://mathewpeet.org/lists/symbols/ here]. For italic Greek characters use

{/Symbol-Oblique [text]}

In emacs, one can use the shorthand <code>M-x ho</code> for Helvetica-Oblique and <code>M-x so</code> for Symbol-Oblique by inserting the following lines into your .emacs file.

(fset 'ho
(lambda (&optional arg) "Keyboard macro." (interactive "p") (kmacro-exec-ring-item (quote ("{/Helvetica-Oblique }^[OD" 0 "%d")) arg)))
(fset 'so
(lambda (&optional arg) "Keyboard macro." (interactive "p") (kmacro-exec-ring-item (quote ("{/Symbol-Oblique }^[OD" 0 "%d")) arg)))

More instructions are available [http://www.gnuplotting.org/manpage-gnuplot-4-6/#Q1-1-42 here].

=== Sample usage script===

The following is a sample gnuplot script. Everything in braces should be replaced. Again, see the [http://www.gnuplot.info/ gnuplot homepage] or appropriate [http://www.gnuplotting.org/ tutorial] for more comprehensive usage.

set term post eps enh color {dashed|solid} lw 2
set output "filename.eps"
set encoding iso_8859_1
set key top right
set format x "%3.2f" #format tics to have max 3 digits and
set format y "%3.2f" #max 2 digits after the decimal
set xlabel "{xLabel}"
set ylabel "{yLabel}"
set xrange [{xmin}:{xmax}]
set yrange [{ymin}:{ymax}]
plot "datafile.dat" using 1:2:3 with errorbars

=== General usage tips ===

<ul>
<li>Journals typically prefer Helvetica as the typeface for figures.</li>
<li>The origin should be marked with '0', not '0.0'. If you produce a figure directly in PDF format, this can be fixed in Adobe Illustrator. However, it is easier to use the [[#Encapsulated PostScript terminal|EPS terminal]] and then edit the resulting .eps file using [http://www.grymoire.com/unix/sed.html sed]:
<pre>
sed -i 's/(0.0)/(0)/g' filename.eps
</pre>
</li>
<li>Gnuplot accepts abbreviated commands: you just have to enter enough characters to uniquely resolve a command. For example, it is acceptable to write
<pre>
plot "file.dat" us 1:2:3 w err
</pre>
instead of
<pre>plot "file.dat" using 1:2:3 with errorbars
</pre>
</li>
<li>Gnuplot can plot compressed data files by sending the data through a pipe:
<pre>
plot '< gzip -dc data.gz'
</pre>
</li>
<li>To plot the average of columns that are spread over multiple files (but have the same number of lines in each file), take advantage of the [[Command-line_interface_on_Linux/UNIX#paste|paste]] command in Unix to combine them in single file. For example, if three files each contain two columns (x y), use
<pre>
paste file1.dat file2.dat file3.dat > combined.dat
</pre>
and then within gnuplot
<pre>
plot "combined.dat" us 1:(($2+$4+$6)/3)
</pre>
This can even be done on the fly by using a pipe:
<pre>
plot '< paste file1.dat file2.dat file3.dat' using 1:(($2+$4+$6)/3)
</pre>
<li>Gnuplot can histogram data without the need for preprocessing:
<pre>
binwidth=[binwidth]
bin(x,width)=width*floor(x/width) + binwidth/2.0
set boxwidth binwidth
plot "file.dat" using (bin($[colnum],binwidth)):(1.0) smooth freq with boxes
</pre>
where the user should replace <tt>[binwidth]</tt> with the desired bin width and <tt>[colnum]</tt> with the column number of the data to be binned.
</ul>

=== Executable version for OS X ===

<ul>

<li>Compiled on OS X 10.10.3 (Yosemite)</li>

<li>Prior to using this, you need to install X11/XQuartz (http://xquartz.macosforge.org/landing/)</li>

<li>Current version is Gnuplot 5.0.1</li>

<li>Includes GNU Readline library and PDF terminal</li>

<li>Installation procedure is not very sophisticated, but the resulting installation will be correct. First download [http://ariadne.ms.northwestern.edu/Download/Gnuplot/gnuplot_osx_501.zip gnuplot_osx_501.zip] and save to <tt>/tmp</tt>. Then:
<pre>
cd /usr/local
sudo unzip /tmp/gnuplot_osx_501.zip
</pre>
</li>

<li>You can now invoke gnuplot from a terminal window</li>

<li>Note: Older version still available: [http://ariadne.ms.northwestern.edu/Download/Gnuplot/gnuplot_osx_465.zip version 4.6.5], compiled on OS X 10.9.3 (Mavericks)</li>
</ul>

Gnuplot

2016-04-10T18:28:41Z

Bernie:

=== Overview ===

Gnuplot is a program for plotting scientific data. The resulting figures can be of very high quality, provided that attention is paid to detail. Impressive examples can be found on the [http://www.gnuplot.info/ gnuplot homepage]. '''This page does not provide a detailed manual for gnuplot''', but instead focuses on specific aspects typically encountered in the CSML when creating figures for manuscripts.

=== PDF terminal ===

The version of gnuplot installed on all [[hardware|local machines]] has been compiled with [http://www.pdflib.com/ PDFlib], which makes it possible to directly produce PDF images. Such figures offer the important advantage that they are fully scalable and yet have a small file size. Choose this terminal via

set term pdf enh [additional options]
set output "filename.pdf"

Sometimes, fine-tuning of the resulting PDF figure is most conveniently done in Adobe Illustrator: each component of the figure will appear as an object that you can move, resize, etc.

=== Encapsulated PostScript terminal ===

Often when the output of a gnupot script requires further formatting it is easiest to do so using the [http://en.wikipedia.org/wiki/Encapsulated_PostScript Encapsulated PostScript] (EPS) terminal. This is essentially the same as the PostScript terminal except that it contains some extra lines that allow it to interface with other programs. Invoke the EPS terminal via

set term post eps enh color [additional options]
set output "filename.eps"

For possible options see the instructions for the PostScript terminal on [http://www.gnuplotting.org/output-terminals/ this page]. One advantage of .eps files is that they can be edited directly in a text editor. See [[#eps-edit|below]] for an example.
Once the eps file has been formatted properly, you can convert it to PDF format using [http://www.ctan.org/pkg/epstopdf epstopdf]. Executing the command

epstopdf filename.eps

will create the PDF file <code>filename.pdf</code>.

=== Formatting labels ===

When choosing for instance an axis label via

set xlabel "textlabel"

it is straightforward to use simple text, however often special formatting is necessary. A brief set of rules follows. For italic script place

{/Helvetica-Oblique [text]}

inside the quotes, where <code>[text]</code> is what you wish to have in italics. All variables should be in italics. For greek letters use

{/Symbol [text]}

where <code>[text]</code> is any character off of the list linked [http://mathewpeet.org/lists/symbols/ here]. For italic Greek characters use

{/Symbol-Oblique [text]}

More instructions are available [http://www.gnuplotting.org/manpage-gnuplot-4-6/#Q1-1-42 here].

=== Sample usage script===

The following is a sample gnuplot script. Everything in braces should be replaced. Again, see the [http://www.gnuplot.info/ gnuplot homepage] or appropriate [http://www.gnuplotting.org/ tutorial] for more comprehensive usage.

set term post eps enh color {dashed|solid} lw 2
set output "filename.eps"
set encoding iso_8859_1
set key top right
set format x "%3.2f" #format tics to have max 3 digits and
set format y "%3.2f" #max 2 digits after the decimal
set xlabel "{xLabel}"
set ylabel "{yLabel}"
set xrange [{xmin}:{xmax}]
set yrange [{ymin}:{ymax}]
plot "datafile.dat" using 1:2:3 with errorbars

=== General usage tips ===

<ul>
<li>Journals typically prefer Helvetica as the typeface for figures.</li>
<li>The origin should be marked with '0', not '0.0'. If you produce a figure directly in PDF format, this can be fixed in Adobe Illustrator. However, it is easier to use the [[#Encapsulated PostScript terminal|EPS terminal]] and then edit the resulting .eps file using [http://www.grymoire.com/unix/sed.html sed]:
<pre>
sed -i 's/(0.0)/(0)/g' filename.eps
</pre>
</li>
<li>Gnuplot accepts abbreviated commands: you just have to enter enough characters to uniquely resolve a command. For example, it is acceptable to write
<pre>
plot "file.dat" us 1:2:3 w err
</pre>
instead of
<pre>plot "file.dat" using 1:2:3 with errorbars
</pre>
</li>
<li>Gnuplot can plot compressed data files by sending the data through a pipe:
<pre>
plot '< gzip -dc data.gz'
</pre>
</li>
<li>To plot the average of columns that are spread over multiple files (but have the same number of lines in each file), take advantage of the [[Command-line_interface_on_Linux/UNIX#paste|paste]] command in Unix to combine them in single file. For example, if three files each contain two columns (x y), use
<pre>
paste file1.dat file2.dat file3.dat > combined.dat
</pre>
and then within gnuplot
<pre>
plot "combined.dat" us 1:(($2+$4+$6)/3)
</pre>
This can even be done on the fly by using a pipe:
<pre>
plot '< paste file1.dat file2.dat file3.dat' using 1:(($2+$4+$6)/3)
</pre>
<li>Gnuplot can histogram data without the need for preprocessing:
<pre>
binwidth=[binwidth]
bin(x,width)=width*floor(x/width) + binwidth/2.0
set boxwidth binwidth
plot "file.dat" using (bin($[colnum],binwidth)):(1.0) smooth freq with boxes
</pre>
where the user should replace <tt>[binwidth]</tt> with the desired bin width and <tt>[colnum]</tt> with the column number of the data to be binned.
</ul>

=== Executable version for OS X ===

<ul>

<li>Compiled on OS X 10.10.3 (Yosemite)</li>

<li>Prior to using this, you need to install X11/XQuartz (http://xquartz.macosforge.org/landing/)</li>

<li>Current version is Gnuplot 5.0.1</li>

<li>Includes GNU Readline library and PDF terminal</li>

<li>Installation procedure is not very sophisticated, but the resulting installation will be correct. First download [http://ariadne.ms.northwestern.edu/Download/Gnuplot/gnuplot_osx_501.zip gnuplot_osx_501.zip] and save to <tt>/tmp</tt>. Then:
<pre>
cd /usr/local
sudo unzip /tmp/gnuplot_osx_501.zip
</pre>
</li>

<li>You can now invoke gnuplot from a terminal window</li>

<li>Note: Older version still available: [http://ariadne.ms.northwestern.edu/Download/Gnuplot/gnuplot_osx_465.zip version 4.6.5], compiled on OS X 10.9.3 (Mavericks)</li>
</ul>

Etiquette: Running Nice on other desktop machines

2016-02-26T01:39:47Z

Bernie:

When using someone else's machine, it's polite to use the command '''nice''' and '''renice'''.

'''Nice''' changes the priority of a task. When using another machine, you want to set your task to a lower priority as to not interfere with the user's tasks. Priority ranges from -20 (highest) to 20 (lowest).

nice [options] [command]

Example:

nice -19 ~/run.py

UNIX priority levels are counted from negative (highest) to positive. Note that <code>nice -n</code> specifies a priority of <code>n</code> whereas <code>nice --n</code> specifies a priority of <code>-n</code>.

'''Renice''' allows you to reset the priority of a process that is already running.

renice [-n] [PID] [options]

Replace [-n] with the desired priority, and [PID] with the process ID number.

Synology diskstation

2015-11-13T20:50:22Z

Bernie: /* Mounting your Synology folder on your Linux workstation via WedDAV */

Pergamon is a Synology diskstation that can be accessed in a number of ways. It offers each group member 1 TB of storage space (this may be expanded in the future).

== Usage scenarios ==

=== Logging in via ssh ===

You have the same username as on the main cluster. Log in from your workstation via 'ssh pergamon'. This is not a full Linux implementation; the main purpose of the shell is to navigate around the file system. Your home folder is '/volume1/homes/USERNAME'.

=== Web access ===

https://pergamon.ms.northwestern.edu:5001

=== Copying data to diskstation via scp ===

You can use scp as you would use it for copying files to any other machine:
<pre>
scp LOCAL_FILES pergamon:
</pre>

=== Copying data to diskstation via rsync ===

You must specify your home folder, i.e., from your workstation do
<pre>
rsync -avz LOCAL_FILES pergamon:/volume1/homes/USERNAME
</pre>

=== Mounting your Synology folder on OS X ===

On your Mac, start Finder, select 'Go' and then 'Connect to Server'. In server address, enter:
<pre>
afp://pergamon.ms.northwestern.edu
</pre>
From the folders offered, select 'home'. This folder is now available in Finder under 'Shared: pergamon' and from the command line on your Mac via '/Volumes/home'.

As an alternative, you can also mount your folder via the WebDAV protocol. This works in the same way, except that you enter the following for the server address:
<pre>
https://pergamon.ms.northwestern.edu:5012
</pre>

=== Accessing your Synology folder on your Linux workstation via WebDAV ===

On Ubuntu, open Nautilus (see e.g. https://www.synology.com/en-global/knowledgebase/tutorials/610#t2_3) and choose '''Connect to server'''. Then enter as ''Server Address'':
<pre>
davs://pergamon.ms.northwestern.edu:5012
</pre>

On openSuSE, open Dolphin (from launcher go to "Computer" and then "Network"). Then choose '''Add Network Folder'''. In the dialog box, enter:
* '''Name:''' Pergamon
* '''User:''' your username
* '''Server:''' pergamon.ms.northwestern.edu
* '''Port:''' 5012
* '''Folder:''' /
* Check '''Use encryption'''. It is possible that the port is then reset to 443. If this happens, change it back to 5012.

Then proceed via '''Save & Connect'''. You will receive a warning about the SSL certificate. Accept this.

=== Mounting your Synology folder on your Linux workstation via WebDAV ===

Prerequisite: the davfs driver must have been installed and configured, see below under 'Enabling davfs'. Then, as regular user, do
<pre>
mount /mnt/pergamon
</pre>

To unmount, do
<pre>
umount /mnt/pergamon
</pre>

This works well, but one problem I observed is that the creation time of folders copied TO the Synology folder is not preserved when using 'cp -p' or 'cp -a'. In principle this should be possible (they are preserved when using rsync), so this appears to be a bug.

=== Mounting your Synology folder on your Linux workstation via NFS ===

This is currently not enabled.

== Administration ==

=== Creating a new user ===

* On the Control Panel, select ''User'', ''Create''.

** Choose the same username as in NIS.

** '''Join groups''': Only ''users''.

** '''Assign shared folder permissions''': ''read/write'' to '''homes''', ''no access'' to all other folders.

** '''User quota settings''': 1000 GB.

** '''Assign application permissions''':

** '''User speed limit setting''': no limit.

* After this, log in as root, and change /etc/passwd, such that for the new users '''/sbin/nologin''' is replaced with '''/bin/sh'''

* Also, while logged in as root, do '''chmod og-rx /volume1/homes/newuser''' (where '''newuser''' is the new username)

=== Enabling davfs on Linux ===

These instructions are for openSuSE 13.1.

* Install davfs2 (see https://software.opensuse.org/package/davfs2)
* Add this line to '''/etc/fstab''':
<pre>
https://pergamon:5012 /mnt/pergamon davfs rw,noauto,user,file_mode=600,dir_mode=700 0 0
</pre>
* Create mount point:
<pre>
mkdir /mnt/pergamon
</pre>

Gnuplot

2015-09-25T23:58:01Z

Bernie: /* General usage tips */

=== Overview ===

Gnuplot is a program for plotting scientific data. The resulting figures can be of very high quality, provided that attention is paid to detail. Impressive examples can be found on the [http://www.gnuplot.info/ gnuplot homepage]. '''This page does not provide a detailed manual for gnuplot''', but instead focuses on specific aspects typically encountered in the CSML when creating figures for manuscripts.

=== PDF terminal ===

The version of gnuplot installed on all [[hardware|local machines]] has been compiled with [http://www.pdflib.com/ PDFlib], which makes it possible to directly produce PDF images. Such figures offer the important advantage that they are fully scalable and yet have a small file size. Choose this terminal via

set term pdf enh [additional options]
set output "filename.pdf"

Sometimes, fine-tuning of the resulting PDF figure is most conveniently done in Adobe Illustrator: each component of the figure will appear as an object that you can move, resize, etc.

=== Encapsulated PostScript terminal ===

Often when the output of a gnupot script requires further formatting it is easiest to do so using the [http://en.wikipedia.org/wiki/Encapsulated_PostScript Encapsulated PostScript] (EPS) terminal. This is essentially the same as the PostScript terminal except that it contains some extra lines that allow it to interface with other programs. Invoke the EPS terminal via

set term post eps enh color [additional options]
set output "filename.eps"

For possible options see the instructions for the PostScript terminal on [http://www.gnuplotting.org/output-terminals/ this page]. One advantage of .eps files is that they can be edited directly in a text editor. See [[#eps-edit|below]] for an example.
Once the eps file has been formatted properly, you can convert it to PDF format using [http://www.ctan.org/pkg/epstopdf epstopdf]. Executing the command

epstopdf filename.eps

will create the PDF file <code>filename.pdf</code>.

=== Formatting labels ===

When choosing for instance an axis label via

set xlabel "textlabel"

it is straightforward to use simple text, however often special formatting is necessary. A brief set of rules follows. For italic script place

{/Helvetica-Oblique [text]}

inside the quotes, where <code>[text]</code> is what you wish to have in italics. All variables should be in italics. For greek letters use

{/Symbol [text]}

where <code>[text]</code> is any character off of the list linked [http://mathewpeet.org/lists/symbols/ here]. For italic Greek characters use

{/Symbol-Oblique [text]}

More instructions are available [http://www.gnuplotting.org/manpage-gnuplot-4-6/#Q1-1-42 here].

=== Sample usage script===

The following is a sample gnuplot script. Everything in braces should be replaced. Again, see the [http://www.gnuplot.info/ gnuplot homepage] or appropriate [http://www.gnuplotting.org/ tutorial] for more comprehensive usage.

set term post eps enh color {dashed|solid} lw 2
set output "filename.eps"
set encoding iso_8859_1
set key top right
set format x "%3.2f" #format tics to have max 3 digits and
set format y "%3.2f" #max 2 digits after the decimal
set xlabel "{xLabel}"
set ylabel "{yLabel}"
set xrange [{xmin}:{xmax}]
set yrange [{ymin}:{ymax}]
plot "datafile.dat" using 1:2:3 with errorbars

=== General usage tips ===

<ul>
<li>Journals typically prefer Helvetica as the typeface for figures.</li>
<li>The origin should be marked with '0', not '0.0'. If you produce a figure directly in PDF format, this can be fixed in Adobe Illustrator. However, it is easier to use the [[#Encapsulated PostScript terminal|EPS terminal]] and then edit the resulting .eps file using [http://www.grymoire.com/unix/sed.html sed]:
<pre>
sed -i 's/(0.0)/(0)/g' filename.eps
</pre>
</li>
<li>Gnuplot accepts abbreviated commands: you just have to enter enough characters to uniquely resolve a command. For example, it is acceptable to write
<pre>
plot "file.dat" us 1:2:3 w err
</pre>
instead of
<pre>plot "file.dat" using 1:2:3 with errorbars
</pre>
</li>
<li>Gnuplot can plot compressed data files by sending the data through a pipe:
<pre>
plot '< gzip -dc data.gz'
</pre>
</li>
<li>To plot the average of columns that are spread over multiple files (but have the same number of lines in each file), take advantage of the [[Command-line_interface_on_Linux/UNIX#paste|paste]] command in Unix to combine them in single file. For example, if three files each contain two columns (x y), use
<pre>
paste file1.dat file2.dat file3.dat > combined.dat
</pre>
and then within gnuplot
<pre>
plot "combined.dat" us 1:(($2+$4+$6)/3)
</pre>
This can even be done on the fly by using a pipe:
<pre>
plot '< paste file1.dat file2.dat file3.dat' using 1:(($2+$4+$6)/3)
</pre>
<li>Gnuplot can histogram data without the need for preprocessing:
<pre>
binwidth=[binwidth]
bin(x,width)=width*floor(x/width) + binwidth/2.0
set boxwidth binwidth
plot "file.dat" using (bin($[colnum],binwidth)):(1.0) smooth freq with boxes
</pre>
where the user should replace <tt>[binwidth]</tt> with the desired bin width and <tt>[colnum]</tt> with the column number of the data to be binned.
</ul>

=== Executable version for OS X ===

<ul>

<li>Compiled on OS X 10.10.3 (Yosemite)</li>

<li>Prior to using this, you need to install X11/XQuartz (http://xquartz.macosforge.org/landing/)</li>

<li>Current version is Gnuplot 5.0.1</li>

<li>Includes GNU Readline library and PDF terminal</li>

<li>Installation procedure is not very sophisticated, but the resulting installation will be correct. First download [http://ariadne.ms.northwestern.edu/Download/Gnuplot/gnuplot_osx_501.zip gnuplot_osx_501.zip] and save to <tt>/tmp</tt>. Then:
<pre>
cd /usr/local
sudo unzip /tmp/gnuplot_osx_501.zip
</pre>
</li>

<li>You can now invoke gnuplot from a terminal window</li>

<li>Note: Older version still available: [http://ariadne.ms.northwestern.edu/Download/Gnuplot/gnuplot_osx_465.zip version 4.6.5], compiled on OS X 10.9.3 (Mavericks)</li>
</ul>

Gnuplot

2015-09-25T22:36:55Z

Bernie: /* General usage tips */

=== Overview ===

Gnuplot is a program for plotting scientific data. The resulting figures can be of very high quality, provided that attention is paid to detail. Impressive examples can be found on the [http://www.gnuplot.info/ gnuplot homepage]. '''This page does not provide a detailed manual for gnuplot''', but instead focuses on specific aspects typically encountered in the CSML when creating figures for manuscripts.

=== PDF terminal ===

The version of gnuplot installed on all [[hardware|local machines]] has been compiled with [http://www.pdflib.com/ PDFlib], which makes it possible to directly produce PDF images. Such figures offer the important advantage that they are fully scalable and yet have a small file size. Choose this terminal via

set term pdf enh [additional options]
set output "filename.pdf"

Sometimes, fine-tuning of the resulting PDF figure is most conveniently done in Adobe Illustrator: each component of the figure will appear as an object that you can move, resize, etc.

=== Encapsulated PostScript terminal ===

Often when the output of a gnupot script requires further formatting it is easiest to do so using the [http://en.wikipedia.org/wiki/Encapsulated_PostScript Encapsulated PostScript] (EPS) terminal. This is essentially the same as the PostScript terminal except that it contains some extra lines that allow it to interface with other programs. Invoke the EPS terminal via

set term post eps enh color [additional options]
set output "filename.eps"

For possible options see the instructions for the PostScript terminal on [http://www.gnuplotting.org/output-terminals/ this page]. One advantage of .eps files is that they can be edited directly in a text editor. See [[#eps-edit|below]] for an example.
Once the eps file has been formatted properly, you can convert it to PDF format using [http://www.ctan.org/pkg/epstopdf epstopdf]. Executing the command

epstopdf filename.eps

will create the PDF file <code>filename.pdf</code>.

=== Formatting labels ===

When choosing for instance an axis label via

set xlabel "textlabel"

it is straightforward to use simple text, however often special formatting is necessary. A brief set of rules follows. For italic script place

{/Helvetica-Oblique [text]}

inside the quotes, where <code>[text]</code> is what you wish to have in italics. All variables should be in italics. For greek letters use

{/Symbol [text]}

where <code>[text]</code> is any character off of the list linked [http://mathewpeet.org/lists/symbols/ here]. For italic Greek characters use

{/Symbol-Oblique [text]}

More instructions are available [http://www.gnuplotting.org/manpage-gnuplot-4-6/#Q1-1-42 here].

=== Sample usage script===

The following is a sample gnuplot script. Everything in braces should be replaced. Again, see the [http://www.gnuplot.info/ gnuplot homepage] or appropriate [http://www.gnuplotting.org/ tutorial] for more comprehensive usage.

set term post eps enh color {dashed|solid} lw 2
set output "filename.eps"
set encoding iso_8859_1
set key top right
set format x "%3.2f" #format tics to have max 3 digits and
set format y "%3.2f" #max 2 digits after the decimal
set xlabel "{xLabel}"
set ylabel "{yLabel}"
set xrange [{xmin}:{xmax}]
set yrange [{ymin}:{ymax}]
plot "datafile.dat" using 1:2:3 with errorbars

=== General usage tips ===

<ul>
<li>Journals typically prefer Helvetica as the typeface for figures.</li>
<li>The origin should be marked with '0', not '0.0'. If you produce a figure directly in PDF format, this can be fixed in Adobe Illustrator. However, it is easier to use the [[#Encapsulated PostScript terminal|EPS terminal]] and then edit the resulting .eps file using [http://www.grymoire.com/unix/sed.html sed]:
<pre>
sed -i 's/(0.0)/(0)/g' filename.eps
</pre>
</li>
<li>Gnuplot accepts abbreviated commands: you just have to enter enough characters to uniquely resolve a command. For example, it is acceptable to write
<pre>
plot "file.dat" us 1:2:3 w err
</pre>
instead of
<pre>plot "file.dat" using 1:2:3 with errorbars
</pre>
</li>
<li>Gnuplot can plot compressed data files by sending the data through a pipe:
<pre>
plot '< gzip -dc data.gz'
</pre>
</li>
<li>To plot the average of columns that are spread over multiple files (but have the same number of lines in each file), take advantage of the [[Command-line_interface_on_Linux/UNIX#paste|paste]] command in Unix to combine them in single file. For example, if three files each contain two columns (x y), use
<pre>
paste file1.dat file2.dat file3.dat > combined.dat
</pre>
and then within gnuplot
<pre>
plot "combined.dat" us 1:(($2+$4+$6)/3)
</pre>
This can even be done on the fly by using a pipe:
<pre>
plot '< paste file1.dat file2.dat file3.dat' using 1:(($2+$4+$6)/3)
</pre>
<li>Gnuplot can histogram data without the need for preprocessing:
<pre>
rint(x)=(x-int(x)>0.9999)?int(x)+1:int(x)
binwidth=[binwidth]
bin(x,width)=width*rint(x/width) + binwidth/2.0
set boxwidth binwidth
plot "file.dat" using (bin($[colnum],binwidth)):1.0 smooth freq with boxes
</pre>
where the user should replace <tt>[binwidth]</tt> with the desired bin width and <tt>[colnum]</tt> with the column number of the data to be binned.
</ul>

=== Executable version for OS X ===

<ul>

<li>Compiled on OS X 10.10.3 (Yosemite)</li>

<li>Prior to using this, you need to install X11/XQuartz (http://xquartz.macosforge.org/landing/)</li>

<li>Current version is Gnuplot 5.0.1</li>

<li>Includes GNU Readline library and PDF terminal</li>

<li>Installation procedure is not very sophisticated, but the resulting installation will be correct. First download [http://ariadne.ms.northwestern.edu/Download/Gnuplot/gnuplot_osx_501.zip gnuplot_osx_501.zip] and save to <tt>/tmp</tt>. Then:
<pre>
cd /usr/local
sudo unzip /tmp/gnuplot_osx_501.zip
</pre>
</li>

<li>You can now invoke gnuplot from a terminal window</li>

<li>Note: Older version still available: [http://ariadne.ms.northwestern.edu/Download/Gnuplot/gnuplot_osx_465.zip version 4.6.5], compiled on OS X 10.9.3 (Mavericks)</li>
</ul>

Notes on Torque

2015-07-08T15:07:41Z

Bernie: /* Special usage notes */

=== Overview ===

The CSML uses Torque with Maui for most of its job scheduling. Please see the Cambridge Dept. of Chemistry's [http://www.ch.cam.ac.uk/computing/maui-and-torque-introduction user guide] for information on submitting and checking jobs, and the associated [http://www.ch.cam.ac.uk/computing/maui-administration admin guide] for more job scheduling tools.

=== General usage ===

=== Special usage notes ===

<ul>
<li>qb is a succinct queue monitoring script that tallies each user's running and queued jobs, total cores used and free, and down nodes (if any). The script was originally written by Daniel Sinkovits and should be available to copy from another user's bin to your own. Maintenance includes keeping the username list up-to-date and adjusting the variable <tt>node_total</tt> (currenlty 37) if a node goes down permanently.
<li>If qb indicates a node is down, check the ganglia of Minotaur or Hydra to find which node it is (down nodes are in red at the top of the list). Grep the output of qstat to find the jobs running on any down nodes with
<pre>
qstat -f | grep -B 3 [node_ID]
</pre>
where <tt>[node_ID]</tt> is the name of the down node, e.g., <tt>h036</tt> on Hydra. Make sure the owners of all jobs on the node have a chance to take note of which of their jobs went down and then restart the down nodes using Microway control.
<li>Change the total cpu time allotted to a job via
<pre>
qalter -l cput=[new_cput] [job_id]
</pre>
where <tt>[new_cput]</tt> is the desired amount of cpu time and <tt>[job_id]</tt> is the job's ID number. Multiple jobs can be changed simultaneously by using <tt>`seq [min_job_id] [max_job_id]`</tt> in the place of <tt>job_id</tt>. This will change all jobs <tt>[min_job_id]</tt> through <tt>[max_job_id]</tt>. Lowering the cpu time requirement of a job can decrease its wait time in the queue, as the scheduler is more likely to be able to use it for backfilling. On the other hand, increasing the cpu time requirement can be used to ensure that a job is able to finish properly (and can be done even while the job is running), but requires root permission.
</li>
<li>Move a queued (i.e., waiting) job to a different queue via
<pre>
qmove [destination] [job_id]
</pre>
where <tt>[destination]</tt> is the new queue (either 'fast' or 'default' for our system) and <tt>[job_id]</tt> is the job ID.
<li>Delete sequence of jobs via
<pre>
qdel `seq [min_job_id] [max_job_id]`
</pre>
where <tt>[min_job_id]</tt> is the first job ID of the sequence of jobs you want to delete and <tt>[max_job_id]</tt> is the last one.
</ul>

Notes on Torque

2015-07-08T15:07:17Z

Bernie: /* Special usage notes */

=== Overview ===

The CSML uses Torque with Maui for most of its job scheduling. Please see the Cambridge Dept. of Chemistry's [http://www.ch.cam.ac.uk/computing/maui-and-torque-introduction user guide] for information on submitting and checking jobs, and the associated [http://www.ch.cam.ac.uk/computing/maui-administration admin guide] for more job scheduling tools.

=== General usage ===

=== Special usage notes ===

<ul>
<li>qb is a succinct queue monitoring script that tallies each user's running and queued jobs, total cores used and free, and down nodes (if any). The script was originally written by Daniel Sinkovits and should be available to copy from any other user's bin to your own. Maintenance includes keeping the username list up-to-date and adjusting the variable <tt>node_total</tt> (currenlty 37) if a node goes down permanently.
<li>If qb indicates a node is down, check the ganglia of Minotaur or Hydra to find which node it is (down nodes are in red at the top of the list). Grep the output of qstat to find the jobs running on any down nodes with
<pre>
qstat -f | grep -B 3 [node_ID]
</pre>
where <tt>[node_ID]</tt> is the name of the down node, e.g., <tt>h036</tt> on Hydra. Make sure the owners of all jobs on the node have a chance to take note of which of their jobs went down and then restart the down nodes using Microway control.
<li>Change the total cpu time allotted to a job via
<pre>
qalter -l cput=[new_cput] [job_id]
</pre>
where <tt>[new_cput]</tt> is the desired amount of cpu time and <tt>[job_id]</tt> is the job's ID number. Multiple jobs can be changed simultaneously by using <tt>`seq [min_job_id] [max_job_id]`</tt> in the place of <tt>job_id</tt>. This will change all jobs <tt>[min_job_id]</tt> through <tt>[max_job_id]</tt>. Lowering the cpu time requirement of a job can decrease its wait time in the queue, as the scheduler is more likely to be able to use it for backfilling. On the other hand, increasing the cpu time requirement can be used to ensure that a job is able to finish properly (and can be done even while the job is running), but requires root permission.
</li>
<li>Move a queued (i.e., waiting) job to a different queue via
<pre>
qmove [destination] [job_id]
</pre>
where <tt>[destination]</tt> is the new queue (either 'fast' or 'default' for our system) and <tt>[job_id]</tt> is the job ID.
<li>Delete sequence of jobs via
<pre>
qdel `seq [min_job_id] [max_job_id]`
</pre>
where <tt>[min_job_id]</tt> is the first job ID of the sequence of jobs you want to delete and <tt>[max_job_id]</tt> is the last one.
</ul>

Notes on Torque

2015-07-08T15:06:52Z

Bernie: /* Special usage notes */

=== Overview ===

The CSML uses Torque with Maui for most of its job scheduling. Please see the Cambridge Dept. of Chemistry's [http://www.ch.cam.ac.uk/computing/maui-and-torque-introduction user guide] for information on submitting and checking jobs, and the associated [http://www.ch.cam.ac.uk/computing/maui-administration admin guide] for more job scheduling tools.

=== General usage ===

=== Special usage notes ===

<ul>
<li>qb is a succinct queue monitoring script that tallies each user's running and queued jobs, total cores used and free, and down nodes (if any). The script was originally written by Daniel Sinkovits and should be available to copy from any other user's directory to your own. Maintenance includes keeping the username list up-to-date and adjusting the variable <tt>node_total</tt> (currenlty 37) if a node goes down permanently.
<li>If qb indicates a node is down, check the ganglia of Minotaur or Hydra to find which node it is (down nodes are in red at the top of the list). Grep the output of qstat to find the jobs running on any down nodes with
<pre>
qstat -f | grep -B 3 [node_ID]
</pre>
where <tt>[node_ID]</tt> is the name of the down node, e.g., <tt>h036</tt> on Hydra. Make sure the owners of all jobs on the node have a chance to take note of which of their jobs went down and then restart the down nodes using Microway control.
<li>Change the total cpu time allotted to a job via
<pre>
qalter -l cput=[new_cput] [job_id]
</pre>
where <tt>[new_cput]</tt> is the desired amount of cpu time and <tt>[job_id]</tt> is the job's ID number. Multiple jobs can be changed simultaneously by using <tt>`seq [min_job_id] [max_job_id]`</tt> in the place of <tt>job_id</tt>. This will change all jobs <tt>[min_job_id]</tt> through <tt>[max_job_id]</tt>. Lowering the cpu time requirement of a job can decrease its wait time in the queue, as the scheduler is more likely to be able to use it for backfilling. On the other hand, increasing the cpu time requirement can be used to ensure that a job is able to finish properly (and can be done even while the job is running), but requires root permission.
</li>
<li>Move a queued (i.e., waiting) job to a different queue via
<pre>
qmove [destination] [job_id]
</pre>
where <tt>[destination]</tt> is the new queue (either 'fast' or 'default' for our system) and <tt>[job_id]</tt> is the job ID.
<li>Delete sequence of jobs via
<pre>
qdel `seq [min_job_id] [max_job_id]`
</pre>
where <tt>[min_job_id]</tt> is the first job ID of the sequence of jobs you want to delete and <tt>[max_job_id]</tt> is the last one.
</ul>

Notes on Torque

2015-06-25T20:47:41Z

Bernie:

=== Overview ===

The CSML uses Torque with Maui for most of its job scheduling. Please see the Cambridge Dept. of Chemistry's [http://www.ch.cam.ac.uk/computing/maui-and-torque-introduction user guide] for information on submitting and checking jobs, and the associated [http://www.ch.cam.ac.uk/computing/maui-administration admin guide] for more job scheduling tools.

=== General usage ===

=== Special usage notes ===

<ul>
<li>If qb indicates a node is down, check the ganglia of Minotaur or Hydra to find which node it is (down nodes are in red at the top of the list). Grep the output of qstat to find the jobs running on any down nodes with
<pre>
qstat -f | grep -B 3 [node_ID]
</pre>
where <tt>[node_ID]</tt> is the name of the down node, e.g., <tt>h036</tt> on Hydra. Make sure the owners of all jobs on the node have a chance to take note of which of their jobs went down and then restart the down nodes using Microway control.
<li>Change the total cpu time allotted to a job via
<pre>
qalter -l cput=[new_cput] [job_id]
</pre>
where <tt>[new_cput]</tt> is the desired amount of cpu time and <tt>[job_id]</tt> is the job's ID number. Multiple jobs can be changed simultaneously by using <tt>`seq [min_job_id] [max_job_id]`</tt> in the place of <tt>job_id</tt>. This will change all jobs <tt>[min_job_id]</tt> through <tt>[max_job_id]</tt>. Lowering the cpu time requirement of a job can decrease its wait time in the queue, as the scheduler is more likely to be able to use it for backfilling. On the other hand, increasing the cpu time requirement can be used to ensure that a job is able to finish properly (and can be done even while the job is running), but requires root permission.
</li>
<li>Move a queued (i.e., waiting) job to a different queue via
<pre>
qmove [destination] [job_id]
</pre>
where <tt>[destination]</tt> is the new queue (either 'fast' or 'default' for our system) and <tt>[job_id]</tt> is the job ID.
<li>Delete sequence of jobs via
<pre>
qdel `seq [min_job_id] [max_job_id]`
</pre>
where <tt>[min_job_id]</tt> is the first job ID of the sequence of jobs you want to delete and <tt>[max_job_id]</tt> is the last one.
</ul>

Hardware

2015-05-02T21:27:29Z

Bernie: /* Checking RAID status */

=== Desktop machines ===

All desktop machines run [http://www.opensuse.org OpenSuSE]. [[Installation instructions for OpenSuSE 13.1]].

[http://ariadne.ms.northwestern.edu/speedtest/ Server connection speed]

[[Etiquette: Running Nice on other desktop machines]]

=== Clusters ===

==== Minotaur ====
* 38 nodes, each containing two 4-core processors (304 cores total). 8 GB memory per node. Processor type: Intel Xeon E5472, 3.0 GHz.
* Jobs are scheduled via [http://www.adaptivecomputing.com/products/open-source/torque/ Torque]/Maui. [[Notes on Torque]].

==== Hydra ====
* 60 nodes, each containing two 6-core processors (720 cores total). 12 GB memory per node. 8 nodes (queue "fast", nodes h001-h008) have Intel Xeon X5690 3.47 GHz processors. 52 nodes (queue "default", nodes h009-h060) have Intel Xeon E5645 2.40 GHz processors.
* Jobs are scheduled via [http://www.adaptivecomputing.com/products/open-source/torque/ Torque]/Maui. [[Notes on Torque]].
* [[General Usage of Hydra]]

==== Quest ====
* Jobs are scheduled via [http://www.adaptivecomputing.com/products/open-source/torque/ Torque]/Moab. [[Notes on Torque]].
* [[General Usage of Quest]]

=== Disk space, backups, and RAID storage ===

==== Disk space allocations and nightly backups ====

Each user has a home directory located on ''ariadne''. This home directory is exported to all desktop machines, so that you see the same home filesystem on each machine. The drive is protected against hardware failure via a [[http://en.wikipedia.org/wiki/RAID_1#RAID_1 RAID-1]] setup. Furthermore, each night all new or modified files on /home are written to tape (located in ariadne). This makes it important not to store temporary data in your home folder, as it would quickly fill up the tape. Since users tend to forget this, a quota system has been enabled on ariadne, restricting each user to 15 GB. To check how much space you are using log on to ariadne and issue the command
<pre>
quota -s
</pre>

In addition, each user has significant additional storage on the scratch partitions. These drives are located in the different desktop machines and protected via RAID-1, but backups are your own responsibility. Note that these partitions are generally only mounted on the desktop machine that contains the corresponding drives. If you need a partition to be exported to a different machine, please ask.

==== Changing the nightly backup tape ====

# Press eject button on tape drive in ariadne.
# Take the tape cartridge out of the drive and put it in its box (should be on top of ariadne). Label the box. Give to Erik.
# Insert cleaning tape (on top of ariadne). It will work for less than a minute and then eject automatically.
# Put cleaning tape back in box on top of ariadne.
# Insert new DDS tape (find in cabinet). Leave empty box on top of ariadne.
# Erik: Update settings in /usr/local/lib/backup, namely ''position'' and ''tapenumber''; update logfile.

==== Recovering data from the nightly backup tape ====

Log files of all nightly backup tapes are located on ariadne, in /usr/local/lib/backup. For privacy reasons, these logfiles are only accessible to root. Once the proper file to be recovered has been identified, insert the corresponding tape into the drive on ariadne and follow these steps (all to be executed as root):

# <tt>cd /</tt> (if you change to a different directory, the recovered file will be placed relative to this directory)
# <tt>/usr/local/bin/tape-rewind</tt> (or <tt>mtst -f /dev/nst0 rewind</tt>)
# <tt>mtst -f /dev/nst0 fsf <position></tt> (see the contents file in /usr/local/lib/backup for the position number)
# <tt>tar xzvf /dev/nst0 <full_file_name_without_leading_slash></tt> This step won't work unless you omit the leading slash; also note that you can specify multiple files, separated by spaces. The 'z' option is necessary because all nightly backups are compressed. For wildcards, use --wildcards and escape '*' and '?'. For example: <tt>tar -x --wildcards -zvf /dev/nst0 \*datafiles\*</tt>
# <tt>/usr/local/bin/tape-rewoffl</tt> (or <tt>mtst -f /dev/nst0 rewoffl</tt>)

==== Archiving data using the LTO tape drive ====

==== Checking RAID status ====

<ul>

<li>Hydra
<ul>
<li>OS is on software RAID (which spans /dev/sda and /dev/sdb). An overview is obtained via
<pre>cat /proc/mdstat</pre>
Detailed information via
<pre>
mdadm --detail /dev/mdX
</pre>
where X = 1, 5, 6, 7. Also see [[Setting up e-mail notifications for Linux Software RAID]].</li>
<li>/home</li>
<li>/archive</li>
</ul>
</li>

<li>Minotaur Web interface. Log in to the head node and use
<pre>
opera http://172.16.0.101
</pre>
RAID status is visible on the top line of the Raid Set Heirarchy table, under Volume State. For drive stability click any channel and find the SMART Attributes at the bottom of the page. Each has two values, Attribute and Threshold (Threshold is in parentheses). An Attribute value lower than Threshold indicates an unstable drive.
</li>

<li>Ariadne RAID-5 controller with 4 drives. Status can be checked by interrogating the controller:
<pre>
/opt/MegaRAID/MegaCli/MegaCli64 -AdpAllInfo -aALL | less
</pre>
In the 'Device Present' section, it is reported if any drives are critical or have failed, and what the state of the RAID is. More detailed information can also be found via
<pre>
/opt/MegaRAID/MegaCli/MegaCli64 -LDPDInfo -aAll | less
</pre>
Directly at the beginning (under 'Adapter #0') it should report 'State: Optimal'</li>

<li>Desktop machines, except pelops Hardware RAID-1. The RAID status is reported upon reboot of a machine. Press Ctrl-C (when prompted) to enter the configuration utility. From within Linux, use (as root):
<pre>
mpt-status -i 0
mpt-status -i 2
</pre>
The second command only applies to machines with a second set of hard drives (achilles, agamemnon, nestor, poseidon) 
To allow regular users to verify the RAID status, the <tt>mpt-status</tt> has been added to <tt>sudo</tt>:
<pre>
sudo mpt-status -i 0
sudo mpt-status -i 2
</pre>
</li>

<li>Pelops: Software RAID (for OS and scratch partitions). See [[#hydra|Hydra]].</li>

</ul>

=== Printers ===
There are two black-and-white laser printers (PS1 and PS2) in the lab, both supporting double-sided printing. For network printing, use luijten-ps1.ms.northwestern.edu and luijten-ps2.ms.northwestern.edu, respectively. For OS X, choose "HP JetDirect" as the protocol.

=== Scanner ===

=== UPS ===

All our UPS units are manufactured by APC, and supported via apcupsd. Installation & configuration instructions:

<ul>

<li>Make sure sure the <tt>apcupsd</tt> package is installed, see [[Installation instructions for OpenSuSE 13.1]].</li>

<li>Connect UPS unit to USB port of the corresponding machine.</li>

<li>In <tt>/etc/apcupsd/apcupsd.conf</tt> edit these lines:
<pre>
UPSCABLE usb
UPSTYPE usb
</pre>
Also, '''comment out''' the <tt>DEVICE</tt> line.
</li>

<li>From command line, do
<pre>
chkconfig apcupsd on
</pre>
</li>

<li>Start the daemon manually:
<pre>
apcupsd
</pre>

<li>Test it:
<pre>
apcaccess
</pre>
This should produce extensive output regarding the UPS unit. 
(Note: this command also works for regular users; in that case use <tt>/usr/sbin/apcaccess</tt>.)

</li>

</ul>

Gnuplot

2015-04-01T22:45:49Z

Bernie:

=== Overview ===

Gnuplot is a program for plotting scientific data. The resulting figures can be of very high quality, provided that attention is paid to detail. Impressive examples can be found on the [http://www.gnuplot.info/ gnuplot homepage]. '''This page does not provide a detailed manual for gnuplot''', but instead focuses on specific aspects typically encountered in the CSML when creating figures for manuscripts.

=== PDF terminal ===

The version of gnuplot installed on all [[hardware|local machines]] has been compiled with [http://www.pdflib.com/ PDFlib], which makes it possible to directly produce PDF images. Such figures offer the important advantage that they are fully scalable and yet have a small file size. Choose this terminal via

set term pdf enh [additional options]
set output "filename.pdf"

Sometimes, fine-tuning of the resulting PDF figure is most conveniently done in Adobe Illustrator: each component of the figure will appear as an object that you can move, resize, etc.

=== Encapsulated PostScript terminal ===

Often when the output of a gnupot script requires further formatting it is easiest to do so using the [http://en.wikipedia.org/wiki/Encapsulated_PostScript Encapsulated PostScript] (EPS) terminal. This is essentially the same as the PostScript terminal except that it contains some extra lines that allow it to interface with other programs. Invoke the EPS terminal via

set term post eps enh color [additional options]
set output "filename.eps"

For possible options see the instructions for the PostScript terminal on [http://www.gnuplotting.org/output-terminals/ this page]. One advantage of .eps files is that they can be edited directly in a text editor. See [[#eps-edit|below]] for an example.
Once the eps file has been formatted properly, you can convert it to PDF format using [http://www.ctan.org/pkg/epstopdf epstopdf]. Executing the command

epstopdf filename.eps

will create the PDF file <code>filename.pdf</code>.

=== Formatting labels ===

When choosing for instance an axis label via

set xlabel "textlabel"

it is straightforward to use simple text, however often special formatting is necessary. A brief set of rules follows. For italic script place

{/Helvetica-Oblique [text]}

inside the quotes, where <code>[text]</code> is what you wish to have in italics. All variables should be in italics. For greek letters use

{/Symbol [text]}

where <code>[text]</code> is any character off of the list linked [http://mathewpeet.org/lists/symbols/ here]. For italic Greek characters use

{/Symbol-Oblique [text]}

More instructions are available [http://www.gnuplotting.org/manpage-gnuplot-4-6/#Q1-1-42 here].

=== Sample usage script===

The following is a sample gnuplot script. Everything in braces should be replaced. Again, see the [http://www.gnuplot.info/ gnuplot homepage] or appropriate [http://www.gnuplotting.org/ tutorial] for more comprehensive usage.

set term post eps enh color {dashed|solid} lw 2
set output "filename.eps"
set encoding iso_8859_1
set key top right
set format x "%3.2f" #format tics to have max 3 digits and
set format y "%3.2f" #max 2 digits after the decimal
set xlabel "{xLabel}"
set ylabel "{yLabel}"
set xrange [{xmin}:{xmax}]
set yrange [{ymin}:{ymax}]
plot "datafile.dat" using 1:2:3 with errorbars

=== General usage tips ===

<ul>
<li>Journals typically prefer Helvetica as the typeface for figures.</li>
<li>The origin should be marked with '0', not '0.0'. If you produce a figure directly in PDF format, this can be fixed in Adobe Illustrator. However, it is easier to use the [[#Encapsulated PostScript terminal|EPS terminal]] and then edit the resulting .eps file using [http://www.grymoire.com/unix/sed.html sed]:
<pre>
sed -i 's/(0.0)/(0)/g' filename.eps
</pre>
</li>
<li>Gnuplot accepts abbreviated commands: you just have to enter enough characters to uniquely resolve a command. For example, it is acceptable to write
<pre>
plot "file.dat" us 1:2:3 w err
</pre>
instead of
<pre>plot "file.dat" using 1:2:3 with errorbars
</pre>
</li>
<li>Gnuplot can plot compressed data files by sending the data through a pipe:
<pre>
plot '< gzip -dc data.gz'
</pre>
</li>
<li>To plot the average of columns that are spread over multiple files (but have the same number of lines in each file), take advantage of the [[Command-line_interface_on_Linux/UNIX#paste|paste]] command in Unix to combine them in single file. For example, if three files each contain two columns (x y), use
<pre>
paste file1.dat file2.dat file3.dat > combined.dat
</pre>
and then within gnuplot
<pre>
plot "combined.dat" us 1:(($2+$4+$6)/3)
</pre>
This can even be done on the fly by using a pipe:
<pre>
plot '< paste file1.dat file2.dat file3.dat' using 1:(($2+$4+$6)/3)
</pre>
</ul>

=== Executable version for OS X ===

<ul>

<li>Compiled on OS X 10.9.3 (Mavericks), but has been tested to also work on OS X 10.10 (Yosemite)</li>

<li>Prior to using this, you need to install X11/XQuartz (http://xquartz.macosforge.org/landing/)</li>

<li>Current version is Gnuplot 4.6.5</li>

<li>Includes GNU Readline library and PDF terminal</li>

<li>Installation procedure is not very sophisticated, but the resulting installation will be correct. First download [http://ariadne.ms.northwestern.edu/Download/Gnuplot/gnuplot_osx_465.zip gnuplot_osx_465.zip] and save to <tt>/tmp</tt>. Then:
<pre>
sudo bash
cd /usr/local
unzip /tmp/gnuplot_osx_465.zip
exit
</pre>
</li>

<li>You can now invoke gnuplot from a terminal window</li>

</ul>

Notes on Torque

2015-01-28T20:31:42Z

Bernie: /* Special usage notes */

=== Overview ===

=== General usage ===

=== Special usage notes ===

<ul>
<li>If qb indicates a node is down, check the ganglia of Minotaur or Hydra to find which node it is (down nodes are in red at the top of the list). Grep the output of qstat to find the jobs running on any down nodes with
<pre>
qstat -f | grep -B 3 [node_ID]
</pre>
where <tt>[node_ID]</tt> is the name of the down node, e.g., <tt>h036</tt> on Hydra. Make sure the owners of all jobs on the node have a chance to take note of which of their jobs went down and then restart the down nodes using Microway control.
<li>Change the total cpu time allotted to a job via
<pre>
qalter -l cput=[new_cput] [job_id]
</pre>
where <tt>[new_cput]</tt> is the desired amount of cpu time and <tt>[job_id]</tt> is the job's ID number. Multiple jobs can be changed simultaneously by using <tt>`seq [min_job_id] [max_job_id]`</tt> in the place of <tt>job_id</tt>. This will change all jobs <tt>[min_job_id]</tt> through <tt>[max_job_id]</tt>. Lowering the cpu time requirement of a job can decrease its wait time in the queue, as the scheduler is more likely to be able to use it for backfilling. On the other hand, increasing the cpu time requirement can be used to ensure that a job is able to finish properly (and can be done even while the job is running), but requires root permission.
</li>
<li>Move a queued (i.e., waiting) job to a different queue via
<pre>
qmove [destination] [job_id]
</pre>
where <tt>[destination]</tt> is the new queue (either 'fast' or 'default' for our system) and <tt>[job_id]</tt> is the job ID.
<li>Delete sequence of jobs via
<pre>
qdel `seq [min_job_id] [max_job_id]`
</pre>
where <tt>[min_job_id]</tt> is the first job ID of the sequence of jobs you want to delete and <tt>[max_job_id]</tt> is the last one.
</ul>

Notes on Torque

2015-01-28T20:30:41Z

Bernie: /* Special usage notes */

=== Overview ===

=== General usage ===

=== Special usage notes ===

<ul>
<li>If qb indicates a node is down, check the ganglia of Minotaur or Hydra to find which node it is (down nodes are in red at the top of the list). Grep the output of qstat to find the jobs running on any down nodes with
<pre>
qstat -f | grep -B 3 [node_ID]
</pre>
where <tt>[node_ID]</tt> is the name of the down node, e.g., <tt>h036</tt> on Hydra. Make sure the owners of all jobs on the node have a chance to take note of which of their jobs went down and then restart the down nodes using Microway control.
<li>Change the total cpu time allotted to a job via
<pre>
qalter -l cput=[new_cput] [job_id]
</pre>
where <tt>[new_cput]</tt> is the desired amount of cpu time and <tt>[job_id]</tt> is the job's ID number. Multiple jobs can be changed simultaneously by using <tt>`seq [min_job_id] [max_job_id]`</tt> in the place of <tt>job_id</tt>. This will change all jobs <tt>[min_job_id]</tt> through <tt>[max_job_id]</tt>. Lowering the cpu time requirement of a job can decrease its wait time in the queue, as the scheduler is more likely to be able to use it for backfilling. On the other hand, increasing the cpu time requirement can be used to ensure that a job is able to finish properly (and can be done even while the job is running), but requires root permission.
</li>
<li>Move a queued (i.e., waiting) job to a different queue via
<pre>
qmove [destination] [job_id]
</pre>
where <tt>[destination]</tt> is the new queue (either 'fast' or 'default' for our system) and <tt>[job_id]</tt> is the job ID.
<li>Delete sequence of jobs via
<pre>
qdel `seq [job_id1] [job_id2]`
</pre>
where <tt>[job_id1]</tt> is the first job ID of the sequence of jobs you want to delete and <tt>[job_id2]</tt> is the last one.
</ul>

Miscellaneous notes

2015-01-28T16:22:15Z

Bernie:

== Introduction ==

This page contains assorted topics relevant to day-to-day operations in the lab. For any questions not covered, please contact the MSE office at (847) 491-3537.

==Item Disposal==

The custodial staff can dispose of most items. Reusable items such as computer supplies can be disposed of via [http://www.northwestern.edu/uservices/office/computer/staff.html University Services]. Follow the appropriate "Schedule a Pick-Up" link for the item you are disposing of.

== Carpet cleaning ==

Schedule a carpet cleaning every 6-12 months by contacting [mailto:d-dennett@northwestern.edu Deirdre Dennett]. An annual reminder for September 19th is set on the group calendar, however there is no need to wait until then.

==Ordering supplies==

Order supplies with the following steps:
*Sign up for an iBuyNU account.
**Call the MSE office at (847) 491-3537.
**Obtain and fill out all necessary forms.
**There is a waiting time of up to two weeks.
*[https://nuportal.northwestern.edu/wps/nuportal/ibuynu/ibuynu.html Log on] and select any items from any suppliers you like.
*"Check out" from the individual suppliers so you have an iBuyNU shopping cart.
*Send a PDF copy of the cart to Erik and request that he forward his approval to msepurchasing@northwestern.edu.
*Use the Assign Cart button to assign your cart to [mailto:boe.welter@northwestern.edu Boe Welter].

Miscellaneous notes

2015-01-19T22:49:32Z

Bernie: Random lab tasks

== Introduction ==

This page contains assorted topics relevant to day-to-day operations in the lab. For any questions not covered, please contact the MSE office at (847) 491-3537.

== Carpet cleaning ==

Schedule a carpet cleaning every 6-12 months by contacting [mailto:d-dennett@northwestern.edu Deirdre Dennett]. An annual reminder for September 19th is set on the group calendar, however there is no need to wait until then.

==Ordering supplies==

Order supplies with the following steps:
*Sign up for an iBuyNU account.
**Call the MSE office at (847) 491-3537.
**Obtain and fill out all necessary forms.
**There is a waiting time of up to two weeks.
*[https://nuportal.northwestern.edu/wps/nuportal/ibuynu/ibuynu.html Log on] and select any items from any suppliers you like.
*"Check out" from the individual suppliers so you have an iBuyNU shopping cart.
*Send a PDF copy of the cart to Erik and request that he forward his approval to msepurchasing@northwestern.edu.
*Use the Assign Cart button to assign your cart to [mailto:boe.welter@northwestern.edu Boe Welter].

User:Bernie

2015-01-19T21:48:15Z

Bernie:

Bernard Beckerman has been in the Luijten group since September 2010 and specializes in simulations of biological many-body systems, especially surfaces and biointerfaces. Bernard has a BS in Physics and Mathematics from NYU where he did research with David Pine on the rheometry of liquid crystals.

Simulations

2014-08-14T19:03:13Z

Bernie: /* LAMMPS Special Usage Notes */

== Introduction ==

Assorted topics relevant to programming particle-based simulation codes and to using these codes for the modeling of a wide range of systems, notably complex fluids.

== Molecular dynamics simulations ==

By and large the CSML uses [http://lammps.sandia.gov/ LAMMPS] for MD, though depending on the application [http://www.ks.uiuc.edu/Research/namd/ NAMD], [http://www.gromacs.org/ GROMACS], or any of [http://en.wikipedia.org/wiki/List_of_software_for_molecular_mechanics_modeling a host of other packages] may be useful. Most issues can be resolved by consulting the [http://lammps.sandia.gov/doc/Manual.html LAMMPS manual], though some common problems are addressed below.

===LAMMPS Special Usage Notes===

<ul>

<li>
By default LAMMPS normalizes the temperature by an amount <math>n_\text{dof} - d</math>, where <math>n_\text{dof}</math> is the system's total number of degrees of freedom and <math>d</math> is the system's dimensionality. Subtracting <math>d</math> accounts for the center-of-mass motion of the system. This leads to an incorrect reported value if the system has a proper frame of reference, e.g., when using a [http://lammps.sandia.gov/doc/fix_langevin.html Langevin thermostat] in which all particles interact with a stationary background solvent. In this case it is necessary to ensure <math>n_\text{dof}</math> is used instead of <math>n_\text{dof} - d</math>. To do this, use [http://lammps.sandia.gov/doc/compute_modify.html compute_modify] as follows

<pre>
compute myTemp all temp
compute_modify myTemp extra 0
thermo_modify temp myTemp
</pre>

As a note, the above only affects the reported temperature. The dynamics are computed correctly regardless.
</li>

</ul>

== Monte Carlo simulations ==

Notes on Torque

2014-06-13T02:30:48Z

Bernie: /* Special usage notes */

=== Overview ===

=== General usage ===

=== Special usage notes ===

<ul>
<li>If qb indicates a node is down, check the ganglia of Minotaur or Hydra to find which node it is (down nodes are in red at the top of the list). Grep the output of qstat to find the jobs running on any down nodes with
<pre>
qstat -f | grep -B 3 [node_ID]
</pre>
where <tt>[node_ID]</tt> is the name of the down node, e.g., <tt>h036</tt> on Hydra. Make sure the owners of all jobs on the node have a chance to take note of which of their jobs went down and then restart the down nodes using Microway control.
<li>Change the total cpu time allotted to a job via
<pre>
qalter -l cput=[new_cput] [job_id]
</pre>
where <tt>[new_cput]</tt> is the desired amount of cpu time and <tt>[job_id]</tt> is the job's ID number. Multiple jobs can be changed simultaneously by using <tt>`seq [min_job_id] [max_job_id]`</tt> in the place of <tt>job_id</tt>. This will change all jobs <tt>[min_job_id]</tt> through <tt>[max_job_id]</tt>. Lowering the cpu time requirement of a job can decrease its wait time in the queue, as the scheduler is more likely to be able to use if for backfilling. On the other hand, increasing the cpu time requirement can be used to ensure that a job is able to finish properly (and can be done even while the job is running), but requires root permission.
</li>
<li>Move a queued (i.e., waiting) job to a different queue via
<pre>
qmove [destination] [job_id]
</pre>
where <tt>[destination]</tt> is the new queue (either 'fast' or 'default' for our system) and <tt>[job_id]</tt> is the job ID.
<li>Delete sequence of jobs via
<pre>
qdel `seq [job_id1] [job_id2]`
</pre>
where <tt>[job_id1]</tt> is the first job ID of the sequence of jobs you want to delete and <tt>[job_id2]</tt> is the last one.
</ul>

Notes on Torque

2014-06-13T02:23:27Z

Bernie: /* Special usage notes */

=== Overview ===

=== General usage ===

=== Special usage notes ===

<ul>
<li>If qb indicates a node is down, check the ganglia of [http://minotaur.research.northwestern.edu/ganglia/index.php Minotaur] or [https://hydra.research.northwestern.edu/ganglia/ Hydra] to find which node it is (down nodes are in red at the top of the list). Grep the output of qstat to find the jobs running on any down nodes with
<pre>
qstat -f | grep -B 3 [node_ID]
</pre>
where <tt>[node_ID]</tt> is the name of the down node, e.g., <tt>h036</tt> on Hydra. Make sure the owners of all jobs on the node have a chance to take note of which of their jobs went down and then restart the down nodes using Microway control.
<li>Change the total cpu time allotted to a job via
<pre>
qalter -l cput=[new_cput] [job_id]
</pre>
where <tt>[new_cput]</tt> is the desired amount of cpu time and <tt>[job_id]</tt> is the job's ID number. Multiple jobs can be changed simultaneously by using <tt>`seq [min_job_id] [max_job_id]`</tt> in the place of <tt>job_id</tt>. This will change all jobs <tt>[min_job_id]</tt> through <tt>[max_job_id]</tt>. Lowering the cpu time requirement of a job can decrease its wait time in the queue, as the scheduler is more likely to be able to use if for backfilling. On the other hand, increasing the cpu time requirement can be used to ensure that a job is able to finish properly (and can be done even while the job is running), but requires root permission.
</li>
<li>Move a queued (i.e., waiting) job to a different queue via
<pre>
qmove [destination] [job_id]
</pre>
where <tt>[destination]</tt> is the new queue (either 'fast' or 'default' for our system) and <tt>[job_id]</tt> is the job ID.
<li>Delete sequence of jobs via
<pre>
qdel `seq [job_id1] [job_id2]`
</pre>
where <tt>[job_id1]</tt> is the first job ID of the sequence of jobs you want to delete and <tt>[job_id2]</tt> is the last one.
</ul>

Simulations

2014-06-12T22:56:58Z

Bernie: /* Molecular dynamics simulations */

== Introduction ==

Assorted topics relevant to programming particle-based simulation codes and to using these codes for the modeling of a wide range of systems, notably complex fluids.

== Molecular dynamics simulations ==

By and large the CSML uses [http://lammps.sandia.gov/ LAMMPS] for MD, though depending on the application [http://www.ks.uiuc.edu/Research/namd/ NAMD], [http://www.gromacs.org/ GROMACS], or any of [http://en.wikipedia.org/wiki/List_of_software_for_molecular_mechanics_modeling a host of other packages] may be useful. Most issues can be resolved by consulting the [http://lammps.sandia.gov/doc/Manual.html LAMMPS manual], though some common problems are addressed below.

===LAMMPS Special Usage Notes===
By default LAMMPS normalizes the temperature by an amount <math>n_\text{dof} - d</math>, where <math>n_\text{dof}</math> is the system's total number of degrees of freedom and <math>d</math> is the system's dimensionality. Subtracting <math>d</math> accounts for the center-of-mass motion of the system. This leads to an incorrect ''reported'' value if the system has a proper frame of reference, e.g., when using a [http://lammps.sandia.gov/doc/fix_langevin.html Langevin thermostat] in which all particles interact with a stationary background solvent. In this case it is necessary to ensure <math>n_\text{dof}</math> is used instead of <math>n_\text{dof} - d</math>. To do this, use the command [http://lammps.sandia.gov/doc/compute_modify.html compute_modify] as follows

<pre>
compute myTemp all temp
compute_modify myTemp extra 0
thermo_modify temp myTemp
</pre>

== Monte Carlo simulations ==

Simulations

2014-06-12T21:59:23Z

Bernie: /* Special Usage Notes */

== Introduction ==

Assorted topics relevant to programming particle-based simulation codes and to using these codes for the modeling of a wide range of systems, notably complex fluids.

== Molecular dynamics simulations ==

===Special Usage Notes===
By default LAMMPS normalizes the temperature by an amount <math>n_\text{dof} - d</math>, where <math>n_\text{dof}</math> is the system's total number of degrees of freedom and <math>d</math> is the system's dimensionality. Subtracting <math>d</math> accounts for the center-of-mass motion of the system. This is incorrect if the system has a proper frame of reference, e.g., when using a Langevin thermostat in which all particles interact with a stationary background solvent. In this case it is necessary to ensure <math>n_\text{dof}</math> is used instead of <math>n_\text{dof} - d</math>. To do this, use the command [http://lammps.sandia.gov/doc/compute_modify.html compute_modify] as follows

<pre>
compute myTemp all temp
compute_modify myTemp extra 0
thermo_modify temp myTemp
</pre>

== Monte Carlo simulations ==

Simulations

2014-06-08T00:14:38Z

Bernie: /* Special Usage Notes */

== Introduction ==

Assorted topics relevant to programming particle-based simulation codes and to using these codes for the modeling of a wide range of systems, notably complex fluids.

== Molecular dynamics simulations ==

===Special Usage Notes===
By default LAMMPS normalizes the temperature by an amount ''dof'' – ''d'', where ''dof'' is the system's total number of degrees of freedom and ''d'' is the system's dimensionality. Subtracting ''d'' accounts for the center-of-mass motion of the system. This is incorrect if the system has a proper frame of reference, e.g., when using a Langevin thermostat in which all particles interact with a stationary background solvent. In this case it is necessary to ensure ''dof'' is used instead of ''dof'' – ''d''. To do this, use the command [http://lammps.sandia.gov/doc/compute_modify.html compute_modify] as follows

<pre>
compute myTemp all temp
compute_modify myTemp extra 0
thermo_modify temp myTemp
</pre>

== Monte Carlo simulations ==

Simulations

2014-06-08T00:03:00Z

Bernie: /* Special Usage Notes */

== Introduction ==

Assorted topics relevant to programming particle-based simulation codes and to using these codes for the modeling of a wide range of systems, notably complex fluids.

== Molecular dynamics simulations ==

===Special Usage Notes===
By default LAMMPS normalizes the temperature by an amount ''dof'' – ''d'', where ''dof'' is the system's total number of degrees of freedom and ''d'' is the dimensionality of the simulation. Subtracting ''d'' accounts for the center-of-mass motion of the system. This is correct except when there is a proper frame of reference for the system, e.g., in a Langevin dynamics simulation in which the implicit background solvent has zero average velocity. In this case it is necessary to use the command [http://lammps.sandia.gov/doc/compute_modify.html compute_modify] to ensure ''dof'' is used instead of ''dof'' – ''d''. For instance one can use

<pre>
compute myTemp all temp
compute_modify myTemp extra 0
thermo_modify temp myTemp
</pre>

== Monte Carlo simulations ==

Simulations

2014-06-08T00:02:25Z

Bernie: /* Special Usage Notes */

== Introduction ==

Assorted topics relevant to programming particle-based simulation codes and to using these codes for the modeling of a wide range of systems, notably complex fluids.

== Molecular dynamics simulations ==

===Special Usage Notes===
By default, LAMMPS normalizes the temperature by an amount ''dof'' – ''d'', where ''dof'' is the system's total number of degrees of freedom and ''d'' is the dimensionality of the simulation. Subtracting ''d'' accounts for the center-of-mass motion of the system. This is correct except when there is a proper frame of reference for the system, e.g., in a Langevin dynamics simulation in which the implicit background solvent has zero average velocity. In this case it is necessary to use the command [http://lammps.sandia.gov/doc/compute_modify.html compute_modify] to ensure ''dof'' is used instead of ''dof'' – ''d''. For instance one can use

<pre>
compute myTemp all temp
compute_modify myTemp extra 0
thermo_modify temp myTemp
</pre>

== Monte Carlo simulations ==

Simulations

2014-06-08T00:02:03Z

Bernie: /* Special Usage Notes */

== Introduction ==

Assorted topics relevant to programming particle-based simulation codes and to using these codes for the modeling of a wide range of systems, notably complex fluids.

== Molecular dynamics simulations ==

===Special Usage Notes===
By default, LAMMPS normalizes the temperature by an amount ''dof'' – ''d'', where ''dof'' is the system's total number of degrees of freedom and ''d'' is the dimensionality of the simulation. Subtracting ''d'' accounts for the center-of-mass motion of the system. This is correct except when there is a proper frame of reference for the system, e.g., in a Langevin dynamics simulation in which the implicit background solvent has zero average velocity. In this case it is necessary to use the command [http://lammps.sandia.gov/doc/compute_modify.html compute_modify] to ensure ''dof'' is used instead of ''dof'' - ''d''. For instance one can use

<pre>
compute myTemp all temp
compute_modify myTemp extra 0
thermo_modify temp myTemp
</pre>

== Monte Carlo simulations ==

Simulations

2014-06-07T23:52:11Z

Bernie: /* Special Usage Notes */

== Introduction ==

Assorted topics relevant to programming particle-based simulation codes and to using these codes for the modeling of a wide range of systems, notably complex fluids.

== Molecular dynamics simulations ==

===Special Usage Notes===
By default, LAMMPS normalizes the temperature by an amount (dof - d), where dof is the system's total number of degrees of freedom and d is the dimensionality of the simulation. Subtracting d accounts for the center-of-mass motion of the system. This is correct except when there is a proper frame of reference for the system, e.g., in a Langevin dynamics simulation in which the implicit background solvent has zero average velocity. In this case it is necessary to use the command [http://lammps.sandia.gov/doc/compute_modify.html compute_modify] to ensure dof is used instead of (dof - d). For instance one can use

<pre>
compute myTemp all temp
compute_modify myTemp extra 0
thermo_modify temp myTemp
</pre>

== Monte Carlo simulations ==

Simulations

2014-06-07T23:51:36Z

Bernie: /* Special Usage Notes */

== Introduction ==

Assorted topics relevant to programming particle-based simulation codes and to using these codes for the modeling of a wide range of systems, notably complex fluids.

== Molecular dynamics simulations ==

===Special Usage Notes===
By default, LAMMPS normalizes the temperature by an amount (dof - d), where dof is the system's total number of degrees of freedom and d is the dimensionality of the simulation. Subtracting d accounts for the center-of-mass motion of the system. This is correct except when there is a proper frame of reference for the system, e.g., in a Langevin dynamics simulation in which the implicit background solvent has zero average velocity. In this case it is necessary to use the command [http://lammps.sandia.gov/doc/compute_modify.html compute_modify] to ensure dof is used instead of (dof - d). An example is as follows.

<pre>
compute myTemp all temp
compute_modify myTemp extra 0
thermo_modify temp myTemp
</pre>

== Monte Carlo simulations ==

Simulations

2014-06-07T23:39:50Z

Bernie: /* Molecular dynamics simulations */

== Introduction ==

Assorted topics relevant to programming particle-based simulation codes and to using these codes for the modeling of a wide range of systems, notably complex fluids.

== Molecular dynamics simulations ==

===Special Usage Notes===
By default, LAMMPS normalizes the temperature by an amount (dof - d), where dof is the system's total number of degrees of freedom and d is the dimensionality of the simulation. (dof - d) is used to account for the center-of-mass motion of the system. This is correct except when there is a proper frame of reference for the system, e.g., in a Langevin dynamics simulation in which the implicit background solvent has zero average velocity. In this case it is necessary to use the command [http://lammps.sandia.gov/doc/compute_modify.html compute_modify] to ensure dof is used instead of (dof - d), as follows.

== Monte Carlo simulations ==

Notes on Torque

2014-05-20T18:09:02Z

Bernie: /*Torque Notes*/

=== Overview ===

=== General usage ===

=== Special usage notes ===

To change, e.g., the total cpu time allotted to a given job, one can use
qalter -l cput=[new_cput] [job_id]
where <tt>[new_cput]</tt> is the desired amount of cpu time and <tt>[job_id]</tt> is the job's ID number. Multiple jobs can be changed simultaneously by using <tt>`seq [min_job_id] [max_job_id]`</tt> in the place of <tt>job_id</tt>. This will change all jobs <tt>[min_job_id]</tt> through <tt>[max_job_id]</tt>. Note that only root can increase cput with this method.

Tools

2014-05-20T17:57:30Z

Bernie:

'''Analysis tools'''

Computer simulations involve the use of a set of analysis tools. Here, we share notes and ideas on programs commonly used in the CSML.

* [[Autocorrelation]]
* [[Generic Analyzer]]
* [[Gnuplot]]

'''Document processing'''

Ideally, we prepare manuscripts in [http://en.wikipedia.org/wiki/LaTeX LaTeX]. Occasionally, especially when collaborating with other research groups, it may be necessary to work in Microsoft Word.

* [[Notes on using LaTeX for manuscripts]]
* [[Notes on using Microsoft Word for manuscripts]]
* [[Journal-specific information]]

'''Job scheduling'''
* [[Notes on Torque]]

Generic Analyzer

2014-05-20T17:54:36Z

Bernie: /* Interpreting GA output */

=== Overview ===

The "generic analyzer" (GA) is a program available on all [[hardware|local machines]], allowing rapid statistical analysis of typical simulation data. It is particularly useful for processing output from Monte Carlo and molecular dynamics simulations.

Data files are expected to be arranged in columns, and the GA will provide the average of each column, along with its standard deviation and correlation coefficient. As a rule of thumb, if the correlation coefficient is above 0.1, the data are considered correlated and the standard deviation is underestimated. This is typical for simulation data when sampling takes place more often than the autocorrelation time. To address this, GA will group the data in blocks and make another pass. The number of data points is now decreased, generally reducing the correlation coefficient.

=== General usage ===

generic_analyzer [options] ''filename'' [columns]

<tt>filename</tt> is a plain text file. Lines starting with a '#' will be ignored (use this feature to insert column descriptions and other information into your simulation data). If <tt>filename</tt> ends with '<tt>.gz</tt>' the file is assumed to be compressed with [http://en.wikipedia.org/wiki/Gzip gzip] and will be decompressed by GA on the fly. Note that this happens in memory; no decompressed version of the file is written to disk. This has the advantage that no additional disk space is required and that no additional time is required to compress the data again after the analysis.

If the number of <tt>columns</tt> is not provided, GA will determine it from the first non-comment line in the file. Otherwise, only the number of columns indicated will be analyzed. Also note the [[#c-option|<tt>-c</tt>]] option below.

To read from standard input instead of a file, specify ''STDIN'' as the filename.

==== Options ====

* <tt id="a-option">-a</tt> ''n'' By default, GA allows up to 107 samples (but note that this number is reduced for files that contain large numbers of columns, see [[#Memory considerations|Memory considerations]] below). This option multiplies the maximum number of entries by ''n''.
* <tt id="c-option">-c</tt> ''n'' Ignore the first ''n'' columns on each line. If the number of columns is specified explicitly on the command-line, these columns are counted ''after'' the ignored columns.
* <tt id="i-option">-i</tt> ''m'' Discard the first ''m'' samples. Note that comment lines are not counted as samples. This option is normally employed to exclude simulation data taken when a system is not equilibrated yet.
* <tt id="o-option">-o</tt> ''filename'' Redirect the data normally written to [[#ana.dat|ana.dat]] (see below) to ''filename''.

=== Interpreting GA output ===

When reading the output for a given column, start at the top, and go down until you have reached the first estimate for which the correlation coefficient is less than 0.1. This line provides the proper estimate of the standard deviation. To help you quickly locate this line, it is marked by a '<' at the end. Note that the correlation time can be different for different columns. The summary printed at the end of the output provides the average of each column along with the correct standard deviation and the number of independent samples. If the data in a certain column are not decorrelated even at the most coarse blocking level, a warning is issued.

The screen output of GA can be redirected to a file, but it must be noted that all output is written to <tt>stderr</tt>. When using the [http://en.wikipedia.org/wiki/Bash_(Unix_shell) bash shell], redirection is achieved via

generic_analyzer file 2> redirected_output


GA also writes a compact version of the analysis results to a file. For ''n'' columns, this file contains a single line of the format 'A1 S1 A2 S2 ... A''n'' S''n''', where A''i'' is the average of column ''i'' and S''i'' the corresponding standard deviation (error of the mean). By default the file is called 'ana.dat', but this can be changed via the [[#o-option|<tt>-o</tt>]] option. Note that a second invocation of GA will not overwrite the file, but instead append another line to it. This format is useful for plotting results in [[gnuplot]], via a command of the form
plot "ana.dat" using 0:1:2 with err
or even

plot "ana.dat" using 0:1:2 with err, "" using 0:3:4, "" using 0:5:6

which will plot the average values of three columns (and their standard deviation) as a function of the line number in 'ana.dat', where each line number represents a different invocation of GA. Note that the standard deviations written into 'ana.dat' are those corresponding to the first decorrelated set of samples for each column (if no decorrelated set exists, the standard error obtained for the most coarse blocking level will be used).

=== Memory considerations ===

By default, GA allows up to 107 samples. If the file contains more samples, use the [[#a-option|<tt>-a</tt>]] option to allocate memory for an integer multiple of this limit.
However, after the number of columns has been determined (either specified explicitly on the command line or automatically from the first sample), the number of entries will be reduced to limit the total memory consumption to 2 GB. If the [[#a-option|<tt>-a</tt>]] option is used, this limit will be increased to an integer multiple of 2 GB.

The total memory requirements can also be reduced by analyzing a file in multiple passes, selecting a subset of the columns in each pass. For example, for a file with 1000 columns, the first 600 columns can be analyzed via
<blockquote>
<tt>generic_analyzer file 600</tt>
</blockquote>
and the remaining 400 columns can be analyzed via
<blockquote>
<tt>generic_analyzer -c 600 file 400</tt>
</blockquote>
(Note that the argument <tt>400</tt> is optional in the second invocation, but would be relevant if the file would be analyzed in more than two passes.)

The maximum number of columns handled by GA is 2048, so for larger numbers of columns the file always must be processed in multiple passes, following the same approach.

=== Special usage notes ===

* To check whether non-equilibrated data at the beginning of a file is affecting the calculated averages, run GA again on the same data file, omitting a fraction of the data via the [[#i-option|<tt>-i</tt>]] option. If this leads to a statistically significant change in the averages, it may indicate that the first part of the file contains samples that are not equilibrated.

* All lines (except comment lines) must contain at least the number of columns specified on the command-line, or the number of columns automatically determined from the first non-comment line. If a line contains more columns, a warning is issued, but the analysis proceeds. On the other hand, if any line contains fewer columns than this number, the analysis is aborted.

* Blank lines are not permitted, not even at the end of a file.

* Although GA tries to detect corruptions in a data file, limitations in the <tt>scanf()</tt> function make it ignore certain non-numerical input. Specifically, an entry that starts with a number followed by a spurious character is truncated. For example, '24j3' is read as '24' (however, scientific notation is parsed correctly; i.e., '5.3e2' is read as '530').

* The ability to read from <tt>stdin</tt> makes it possible to process simulation data and directly pipe the results into GA. For example: <tt>awk unprocessed_data | ga STDIN</tt>

=== Algorithm ===

(coming soon)

=== Download binary versions (Linux and OS X) ===

The current version is dated 03/28/2014. It is strongly recommend that you upgrade from any earlier version. You can download binary versions of this program here, but note that this was created for internal lab use - we cannot provide any support.

* [http://ariadne.ms.northwestern.edu/Download/Generic_analyzer/ga.zip Linux executable (recommended version; compiled on OpenSuSE 13.1, 64-bit)]
* [http://ariadne.ms.northwestern.edu/Download/Generic_analyzer/ga_static.zip Linux executable (try if regular version does not work; static binary that should be suitable for most x86_64 Linux installations)]
* [http://ariadne.ms.northwestern.edu/Download/Generic_analyzer/ga_osx.zip OS X executable (compiled on OS X 10.9.2)]

Gnuplot

2014-05-20T17:52:38Z

Bernie: /* Encapsulated PostScript terminal */

Gnuplot

2014-05-20T17:52:18Z

Bernie: /* Formatting labels */

Gnuplot

2014-05-20T17:51:29Z

Bernie: /* Sample usage script */

Gnuplot

2014-05-20T17:50:12Z

Bernie: /* Encapsulated PostScript terminal */

Gnuplot

2014-05-20T17:49:15Z

Bernie: /* PDF terminal */

Gnuplot

2014-05-17T20:17:15Z

Bernie: /* Encapsulated PostScript terminal */

Gnuplot

2014-05-17T20:16:54Z

Bernie: /* Using symbols in labels */

Gnuplot

2014-05-17T20:03:10Z

Bernie: /* Encapsulated PostScript terminal */

Gnuplot

2014-05-17T20:02:36Z

Bernie:

Gnuplot

2014-05-17T19:41:57Z

Bernie: /* Encapsulated PostScript terminal */

Gnuplot

2014-05-17T19:41:29Z

Bernie: /* Encapsulated PostScript terminal */

Gnuplot

2014-05-17T19:40:26Z

Bernie: /* Enhanced PostScript terminal */

Generic Analyzer

2014-05-16T18:50:06Z

Bernie: /* Special usage notes */

=== Overview ===

The "generic analyzer" (GA) is a program available on all [[hardware|local machines]], allowing rapid statistical analysis of typical simulation data. It is particularly useful for processing output from Monte Carlo and molecular dynamics simulations.

Data files are expected to be arranged in columns, and the GA will provide the average of each column, along with its standard deviation and correlation coefficient. As a rule of thumb, if the correlation coefficient is above 0.1, the data are considered correlated and the standard deviation is underestimated. This is typical for simulation data when sampling takes place more often than the autocorrelation time. To address this, GA will group the data in blocks and make another pass. The number of data points is now decreased, generally reducing the correlation coefficient.

=== General usage ===

generic_analyzer [options] ''filename'' [columns]

<tt>filename</tt> is a plain text file. Lines starting with a '#' will be ignored (use this feature to insert column descriptions and other information into your simulation data). If <tt>filename</tt> ends with '<tt>.gz</tt>' the file is assumed to be compressed with [[http://en.wikipedia.org/wiki/Gzip gzip]] and will be decompressed by GA on the fly. Note that this happens in memory; no decompressed version of the file is written to disk. This has the advantage that no additional disk space is required and that no additional time is required to compress the data again after the analysis.

If the number of <tt>columns</tt> is not provided, GA will determine it from the first non-comment line in the file. Otherwise, only the number of columns indicated will be analyzed. Also note the [[#c-option|<tt>-c</tt>]] option below.

==== Options ====

* <tt id="a-option">-a</tt> ''n'' By default, GA allows up to 107 samples (but note that this number is reduced for files that contain large numbers of columns, see [[#Memory considerations|Memory considerations]] below). This option multiplies the maximum number of entries by ''n''.
* <tt id="c-option">-c</tt> ''n'' Ignore the first ''n'' columns on each line. If the number of columns is specified explicitly on the command-line, these columns are counted ''after'' the ignored columns.
* <tt id="i-option">-i</tt> ''m'' Discard the first ''m'' samples. Note that comment lines are not counted as samples. This option is normally employed to exclude simulation data taken when a system is not equilibrated yet.
* <tt id="o-option">-o</tt> ''filename'' Redirect the data normally written to [[#ana.dat|ana.dat]] (see below) to ''filename''.

=== Interpreting GA output ===

When reading the output for a given column, start at the top, and go down until you have reached the first estimate for which the correlation coefficient is less than 0.1. This line provides the proper estimate of the standard deviation. To help you quickly locate this line, it is marked by a '<' at the end. Note that the correlation time can be different for different columns. The summary printed at the end of the output provide the average of each column along with the correct standard deviation and the number of independent samples. If the data in a certain column are not decorrelated even at the most coarse blocking level, a warning is issued.

The screen output of GA can be redirected to a file, but it must be noted that all output is written to <tt>stderr</tt>. When using the [[http://en.wikipedia.org/wiki/Bash_(Unix_shell) bash shell]], redirection is achieved via <blockquote><tt>generic_analyzer file 2> redirected_output</tt></blockquote>


GA also writes a compact version of the analysis results to a file. For ''n'' columns, this file contains a single line of the format 'A1 S1 A2 S2 ... A''n'' S''n''', where A''i'' is the average of column ''i'' and S''i'' the corresponding standard deviation (error of the mean). By default the file is called 'ana.dat', but this can be changed via the [[#o-option|<tt>-o</tt>]] option. Note that a second invocation of GA will not overwrite the file, but instead append another line to it. This format is useful for plotting results in [[gnuplot]], via a command of the form <blockquote><tt>plot "ana.dat" using 0:1:2 with err</tt></blockquote> or even <blockquote><tt>plot "ana.dat" using 0:1:2 with err, "" using 0:3:4, "" using 0:5:6</tt></blockquote> which will plot the average values of three columns (and their standard deviation) as a function of the line number in 'ana.dat', where each line number represents a different invocation of GA. Note that the standard deviations written into 'ana.dat' are those corresponding to the first decorrelated set of samples for each column (if no decorrelated set exists, the standard error obtained for the most coarse blocking level will be used).

=== Memory considerations ===

By default, GA allows up to 107 samples. If the file contains more samples, use the [[#a-option|<tt>-a</tt>]] option to allocate memory for an integer multiple of this limit.
However, after the number of columns has been determined (either specified explicitly on the command line or automatically from the first sample), the number of entries will be reduced to limit the total memory consumption to 2 GB. If the [[#a-option|<tt>-a</tt>]] option is used, this limit will be increased to an integer multiple of 2 GB.

The total memory requirements can also be reduced by analyzing a file in multiple passes, selecting a subset of the columns in each pass. For example, for a file with 1000 columns, the first 600 columns can be analyzed via
<blockquote>
<tt>generic_analyzer file 600</tt>
</blockquote>
and the remaining 400 columns can be analyzed via
<blockquote>
<tt>generic_analyzer -c 600 file 400</tt>
</blockquote>
(Note that the argument <tt>400</tt> is optional in the second invocation, but would be relevant if the file would be analyzed in more than two passes.)

The maximum number of columns handled by GA is 2048, so for larger numbers of columns the file always must be processed in multiple passes, following the same approach.

=== Special usage notes ===

* To check whether non-equilibrated data at the beginning of a file is affecting the calculated averages, run GA again on the same data file, omitting a fraction of the data via the [[#i-option|<tt>-i</tt>]] option. If this leads to a statistically significant change in the averages, it may indicate that the first part of the file contains samples that are not equilibrated.

* All lines (except comment lines) must contain at least the number of columns specified on the command-line, or the number of columns automatically determined from the first non-comment line. If a line contains more columns, a warning is issued, but the analysis proceeds. On the other hand, if any line contains fewer columns than this number, the analysis is aborted.

* Blank lines are not permitted, not even at the end of a file.

* Although GA tries to detect corruptions in a data file, limitations in the <tt>scanf()</tt> function make it ignore certain non-numerical input. Specifically, an entry that starts with a number followed by a spurious character is truncated. For example, '24j3' is read as '24' (scientific notation however is read correctly, that is 5.3e2 is read as 530).

=== Algorithm ===

(coming soon)

=== Download binary versions (Linux and OS X) ===

The current version is dated 03/28/2014. It is strongly recommend that you upgrade from any earlier version. You can download binary versions of this program here, but note that this was created for internal lab use - we cannot provide any support.

* [http://ariadne.ms.northwestern.edu/Download/Generic_analyzer/ga.zip Linux executable (recommended version; compiled on OpenSuSE 13.1, 64-bit)]
* [http://ariadne.ms.northwestern.edu/Download/Generic_analyzer/ga_static.zip Linux executable (try if regular version does not work; static binary that should be suitable for most x86_64 Linux installations)]
* [http://ariadne.ms.northwestern.edu/Download/Generic_analyzer/ga_osx.zip OS X executable (compiled on OS X 10.9.2)]

Generic Analyzer

2014-05-16T18:44:04Z

Bernie: /* Options */

=== Overview ===

The "generic analyzer" (GA) is a program available on all [[hardware|local machines]], allowing rapid statistical analysis of typical simulation data. It is particularly useful for processing output from Monte Carlo and molecular dynamics simulations.

Data files are expected to be arranged in columns, and the GA will provide the average of each column, along with its standard deviation and correlation coefficient. As a rule of thumb, if the correlation coefficient is above 0.1, the data are considered correlated and the standard deviation is underestimated. This is typical for simulation data when sampling takes place more often than the autocorrelation time. To address this, GA will group the data in blocks and make another pass. The number of data points is now decreased, generally reducing the correlation coefficient.

=== General usage ===

generic_analyzer [options] ''filename'' [columns]

<tt>filename</tt> is a plain text file. Lines starting with a '#' will be ignored (use this feature to insert column descriptions and other information into your simulation data). If <tt>filename</tt> ends with '<tt>.gz</tt>' the file is assumed to be compressed with [[http://en.wikipedia.org/wiki/Gzip gzip]] and will be decompressed by GA on the fly. Note that this happens in memory; no decompressed version of the file is written to disk. This has the advantage that no additional disk space is required and that no additional time is required to compress the data again after the analysis.

If the number of <tt>columns</tt> is not provided, GA will determine it from the first non-comment line in the file. Otherwise, only the number of columns indicated will be analyzed. Also note the [[#c-option|<tt>-c</tt>]] option below.

==== Options ====

* <tt id="a-option">-a</tt> ''n'' By default, GA allows up to 107 samples (but note that this number is reduced for files that contain large numbers of columns, see [[#Memory considerations|Memory considerations]] below). This option multiplies the maximum number of entries by ''n''.
* <tt id="c-option">-c</tt> ''n'' Ignore the first ''n'' columns on each line. If the number of columns is specified explicitly on the command-line, these columns are counted ''after'' the ignored columns.
* <tt id="i-option">-i</tt> ''m'' Discard the first ''m'' samples. Note that comment lines are not counted as samples. This option is normally employed to exclude simulation data taken when a system is not equilibrated yet.
* <tt id="o-option">-o</tt> ''filename'' Redirect the data normally written to [[#ana.dat|ana.dat]] (see below) to ''filename''.

=== Interpreting GA output ===

When reading the output for a given column, start at the top, and go down until you have reached the first estimate for which the correlation coefficient is less than 0.1. This line provides the proper estimate of the standard deviation. To help you quickly locate this line, it is marked by a '<' at the end. Note that the correlation time can be different for different columns. The summary printed at the end of the output provide the average of each column along with the correct standard deviation and the number of independent samples. If the data in a certain column are not decorrelated even at the most coarse blocking level, a warning is issued.

The screen output of GA can be redirected to a file, but it must be noted that all output is written to <tt>stderr</tt>. When using the [[http://en.wikipedia.org/wiki/Bash_(Unix_shell) bash shell]], redirection is achieved via <blockquote><tt>generic_analyzer file 2> redirected_output</tt></blockquote>


GA also writes a compact version of the analysis results to a file. For ''n'' columns, this file contains a single line of the format 'A1 S1 A2 S2 ... A''n'' S''n''', where A''i'' is the average of column ''i'' and S''i'' the corresponding standard deviation (error of the mean). By default the file is called 'ana.dat', but this can be changed via the [[#o-option|<tt>-o</tt>]] option. Note that a second invocation of GA will not overwrite the file, but instead append another line to it. This format is useful for plotting results in [[gnuplot]], via a command of the form <blockquote><tt>plot "ana.dat" using 0:1:2 with err</tt></blockquote> or even <blockquote><tt>plot "ana.dat" using 0:1:2 with err, "" using 0:3:4, "" using 0:5:6</tt></blockquote> which will plot the average values of three columns (and their standard deviation) as a function of the line number in 'ana.dat', where each line number represents a different invocation of GA. Note that the standard deviations written into 'ana.dat' are those corresponding to the first decorrelated set of samples for each column (if no decorrelated set exists, the standard error obtained for the most coarse blocking level will be used).

=== Memory considerations ===

By default, GA allows up to 107 samples. If the file contains more samples, use the [[#a-option|<tt>-a</tt>]] option to allocate memory for an integer multiple of this limit.
However, after the number of columns has been determined (either specified explicitly on the command line or automatically from the first sample), the number of entries will be reduced to limit the total memory consumption to 2 GB. If the [[#a-option|<tt>-a</tt>]] option is used, this limit will be increased to an integer multiple of 2 GB.

The total memory requirements can also be reduced by analyzing a file in multiple passes, selecting a subset of the columns in each pass. For example, for a file with 1000 columns, the first 600 columns can be analyzed via
<blockquote>
<tt>generic_analyzer file 600</tt>
</blockquote>
and the remaining 400 columns can be analyzed via
<blockquote>
<tt>generic_analyzer -c 600 file 400</tt>
</blockquote>
(Note that the argument <tt>400</tt> is optional in the second invocation, but would be relevant if the file would be analyzed in more than two passes.)

The maximum number of columns handled by GA is 2048, so for larger numbers of columns the file always must be processed in multiple passes, following the same approach.

=== Special usage notes ===

* To check whether non-equilibrated data at the beginning of a file is affecting the calculated averages, run GA again on the same data file, omitting a fraction of the data via the [[#i-option|<tt>-i</tt>]] option. If this leads to a statistically significant change in the averages, it may indicate that the first part of the file contains samples that are not equilibrated.

* All lines (except comment lines) must contain at least the number of columns specified on the command-line, or the number of columns automatically determined from the first non-comment line. If a line contains more columns, a warning is issued, but the analysis proceeds. On the other hand, if any line contains fewer columns than this number, the analysis is aborted.

* Blank lines are not permitted, not even at the end of a file.

* Although GA tries to detect corruptions in a data file, limitations in the <tt>scanf()</tt> function make it ignore certain non-numerical input. Specifically, an entry that starts with a number followed by a spurious character is truncated. For example, '24j3' is read as '24'.

=== Algorithm ===

(coming soon)

=== Download binary versions (Linux and OS X) ===

The current version is dated 03/28/2014. It is strongly recommend that you upgrade from any earlier version. You can download binary versions of this program here, but note that this was created for internal lab use - we cannot provide any support.

* [http://ariadne.ms.northwestern.edu/Download/Generic_analyzer/ga.zip Linux executable (recommended version; compiled on OpenSuSE 13.1, 64-bit)]
* [http://ariadne.ms.northwestern.edu/Download/Generic_analyzer/ga_static.zip Linux executable (try if regular version does not work; static binary that should be suitable for most x86_64 Linux installations)]
* [http://ariadne.ms.northwestern.edu/Download/Generic_analyzer/ga_osx.zip OS X executable (compiled on OS X 10.9.2)]