ssu-prep(1)

NAME
ssu-prep - prepare SSU rRNA sequences for parallel ssu-align jobs

SYNOPSIS

Add a prefix and suffix to ssu-align job commands as defined in <prefix-suffix-file>:

ssu-prep [options] seqfile output-dir n prefix-suffix-file

Run all jobs in parallel on one machine with <n> cores:

ssu-prep -x [options] seqfile output-dir n

Omit prefixes and suffixes from ssu-align job commands, to be manually added later:

ssu-prep -y [options] seqfile output-dir n

DESCRIPTION

ssu-prep reads the unaligned FASTA-formatted seqfile, splits it up into n smaller sequence files and creates a shell script that will execute n ssu-align jobs in parallel, each processing one of the small sequence files. The results of all jobs will automatically be merged together by the final job, giving the same results as if a single ssu-align job was run for the complete seqfile. The advantage of using ssu-prep is that it allows the n smaller jobs to be run simultaneously, in parallel, either on a multi-core machine or on a compute cluster, so the final result is obtained up to (nearly) n times faster. The merging is performed by an internal call to the ssu-merge script. See the ssu-merge manual page for more information.

All of the ssu-align options (see the ssu-align manual page) are available to ssu-prep. Any of these options that are set on the ssu-prep command line will identically set in the ssu-align job commands in the output shell script. The output shell script will be named <output-dir>.ssu-align.sh and can be executed with the command sh <output-dir>.ssu-align.sh.

The smaller sequence files created by ssu-prep will be named as seqfile.<i> where <i> is a number ranging from 1 to n. By default, ssu-prep will try to divide the sequences such that, as nearly as possible, all jobs process the same number of nucleotides, with the restriction that sequences are never reordered (by concatenating the n smaller sequence files in numerical order, the original seqfile would be reproduced).

ssu-prep will create an output directory called output-dir into which all output files will be placed, including the small sequence files. When the ssu-align jobs are run, each one will create a subdirectory within output-dir called output-dir.<i> and place its output files there. When the n jobs are merged, these directories will be emptied and removed (unless the --keep-merge option is set).

CUSTOMIZING PARALLELIZATION FOR THE SSU-ALIGN JOBS USING PREFIXES AND SUFFIXES
The main purpose of ssu-prep is to allow simple parallelization of large ssu-align jobs by breaking large jobs into many smaller ones, each of which will run separately. Currently, the SSU-ALIGN package does not support more respectable paralellization strategies such as MPI, nor does it use threads.

In order to allow jobs to run in parallel, ssu-prep allows the user to set a fixed prefix and suffix string to be added before and after each ssu-align job command. The three different usages listed in the SYNOPSIS above control how this prefix and suffix are defined.

In the first case, if neither the -x nor the -y options are used, the prefix and suffix are defined in the <prefix-suffix-file> as discussed in FORMAT OF THE PREFIX-SUFFIX-FILE below.

Secondly, if -x is used the job commands will be run all at once on a single multi-core machine, and so no prefix or suffix is necessary. (Actually, in this case the suffix used will be ' > /dev/null & ' for all but the final job, which makes the jobs silently and simultaneously run in the background.)

Finally, if the user specifies -y, no prefix or suffix will be added to the ssu-align job commands, but the user should manually add their own later. The -y option is meant to be useful for users who want the jobs to have prefixes and suffixes that are not constant, or that are more complex than those possible by the simplistic use of a two-line prefix-suffix-file. For example, the ssu-align commands might need to be preceded by a 'ssh <host>;' command, but where <host> is different for each command.

FORMAT OF THE PREFIX-SUFFIX-FILE
The prefix-suffix-file must contain exactly two non-comment lines (comment lines begin with '#'). The first line will become a prefix for all ssu-align commands and the second line will become a suffix. For example, if the first line is:

qsub -N ssu-align -o /dev/null -b y -j y -cwd -V '

And the second line is:

'

Then a resulting ssu-align job command in the output shell script might be:

qsub -N ssu-align -o /dev/null -b y -j y -cwd -V ' ssu-align foo/foo.1/my.fa.1 foo/foo.1 '

In contrast, if the -y option is used, specifying no prefix or suffix, the same job command would appear as:

ssu-align foo/foo.1/my.fa.1 foo/foo.1

in the output shell script.

SPECIFYING OPTIONS FOR THE PARALLEL ALIGNMENT JOBS
All ssu-align options can be set on the ssu-prep command line, and they will be added to the ssu-align command-line of the resulting parallel jobs in the output shell script. See the ssu-align manual page for a list of available options. Additionally, ssu-prep specific options are listed below.

OPTIONS

-h  Print brief help; includes version number and summary of all options.

-x  Specify that all n jobs in should be run in parallel on a single machine that has n processors (cores). See the CUSTOMIZING PARALLELIZATION section above for more details. Note that ssu-prep does not check to make sure your computer actually has n processors.

-y  Specify that the ssu-align jobs should have no prefix or suffix. The user should manually add them to the output shell script later in order to ensure the jobs actually run in parallel. See the CUSTOMIZING PARALLELIZATION section above for more details.

-f  Allow output directory output-dir to be deleted and overwritten. Without this option, if the output-dir directory already exists, ssu-prep exits with an error.

-q  Specify that the third ssu-prep command-line argument, n, specifies the desired number of sequences per job, not the number of total jobs. The number of total jobs will be dependent on the number of sequences in seqfile.

-e  Specify that each job should process the same number of sequences, instead of the same number of total nucleotides, which is the default behavior.

--no-bash
    Specify that the output shell script that will submit/run all n jobs not be written in BASH shell syntax, which it is by default.

--no-merge
    Specify that the final job should not automatically merge the results of all jobs. The user can still manually merge the jobs using the ssu-merge program after all jobs are finished running.

--keep-merge
    Do not automatically remove the smaller per-job output files once they have successfully been merged together. If this option is selected, a shell script called output-dir.cleanup.sh will be created that can be used to remove all the smaller per-job output files at a later time.

SEE ALSO
See ssu(1) for a master man page with a list of all the individual man pages for programs in the SSU-ALIGN package.

For complete documentation, see the user's guide that came with your SSU-ALIGN distribution (Userguide.pdf).

COPYRIGHT
Copyright (C) 2016 Howard Hughes Medical Institute.
Freely distributed under the BSD open source license.

AUTHOR
Eric Nawrocki
HHMI Janelia Farm Research Campus
19700 Helix Drive
Ashburn VA 20147 USA
http://eddylab.org/