# Referee README

### This page contains all info about the Referee program including its inputs, options, and outputs.

## Installation

Clone or download the github repo: Referee github

The only dependency is Python 2.7 or higher. You may want to add the Referee folder to your $PATH variable for ease of use!

## Usage

These are the general steps for scoring your genome:

1. Using any applicable software, map the reads from which you constructed your genome back to the finished assembly. (A BAM file is usable by ANGSD for calculating genotype likelihoods in the next step)

2. Compile a pileup file for Referee to calculate genotype likelihoods OR pre-calculate genotype log-likelihoods for all 10 genotypes at every position in the genome (we recommend ANGSD for this).

3. Score your genome with one of the following Referee commands:

       python referee.py -gl [genotype likelihood file] -ref [reference genome FASTA file] --pileup

   Alternatively, if you have multiple genotype likelihood files you wish to score with the same reference genome, you could put the paths to each file in a text file with one file path per line for Referee to score them all:

       python referee.py -i [text file with paths to genotype likelihood files] -ref [reference genome FASTA file] --pileup

   If you have pre-calculated genotype likelihoods as input, exclude the `--pileup` flag.

### Input

There are two main inputs for the program:

1. A genotype log-likelihood file (`-gl`) or files (`-i`). File(s) can be either pre-calculated genotype log-likelihoods in a certain format (see below), or a pileups from which Referee will calculate genotype likelihoods. See the walkthrough for more info.

   If you use a pileup file as input, be sure to use the `--pileup` flag.

   If you have pre-calculated genotype log-likelihoods, they must be formatted in a tab delimited file with the following columns and no column headers:

       Scaffold ID  Position    AA  AC  AG  AT  CC  CG  CT  GG  GT  TT

   If your input has a different column ordering, you will not get accurate scores!

   For example, the following output snippet from ANGSD is acceptable:

       scaffold_0	5	0.000000	-0.693147	-0.693147	-0.693147	-15.374639	-15.374639	-15.374639	-15.374639	-15.374639	-15.374639
       scaffold_0	6	0.000000	-1.386294	-1.386294	-1.386294	-30.519020	-30.519020	-30.519020	-30.519020	-30.519020	-30.519020
       scaffold_0	7	-30.288761	-30.288761	-1.386294	-30.288761	-30.288761	-1.386294	-30.288761	0.000000	-1.386294	-30.288761
       scaffold_0	8	-27.986172	-1.386293	-27.986172	-27.986172	0.000000	-1.386293	-1.386293	-27.986172	-27.986172	-27.986172
       scaffold_0	9	-27.755912	-1.386292	-27.755912	-27.755912	0.000000	-1.386292	-1.386292	-27.755912	-27.755912	-27.755912
       scaffold_0	10	-8.689986	0.000000	-10.076280	-10.076280	-29.821151	-30.514277	-30.514277	-40.590558	-40.590558	-40.590558

   Note that ANGSD also scales the log likelihoods by subtracting the highest likelihood from each likelihood. This has no effect on Referee's scoring.

   If you have multiple genotype log-likelihood files from the same genome, you can put the paths to those files in a text file and give that to Referee as input through `-i` instead of `-gl`

   Example `-i` input:

       /path/to/gl/file1.txt
       /path/to/gl/file2.txt
       /path/to/gl/file3.txt
       /path/to/gl/file4.txt

2. A reference FASTA file containing the sequences used to calculate genotype log-likelihoods (`-ref`). The FASTA headers in this file must match those in the first column of the ANGSD or pileup file(s) specified with `-i` or `-gl`. By default, Referee will trim the FASTA headers at the first occurrence of a space character, so be sure to account for this. I admit this is a shaky workaround, but given the non-standard nature of FASTA files its what I came up with. Please contact me if you would like some other header format implemented.

### Output

With no specification, Referee will create one output file and one log file. A FASTQ output file can be created with the `--fastq` flag and a Bed file can be created with the `--bed` flag.

By default, all outputs created by Referee will be files beginning with `referee-out-[start date]-[start time]-[random 6 char string]`. To change this, use the `-o` option. For the following examples we assume `-o ref-out` has beend specified.

1. #### ref-out.log

   This is a log file containing information about the Referee run, including the options used and specified inputs and outputs. It also contains runtime and memory usage (if the Python module `psutil` is available) info for each step. If you have many inputs or use many processors these runtime statistics can be distracting, so you can disable with `--quiet`

2. #### ref-out.txt

   This is a tab delimited output file containing the Referee scores for every position in the input reference genome. This file has the following columns:

       Scaffold ID  Position    Referee score

   Example:

       scaffold_0	5	0
       scaffold_0	6	13
       scaffold_0	7	13
       scaffold_0	8	12
       scaffold_0	9	12
       scaffold_0	10	13

   If you specify the `--correct` option, then Referee will also output higher scoring bases for positions that score 0. This file will have two extra columns:

       Scaffold ID  Position    Referee score   Corrected base    Referee score for corrected base

   Example with `--correct` and one position with a better scoring base (position 5):

       scaffold_0	5	0	A	6
       scaffold_0	6	13
       scaffold_0	7	13
       scaffold_0	8	12
       scaffold_0	9	12
       scaffold_0	10	13

3. #### ref-out.fq

   If `--fastq` is specified Referee will create a FASTQ file with the reference genome annotated with Referee's scores Referee scores are encoded as ASCII characters with the following method:

   `FASTQ score = ascii(Integer score + 35)`

   For example, the ASCII character `S` corresponds to the decimal 83. That means the score at this position was 83 - 35 = 48.

   Example FASTQ output:

       @scaffold_0 1:40 length=40
       GGTGTAGCCAGAGAGTAAANAATATGGTGAAGCCAGAGAG
       +
       !!!!#00//0442.45=CK"CKKLLKLKLKSRRRSSRSSS

   If the `--correct` option is specified, corrected bases will be lower case. All others should be upper case.

4. #### ref-out-bed-files/[scaffold ID].bed

   If `--bed` is specified Referee will create Bed files to visualize Referee scores in genome browsers. Referee will create one Bed file for every scaffold present in the input genotype likelihood file(s) and place them in the directory `ref-out-bed-files/`.

## Options Table

Option | Description
--- | ---
`-gl` | A single pileup file or a single file containing log genotype likelihoods for every site in your genome with reads mapped to it. Can be gzip compressed or not. If using pre-calculated log likelihoods, see the important information below regarding the order of the columns in the file. Note: Only one of `-gl` or `-i` can be specified.
`-i` | A file containing paths to multiple pileup files or multiple genotype log likelihood files. One file path per line. Note: Only one of `-gl` or `-i` can be specified.
`-ref` | A FASTA formatted file containing the genome you wish to score. Can be gzip compressed or not. FASTA headers must match the sequence IDs in column one of the pileup or genotype log likelihood file.
`-o` | Referee will create at least 2 output files: a tab delimited score file and a log file. Use this option to specify a prefix for these file names. Otherwise, they will default to `referee-out-[date]-[time]-[random string]`. If `-i` is specified, this will be the name of the output directory.
`--pileup` | If this option is set, Referee will read the input file(s) in pileup format and use this info to calculate genotype likelihoods prior to the reference quality score.
`--mapq` | If pileup file(s) are given as input, set this to incorporate mapping quality into Referee's quality score calculation. Mapping quality can be output by samtools mpileup with the `-s` option, and will appear in the 7th column of the file. If `--mapq` is not set, mapping qualities will be ignored even if they are present.
`--fastq` | With this option, Referee scores will also be output in FASTQ format. Scores will be converted to ASCII characters: score + 35 = ASCII char. Note 1: If `--correct` is set, corrected bases will appear as lower case. Note 2: This option cannot be set with `--mapped`.
`--bed` | Referee can output scores in binned BED format for visualizing tracks of scores in most genome browsers. One `.bed` file will be created for each scaffold scored and these will be placed in a directory ending with -bed-files. Note: This option cannot be set with `--mapped`.
`--haploid` | Set this option if your input sequencing data comes from a haploid species. Referee will limit it's likelihood calculations to single base states. Note: This option can only be used with an input `--pileup` file.
`--correct` | With this option, sites where reads do not support the called reference base (score <= 0) will have a higher scoring base suggested. In the tab delimited output, the corrected base and score are reported in additional columns. In FASTQ output, the corrected positions are indicated by lower case bases.
`--mapped` | Only report scores for sites with reads mapped to them. Note: This option cannot be set with `--fastq` or `--bed`.
`--quiet` | Set this option to prevent Referee from printing out runtime statistics for each step.
`-p` | The number of processes Referee can use.