# Prophane: Annotate your metaproteomic search results

Prophane is accessible as a web service via ([https://www.prophane.de](https://www.prophane.de)) 
but can also be installed locally. The remainder of this readme explains the usage on a local machine.

## System requirements

- operating system: linux
- RAM: at least 13 GB
- disk space: depends on the installed databases (*e.g.* ~5 GB for PFAMs DB, ~190 GB for ncbi nr)

## Installation

Install time strongly depends on your available bandwidth. The Prophane tool should be installed in
around 15 minutes. Setup of databases can take multiple hours for the large DBs 
(*i.e.* ncbi_nr, uniprot tremble/complete, eggnog)

download Miniconda: 

    wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
    
install Miniconda: 

    bash Miniconda3-latest-Linux-x86_64.sh
    
Prophane: (you might have to install git first: `conda install git`)

    git clone https://gitlab.com/s.fuchs/prophane.git
    
Setup Prophane environment:

    cd prophane
    ./setup.sh
    conda activate prophane

### Initialization

Execute `./prophane.py init {DATABASE DIRECTORY}`, where `DATABASE DIRECTORY` is a non-existing/empty directory, that 
has enough free space to hold the desired databases (can exceed several hundred GBs). Prophane will write this path into
a config file.

## Run

You need:
- the protein groups from a proteomic search result
- the fasta file containing accessions and sequences that was provided to the proteomic search engine
- optionally: a taxmap to restrict the taxonomic annotation of prophane

An example config file can be found in [templates/config/config.yaml](templates/config/config.yaml). 

All entries of the config file are explained [here](templates/config/README.md).

With config file set up, execute in the prophane directory:

    ./prophane.py run path/to/your/config.yaml

### Running Example Data

This test run will execute a couple of tests, including one full test of the Prophane pipeline on 
a small example dataset (incl. test databases).
Execution time on a desktop computer is 10-20 minutes. Expected output of the full test can be 
found in `tests/resources/test_full_analysis_and_file_presence/expected-results`.

In the prophane directory, execute:

    ./test.sh    

The first printed line of the test script will show the output directory.

## INPUT FILES
So far, Prophane supports proteomic search result data provided in a:
* [generic input format](templates/config/README.md#generic-input-format)
* [mzTab format](templates/config/README.md#data-in-mzIdentML1.2.0-exchange-format)
* [mzIdent1.2 format](templates/config/README.md#data-in-mzTab1.0.0-exchange-format)
  or produced by the search software:
* [Scaffold](templates/config/README.md#data-provided-by-scaffold)
* MetaProteome Analyzer in [single](templates/config/README.md#data-provided-by-metaproteome-analyzer)
  or [sample comparison](templates/config/README.md#data-provided-by-metaproteome-analyzer-in-multisample-format) format
* [Proteome Discoverer](templates/config/README.md#data-provided-by-proteome-discoverer) protein group output

## Folder structure
Location is given relative to the 'output_dir' in job-specific config.yaml.
- **annot_type:** annotation type of task, fun: functional annotation, tax: taxonomic annotation
- **db_type:** type of database that is used for annotation (e.g. ncbi_nr, eggnog, ...)
- **db_version:** version hash of database that is used for annotation
- **n:** protein group number
- **file:** any file
- **taskid:** task number (order of appearance in job config)
- **tool:** annotation tool used for the task, e.g. diamond, hmmer, emapper

| location                            | content                                                                                                                                                                                    | filetype |
| -----------------------------       | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------- |
| summary.txt                         | Main result file: summary table of all analyses                                                                                                                                            | TSV      |
| job_info                            | TXT file to store all job information and parameter                                                                                                                                        | TXT      |
| algn/algn/pg.{n}.mafft              | aligned sequences of protein group n, original ouput of MAFFT                                                                                                                              | FASTA    |
| algn/report/mafft.{n}.txt           | minimum and maximum sequence length and relative pairwise identity in protein group n                                                                                                      | TSV      |
| pgs/protein_groups.yaml             | protein group information: accessions, quantitation, samples                                                                                                                               | YAML     |
| plots/plot_of_{annot_type}_annot_by_{tool}_on_{db_type}.v{db_version}.task{taskid}.html            | interactive krona plots                                                                                                     | HTML     |
| seqs/all.faa                        | all accessions from the proteomic search result and their respective protein sequences                                                                                                     | FASTA    |
| seqs/missing_taxa.faa               | FASTA file containing sequences of taxonomically undefined proteins (those not covered by the optionally provided taxmap input file. In most cases, this file is the same as seqs/all.faa) | FASTA    |
| seqs/ambiguous_sequences.txt        | Usually not created; accessions with ambiguous sequence information, to help identify issues with input-fasta files.                                                                       | TXT      |
| seqs/missing_sequences.txt          | Usually not created; accessions, which could not be found in provided input FASTA.                                                                                                         | TXT      |
| segs/pgs/pg.{n}.faa                 | sequence information of protein group n                                                                                                                                                    | FASTA    |
| tasks/quant.tsv                     | quantification information for each taxonomic or functional lowest common ancestor (LCA)                                                                                                   | TSV      |
| tasks/{annot_type}_annot_by_{tool}_on_{db_type}.v{db_version}.task{taskid}.best_hits            | best hits extracted from raw annotation result                                                                                                                                                   | TSV      |
| tasks/{annot_type}_annot_by_{tool}_on_{db_type}.v{db_version}.task{taskid}.lca                  | LCA of each protein group                                                                                                                                                                  | TSV      |
| tasks/{annot_type}_annot_by_{tool}_on_{db_type}.v{db_version}.task{taskid}.log                  | log of annotation command                                                                                                                                                                  | TXT      |
| tasks/{annot_type}_annot_by_{tool}_on_{db_type}.v{db_version}.task{taskid}.map                  | respective accession-annotation linking                                                                                                                                                    | TSV      |
| tasks/{annot_type}_annot_by_{tool}_on_{db_type}.v{db_version}.task{taskid}.quant                | LCA of each sample and replicate and their calculated quantification                                                                                                                       | TSV      |
| tasks/{annot_type}_annot_by_{tool}_on_{db_type}.v{db_version}.task{taskid}.result               | raw annotation result                                                                                                                                                                      | TXT/TSV  |
| tasks/{annot_type}_annot_by_{tool}_on_{db_type}.v{db_version}.task{taskid}.result-cmd.txt       | annotation command, as executed in the shell                                                                                                                                               | TXT      |
| tasks/{annot_type}_annot_by_{tool}_on_{db_type}.v{db_version}.task{taskid}.xml                  | input for creation of krona plot                                                                                                                                                           | XML      |
| tasks/{annot_type}_annot_by_{tool}_on_{db_type}.v{db_version}.task{taskid}.yaml                 | annotation task parameters (full path to db, annotation algorithm, command line parameters, shortname of task, annotation type)                                                            | YAML     |
| {file}.benchmark.txt                | ressources (io, mem, time) utilized to create {file}                                                                                                                                       | TXT/TSV  |
| tax/taxmap.txt                      | taxonomic information extracted from user-defined taxmaps (see section 2)                                                                                                                  | TSV      |
| tax/missing_taxa_map.txt            | accessions without any taxonomic data/prediction                                                                                                                                           | TXT      |
| tax/ambiguous_taxa.txt              | Usually not created; accessions with ambiguous taxonomies extracted from user-defined taxmaps                                                                                              | TXT      |

## Tasks
### mafft
Multiple alignment of all group members.

### Taxonomic tasks
Diamontblast of identified proteins against a selected protein database (Uniprot complete, Uniprot TrEMBL, Swissprot or 
NCBI non redundant database). The required tasks are defined in the config file or on the website under expert settings.
See https://github.com/bbuchfink/diamond/raw/master/diamond_manual.pdf for additional DIAMOND parameters details.  
Possible combination of parameters:  

| Database   | Database full name     | algorithm        | mandatory param: default value |  optional parameters: default value or (type)                                                                                                                                                                                                                                                                                                                                                                                          |
| ---------- | ---------------------- | ---------------- | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| uniprot_sp | Uniprot Swissprot      | 'diamond blastp' | evalue: 0.001        | algo: 0, band: , block-size: 2.0, comp-based-stats: 1, dbsize: 40000000, frameshift: 0, freq-sd: , gapextend: 0, gapopen: 0, gapped-xdrop: , hit-band: , hit-score: , id: 0.0, id2: , index-mode: 0, masking: 1, max-hsps: 1, rank-ratio: '', rank-ratio2: '', sensitive: '', shape-mask: , matrix: 'BLOSUM62', strand: both, unapped-score: , window: , shapes: 0, xdrop: , query-cover: 0, more-sensitive: '', min-score: 20 |
| uniprot_tr | Uniprot TrEMBL        | 'diamond blastp' | evalue: 0.001        | algo: 0, band: , block-size: 2.0, comp-based-stats: 1, dbsize: 40000000, frameshift: 0, freq-sd: , gapextend: 0, gapopen: 0, gapped-xdrop: , hit-band: , hit-score: , id: 0.0, id2: , index-mode: 0, masking: 1, max-hsps: 1, rank-ratio: '', rank-ratio2: '', sensitive: '', shape-mask: , matrix: 'BLOSUM62', strand: both, unapped-score: , window: , shapes: 0, xdrop: , query-cover: 0, more-sensitive: '', min-score: 20 |
| uniprot_complete | Uniprot Complete (concatenated Swissprot and Trembl)     | 'diamond blastp' | evalue: 0.001 | algo: 0, band: , block-size: 2.0, comp-based-stats: 1, dbsize: 40000000, frameshift: 0, freq-sd: , gapextend: 0, gapopen: 0, gapped-xdrop: , hit-band: , hit-score: , id: 0.0, id2: , index-mode: 0, masking: 1, max-hsps: 1, rank-ratio: '', rank-ratio2: '', sensitive: '', shape-mask: , matrix: 'BLOSUM62', strand: both, unapped-score: , window: , shapes: 0, xdrop: , query-cover: 0.0, more-sensitive: '', min-score: 20 |
| ncbi_nr     | Uniprot Trembl        | 'diamond blastp' | evalue: 0.001        | algo: 0, band: , block-size: 2.0, comp-based-stats: 1, dbsize: 40000000, frameshift: 0, freq-sd: , gapextend: 0, gapopen: 0, gapped-xdrop: , hit-band: , hit-score: , id: 0.0, id2: , index-mode: 0, masking: 1, max-hsps: 1, rank-ratio: '', rank-ratio2: '', sensitive: '', shape-mask: , matrix: 'BLOSUM62', strand: both, unapped-score: , window: , shapes: 0, xdrop: , query-cover: 0, more-sensitive: '', min-score: 20 |

### Functional tasks
Functional annotation based on different databases (Tigrfams, Eggnog, PFAM, FOAM, dbcan, Resfam) by the programs 
hmmsearch, hmmscan, or emapper. The required tasks are defined in the config file or on the website under expert settings.  
See https://github.com/eggnogdb/eggnog-mapper/wiki for additional EMAPPER parameters details.  
Possible combination of parameters:  

| Database    | algorithm            | mandatory param     | optional parameters: default value or (type)                                                                                                                                                                                                                           |
| ----------- | -------------------- | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| eggnog      | emapper              | m: diamond OR hmmer | gapextend: 0, gapopen: 0, go_evidence: 'experimental', gguessdb: 131567, hmm_maxhits:1, hmm_maxseqlen: 5000, hmm_qcov: 0, hmm_score: 20.0, query-cover: 0, seed_ortholog_score: 60.0, subject-cover: 0, target_orthologs: one2one, tax_scope: 131567, Z: 40000000 |
| pfams       | hmmscan OR hmmsearch |                     | E: 10, T: 0.0, domE: 10, domT:0.0, incE: 10, incT: 0.0,  incdomE: 10, incomT: 0.0, cut_ga: '', cut_nc: '', cut_tc: '', max: '', F1: 0.02, F2: 0.001, F3: 0.00001, nobias: '', nonull2: '', domZ: (int), seed: 42, Z: (int)                                                             |
| tigrfams    | hmmscan OR hmmsearch |                     | E: 10, T: 0.0, domE: 10, domT:0.0, incE: 10, incT: 0.0,  incdomE: 10, incomT: 0.0, cut_nc: '', cut_tc: '', max: '', F1: 0.02, F2: 0.001, F3: 0.00001, nobias: '', nonull2: '', domZ: (int), seed: 42, Z: (int)                                               |
| foam        | hmmscan OR hmmsearch |                     | E: 10, T: 0.0, domE: 10, domT:0.0, incE: 10, incT: 0.0,  incdomE: 10, incomT: 0.0, cut_nc: '', cut_tc: '', max: '', F1: 0.02, F2: 0.001, F3: 0.00001, nobias: '', nonull2: '', domZ: (int), seed: 42, Z: (int)                                         |
| dbcan       | hmmscan OR hmmsearch |                     | E: 10, T: 0.0, domE: 10, domT:0.0, incE: 10, incT: 0.0,  incdomE: 10, incomT: 0.0, cut_nc: '', cut_tc: '', max: '', F1: 0.02, F2: 0.001, F3: 0.00001, nobias: '', nonull2: '', domZ: (int), seed: 42, Z: (int)                                     |
| resfam_full | hmmscan OR hmmsearch |                     | E: 10, T: 0.0, domE: 10, domT:0.0, incE: 10, incT: 0.0,  incdomE: 10, incomT: 0.0, cut_nc: '', cut_tc: '', max: '', F1: 0.02, F2: 0.001, F3: 0.00001, nobias: '', nonull2: '', domZ: (int), seed: 42, Z: (int)                                        |
| resfam_core | hmmscan OR hmmsearch |                     | E: 10, T: 0.0, domE: 10, domT:0.0, incE: 10, incT: 0.0,  incdomE: 10, incomT: 0.0, cut_nc: '', cut_tc: '', max: '', F1: 0.02, F2: 0.001, F3: 0.00001, nobias: '', nonull2: '', domZ: (int), seed: 42, Z: (int)                |

### Quantification
The quantification is based on the number of spectral counts per group.

### Determination of lowest common ancestors (LCA) for task results
Two methods:  
group LCA: choose based on protein/spectra count for all result of a task of a group the one with highest support (number of assigned proteins/spectra) above a given threshold (default=0.5001).  
democratic LCA: selects the result that occurs most frequently (has highest number of assigned proteins/spectra) in the entire task results, independent of the spectra support within the group.

## Results
### summary.txt
The summary.txt file shows the results of all tasks per protein group. All members of a group were assigned the same 
protein group number (column '#pg').  
The member rows contain the results of the analyses per protein (several are possible, if the task did not yield any
results for a protein, it is marked as 'unclassified').
The group rows contain summarized information about all group members.  
Protein information columns:  
All proteins assigned to one group are displayed in group_rows in the column 'members_identifier', and their number in 
the column 'members_count'. The minimum and maximum sequence length of the proteins of this group and their minimum and maximum
relative pairwise identity calculated by the multiple alignment program mafft are shown in the columns 'min_seqlen', 
'max_seqlen', 'min_rel_pairw_ident' and 'max_rel_pairw_ident'.  
Task columns:  
The subsequent columns show the results of the taxonomic and functional tasks and following the name schema 
'task_{tasknumber}::{task_name}::{task_level}'. The group rows contain the weighted lowest common ancestor (LCA) for 
every task, which was generated according to the settings defined in the config file. The member rows shows the (possibly) 
different results for the task.  
LCA-support columns:  
The LCA-support columns (e.g. task_0::{task}::{level}::lca-support) define the support for the LCA, i.e. the number of 
proteins or spectra (if available) supporting the LCA, and the total number of identified proteins/spectra for the whole
group (e.g. 130/150). If the first value is in brackets (e.g. (60)/150)), no LCA with support above a given threshold could be identified.
The value in brackets corresponds to the highest possible support. The LCA support value 0/0 is set, if no lineage 
information is available (corresponds to 'unclassified' entries).  
Quantification result columns:   
The 'raw_quant::sample (mean)' resp. 'raw_quant::sample {sample_name}::replicate {numer}' columns correspond to the number
of identified proteins (or spectra if available) per protein group resp. per sample. 
The 'quant::sample (mean)' resp. 'quant::sample {sample_name}::replicate {numer}' columns correspond to the relative share 
of the protein group resp. member proteins in the total identified proteins (or spectra if available).  
Spectra columns:
If the spectra information are available (e.g. possible for mzTab and mzIdent1.2), they are shown in the columns '{sample_name}:: spectra_IDs'

###Krona Plots
For the purpose of visualization, krona plots are generated for each task, 
which represent the results of the lowest common ancestors per group and their quantification results.

