# README: JOB CONFIG FILES IN PROPHANE

## INTRODUCTION
Config files (YAML) are used to define input/output paths, analysis tasks and
parameters used for your analysis.  
Ideally, the config file is named *config.yaml* and is located in the
*path/to/my_job/input* directory.

An exemplary config file can be found [here](templates/config/config.yaml)


## GENERAL INFORMATION
```yaml
#general
general:
  job_name: "demonstration job"
  job_comment: "this job is for demonstration use"
```

General job information is stored in the *general* section.

| 1st level key | 2nd level key | value                |
| --------------| --------------| -------------------- |
| general       |job_name       | name of your job     |
| general       |job_comment    | comment on your job  |

## INPUT FILES
So far, Prophane supports proteomic search result data provided in a:  
* [generic input format](#generic-input-format)
* [mzTab format](#data-in-mzIdentML1.2.0-exchange-format)
* [mzIdent1.2 format](#data-in-mzTab1.0.0-exchange-format)
or produced by the search software: 
* [Scaffold](#data-provided-by-scaffold)
* MetaProteome Analyzer in [single](#data-provided-by-metaproteome-analyzer)
or [sample comparison](#data-provided-by-metaproteome-analyzer-in-multisample-format) format
* [Proteome Discoverer](#data-provided-by-proteome-discoverer) protein group output  

Additionally, Prophane needs a data format-specific style
information file (which can be found in the prophane style folder), a plain or gzipped FASTA file
(containing sequences of all identified proteins) and, optionally, a so called
taxmap (tab-delimited text file containing taxonomic information).

Input options for different search result data types (Generic, Scaffold, Metaproteome 
Analyzer) are explained below.


### Generic input format
```yaml
#input
input:
  search_result: /path/to/test_job/input/generic_table.txt
  report_style: /path/to/prophane/styles/generic.yaml
  fastas:
    - /path/to/jobdir/targetdb.fasta[.gz]
  taxmaps: []
```
A generic table format (tab separated values) that is designed to be easy to 
recreate. An example table can be found [here](templates/input/generic_table.txt).

| 1st level key | 2nd level key | value                                                                  | comment                                 |
| --------------| ------------- | ---------------------------------------------------------------------- | --------------------------------------- |
| input         | search_result | generic search result table                                            | use absolute path                       |
| input         | report_style  | input format style file (YAML)                                         | use absolute path                       |
| input         | fastas        | list of files (FASTA) containing accession-based sequence information  | use absolute paths<br>use *[]* if empty |
| input         | taxmaps       | list of files (TSV) containing accession-based taxonomy information    | use absolute paths<br>use *[]* if empty |

### Data in mzIdentML1.2.0 exchange format
```yaml
#input
input:
  search_result: /path/to/jobdir/input/metaprotein_results.mzid
  report_style: /path/to/prophane3/styles/mzident_1_2_0.yaml
  fastas:
    - /path/to/jobdir/targetdb.fasta[.gz]
  taxmaps: []
```

Data provided in the mzIdentML exchange format can be processed by prophane. Unlike the mzIdent 1.1.0 version, similar proteins are summarized in the 'ProteinAmbiguityGroup' sections. All input files have to be defined in the *input*
section. Sequence accession IDs should not contain any pipes ('|') to allow prophane a consistent processing of the data. Sequence IDs containing 2 or more '|', like Uniprot protein accessions, will be processed, e.g. from sp|O70558|SPR2G_MOUSE to 070558.

| 1st level key | 2nd level key            | value                                                                  | comment                                 |
| ------------- | ------------------------ | ---------------------------------------------------------------------- | --------------------------------------- |
| input         | search_result            | mzIdentML1.2.0 file (MZID)                                             | use absolute path                       |
| input         | report_style             | input format style file (YAML)                                         | use absolute path                       |
| input         | fastas                   | list of files (FASTA) containing accession-based sequence information  | use absolute paths; use *[]* if empty   |
| input         | taxmaps                  | list of files (TSV) containing accession-based taxonomyinformation     | use absolute paths; use *[]* if empty   |

### Data in mzTab1.0.0 exchange format
```yaml
#input
input:
  search_result: /path/to/jobdir/input/metaprotein_results.mztab
  report_style: /path/to/prophane3/styles/mztab_1_0_0.yaml
  fastas:
    - /path/to/jobdir/targetdb.fasta[.gz]
  taxmaps: []
```

Data provided in the mzTab exchange format can be processed by prophane. Protein group information must be provided in the ambiguity_members column. All input files have to be defined in the *input*
section.
To distinguish between different samples, the entries in the column spectra_ref in the PSM section must be in the following format: ms_run[1-n]:{SPECTRA_REF}. This format must also be kept if only one ms run is analyzed.
Sequence accession IDs should not contain any pipes ('|') to allow prophane a consistent processing of the data.
Sequence IDs containing 2 or more '|', like Uniprot protein accessions, will be processed, e.g. from sp|O70558|SPR2G_MOUSE to 070558.

| 1st level key | 2nd level key            | value                                                                  | comment                                 |
| ------------- | ------------------------ | ---------------------------------------------------------------------- | --------------------------------------- |
| input         | search_result            | mzTab1.0.0 file (MZTAB)                                             | use absolute path                       |
| input         | report_style             | input format style file (YAML)                                         | use absolute path                       |
| input         | fastas                   | list of files (FASTA) containing accession-based sequence information  | use absolute paths; use *[]* if empty   |
| input         | taxmaps                  | list of files (TSV) containing accession-based taxonomyinformation     | use absolute paths; use *[]* if empty   |

### Data provided by SCAFFOLD
```yaml
#input
input:
  search_result: /path/to/jobdir/input/Protein_Report.xls
  report_style: /path/to/prophane3/styles/scaffold_4_8_x.yaml
  fastas:
    - /path/to/jobdir/targetdb.fasta[.gz]
  taxmaps: []
```
Data provided by Scaffold can be used by exporting Scaffold's protein report
(TSV/XLS). All relevant input files have to be defined in the *input* section.

| 1st level key | 2nd level key | value                                                                  | comment                                 |
| --------------| --------------| ---------------------------------------------------------------------- | --------------------------------------- |
| input         | search_result | Scaffold's protein report                                              | use absolute path                       |
| input         | report_style  | input format style file (YAML)                                         | use absolute path                       |
| input         | fastas        | list of files (FASTA) containing accession-based sequence information  | use absolute paths<br>use *[]* if empty |
| input         | taxmaps       | list of files (TSV) containing accession-based taxonomic information   | use absolute paths<br>use *[]* if empty |

### Data provided by METAPROTEOME ANALYZER
```yaml
#input
input:
  search_result: /path/to/jobdir/input/metaproteins.txt
  report_style: /path/to/prophane3/styles/mpa_1_8_x.yaml
  fastas:
    - /path/to/jobdir/targetdb.fasta[.gz]
  taxmaps: []
```

Data provided by Metaproteome Analyzer (MPA) can be used by exporting
metaprotein report (CSV). All input files have to be defined in the *input*
section.

| 1st level key | 2nd level key            | value                                                                  | comment                                 |
| ------------- | ------------------------ | ---------------------------------------------------------------------- | --------------------------------------- |
| input         | search_result            | MPA's metaprotein report (CSV)                                         | use absolute path                       |
| input         | report_style             | input format style file (YAML)                                         | use absolute path                       |
| input         | fastas                   | list of files (FASTA) containing accession-based sequence information  | use absolute paths; use *[]* if empty   |
| input         | taxmaps                  | list of files (TSV) containing accession-based taxonomyinformation     | use absolute paths; use *[]* if empty   |


### Data provided by METAPROTEOME ANALYZER in multisample format
```yaml
#input
input:
  search_result: /path/to/jobdir/input/metaproteins_multisample.txt
  report_style: /path/to/prophane3/styles/mpa_server_multisample.yaml
  fastas:
    - /path/to/jobdir/targetdb.fasta[.gz]
  taxmaps: []
```

Data provided by Metaproteome Analyzer (MPA) can be used by exporting
metaprotein multisample report (CSV). All input files have to be defined in the *input*
section.

| 1st level key | 2nd level key            | value                                                                  | comment                                 |
| ------------- | ------------------------ | ---------------------------------------------------------------------- | --------------------------------------- |
| input         | search_result            | MPA's metaprotein report (CSV)                                         | use absolute path                       |
| input         | report_style             | input format style file (YAML)                                         | use absolute path                       |
| input         | fastas                   | list of files (FASTA) containing accession-based sequence information  | use absolute paths; use *[]* if empty   |
| input         | taxmaps                  | list of files (TSV) containing accession-based taxonomy information     | use absolute paths; use *[]* if empty   |

### Data provided by Proteome Discoverer
```yaml
#input
input:
  search_result: /path/to/jobdir/input/pd_protein_group.xlsx
  report_style: /path/to/prophane3/styles/proteome_discoverer_2_5.yaml
  fastas:
    - /path/to/jobdir/targetdb.fasta[.gz]
  taxmaps: []
```

Data provided by Proteome Discoverer can be used by following steps:  
1. check for filters set under "Display Folters -> Proteins", and remove the filter for master proteins
2. go to Protein Groups table and request "Show associated tables -> Proteins"
3. export this record as follows: File -> Export -> to Microsoft Excel
Abundance of the master protein is used for the protein group abundances. If multiple abundances are available:  
   order of selected abundances: 'Abundance: sample' (=raw abundance), 'Abundances (Scaled): sample', 'Abundances (Normalized): sample',  
If the master_protein row of a protein group contains no values for all samples in the abundance columns, these protein groups are ignored.


| 1st level key | 2nd level key            | value                                                                  | comment                                 |
| ------------- | ------------------------ | ---------------------------------------------------------------------- | --------------------------------------- |
| input         | search_result            | Proteome Discoverer protein group report (xlsx)                                         | use absolute path                       |
| input         | report_style             | input format style file (YAML)                                         | use absolute path                       |
| input         | fastas                   | list of files (FASTA) containing accession-based sequence information  | use absolute paths; use *[]* if empty   |
| input         | taxmaps                  | list of files (TSV) containing accession-based taxonomy information     | use absolute paths; use *[]* if empty   |

## OUTPUT DIRECTORY
```yaml
#output directory
output_dir: '/path/to/test_job/'
```

The output directory will contain all working and result data.
Please consider, that existing files might be overwritten!

| 1st level key | value                    | comment                                 |
| --------------| ------------------------ | --------------------------------------- |
| output_dir    | name of output directory | use absolute path                       |


## TASKS
```
#tasks
tasks:
  - ...
  - ...
  - ...
```

All tasks have to be defined in the *tasks* section. The different single tasks
are defined as as list. There is no limit in number of tasks.

| 1st level key | value                                 | comment           |
| ------------- | ------------------------------------- | ----------------- |
| tasks         | list of task elements                 | use absolute path |


### Database types
Prophane provides different "built-in databases" defined by an database type.
To show a list of all databases available on your system use `run.sh --list-dbs`.

| db_type               | annotation | databse description                                 |
| --------------------- | ---------- | --------------------------------------------------- |
| ncbi_nr               | taxonomic  | NCBI NR database                                    |
| uniprot_tr            | taxonomic  | Uniprot Trembl                                      |
| uniprot_sp            | taxonomic  | Uniprot Swissprot                                   |
| uniprot_complete      | taxonomic  | Uniprot Trembl & Swissprot                          |
| tigrfams              | functional | TIGRFAMs                                            |
| pfams                 | functional | PFAMs                                               |
|eggnog                 | functional | Eggnog DB version 4 or version 5                                          |
                                          |

### TAXONOMIC prediction using DIAMOND
```yaml
  - prog: 'diamond blastp'
    shortname: tax_from_trembl_20180808_qcover90
    db_type: uniprot_tr
    type: taxonomic
    params:
      evalue: 0.01
      query-cover: 0.9
```

DIAMOND BLASTP can be used to transfer taxonomic data from a sequence database
such as NCBI Protein NR or Uniprot based on sequence homology. DIAMOND BLASTP
has a lower sensitivity than NCBI BLASTP but is much faster.

|key           | value                                                                                                                 | comment                                          |
| ------------ | --------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ |
| prog         | 'diamond blastp'                                                                                                      |                                                  |
| shortname    | short task description                                                                                                | avoid spaces, commas, tabulators, and semicolons |
| type         | 'taxonomic'                                                                                                           |                                                  |
| db_type      | db_type to search against (see 4.1)                                                                                   |                                                  |
| params       | additional DIAMOND parameters<br>(see https://github.com/bbuchfink/diamond/raw/master/diamond_manual.pdf for details) | syntax<br>option: value<br>(use '' for no value) |


### FUNCTIONAL prediction using HMMSEARCH or HMMSCAN
```yaml
  - prog: hmmscan
    shortname: fun_from_TIGRFAMs_15_cut_tc
    db_type: tigrfams
    type: functional
    params:
      cut_tc: ""
```

HMMSEARCH or HMMSCAN can be used to transfer functional data from a HMM library
such as TIGRFAMs or PFAMs based on sequence homology.

| key           | value                                                                                                               | comment                                          |
| ------------- | --------------------------------------------------------------------------------------------------------------------| ------------------------------------------------ |
| prog          | program to use                                                                                                      | 'hmmsearch' or 'hmmscan'                         |
| shortname     | short task description                                                                                              | avoid spaces, commas, tabulators, and semicolons |
| type          | 'functional'                                                                                                        |                                                  |
| db_type       | db_type to search against (see 4.1)                                                                                 |                                                  |
| params        | additional HMMSEARCH/HMMSCAN parameters<br>(see http://eddylab.org/software/hmmer3/3.1b2/Userguide.pdf for details) | syntax<br>option: value<br>(use '' for no value) |


### FUNCTIONAL prediction using EMAPPER
```yaml
  - prog: emapper
    shortname: fun_from_eggNog_5.1.0
    db_type: eggnog
    type: functional
    params:
      m: diamond
```

EMAPPER can be used to transfer functional data from eggNog based on sequence
homology. Emapper version 1 is used for eggnog databases of version <= version 4 and emapper version 2 for eggnog 
databases >= version 5. 

| key       | value                                                                                          | comment                                          |
| --------- | ---------------------------------------------------------------------------------------------- | ------------------------------------------------ |
| prog      | 'emapper'                                                                                      |                                                  |
| shortname | short task description                                                                         | avoid spaces, commas, tabulators, and semicolons |
| type      | 'functional'                                                                                   |                                                  |
| db_type   | db_type to search against (see 4.1)                                                            |                                                  |
| params    | additional EMAPPER parameters (see https://github.com/eggnogdb/eggnog-mapper/wiki for details) | syntax<br>option: value<br>(use '' for no value) |

### ACCESSION based annotation using user provided mapping table
```yaml
  - prog: acc2annot_mapper
    type: taxonomic
    shortname: fun_from_my_file
    db_type: custom_map
    params:
      path: /path/to/accession2annotation_map.tsv
```

The acc2annot_mapper can be used to transfer any kind of annotation (functional or taxonomic) based on the protein 
accessions. Annotations are retrieved from a user-provided tab-separated table (tsv) file. Annotation level names are 
based on the column names of the user provided table (= column names).

Example table:

```tsv
acc	level1	level2	level3
KZK33282.1	Bacteria	Firmicutes	Bacilli
NP_268346.1	Bacteria	Firmicutes	Bacilli
WP_003131952.1	Bacteria	Firmicutes	Bacilli
PKC80086.1	Bacteria	Actinobacteria	Bifidobacteriales
XP_963622.1	Eukaryota	Ascomycota	Sordariomycetes
WP_000665196.1	Bacteria	Proteobacteria	Gammaproteobacteria
WP_000665196.1	Bacteria	Proteobacteria	Gammaproteobacteria group
```

| key       | value                                                                                          | comment                                          |
| --------- | ---------------------------------------------------------------------------------------------- | ------------------------------------------------ |
| prog      | 'acc2annot_mapper'                                                                                      |                                                  |
| shortname | short task description                                                                         | avoid spaces, commas, tabulators, and semicolons |
| type      | 'functional' OR 'taxonomic'                                                                                  |                                                  |
| db_type   | 'custom_map'                                                         |                                                  |
| params    | `path` path to accession map file in tsv-format |  |

## SAMPLES & SAMPLEGROUPS 
```yaml
#samples
sample_groups:
    control:
      - "controls::R1"
      - "controls::R2"
      - "controls::R3"
    treated:
      - "treated::R1"
      - "treated::R2"
      - "treated::R3"
```

**Optional**

Samples can be assigned to sample groups in this section. For each sample group, the mean and standard
deviation of the quantification values of its associated samples are calculated. Calculated values are 
reported in the summary table. If no groups are specified, quantification is carried out for each sample separately.

| source                    | sample columns in input report                                    | comment                                                                                    |
| ------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| GENERIC table             | *sample category*, *sample name*                                  | values of both columns have to be fused using *::* as separator (e.g. `sample1::replicateA`) |
| SCAFFOLD's Protein Report | *Biological sample category*, *Biological sample name*            | values of both columns have to be fused using *::* as separator (e.g. `sample1::replicateA`) |
| MPA's Metaprotein Report  | *sample*                                                          | if column 'sample' is present in given report, content of this sample will be used. Else, use `sample1`. |
| MPA Multiprotein Report   | *EXP:{sample_1}*, *EXP:{sample_2}*, ... , *EXP:{sample_n}*        | sample names are extracted from column headers that start with "EXP:", with the string after the colon (':') being the sample name |

The `sample_groups` names are arbitrary and can be chosen by the user as desired.
Sample names that belong to each sample_group have to be defined as described in the table above.
Sample (groups) have to be defined as list.

| 1st level key | 2nd level key         | list values                                           |
| ------------- | --------------------- | ----------------------------------------------------- | 
| sample_groups | name of sample group  | sample name assigned to the respective sample group   |



## Recognition of DECOY sequences
```yaml
#decoy
decoy_regex: DECOY$
```

A pattern of decoy accessions can be defined in the *decoy* section.
Matching accessions are excluded from the analysis.

| 1st level key | value                                           | comment                |
| ------------- | ----------------------------------------------- | ---------------------- |
| decoy_regex   | regex exclusively matching to  decoy accessions | use python regex style |

## 7. NSAF calculation method

```yaml
#quant_method
quant_method: max_nsaf
```

The method of NSAF calculation can be defined in the *quant_method* section
(default is *max_quant*).

| 1st level key | value       | comment                                                                                                     |
| ------------- | ----------- | ----------------------------------------------------------------------------------------------------------- |
| quant_method  | *max_nsaf*  | NSAF are calculated. Normalization is based on the largest sequence length of the respective protein group  |
| quant_method  | *min_nsaf*  | NSAF are calculated. Normalization is based on the smallest sequence length of the respective protein group |
| quant_method  | *mean_nsaf* | NSAF are calculated. Normalization is based on the averaged sequence length of the respective protein group |
| quant_method  | *raw*       | unprocessed spectral counts are shown. No NSAF are calculated.                                             |


## Additional Information
```yaml
#additional
seqdb:
  - /path/to/dbs/skip/skip.fa
taxmaps:
  - /path/to/dbs/skip/skip.map
taxlevel:
  - superkingdom
  - class
  - order
  - family
  - genus
  - species
```

Information in the *Additional Information* is required by Prophane. Only paths
should be adapted to fit to the locally installed database.

Mandatory:
- seqdb
- taxmaps


| 1st level key | value      | comment           |
| ------------- | ---------- | ----------------- |
| seqdb         | empty file | use absolute path |
| taxmaps       | empty file | use absolute path |
