# README: JOB CONFIG FILES IN PROPHANE

## INTRODUCTION
Config files (YAML) are used to define input/output paths, analysis tasks and
parameters used for your analysis.  
Ideally, the config file is named *config.yaml* and is located in the
*path/to/my_job/input* directory.

An exemplary config file can be found [here](templates/config/config.yaml)


## GENERAL INFORMATION
```yaml
#general
general:
  job_name: "demonstration job"
  job_comment: "this job is for demonstration use"
```

General job information is stored in the *general* section.

| 1st level key | 2nd level key | value                |
| --------------| --------------| -------------------- |
| general       |job_name       | name of your job     |
| general       |job_comment    | comment on your job  |

## INPUT FILES
So far, Prophane supports proteomic search result data provided in a 
[generic input format](#generic-input-format), 
or produced by the search software [Scaffold](#data-provided-by-scaffold), 
or the MetaProteome Analyzer in [single](#metaproteome-analyzer)
or [sample comparison](#data-provided-by-metaproteome-analyzer-in-multisample-format) format.
Additionally, Prophane needs a data format-specific style
information file (which can be found in the prophane style folder), a FASTA file
(containing sequences of all identified proteins) and, optionally, a so called
taxmap (tab-delimited text file containing taxonomic information).

Input options for different search result data types (Generic, Scaffold, Metaproteome 
Analyzer) are explained below.


### Generic input format
```yaml
#input
input:
  search_result: /path/to/test_job/input/generic_table.txt
  report_style: /path/to/prophane/styles/generic.yaml
  fastas:
    - /path/to/jobdir/targetdb.fasta
  taxmaps: []
```
A generic table format (tab separated values) that is designed to be easy to 
recreate. An example table can be found [here](templates/input/generic_table.txt).

| 1st level key | 2nd level key | value                                                                  | comment                                 |
| --------------| ------------- | ---------------------------------------------------------------------- | --------------------------------------- |
| input         | search_result | generic search result table                                            | use absolute path                       |
| input         | report_style  | input format style file (YAML)                                         | use absolute path                       |
| input         | fastas        | list of files (FASTA) containing accession-based sequence information  | use absolute paths<br>use *[]* if empty |
| input         | taxmaps       | list of files (TSV) containing accession-based taxonomy information    | use absolute paths<br>use *[]* if empty |


### Data provided by SCAFFOLD
```yaml
#input
input:
  search_result: /path/to/jobdir/input/Protein_Report.xls
  report_style: /path/to/prophane3/styles/scaffold_4_8_x.yaml
  fastas:
    - /path/to/jobdir/targetdb.fasta
  taxmaps: []
```
Data provided by Scaffold can be used by exporting Scaffold's protein report
(TSV/XLS). All relevant input files have to be defined in the *input* section.

| 1st level key | 2nd level key | value                                                                  | comment                                 |
| --------------| --------------| ---------------------------------------------------------------------- | --------------------------------------- |
| input         | search_result | Scaffold's protein report                                              | use absolute path                       |
| input         | report_style  | input format style file (YAML)                                         | use absolute path                       |
| input         | fastas        | list of files (FASTA) containing accession-based sequence information  | use absolute paths<br>use *[]* if empty |
| input         | taxmaps       | list of files (TSV) containing accession-based taxonomic information   | use absolute paths<br>use *[]* if empty |

### Data provided by METAPROTEOME ANALYZER
```yaml
#input
input:
  search_result: /path/to/jobdir/input/metaproteins.txt
  report_style: /path/to/prophane3/styles/mpa_1_8_x.yaml
  fastas:
    - /path/to/jobdir/targetdb.fasta
  taxmaps: []
```

Data provided by Metaproteome Analyzer (MPA) can be used by exporting
metaprotein report (CSV). All input files have to be defined in the *input*
section.

| 1st level key | 2nd level key            | value                                                                  | comment                                 |
| ------------- | ------------------------ | ---------------------------------------------------------------------- | --------------------------------------- |
| input         | search_result            | MPA's metaprotein report (CSV)                                         | use absolute path                       |
| input         | report_style             | input format style file (YAML)                                         | use absolute path                       |
| input         | fastas                   | list of files (FASTA) containing accession-based sequence information  | use absolute paths; use *[]* if empty   |
| input         | taxmaps                  | list of files (TSV) containing accession-based taxonomyinformation     | use absolute paths; use *[]* if empty   |


### Data provided by METAPROTEOME ANALYZER in multisample format
```yaml
#input
input:
  search_result: /path/to/jobdir/input/metaproteins_multisample.txt
  report_style: /path/to/prophane3/styles/mpa_server_multisample.yaml
  fastas:
    - /path/to/jobdir/targetdb.fasta
  taxmaps: []
```

Data provided by Metaproteome Analyzer (MPA) can be used by exporting
metaprotein multisample report (CSV). All input files have to be defined in the *input*
section.

| 1st level key | 2nd level key            | value                                                                  | comment                                 |
| ------------- | ------------------------ | ---------------------------------------------------------------------- | --------------------------------------- |
| input         | search_result            | MPA's metaprotein report (CSV)                                         | use absolute path                       |
| input         | report_style             | input format style file (YAML)                                         | use absolute path                       |
| input         | fastas                   | list of files (FASTA) containing accession-based sequence information  | use absolute paths; use *[]* if empty   |
| input         | taxmaps                  | list of files (TSV) containing accession-based taxonomyinformation     | use absolute paths; use *[]* if empty   |

## OUTPUT DIRECTORY
```yaml
#output directory
output_dir: '/path/to/test_job/'
```

The output directory will contain all working and result data.
Please consider, that existing files might be overwritten!

| 1st level key | value                    | comment                                 |
| --------------| ------------------------ | --------------------------------------- |
| output_dir    | name of output directory | use absolute path                       |


## TASKS
```
#tasks
tasks:
  - ...
  - ...
  - ...
```

All tasks have to be defined in the *tasks* section. The different single tasks
are defined as as list. There is no limit in number of tasks.

| 1st level key | value                                 | comment           |
| ------------- | ------------------------------------- | ----------------- |
| tasks         | list of task elements                 | use absolute path |


### Database types
Prophane provides different "built-in databases" defined by an database type.
To show a list of all databases available on your system use `run.sh --list-dbs`.

| db_type               | annotation | databse description                                 |
| --------------------- | ---------- | --------------------------------------------------- |
| ncbi_nr               | taxonomic  | NCBI NR database                                    |
| uniprot_tr            | taxonomic  | Uniprot Trembl                                      |
| uniprot_sp            | taxonomic  | Uniprot Swissprot                                   |
| uniprot_complete      | taxonomic  | Uniprot Trembl & Swissprot                          |
| tigrfams              | functional | TIGRFAMs                                            |
| pfams                 | functional | PFAMs                                               |
|eggnog                 | functional | Eggnog                                              |

### TAXONOMIC prediction using DIAMOND
```yaml
  - prog: 'diamond blastp'
    shortname: tax_from_trembl_20180808_qcover90
    db_type: uniprot_tr
    type: taxonomic
    params:
      evalue: 0.01
      query-cover: 0.9
```

DIAMOND BLASTP can be used to transfer taxonomic data from a sequence database
such as NCBI Protein NR or Uniprot based on sequence homology. DIAMOND BLASTP
has a lower sensitivity than NCBI BLASTP but is much faster.

|key           | value                                                                                                                 | comment                                          |
| ------------ | --------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ |
| prog         | 'diamond blastp'                                                                                                      |                                                  |
| shortname    | short task description                                                                                                | avoid spaces, commas, tabulators, and semicolons |
| type         | 'taxonomic'                                                                                                           |                                                  |
| db_type      | db_type to search against (see 4.1)                                                                                   |                                                  |
| params       | additional DIAMOND parameters<br>(see https://github.com/bbuchfink/diamond/raw/master/diamond_manual.pdf for details) | syntax<br>option: value<br>(use '' for no value) |


### FUNCTIONAL prediction using HMMSEARCH or HMMSCAN
```yaml
  - prog: hmmscan
    shortname: fun_from_TIGRFAMs_15_cut_tc
    db_type: tigrfams
    type: functional
    params:
      cut_tc: ""
```

HMMSEARCH or HMMSCAN can be used to transfer functional data from a HMM library
such as TIGRFAMs or PFAMs based on sequence homology.

| key           | value                                                                                                               | comment                                          |
| ------------- | --------------------------------------------------------------------------------------------------------------------| ------------------------------------------------ |
| prog          | program to use                                                                                                      | 'hmmsearch' or 'hmmscan'                         |
| shortname     | short task description                                                                                              | avoid spaces, commas, tabulators, and semicolons |
| type          | 'functional'                                                                                                        |                                                  |
| db_type       | db_type to search against (see 4.1)                                                                                 |                                                  |
| params        | additional HMMSEARCH/HMMSCAN parameters<br>(see http://eddylab.org/software/hmmer3/3.1b2/Userguide.pdf for details) | syntax<br>option: value<br>(use '' for no value) |


### FUNCTIONAL prediction using EMAPPER
```yaml
  - prog: emapper
    shortname: fun_from_eggNog_4.5.1
    db_type: eggnog
    type: functional
    db: /path/to/dbs/eggnog/4_5_1/eggnog_4_5_1
    params:
      m: diamond
```

EMAPPER can be used to transfer functional data from eggNog based on sequence
homology.

| key       | value                                                                                          | comment                                          |
| --------- | ---------------------------------------------------------------------------------------------- | ------------------------------------------------ |
| prog      | 'emapper'                                                                                      |                                                  |
| shortname | short task description                                                                         | avoid spaces, commas, tabulators, and semicolons |
| type      | 'functional'                                                                                   |                                                  |
| db_type   | db_type to search against (see 4.1)                                                            |                                                  |
| params    | additional EMAPPER parameters (see https://github.com/eggnogdb/eggnog-mapper/wiki for details) | syntax<br>option: value<br>(use '' for no value) |

## SAMPLES & SAMPLEGROUPS 
```yaml
#samples
sample_groups:
    control:
      - "controls::R1"
      - "controls::R2"
      - "controls::R3"
    treated:
      - "treated::R1"
      - "treated::R2"
      - "treated::R3"
```

**Optional**

Samples can be assigned to sample groups in this section. For each sample group, the mean and standard
deviation of the quantification values of its associated samples are calculated. Calculated values are 
reported in the summary table. If no groups are specified, quantification is carried out for each sample separately.

| source                    | sample columns in input report                                    | comment                                                                                    |
| ------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| GENERIC table             | *sample category*, *sample name*                                  | values of both columns have to be fused using *::* as separator (e.g. `sample1::replicateA`) |
| SCAFFOLD's Protein Report | *Biological sample category*, *Biological sample name*            | values of both columns have to be fused using *::* as separator (e.g. `sample1::replicateA`) |
| MPA's Metaprotein Report  | *sample*                                                          | if column 'sample' is present in given report, content of this sample will be used. Else, use `sample1`. |
| MPA Multiprotein Report   | *EXP:{sample_1}*, *EXP:{sample_2}*, ... , *EXP:{sample_n}*        | sample names are extracted from column headers that start with "EXP:", with the string after the colon (':') being the sample name |

The `sample_groups` names are arbitrary and can be chosen by the user as desired.
Sample names that belong to each sample_group have to be defined as described in the table above.
Sample (groups) have to be defined as list.

| 1st level key | 2nd level key         | list values                                           |
| ------------- | --------------------- | ----------------------------------------------------- | 
| sample_groups | name of sample group  | sample name assigned to the respective sample group   |



## Recognition of DECOY sequences
```yaml
#decoy
decoy_regex: DECOY$
```

A pattern of decoy accessions can be defined in the *decoy* section.
Matching accessions are excluded from the analysis.

| 1st level key | value                                           | comment                |
| ------------- | ----------------------------------------------- | ---------------------- |
| decoy_regex   | regex exclusively matching to  decoy accessions | use python regex style |

## 7. NSAF calculation method

```yaml
#quant_method
quant_method: max_nsaf
```

The method of NSAF calculation can be defined in the *quant_method* section
(default is *max_quant*).

| 1st level key | value       | comment                                                                                                     |
| ------------- | ----------- | ----------------------------------------------------------------------------------------------------------- |
| quant_method  | *max_nsaf*  | NSAF are calculated. Normalization is based on the largest sequence length of the respective protein group  |
| quant_method  | *min_nsaf*  | NSAF are calculated. Normalization is based on the smallest sequence length of the respective protein group |
| quant_method  | *mean_nsaf* | NSAF are calculated. Normalization is based on the averaged sequence length of the respective protein group |
| quant_method  | *raw*       | unprocessed spectral counts are shown. No NSAF are calculated.                                             |


## Additional Information
```yaml
#additional
seqdb:
  - /path/to/dbs/skip/skip.fa
taxmaps:
  - /path/to/dbs/skip/skip.map
taxlevel:
  - superkingdom
  - class
  - order
  - family
  - genus
  - species
```

Information in the *Additional Information* is required by Prophane. Only paths
should be adapted to fit to the locally installed database.

Mandatory:
- seqdb
- taxmaps


| 1st level key | value      | comment           |
| ------------- | ---------- | ----------------- |
| seqdb         | empty file | use absolute path |
| taxmaps       | empty file | use absolute path |
