# README: DATABASES IN PROPHANE

- [Taxonomic DBs](#taxonomic-dbs)
    - [NCBI NR](#ncbi-nr)
    - [UNIPROT SWISSPROT](#uniprot-swissprot)
    - [UNIPROT TREMBL](#uniprot-trembl)
    - [UNIPROT COMPLETE](#uniprot-complete)
- [Helper DBs for taxonomic DBs](#helper-dbs-for-taxonomic-dbs)
    - [NCBI_ACC2TAX](#ncbi_acc2tax)
    - [NCBI_TAXDUMP](#ncbi_taxdump)
    - [UNIPROT_TAX](#uniprot_tax)
- [Functional DBs](#functional-dbs)
    - [dbCAN/CAzY](#dbcancazy)
    - [EGGNOG](#eggnog)
    - [foam](#foam)
    - [PFAMs](#pfams)
    - [Resfams (Core/Full)](#resfams)
    - [TIGRFAMs](#tigrfams)

Description for database schema version 6

All databases are located in the same, dedicated directory. Each database type is placed in a dedicated subdirectory within 
this directory, with the database type as name. Each type-specific directory, can hold multiple versions in 
subdirectories named according to the version. To use a specific version in an annotation task, provide the 'db_version'
 config key as documented in the [config template](templates/config/config.yaml).

Example folder structure:

    - db_base_dir
      - ncbi_accession2taxid
        - 2019-08-25
        - 2020-09-30
      - eggnog
        - 4.5.1
      - ncbi_nr
        - 2019-08-25
        - 2020-09-30
      - ncbi_taxdump
        - 2019-08-25
        - 2020-09-30

Each version subfolder contains a `db_config.yaml` that holds the required info for the respective database.

A template for the db_config.yaml of each database is located in the directory [utils/databases](../utils/databases).
Filename: `{db_type}.yaml.template`

Supported versions for each database are collected in the res file: `utils/databases/{db_type}_res.yaml`. (Note: the 
version key `current_date` means that the current date in the format yyyy-mm-dd will be used as the version).

## Taxonomic DBs

### NCBI NR

subdirectory name: `ncbi_nr`

dependencies:
- [ncbi_acc2tax](#ncbi_acc2tax)
- [ncbi_taxdump](#ncbi_taxdump)

Different versions of NCBI NR databases can be stored in the *ncbi_nr* database directory.
For each version a separate subfolder has to be created. The name of the subfolder has to be the *version* entry of
the database configuration yaml.
The database has to be downloaded from the source and stored as compressed archive under the filename given in the
database configuration yaml. Additionally, a yaml file has to be provided
with the following mandatory information:

| key | value |
| --- | --- |
| name | database name |
| type | 'ncbi_nr' |
| version | database version (we recommend to use the download date) |
| comment	|	comment on the database shown in the job info, when the database has been used |
| files | dictionary of files directly associated with the db |
| | fasta_gz: relative path to compressed fasta file |
| required_dbs | dictionary of required dbs and respective versions |
| | taxdump: dict with single entry: version -> required db version. Use version of [ncbi_taxdump db](#ncbi_taxdump) |
| | accession2taxid: dict with single entry: version -> required db version. Use version of [ncbi_accession2taxid db](#ncbi_acc2tax) |
| acc_regexp  | regular expression to extract accessions as first reference from FASTA header of the database |
| acc_hit_regexp | regular expression to extract accessions for BLAST hit columns |
| lca_level | number of taxonomic levels considered by the linked tax map |
| scope | scope shown in the job info, when the database has been used (use tax for taxonomic) |
| db_schema_version | schema version of database yaml, see top of this document |

Prophane will create following files when the database is used for the first time by a trigger task:

| file extension | trigger task | content |
| --- | --- | --- |
| *.dmnd* | diamond blastP | diamond blastp database |
| *.dmnd.log* | diamond blastP | information on diamond database creation |
| *.map.gz* | any | taxonomic map |
| *.unmpapped.log* | any | accession not linked to any taxonomic information (not present in taxdump) |

### UNIPROT SWISSPROT

subdirectory name:: `uniprot_sp`

dependencies:
- [uniprot_tax](#uniprot_tax)

Different versions of SWISSPROT databases can be stored in the *uniprot_sp* database directory.
For each version a separate subfolder has to be created. The name of the subfolder has to be the *version* entry of
the database configuration yaml.
The database has to be downloaded from the source and stored as compressed archive
named *uniprot_sprot.fasta.gz*. Additionally a yaml file has to be provided
with the following mandatory information:

| key | value |
| --- | --- |
| name | database name |
| type | 'uniprot_sp' |
| version | database version (we recommend to use the download date) |
| comment	|	comment on the database shown in the job info, when the database has been used |
| files | dictionary of files directly associated with the db |
| | fasta_gz: relative path to compressed fasta file |
| required_dbs | dictionary of required dbs and respective versions |
| | uniprot_tax: dict with single entry: version -> required db version. Use version of [uniprot_tax db](#uniprot_tax) |
| lca_level | number of taxonomic levels considered by the linked tax map |
| scope | scope shown in the job info, when the database has been used (use tax for taxonomic) |
| tag |  database label used in respective FASTA header |
| db_schema_version | schema version of database yaml, see top of this document |

Prophane will create following files when the database is used for the first time by a trigger task:

| file extension | trigger task | content |
| --- | --- | --- |
| *.dmnd* | diamond blastP | diamond blastp database |
| *.dmnd.log* | diamond blastP | information on diamond database creation |
| *.map.gz* | any | taxonomic map |
| *.unmpapped.log* | any | accession not linked to any taxonomic information (not present in uniprot_tax) |

### UNIPROT TREMBL

subdirectory name:: `uniprot_tr`

dependencies:
- [uniprot_tax](#uniprot_tax)

Different versions of TREMBL databases can be stored in the *uniprot_tr* database directory.
For each version a separate subfolder has to be created. The name of the subfolder has to be the *version* entry of
the database configuration yaml.
The database has to be downloaded from the source and stored as compressed archive
named *uniprot_trembl.fasta.gz*. Additionally a yaml file has to be provided
with the following mandatory information:

| key | value |
| --- | --- |
| name | database name |
| type | 'uniprot_tr' |
| version | database version (we recommend to use the download date) |
| comment	|	comment on the database shown in the job info, when the database has been used |
| files | dictionary of files directly associated with the db |
| | fasta_gz: relative path to compressed fasta file |
| required_dbs | dictionary of required dbs and respective versions |
| | uniprot_tax: dict with single entry: version -> required db version. Use version of [uniprot_tax db](#uniprot_tax) |
| lca_level | number of taxonomic levels considered by the linked tax map |
| scope | scope shown in the job info, when the database has been used (use tax for taxonomic) |
| tag |  database label used in respective FASTA header |
| db_schema_version | schema version of database yaml, see top of this document |

Prophane will create following files when the database is used for the first time by a trigger task:

| file extension | trigger task | content |
| --- | --- | --- |
| *.dmnd* | diamond blastP | diamond blastp database |
| *.dmnd.log* | diamond blastP | information on diamond database creation |
| *.map.gz* | any | taxonomic map |
| *.unmpapped.log* | any | accession not linked to any taxonomic information (not present in uniprot_tax) |

### UNIPROT COMPLETE

subdirectory name: `uniprot_complete`

dependencies:
- [uniprot_tax](#uniprot_tax)
- [uniprot swissprot](#uniprot-swissprot)
- [uniprot trembl](#uniprot-trembl)

The "UNIPROT COMPLETE" database uses the SWISSPROT and TREMBL
fasta files.
Different versions of the database can be stored in the *uniprot_complete* database directory.
For each version a separate subfolder has to be created. The name of the subfolder has to be the *version* entry of
the database configuration yaml.
A yaml file has to be provided with the following mandatory information:

| key | value |
| --- | --- |
| name | database name |
| type | 'uniprot_complete' |
| version | database version (we recommend to use the download date) |
| comment	|	comment on the database shown in the job info, when the database has been used |
| files | dictionary of files directly associated with the db |
| | empty list (`[]`), required files are in required_dbs |
| required_dbs | dictionary of required dbs and respective versions |
| | uniprot_tax: dict with single entry: version -> required db version. Use version of [uniprot_tax db](#uniprot_tax) |
| | uniprot_sp: dict with single entry: version -> required db version. Use version of [uniprot swissprot db](#uniprot-swissprot) |
| | uniprot_tr: dict with single entry: version -> required db version. Use version of [uniprot trembl db](#uniprot-trembl) |
| acc_hit_regexp | regular expression to extract accessions for BLAST hit columns |
| lca_level | number of taxonomic levels considered by the linked tax map |
| scope | scope shown in the job info, when the database has been used (use tax for taxonomic) |
| tag |  database label used in respective FASTA header |
| db_schema_version | schema version of database yaml, see top of this document |

Prophane will create following files when the database is used for the first time by a trigger task:

| file extension | trigger task | content |
| --- | --- | --- |
| *.dmnd* | diamond blastP | diamond blastp database |
| *.dmnd.log* | diamond blastP | information on diamond database creation |
| *.map.gz* | any | taxonomic map |
| *.unmpapped.log* | any | accession not linked to any taxonomic information (not present in uniprot_tax) |

## Helper DBs for taxonomic DBs

### NCBI_ACC2TAX

subdirectory name: `ncbi_accession2taxid`

Prophane needs files that link NCBI's or PDB's protein accessions to taxIDs. Please consider,
that databases provided by NCBI should be linked to these files only when downloaded at
the same day else 100% accession linking cannot be guaranteed (see *.unmpapped.log* file of the respective
database to check unlinked accessions, see section [NCBI NR](#ncbi-nr).
The link between databases and these files has
to be defined in the database info yaml files (see section 
[NCBI NR](#ncbi-nr)).
The files have to be downloaded from the source and stored as compressed gz archive.
Different versions of these files can be stored in the *accession2taxid* database directory.
For each version a separate subfolder has to be created. The name of the subfolder has to be the *version* entry of
the database configuration yaml. Additionally, a yaml file has to be provided
with the following mandatory information:

| key | value |
| --- | --- |
| name | database name |
| type | 'accession2taxid' |
| version | database version (we recommend to use the download date) |
| comment	| comment on the database shown in the job info, when the database has been used |
| files | dictionary of files directly associated with the db |
| | prot2taxid: relative path to prot.accession2taxid.gz |
| | pdb2taxid: relative path to pdb.accession2taxid.gz |
| scope | 'helper_db' |
| db_schema_version | schema version of database yaml, see top of this document |

### NCBI_TAXDUMP

subdirectory name: `ncbi_taxdump`

NCBI's taxdump archive is used to link NCBI's taxIDs to taxonomic annotation. Please consider
that databases provided by NCBI as well as ncbi_acc2tax files should be linked to this archive only when 
downloaded at the same day else 100% taxID linking cannot be guaranteed. The link between databases, ncbi_acc2tax files and 
taxdump archive has to be defined in the database info yaml files 
(see [NCBI NR](#ncbi-nr)).
Different versions of these files can be stored in the *taxdump* database directory.
For each version a separate subfolder has to be created. The name of the subfolder has to be the *version* entry of
the database configuration yaml.
Additionally, a yaml file has to be provided with the following mandatory information:

| key | value |
| --- | --- |
| name | taxdump name |
| type | 'taxdump' |
| version | taxdump version (we recommend to use the download date) |
| comment	|	comment on taxdump shown in the job info, when the database |
| files | dictionary of files directly associated with the db |
| | taxdump: relative path to taxdump.tar.gz |
| scope | 'helper_db' |
| db_schema_version | schema version of database yaml, see top of this document |

### UNIPROT_TAX

subdirectory name: `uniprot_tax`

Uniprot taxonomic data is used to link Uniprot protein accessions to NCBI's taxonomic annotation. Please consider, that 
databases provided by UNIPROT should be linked to this file only when downloaded at the same day else 100% accession 
linking cannot be guaranteed. The link between databases and this taxonomic data has to be defined in the database info 
yaml files 
(see [UNIPROT SWISSPROT](#uniprot-swissprot), 
[UNIPROT TREMBL](#uniprot-trembl), [UNIPROT COMPLETE](#uniprot-complete)).
Different versions of these files can be stored in the *uniprot_tax* directory.
For each version a separate subfolder has to be created. The name of the subfolder has to be the *version* entry of
the database configuration yaml.
Additionally a yaml file has to be provided with the following mandatory information:

| key | value |
| --- | --- |
| name | database name |
| type | 'uniprot_tax' |
| version | database version (we recommend to use the download date) |
| comment	|	comment on database shown in the job info |
| files | dictionary of files directly associated with the db |
| | idmap: relative path to idmapping.dat.gz |
| | uniprot_tax_txt: relative path to uniprot_tax.txt |
| scope | 'helper_db' |
| db_schema_version | schema version of database yaml, see top of this document |

Prophane will create following files when the database is used for the first time by a trigger task:

| file extension | trigger task | content |
| --- | --- | --- |
| *.map.gz* | any | taxonomic map |
| *.unmpapped.log* | any | accession not linked to any taxonomic information |

## Functional DBs

### dbCAN/CAzY

subdirectory name: `dbcan`

Setup is automated, yaml-template and links to resource files are in db-specific files in databases folder.

### EGGNOG

subdirectory name: `eggnog`

The eggnog database version 5 works with emapper version 2, the eggnog database version 4 with
emapper version 1. The files containing functional annotation data have to be downloaded
from the source without changing the compression level.
Additionally, a yaml file has to be provided with the following mandatory information:

| key | value |
| --- | --- |
| name | database name |
| type | 'eggnog' |
| version | eggnog version |
| comment	|	comment on the database shown in the job info, when the database has been used |
| files | dictionary of files directly associated with the db |
| | emapper_data: relative path to downloaded eggnog database |
| | fun_cats: relative path to  file containing data on functional categories |
| lca_level | number of functional levels considered by the database |
| scope | scope shown in the job info, when the database has been used (use func for functional) |
| db_schema_version | schema version of database yaml, see top of this document |

Prophane will create following files when the eggnog database version 4 is used for the first time by a trigger task 
(not for eggnog database version 5):

| file extension | trigger task | content |
| --- | --- | --- |
| *.map.gz* | emapper | functional map |

### FOAM

subdirectory name: `foam`

Setup is automated, yaml-template and links to resource files are in db-specific files in databases folder.


### PFAMs

subdirectory name: `pfams`

Different versions of PFAMs can be stored in the *pfams* database directory.
For each version a separate subfolder has to be created. The name of the subfolder has to be the *version* entry of
the database configuration yaml.
The library and files containing functional annotation data have to be downloaded
from the source without changing the compression level. Only the *Pfam-A.hmm* file has to be stored as both gz archive and flat file. Please consider, that some of the files have
to be renamed in order to start with the same prefix (here: *Pfam-A*).
Additionally a yaml file has to be provided with the following mandatory information:

| key | value |
| --- | --- |
| name | database name |
| type | 'pfams' |
| version | pfams version |
| comment	|	comment on the database shown in the job info, when the database has been used |
| files | dictionary of files directly associated with the db |
| | hmm_lib: relative path to Pfam-A.hmm |
| lca_level | number of functional levels considered by the database |
| scope | scope shown in the job info, when the database has been used (use func for functional) |
| db_schema_version | schema version of database yaml, see top of this document |

Prophane will create following files when the database is used for the first time by a trigger task:

| file extension | trigger task | content |
| --- | --- | --- |
| *.h3f*, *.h3i*,*.h3m*, *.h3p* | hmmsearch | hmmer database |
| *.LIB.log* | hmmsearch | information on hmmer database creation |
| *.map.gz* | hmmsearch | functional map |

### Resfams

subdirectory name: `resfams_core`  
OR  
subdirectory name: `resfams_full`

Setup is automated, yaml-template and links to resource files are in db-specific files in databases folder.


### TIGRFAMs

subdirectory name: `tigrfams`

Different versions of TIGRFAMs can be stored in the *tigrfams* database directory.
For each version a separate subfolder has to be created. The name of the subfolder has to be the *version* entry of
the database configuration yaml.
The library and files containing functional annotation data have to be downloaded
from the source without changing the compression level. Please consider, that some of the files have
to be renamed in order to start with the same prefix (here: *TIGRFAMs_15.0_*).
Additionally a yaml file has to be provided with the following mandatory information:

| key | value |
| --- | --- |
| name | database name |
| type | 'tigrfams' |
| version | tigrfams version |
| comment	|	comment on the database shown in the job info, when the database has been used |
| files | dictionary of files directly associated with the db |
| | hmm_lib: relative path to TIGRFAMs_{version}_HMM.LIB |
| lca_level | number of functional levels considered by the database |
| scope | scope shown in the job info, when the database has been used (use func for functional) |
| db_schema_version | schema version of database yaml, see top of this document |

Prophane will create following files when the database is used for the first time by a trigger task:

| file extension | trigger task | content |
| --- | --- | --- |
| *.h3f*, *.h3i*,*.h3m*, *.h3p* | hmmsearch | hmmer database |
| *.LIB.log* | hmmsearch | information on hmmer database creation |
| *.map.gz* | hmmsearch | functional map |
