# README: DATABASES IN PROPHANE

- [NCBI NR](#ncbi-nr)
- [UNIPROT SWISSPROT](#uniprot-swissprot)
- [UNIPROT TREMBL](#uniprot-trembl)
- [UNIPROT COMPLETE](#uniprot-complete)
- [ACC2TAX](#acc2tax)
- [TAXDUMP](#taxdump)
- [UNIPROT_TAX](#uniprot_tax)
- [TIGRFAMs](#tigrfams)
- [PFAMs](#pfams)
- [EGGNOG](#eggnog)

Description for database schema version 3

## NCBI NR
>>>
directory: *path/to/dbs/ncbi_nr/<dl_date>/*

info file: *nr.yaml*

usage: taxonomic assignment

sources [md5sum] -> destination [-> flat file] :
	- ftp://ftp.ncbi.nlm.nih.gov/blast/db/FASTA/nr.gz ftp://ftp.ncbi.nlm.nih.gov/blast/db/FASTA/nr.gz.md5 -> *nr.gz* -> *nr*

dependencies:
	- [acc2tax](#acc2tax)
	- [taxdump](#taxdump)
>>>

Different versions of NCBI NR databases can be stored in the *ncbi_nr* database directory.
For each version a separate subfolder has to be created. The name of the subfolder has to be the *version* entry of
the database configuration yaml.
The database has to be downloaded from the source and stored as compressed archive under the filename given in the
database configuration yaml. Additionally a yaml file has to be provided
with the following mandatory information:

| key | value |
| --- | --- |
| name | database name |
| type | 'ncbi_nr' |
| version | database version (we recommend to use the download date) |
| comment	|	comment on the database shown in the job info, when the database has been used |
| files | dictionary of files directly associated with the db |
| | fasta_gz: relative path to compressed fasta file |
| required_dbs | dictionary of required dbs and respective versions |
| | taxdump: dict with single entry: version -> required db version. Use version of [taxdump db](#taxdump) |
| | accession2taxid: dict with single entry: version -> required db version. Use version of [accession2taxid db](#acc2tax) |
| acc_regexp  | regular expression to extract accessions as first reference from FASTA header of the database |
| acc_hit_regexp | regular expression to extract accessions for BLAST hit columns |
| lca_level | number of taxonomic levels considered by the linked tax map |
| scope | scope shown in the job info, when the database has been used (use tax for taxonomic) |
| db_schema_version | schema version of database yaml, see top of this document |

Prophane will create following files when the database is used for the first time by a trigger task:

| file extension | trigger task | content |
| --- | --- | --- |
| *.dmnd* | diamond blastP | diamond blastp database |
| *.dmnd.log* | diamond blastP | information on diamond database creation |
| *.map* | any | taxonomic map |
| *.unmpapped.log* | any | accession not linked to any taxonomic information (not present in taxdump) |

## UNIPROT SWISSPROT
>>>
directory: *path/to/dbs/uniprot_sp/<dl_date>/*

info file: *uniprot_sprot.yaml*

usage: taxonomic assignment

sources [md5sum] -> destination [-> flat file] :
	- ftp://ftp.uniprot.org/pub/databases/uniprot/current_release/knowledgebase/complete/uniprot_sprot.fasta.gz
	-> *uniprot_sprot.fasta.gz* -> *uniprot_sprot.fasta*

dependencies:
	- [acc2tax](#acc2tax)
	- [taxdump](#taxdump)
	- [uniprot_tax](#uniprot_tax)
>>>

Different versions of SWISSPROT databases can be stored in the *uniprot_sp* database directory.
For each version a separate subfolder has to be created. The name of the subfolder has to be the *version* entry of
the database configuration yaml.
The database has to be downloaded from the source and stored as compressed archive
named *uniprot_sprot.fasta.gz*. Additionally a yaml file has to be provided
with the following mandatory information:

| key | value |
| --- | --- |
| name | database name |
| type | 'uniprot_sp' |
| version | database version (we recommend to use the download date) |
| comment	|	comment on the database shown in the job info, when the database has been used |
| files | dictionary of files directly associated with the db |
| | fasta_gz: relative path to compressed fasta file |
| required_dbs | dictionary of required dbs and respective versions |
| | accession2taxid: dict with single entry: version -> required db version. Use version of [accesseion2taxid db](#acc2tax) |
| | taxdump: dict with single entry: version -> required db version. Use version of [taxdump db](#taxdump) |
| | uniprot_tax: dict with single entry: version -> required db version. Use version of [uniprot_tax db](#uniprot_tax) |
| lca_level | number of taxonomic levels considered by the linked tax map |
| scope | scope shown in the job info, when the database has been used (use tax for taxonomic) |
| tag |  database label used in respective FASTA header |
| db_schema_version | schema version of database yaml, see top of this document |

Prophane will create following files when the database is used for the first time by a trigger task:

| file extension | trigger task | content |
| --- | --- | --- |
| *.dmnd* | diamond blastP | diamond blastp database |
| *.dmnd.log* | diamond blastP | information on diamond database creation |
| *.perf ...* | ncbi blastP | ncbi blast database |
| *.log* | ncbi blastP | information on ncbi blast database creation |
| *.map* | any | taxonomic map |
| *.unmpapped.log* | any | accession not linked to any taxonomic information (not present in taxdump) |

## UNIPROT TREMBL
>>>
directory: *path/to/dbs/uniprot_tr/<dl_date>/*

info file: *uniprot_trembl.yaml*

usage: taxonomic assignment

sources [md5sum] -> destination [-> flat file] :
	-  ftp://ftp.uniprot.org/pub/databases/uniprot/current_release/knowledgebase/complete/uniprot_trembl.fasta.gz
	-> *uniprot_trembl.fasta.gz* -> *uniprot_trembl.fasta*

dependencies:
	- [acc2tax](#acc2tax)
	- [taxdump](#taxdump)
	- [uniprot_tax](#uniprot_tax)
>>>

Different versions of TREMBL databases can be stored in the *uniprot_tr* database directory.
For each version a separate subfolder has to be created. The name of the subfolder has to be the *version* entry of
the database configuration yaml.
The database has to be downloaded from the source and stored as compressed archive
named *uniprot_trembl.fasta.gz*. Additionally a yaml file has to be provided
with the following mandatory information:

| key | value |
| --- | --- |
| name | database name |
| type | 'uniprot_tr' |
| version | database version (we recommend to use the download date) |
| comment	|	comment on the database shown in the job info, when the database has been used |
| files | dictionary of files directly associated with the db |
| | fasta_gz: relative path to compressed fasta file |
| required_dbs | dictionary of required dbs and respective versions |
| | accession2taxid: dict with single entry: version -> required db version. Use version of [accesseion2taxid db](#acc2tax) |
| | taxdump: dict with single entry: version -> required db version. Use version of [taxdump db](#taxdump) |
| | uniprot_tax: dict with single entry: version -> required db version. Use version of [uniprot_tax db](#uniprot_tax) |
| lca_level | number of taxonomic levels considered by the linked tax map |
| scope | scope shown in the job info, when the database has been used (use tax for taxonomic) |
| tag |  database label used in respective FASTA header |
| db_schema_version | schema version of database yaml, see top of this document |

Prophane will create following files when the database is used for the first time by a trigger task:

| file extension | trigger task | content |
| --- | --- | --- |
| *.dmnd* | diamond blastP | diamond blastp database |
| *.dmnd.log* | diamond blastP | information on diamond database creation |
| *.map* | any | taxonomic map |
| *.unmpapped.log* | any | accession not linked to any taxonomic information (not present in taxdump) |

## UNIPROT COMPLETE
>>>
directory: *path/to/dbs/uniprot_complete/<dl_date>/*

info file: *uniprot_complete.yaml*

usage: taxonomic assignment

sources [md5sum] -> destination [-> flat file] :
	- ftp://ftp.uniprot.org/pub/databases/uniprot/current_release/knowledgebase/complete/uniprot_sprot.fasta.gz ftp://ftp.uniprot.org/pub/databases/uniprot/current_release/knowledgebase/complete/uniprot_trembl.fasta.gz
	-> *uniprot_complete.fasta.gz* -> *uniprot_complete.fasta*

dependencies:
	- [acc2tax](#acc2tax)
	- [taxdump](#taxdump)
	- [uniprot_tax](#uniprot_tax)
>>>

A "UNIPROT COMPLETE" database can be generated by concatenating the SWISSPROT and TREMBL
fasta files.
Different versions of the database can be stored in the *uniprot_complete* database directory.
For each version a separate subfolder has to be created. The name of the subfolder has to be the *version* entry of
the database configuration yaml.
The database has to be downloaded from the source and stored as compressed archive
named *uniprot_complete.fasta.gz*. Additionally a yaml file has to be provided
with the following mandatory information:

| key | value |
| --- | --- |
| name | database name |
| type | 'uniprot_complete' |
| version | database version (we recommend to use the download date) |
| comment	|	comment on the database shown in the job info, when the database has been used |
| files | dictionary of files directly associated with the db |
| | fasta_gz: relative path to compressed fasta file |
| required_dbs | dictionary of required dbs and respective versions |
| | accession2taxid: dict with single entry: version -> required db version. Use version of [accesseion2taxid db](#acc2tax) |
| | taxdump: dict with single entry: version -> required db version. Use version of [taxdump db](#taxdump) |
| | uniprot_tax: dict with single entry: version -> required db version. Use version of [uniprot_tax db](#uniprot_tax) |
| acc_hit_regexp | regular expression to extract accessions for BLAST hit columns |
| lca_level | number of taxonomic levels considered by the linked tax map |
| scope | scope shown in the job info, when the database has been used (use tax for taxonomic) |
| tag |  database label used in respective FASTA header |
| db_schema_version | schema version of database yaml, see top of this document |

Prophane will create following files when the database is used for the first time by a trigger task:

| file extension | trigger task | content |
| --- | --- | --- |
| *.dmnd* | diamond blastP | diamond blastp database |
| *.dmnd.log* | diamond blastP | information on diamond database creation |
| *.perf ...* | ncbi blastP | ncbi blast database |
| *.log* | ncbi blastP | information on ncbi blast database creation |
| *.map* | any | taxonomic map |
| *.unmpapped.log* | any | accession not linked to any taxonomic information (not present in taxdump) |

## ACC2TAX
>>>
directory: *path/to/dbs/accession2taxid/<dl_date>/*

usage: taxonomic assignment

sources [md5sum] -> destination [-> flat file] :
	- https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/accession2taxid/prot.accession2taxid.gz [https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/accession2taxid/prot.accession2taxid.gz.md5] -> *prot.accession2taxid.gz*
	- https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/accession2taxid/pdb.accession2taxid.gz [https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/accession2taxid/pdb.accession2taxid.gz.md5] -> *pdb.accession2taxid.gz*
>>>

Prophane needs files that link NCBI's or PDB's protein accessions to taxIDs. Please consider,
that databases provided by NCBI or UNIPROT should be linked to these files only when downloaded at
the same day else 100% accession linking cannot be guaranteed (see *.unmpapped.log* file of the respective
database to check unlinked accessions, see sections 
([NCBI NR](#ncbi-nr), [UNIPROT SWISSPROT](#uniprot-swissprot), 
[UNIPROT TREMBL](#uniprot-trembl), [UNIPROT COMPLETE](#uniprot-complete))
). 
The link between databases and these files has
to be defined in the database info yaml files (see section 
([NCBI NR](#ncbi-nr), [UNIPROT SWISSPROT](#uniprot-swissprot), 
[UNIPROT TREMBL](#uniprot-trembl), [UNIPROT COMPLETE](#uniprot-complete))).
The files have to be downloaded from the source and stored as compressed gz archive.
Different versions of these files can be stored in the *uniprot_complete* database directory.
For each version a separate subfolder has to be created. The name of the subfolder has to be the *version* entry of
the database configuration yaml. Additionally a yaml file has to be provided
with the following mandatory information:

| key | value |
| --- | --- |
| name | database name |
| type | 'accession2taxid' |
| version | database version (we recommend to use the download date) |
| comment	| comment on the database shown in the job info, when the database has been used |
| files | dictionary of files directly associated with the db |
| | prot2taxid: relative path to prot.accession2taxid.gz |
| | pdb2taxid: relative path to pdb.accession2taxid.gz |
| scope | 'helper_db' |
| db_schema_version | schema version of database yaml, see top of this document |

## TAXDUMP
>>>
directory: *path/to/dbs/taxdump/<dl_date>/*

usage: taxonomic assignment

info file: *taxdump.yaml*

sources [md5sum] -> destination [-> flat file] :
	- https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump.tar.gz [https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump.tar.gz.md5] -> *taxdump.tar.gz*
>>>

NCBI's taxdump archive is used to link NCBI's taxIDs to taxonomic annotation. Please consider
that databases provided by NCBI or UNIPROT as well as acc2tax files should be linked to this archive only when downloaded at the same day else 100% taxID linking cannot be guaranteed. The link between databases, acc2tax files and taxdump archive has to be defined in the database info yaml files 
([NCBI NR](#ncbi-nr), [UNIPROT SWISSPROT](#uniprot-swissprot), 
[UNIPROT TREMBL](#uniprot-trembl), [UNIPROT COMPLETE](#uniprot-complete)).
Different versions of these files can be stored in the *taxdump* database directory.
For each version a separate subfolder has to be created. The name of the subfolder has to be the *version* entry of
the database configuration yaml.
Additionally a yaml file has to be provided with the following mandatory information:

| key | value |
| --- | --- |
| name | taxdump name |
| type | 'taxdump' |
| version | taxdump version (we recommend to use the download date) |
| comment	|	comment on taxdump shown in the job info, when the database |
| files | dictionary of files directly associated with the db |
| | taxdump: relative path to taxdump.tar.gz |
| scope | 'helper_db' |
| db_schema_version | schema version of database yaml, see top of this document |

## UNIPROT_TAX
>>>
directory: *path/to/dbs/uniprot_tax/<dl_date>/*

usage: taxonomic assignment

sources [md5sum] -> destination [-> flat file] :
	- ftp://ftp.uniprot.org/pub/databases/uniprot/current_release/knowledgebase/idmapping/idmapping.dat.gz -> *idmapping.dat.gz*
	- https://www.uniprot.org/taxonomy/?sort=&desc=&compress=no&query=&fil=&force=no&preview=true&format=tab&columns=id -> *uniprot_tax.txt*
>>>

Uniprot taxonomic data is used to link Uniprot protein accessions to NCBI's taxonomic annotation. Please consider, that databases provided by UNIPROT should be linked to this files only when downloaded at the same day else 100% accession linking cannot be guaranteed. The link between databases and this taxonomic data has to be defined in the database info yaml files 
([NCBI NR](#ncbi-nr), [UNIPROT SWISSPROT](#uniprot-swissprot), 
[UNIPROT TREMBL](#uniprot-trembl), [UNIPROT COMPLETE](#uniprot-complete)).
Different versions of these files can be stored in the *uniprot_tax* directory.
For each version a separate subfolder has to be created. The name of the subfolder has to be the *version* entry of
the database configuration yaml.
Additionally a yaml file has to be provided with the following mandatory information:

| key | value |
| --- | --- |
| name | database name |
| type | 'uniprot_tax' |
| version | database version (we recommend to use the download date) |
| comment	|	comment on database shown in the job info |
| files | dictionary of files directly associated with the db |
| | idmap: relative path to idmapping.dat.gz |
| | uniprot_tax_txt: relative path to uniprot_tax.txt |
| scope | 'helper_db' |
| db_schema_version | schema version of database yaml, see top of this document |

Prophane will create following files when the database is used for the first time by a trigger task:

| file extension | trigger task | content |
| --- | --- | --- |
| *.map* | any | taxonomic map |
| *.unmpapped.log* | any | accession not linked to any taxonomic information |

## TIGRFAMs
>>>
directory: *path/to/dbs/tigrfams/<release>/*

usage: functional assignment

info file: *TIGRFAMs_15.0_HMM.yaml* (replace *15.0* by current release)

sources [md5sum] -> destination [-> flat file] :
	- ftp://ftp.jcvi.org/pub/data/TIGRFAMs/TIGRFAMs_15.0_HMM.LIB.gz -> *TIGRFAMs_15.0_HMM.LIB.gz* (replace *15.0* by current release)
	- ftp://ftp.jcvi.org/pub/data/TIGRFAMs/TIGRFAMs_15.0_HMM.tar.gz -> *TIGRFAMs_15.0_HMM.tar.gz* (replace *15.0* by current release)
	- ftp://ftp.jcvi.org/pub/data/TIGRFAMs/TIGRFAMs_15.0_INFO.tar.gz -> *TIGRFAMs_15.0_INFO.tar.gz* (replace *15.0* by current release)
	- ftp://ftp.jcvi.org/pub/data/TIGRFAMs/TIGRFAMS_ROLE_LINK -> *TIGRFAMs_15.0_ROLE_LINK* (replace *15.0* by current release)
	- ftp://ftp.jcvi.org/pub/data/TIGRFAMs/TIGR_ROLE_NAMES -> *TIGRFAMs_15.0_ROLE_NAMES* (replace *15.0* by current release)
>>>

Different versions of TIGRFAMs can be stored in the *tigrfams* database directory.
For each version a separate subfolder has to be created. The name of the subfolder has to be the *version* entry of
the database configuration yaml.
The library and files containing functional annotation data have to be downloaded
from the source without changing the compression level. Please consider, that some of the files have
to be renamed in order to start with the same prefix (here: *TIGRFAMs_15.0_*).
Additionally a yaml file has to be provided with the following mandatory information:

| key | value |
| --- | --- |
| name | database name |
| type | 'tigrfams' |
| version | tigrfams version |
| comment	|	comment on the database shown in the job info, when the database has been used |
| files | dictionary of files directly associated with the db |
| | hmm_lib: relative path to TIGRFAMs_{version}_HMM.LIB |
| lca_level | number of functional levels considered by the database |
| scope | scope shown in the job info, when the database has been used (use func for functional) |
| db_schema_version | schema version of database yaml, see top of this document |

Prophane will create following files when the database is used for the first time by a trigger task:

| file extension | trigger task | content |
| --- | --- | --- |
| *.h3f*, *.h3i*,*.h3m*, *.h3p* | hmmsearch | hmmer database |
| *.LIB.log* | hmmsearch | information on hmmer database creation |
| *.map* | hmmsearch | functional map |

## PFAMs
>>>
directory: *path/to/dbs/pfams/<release>/*

usage: functional assignment

info file: *Pfam-A.yaml*

sources [md5sum] -> destination [-> flat file] :
	- ftp://ftp.ebi.ac.uk/pub/databases/Pfam/current_release/Pfam-A.hmm.gz -> *Pfam-A.hmm.gz* -> *Pfam-A.hmm*
	- ftp://ftp.ebi.ac.uk/pub/databases/Pfam/current_release/database_files/clan.txt.gz -> *clan.txt.gz*
	- ftp://ftp.ebi.ac.uk/pub/databases/Pfam/current_release/Pfam-A.clans.tsv.gz -> *Pfam-A.clans.tsv.gz*
>>>

Different versions of PFAMs can be stored in the *pfmas* database directory.
For each version a separate subfolder has to be created. The name of the subfolder has to be the *version* entry of
the database configuration yaml.
The library and files containing functional annotation data have to be downloaded
from the source without changing the compression level. Only the *Pfam-A.hmm* file has to be stored as both gz archive and flat file. Please consider, that some of the files have
to be renamed in order to start with the same prefix (here: *Pfam-A*).
Additionally a yaml file has to be provided with the following mandatory information:

| key | value |
| --- | --- |
| name | database name |
| type | 'pfams' |
| version | pfams version |
| comment	|	comment on the database shown in the job info, when the database has been used |
| files | dictionary of files directly associated with the db |
| | hmm_lib: relative path to Pfam-A.hmm |
| lca_level | number of functional levels considered by the database |
| scope | scope shown in the job info, when the database has been used (use func for functional) |
| db_schema_version | schema version of database yaml, see top of this document |

Prophane will create following files when the database is used for the first time by a trigger task:

| file extension | trigger task | content |
| --- | --- | --- |
| *.h3f*, *.h3i*,*.h3m*, *.h3p* | hmmsearch | hmmer database |
| *.LIB.log* | hmmsearch | information on hmmer database creation |
| *.map* | hmmsearch | functional map |

## EGGNOG
>>>
directory: *path/to/dbs/eggnog/<release>/*

usage: functional assignment

info file: *eggnog_<release>.yaml*

sources [md5sum] -> destination [-> flat file] :
	- http://eggnogdb.embl.de/download/latest/eggnog4.functional_categories.txt -> *eggnog4.functional_categories.txt* (replace *4* by current main release number)
	- http://eggnogdb.embl.de/download/latest/all_OG_annotations.tsv.gz -> *all_OG_annotations.tsv.gz*gz*
>>>

To download and setup the database itself, you need to run following commands
```
conda env install --name emapper --file path/to/prophane/envs/emapper.yaml
source activate emapper
download_eggnog_data.py -y -f --data_dir path/to/dbs/eggnog/<release>/data euk bact arch viruses
source deactivate
conda env remove --name emapper
```

Different versions of the EGGNOG database can be stored in the *eggnog* database directory.
For each version a separate subfolder has to be created. The name of the subfolder has to be the *version* entry of
the database configuration yaml.
The files containing functional annotation data have to be downloaded
from the source without changing the compression level.
Additionally a yaml file has to be provided with the following mandatory information:

| key | value |
| --- | --- |
| name | database name |
| type | 'eggnog' |
| version | pfams version |
| comment	|	comment on the database shown in the job info, when the database has been used |
| files | dictionary of files directly associated with the db |
| | emapper_data: relative path to downloaded eggnog database |
| | og_annotations: relative path to file containing OG annotation data |
| | fun_cats: relative path to  file containing data on functional categories |
| lca_level | number of functional levels considered by the database |
| scope | scope shown in the job info, when the database has been used (use func for functional) |
| db_schema_version | schema version of database yaml, see top of this document |

Prophane will create following files when the database is used for the first time by a trigger task:

| file extension | trigger task | content |
| --- | --- | --- |
| *.map* | emapper | functional map |

## SKIP
>>>
directory: *path/to/dbs/skip*

usage: skipping of functional or taxonomic assignments

sources [md5sum] -> destination [-> flat file] :
	- "-" -> skip.fa
	- "-" -> skip.map
>>>

source: -
directory: *path/to/dbs/skip*
usage: functional and taxonomic assignment
mandatory files:
	- *skip.fa*
	- *skip.map*

To skip tasks, if necessary, Prophane needs an empty database configuration stored in the
*path/to/dbs/skip* directory
All files have to be empty (no content).
