# ChewieSnake: A pipeline for cgMLST allele calling and allele hashing

ChewieSnake is  a snakemake workflow that performs cgMLST allele calling for a set of assembled genomes using [chewBBACA](https://www.ncbi.nlm.nih.gov/pubmed/29543149). 

Subsequently, the workflow converts the resulting allele profiles into a **nomenclature-free** allele profile (follwing the idea of [sistr](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0147101)).

The allele profiles of all input genomes are converted into an allele distance matrix, a minimum spanning tree and a neighbor-joining tree using [grapetree](https://genome.cshlp.org/content/28/9/1395).

The allele numbers in the **nomenclature-free** allele profiles are derived from the hashed allele sequences. Therefore they can be directly compared between two or more laboratories without the need of an explicit exchange of the genome data, allele nomenclature or allele sequences.

## Details
* snakemake: Snakemake is a workflow engine that allows a reproducible and automated data analysis of a large number of files. Whenever new files are added, allele profiles need only be computed for these files
* chewBBACA: An open source cgMLST suite
* hashing: Module that converts allele numbers to hashed allele numbers (Inspired from sistr)
* grapetree: A cgMLST analysis and visualization tool from Enterobase

## News and new features
* single conda env (and bioconda package as well as docker container)
* hashID as a unique Sequence Type of a sample
* chewieSnake_join workflow (see test below) 

## Table of contents

* [Installation and set-up](https://gitlab.com/bfr_bioinformatics/chewieSnake/blob/master/README.md#installation-and-set-up)
* [Usage (assembly-based version)](https://gitlab.com/bfr_bioinformatics/chewieSnake/blob/master/README.md#usage-assembly-based-version)
* [Usage (read-based flavour)](https://gitlab.com/bfr_bioinformatics/chewieSnake/blob/master/README.md#usage-read-based-flavour)
* [Output description](https://gitlab.com/bfr_bioinformatics/chewieSnake/blob/master/README.md#output-description)
* [Test execution on Salmonella test data](https://gitlab.com/bfr_bioinformatics/chewieSnake/blob/master/README.md#test-execution-on-salmonella-test-data)
* [Test execution on chewieSnake join](https://gitlab.com/bfr_bioinformatics/chewieSnake/blob/master/README.md#test-of-chewiesnake_join)



## Installation and set-up

### Clone this repository

```
    cd /path/to/
    git clone https://gitlab.com/bfr_bioinformatics/chewieSnake
```

### Install the conda environment

```
    conda env create -c conda-forge -n chewiesnakes -f path/2/rep/chewiesnake.yaml
```

#### Alternatives

##### Install the BioConda environment

```
*** coming soon
```

##### Install the Docker container

Prerequisite: Install the Docker engine for your favourite operating system (e.g. [Ubuntu Linux](https://docs.docker.com/engine/install/ubuntu/)).

Download the latest version of chewieSnake from Docker Hub and note down the Docker Image ID on your system  (hereafter refered as $docker_image_id) with the shell commands:
```
docker pull bfrbioinformatics/chewiesnake:latest
docker image list | grep "chewiesnake" | grep "latest" | awk '{ print $3 }'
```

To process data and write results, Docker needs a *volume mapping* from a host directory containing your sequence data (hereafter called `</your/data/directory>`) to the Docker container (`/chewieSnake/analysis`). Your sample list (`samples.tsv`) needs to be located within `</your/data/directory>` and contain relative paths to your NGS reads or assembly files in the same or another child directory. 
You may generate a Docker-compatible sample list in your host directory (`</your/data/directory>/assemblies/samples.tsv`) by executing the `create_sampleSheet.sh` from the container with the following commands:
```
host:/your/data/directory$ ls assemblies/
sample_1.fasta   sample_2.fasta   sample_3.fasta

docker run --rm -v </your/data/directory>:/chewieSnake/analysis -e LOCAL_USER_ID=$(id -u $USER) \
   --entrypoint bash \
   $docker_image_id \
   /chewieSnake/scripts/create_sampleSheet.sh \
     --mode assembly \
     --fastxDir /chewieSnake/analysis/assemblies \
     --outDir /chewieSnake/analysis/assemblies
```

With the following command, chewieSnake is started within the Docker container and will process any option appended:
```
docker run --rm -v </your/data/directory>:/chewieSnake/analysis -e LOCAL_USER_ID=$(id -u $USER) \
  $docker_image_id \
  --condaprefix /opt/conda/envs \
  --sample_list /chewieSnake/analysis/assemblies/samples.tsv \
  --working_directory /chewieSnake/analysis \
  --<any_other_chewieSnake_option>
```
*Note:* The container path `/chewieSnake/analysis` is fixed and may not be altered. Any subdirectories of `</your/data/directory>` will be available as subdirectories under `/chewieSnake/analysis/`. Our container is able to write results with the Linux user and group ID  of your choice (`UID` and `GID`, respectively) to blend into your host file permission setup. With the above option `-e LOCAL_USER_ID==$(id -u $USER)` the UID of the currently executing user is inherited, change it according to your needs.


### Install cgMLST schema

Different options exist for installing a cgMLST schema

#### Get a schema from Enterobase, bigsDB, cgMLST.org, Innuendo and prepare with chewBBACA

see [below](https://gitlab.com/bfr_bioinformatics/chewieSnake/-/blob/master/README.md#cgmlst-databases) for available ressources and the [chewBBACA wiki](https://github.com/B-UMMI/chewBBACA/wiki/1.-Schema-Creation) for instructions how to prepare a scheme.


#### Create your own schema from scratch

see [chewBBACA wiki](https://github.com/B-UMMI/chewBBACA/wiki/3.-cgMLST-definition) for details

#### Retrieve already prepared schemas

Selected publically available cgMLST schemas for Salmonella enterica , Listeria monocytogenes, Escherichia coli and Campylobacter jejuni have been prepared for usage with chewbbaca / chewieSnake and can be found [here](https://databay.bfrlab.de/d/7d7ff2e1ef5d42098f67/).

Please download and unzip the schema of your choice. Specify the full path to the schema's directory when executing chewieSnake. 


#### Retrieve schemas from Innuendo

see [link](http://chewbbaca.online/) for available schemas

### Create sample sheet
  
#### For assembly fasta files
  
All data must be supplied in a sample sheet (samples.tsv). It requires the following entries:

```
sample    assembly
sample_1    path/2/assembly_1.fasta
sample_2    path/2/assembly_1.fasta
```

The sample sheet can be updated and analysis will be run only on new samples.
The sample sheets can be created for all fasta files in a directory with the script create_sampleSheet.sh

```
cd $testDIR/assemblies
path/2/chewieSnake/scripts/create_sampleSheet.sh --mode assembly
```

#### For raw fastq files (for read-based flavor)

All data must be supplied in a sample sheet (samples.tsv). It requires the following entries:

```
sample    fq1   fq2
sample_1    path/2/sample1_R1.fastq.gz  path/2/sample1_R2.fastq.gz
sample_2    path/2/sample2_R1.fastq.gz  path/2/sample2_R2.fastq.gz
```

Again, the sample sheet can be created with the helper script:

```
cd $testDIR/assemblies
path/2/chewieSnake/scripts/create_sampleSheet.sh --mode flex ## using a flexible scheme where the part before the first '_' denotes the sample name sample_*_R[12]_*.fastq.gz
path/2/chewieSnake/scripts/create_sampleSheet.sh --mode illumina ## using the illulima naming scheme sample_S[0-9]_L001_R[12]_001.fastq.gz
path/2/chewieSnake/scripts/create_sampleSheet.sh --mode ncbi ## using ncbi style names: sample_[12].fastq.gz
```


### Prodigal training file (provided by CHEWBBACA)

* chewbbace provides for a number of species a prodigal training file: It is located at `path/to/chewieSnake/chewBBACA/CHEWBBACA/prodigal_training_files`
* see also [here](https://gitlab.com/bfr_bioinformatics/chewieSnake/tree/master/chewBBACA/CHEWBBACA/prodigal_training_files) for the available training files

## Usage (assembly-based version)

The execution needs the path to the config file and to the Snakefile of this repository (`$mydir/bakcharak/Snakefile`). Furthermore you can specify the number of threads.

### Step 1: Activate conda (installation see above)

```
	source activate snakemake
```

### Using the wrapper 

You can use the python wrapper `chewieSnake.py` for creating the config file and subsequent execution of chewieSnake

```
chewieSnake.py  -t THREADS -l SAMPLE_LIST -d WORKING_DIRECTORY -c CONDAPREFIX --scheme SCHEME --prodigal PRODIGAL --report [--dryrun]
```

* Execution will create a config file config.yaml in your WORKING_DIRECTORY
* Use flag `--dryrun` to test command and write config file
* THREADS: Number of cpus to use
* SAMPLE_LIST: sample sheet as described above
* WORKING_DIRECTORY: where the result files should be stored
* CONDAPREFIX: where conda dependencies should be stored (important for efficient re-usage such that software dependencies need to be installed only once)
* SCHEME: Location of cgMLST/wgMLST scheme as described above
* PRODIGAL: Location of prodigal trainining file as described above




#### Details of wrapper

<details><summary>help file</summary>
<p>

```
$ ./chewieSnake.py --help
usage: chewieSnake.py [-h] -l SAMPLE_LIST -d WORKING_DIRECTORY
                      [--condaprefix CONDAPREFIX] [--reads] [--scheme SCHEME]
                      [--prodigal PRODIGAL] [--bsr_threshold BSR_THRESHOLD]
                      [--distance_method DISTANCE_METHOD]
                      [--clustering_method CLUSTERING_METHOD]
                      [--distance_threshold DISTANCE_THRESHOLD]
                      [--address_range ADDRESS_RANGE] [--report]
                      [--comparison] [--comparison_db COMPARISON_DB]
                      [--joining_threshold JOINING_THRESHOLD]
                      [--min_trimmed_length MIN_TRIMMED_LENGTH]
                      [--assembler ASSEMBLER]
                      [--shovill_output_options SHOVILL_OUTPUT_OPTIONS]
                      [--shovill_extraopts SHOVILL_EXTRAOPTS]
                      [--shovill_modules SHOVILL_MODULES]
                      [--shovill_depth SHOVILL_DEPTH]
                      [--shovill_tmpdir SHOVILL_TMPDIR]
                      [--threads_sample THREADS_SAMPLE] [-t THREADS] [-n] [-f]
                      [--forceall] [--unlock] [--logdir LOGDIR]

optional arguments:
  -h, --help            show this help message and exit
  -l SAMPLE_LIST, --sample_list SAMPLE_LIST
                        List of samples to analyze, as a two column tsv file
                        with columns sample and assembly. Can be generated
                        with provided script create_sampleSheet,sh
  -d WORKING_DIRECTORY, --working_directory WORKING_DIRECTORY
                        Working directory where results are saved
  --condaprefix CONDAPREFIX
                        Path of default conda environment, enables recycling
                        built environments, default is in folder conda_env in
                        repository directory.
  --reads               Input data is reads. Assemble (using shovill) prior to
                        allele calling (default is contigs)
  --scheme SCHEME       Path to scheme_dir of the chewieSnake pipeline,
                        (default: /cephfs/abteilung4/deneke/ref/chewbakka/ente
                        robase_senterica_cgmlst)
  --prodigal PRODIGAL   Path to prodigal_training_file (e.g. provided in
                        chewbbaca, path/to/chewieSnake/chewBBACA/CHEWBBACA/pro
                        digal_training_files, default: path/to/chewieSnake/che
                        wBBACA/CHEWBBACA/prodigal_training_files/Salmonella_en
                        terica.trn)
  --bsr_threshold BSR_THRESHOLD
                        blast scoring ratio threshold to use , default = 0.6
  --distance_method DISTANCE_METHOD
                        Grapetree distance method; default = 3
  --clustering_method CLUSTERING_METHOD
                        The agglomeration method to be used for hierarchical
                        clustering. This should be (an unambiguous
                        abbreviation of) one of "ward.D", "ward.D2", "single",
                        "complete", "average" (= UPGMA), "mcquitty" (= WPGMA),
                        "median" (= WPGMC) or "centroid" (= UPGMC); default =
                        single
  --distance_threshold DISTANCE_THRESHOLD
                        A single distance threshold for the exctraction of
                        sub-trees; default = 10
  --address_range ADDRESS_RANGE
                        A comma separated set of cutoff values for defining
                        the clustering address (Default:
                        1,5,10,20,50,100,200,1000)
  --report              Create html report
  --comparison          Compare these results to pre-computed allele database
  --comparison_db COMPARISON_DB
                        Path to pre-computed allele database for comparison
  --joining_threshold JOINING_THRESHOLD
                        A distance threshold for joining data with
                        comparsion_db; default = 30
  --min_trimmed_length MIN_TRIMMED_LENGTH
                        Minimum length of a read to keep, default = 15
  --assembler ASSEMBLER
                        Assembler to use in shovill, choose from megahit
                        velvet skesa spades (default: spades)
  --shovill_output_options SHOVILL_OUTPUT_OPTIONS
                        Extra output options for shovill (default: "")
  --shovill_extraopts SHOVILL_EXTRAOPTS
                        Extra options for shovill (default: "")
  --shovill_modules SHOVILL_MODULES
                        Module options for shovill, choose from --noreadcorr
                        --trim --nostitch --nocorr --noreadcorr (default: "--
                        noreadcorr")
  --shovill_depth SHOVILL_DEPTH
                        Sub-sample --R1/--R2 to this depth. Disable with
                        --depth 0 (default: 100)
  --shovill_tmpdir SHOVILL_TMPDIR
                        Fast temporary directory (default: "")
  --threads_sample THREADS_SAMPLE
                        Number of Threads to use per sample, default = 10
  -t THREADS, --threads THREADS
                        Number of Threads to use. Note that samples can only
                        be processed sequentially due to the required database
                        access. However the allele calling can be executed in
                        parallel for the different loci, default = 10
  -n, --dryrun          Snakemake dryrun. Only calculate graph without
                        executing anything
  -f, --force           Snakemake force. Force recalculation of all steps
  --forceall            Snakemake force. Force recalculation of all steps
  --unlock              unlock snakemake
  --logdir LOGDIR       Path to directory whete .snakemake output is to be
                        saved
```


</p>
</details>


#### Explicit call of the snakemake workflow (skipping the wrapper)


```
    conda activate snakemake
    snakemake -p -s /path/to/chewieSnake/Snakefile --cores 20 -f all --keep-going --configfile /path/to/config.yaml
```

See the [Snakemake documentation](https://snakemake.readthedocs.io) for further details.


### Usage (read-based flavour)

To call the read-based flavour is very similar. Simply put `--reads` to your command. Make sure that the provided sample sheet contains the raw reads. More options are available for trimming and assembly (see below).

```
chewieSnake.py -t THREADS --reads -l SAMPLE_LIST -d WORKING_DIRECTORY -c CONDAPREFIX --scheme SCHEME --prodigal PRODIGAL --report [--comparison_db] [--dryrun] 
```

* Execution will create a config file config.yaml in your WORKING_DIRECTORY
* Use flag `--dryrun` to test command and write config file
* THREADS: Number of cpus to use
* SAMPLE_LIST: sample sheet as described above
* WORKING_DIRECTORY: where the result files should be stored
* CONDAPREFIX: where conda dependencies should be stored (important for efficient re-usage such that software dependencies need to be installed only once)
* SCHEME: Location of cgMLST/wgMLST scheme as described above
* PRODIGAL: Location of prodigal trainining file as described above


### Comparing dataset to a pre-computed allele database

Using the `--comparison` allows the comparison of the current (query) data with previously computed data. Modify the above command by adding the parameters

*  --comparison          Compare query results to pre-computed allele database
*   --comparison_db COMPARISON_DB Path to pre-computed allele profile file for comparison
*   --joining_threshold At which allele distance should query and comparison db sample's be joined (relevant for joined clustering)


E.g.:

```
chewieSnake.py -t THREADS --reads -l SAMPLE_LIST -d WORKING_DIRECTORY -c CONDAPREFIX --scheme SCHEME --prodigal PRODIGAL --report [--comparison_db] [--dryrun] 
```



## Output description

* cgmlst/allele_profiles.tsv: The combined and hashed allele profiles

* cgmlst/distance_matrix.tsv: The allele distance matrix computed with grapetree (grapetree's can be changed with --distance_method ); in grapetree format

* cgmlst/grapetree_mstree: The minimum spanning tree provided by grapetree

* reports/allele_distance_matrix.tsv: allele distance matrix reformated for use in excel etc

* repors/report.html: Interactive html report containing allele statistics, allele distance table, allele distance matrix, cluster types and cluster address, and visualization of the minimum-spanning tree

* Reports for the provided test data are available in this repo:
  * [cgMLST report](https://bfr_bioinformatics.gitlab.io/chewieSnake/report_chewiesnake.html)
  * [cgMLST comparison report](https://bfr_bioinformatics.gitlab.io/chewieSnake/report_chewiesnake_comparison.html)

### Extra output for comparison_db option

* reports/cgmlst_comparison_report.html: Interactive clustering report including a comparison and clustering with the allele database specified in --comparison_db (see example [here](https://bfr_bioinformatics.gitlab.io/chewieSnake/report_chewiesnake_comparison.html))

* cgmlst_joined/match_summary.txt Summary of all matches below joining threshold between query and comparison data

* cgmlst_joined/samples_related_meta.tsv Table of query and matched comparison db samples

* cgmlst_joined/distance_matrix_filtered.tsv Distance matrix of matching data

* cgmlst_joined/grapetree_mstree_filtered.tre Minimum spanning tree of matching data



## Test execution on Salmonella test data



0. Define paths of data and software
```
chewiesnake_path= "" #path/2/chewieSnake
testDIR=path/to/testdata
```

1. Download Salmonella cgMLST scheme from [here](https://databay.bfrlab.de/d/7d7ff2e1ef5d42098f67/) (see above):

```
    mkdir -p $chewiesnake_path/schema && cd $chewiesnake_path/schema
    wget https://databay.bfrlab.de/f/ff459d8ea2c34647a79a/?dl=1
    unzip Salmonella_enterobase_senterica_cgmlst.zip
```

You should now have a folder `enterobase_senterica_cgmlst`


2. Download and extract the test data

```sh
cd $testDIR
wget -O assemblies.tar.gz https://databay.bfrlab.de/f/7a82653915534d628cfa/?dl=1
tar -xzvf assemblies.tar.gz
```

NOTE the data is also contained in testdata/chewiesnake/data

3. Create sample sheet

```

cd $testDIR/assemblies
$chewiesnake_path/scripts/create_sampleSheet.sh --mode assembly
```

A file samples.tsv is returned.

4. Make a dry-run

```
conda activate snakemake
$chewiesnake_path/chewieSnake.py  -t 10 -l $testDIR/assemblies/samples.tsv -d $testDIR/chewieSnake --scheme $chewiesnake_path/schema/enterobase_senterica_cgmlst --prodigal $chewiesnake_path/chewBBACA/CHEWBBACA/prodigal_training_files/Salmonella_enterica.trn --dryrun
```

The screen now displays a number of rules to be executed.

5. Run

```
conda activate snakemake
$chewiesnake_path/chewieSnake.py  -t 10 -l $testDIR/assemblies/samples.tsv -d $testDIR/chewieSnake --scheme $chewiesnake_path/schema/enterobase_senterica_cgmlst --prodigal $chewiesnake_path/chewBBACA/CHEWBBACA/prodigal_training_files/Salmonella_enterica.trn
```
If the final output reads "Workflow finished, no error" everything went fine.


## Test of chewieSnake_join


Test data is supplied under https://gitlab.com/bfr_bioinformatics/chewieSnake/-/tree/master/testdata/chewiesnake_join/data


1. create a allele sheet

```
./create_alleledbSheet.sh {path2repo}/testdata/chewiesnake_join/data
```

This creates an allele sheet which is a tabular file with columns sender  profiles        statistics      timestamps

* The values in sender correspond to the desired partner names
* The paths in profiles are the paths to the (hashed) alelle profiles of each partner
* The paths in profiles are the paths to the allele statistics of each partner. These files must exist but can be empty
* The paths in profiles are the paths to time stamps of each partner (produced by chewieSnake) but can also describe the sampling dates in the format YYYY-MM-DD (e.g. 2020-01-01)

The resulting tab separated file `samples.tsv` is the input for the chewieSnake_join workflow.

2. Call the pipeline

* Please adapt all paths to your repo path
* activate your conda environment


Call:

```
./chewieSnake_join.py -l {path2repo}/testdata/chewiesnake_join/data/samples.tsv -d {path2repo}/testdata/chewiesnake_join/test_results --cluster --serovar_info {path2repo}/testdata/chewiesnake_join/sample_serovars.tsv --external_cluster_names {path2repo}/testdata/chewiesnake_join/epiclusters.tsv  --threads 2 -n
```

Example reports for chewieSnake join (for the provided test data) can be found here:
* [overview report](https://bfr_bioinformatics.gitlab.io/chewieSnake/report_chewiesnake_join.html)
* [cluster report](https://bfr_bioinformatics.gitlab.io/chewieSnake/clustering/CT_0004/clusterreport.html)


## Authors

* Carlus Deneke (@Deneke)

## References
* snakemake: https://snakemake.readthedocs.io/en/stable/
Köster, Johannes and Rahmann, Sven. “Snakemake - A scalable bioinformatics workflow engine”. Bioinformatics 2012.
* chewBBACA: https://github.com/B-UMMI/chewBBACA
Silva M, Machado M, Silva D, Rossi M, Moran-Gilad J, Santos S, Ramirez M, Carriço J. 15/03/2018. M Gen 4(3): doi:10.1099/mgen.0.000166
* sistr: https://github.com/phac-nml/sistr_cmd
Citation: The Salmonella In Silico Typing Resource (SISTR): an open web-accessible tool for rapidly typing and subtyping draft Salmonella genome assemblies. Catherine Yoshida, Peter Kruczkiewicz, Chad R. Laing, Erika J. Lingohr, Victor P.J. Gannon, John H.E. Nash, Eduardo N. Taboada. PLoS ONE 11(1): e0147101. doi: 10.1371/journal.pone.0147101
* grapetree: https://github.com/achtman-lab/GrapeTree
Z Zhou, NF Alikhan, MJ Sergeant, N Luhmann, C Vaz, AP Francisco, JA Carrico, M Achtman (2018) "GrapeTree: Visualization of core genomic relationships among 100,000 bacterial pathogens", Genome Res; doi: https://doi.org/10.1101/gr.232397.117


## cgmlst databases
* enterobase: http://enterobase.warwick.ac.uk/
* chewBBACA: http://chewbbaca.online/
* cgmlst.org: https://www.cgmlst.org/ncs
* bigsdb: https://bigsdb.pasteur.fr/



