# AQUAMIS - Assembly-based QUAlity assessment for Microbial Isolate Sequencing

[![DOI](https://img.shields.io/static/v1?label=DOI&message=10.3390/genes12050644&color=blue)](https://doi.org/10.3390/genes12050644)
[![Bioconda](https://img.shields.io/conda/vn/bioconda/aquamis?label=BioConda&color=green)](https://anaconda.org/bioconda/aquamis)
[![BioConda Install](https://img.shields.io/conda/d/bioconda/aquamis?label=BioConda%20Install&color=green)](https://anaconda.org/bioconda/aquamis)
[![DockerHub](https://img.shields.io/docker/v/bfrbioinformatics/aquamis/latest?label=Docker)](https://hub.docker.com/r/bfrbioinformatics/aquamis)
[![DockerHub Pulls](https://img.shields.io/docker/pulls/bfrbioinformatics/aquamis?label=Docker%20Pulls)](https://hub.docker.com/r/bfrbioinformatics/aquamis)
[![DockerHub Image Size](https://img.shields.io/docker/image-size/bfrbioinformatics/aquamis/latest?label=Docker%20Image%20Size)](https://hub.docker.com/r/bfrbioinformatics/aquamis)

[TOC]

## 📖 Description

AQUAMIS is a pipeline for routine assembly and quality assessment of microbial isolate sequencing experiments.
It is based on Snakemake and includes the following tools:

* fastp for read trimming and read QC
* shovill (based on Spades) for de-novo assembly
* mash for reference search and species determination
* QUAST v5 (including BUSCO) for assembly QC
* confindr for inter and intra genus contamination analysis
* kraken2 for read and assembly based taxonomic profiling

It will read untrimmed fastq data from your Illumina or IonTorrent sequencing experiments as paired .fastq.gz-files.
These are then trimmed, assembled and polished.
Besides generating ready-to-use contigs, AQUAMIS will select the closest reference genome from NCBI RefSeq and produce an intuitive, detailed report on your data and assemblies to evaluate its reliability for further analyses.
It relies on reference-based and reference-free measures such as coverage depth, gene content, genome completeness and contamination, assembly length and many more.
Based on the experience from thousands of sequencing experiments, threshold sets for different species have been defined to detect potentially poor results.

[Analysis results](Results) are provided in [JSON format](JSON output) for database import and as a convenient interactive AQUAMIS HTML report for fast QC assessment ([example of a test dataset report](https://bfr_bioinformatics.gitlab.io/AQUAMIS/report_test_data/assembly_report.html)).

### Latest News

* [2023-09-14] AQUAMIS v1.4 released, see [Release Notes](RELEASE_NOTES.md).
* [2023-09-06] New conda installations pull updated versions of confindr and quast (see [conda_env_comparison](resources/conda_env_comparison_fromHistory_20230906.tsv)) which introduced breaking changes and required an update of parse_json.py. [FIXED]
* [2023-05-10] We switched the Docker deployment strategy to docker-compose. Current v1.3.12; v1.3.7 was patched alongside.
* [2023-05-05] Confindr: the Augustus module shipped through our `aquamis_setup.sh` script is not compatible with Ubuntu 22.04 (jammy) and later due to compiling issues.
  Tested and working Linux OS are Debian stretch, buster, bullseye; Ubuntu xenial, bionic, focal; Suse LES 12.

### Standards and Distribution

AQUAMIS is used by many German and European Public Health & Food Safety institutes and is installed on a wide variety of computing platforms from laptops, servers, kubernetes instances, HPCs and SLURM-managed supercomputers.  
Its QC assessment is based on the [ISO standard 23418:2022](https://www.iso.org/standard/75509.html) and the software is accredited for usage by the National Reference Laboratories at the [German Federal Institute for Risk Assessment (BfR)](https://www.bfr.bund.de/en/home.html).

## 📜 Citation

> Deneke, C.; Brendebach, H.; Uelze, L.; Borowiak, M.; Malorny, B.; Tausch, S.H.
> Species-Specific Quality Control, Assembly and Contamination Detection in Microbial Isolate Sequences with AQUAMIS
> Genes 2021, 12, 644. [doi:10.3390/genes12050644](https://doi.org/10.3390/genes12050644)

## 🌍 Website

The AQUAMIS project website is [https://gitlab.com/bfr_bioinformatics/AQUAMIS](https://gitlab.com/bfr_bioinformatics/AQUAMIS).

There, you can find the latest version, source code and documentation.

You can find an example AQUAMIS report [here](https://bfr_bioinformatics.gitlab.io/AQUAMIS/report_test_data/assembly_report.html).

## 🏗 Installation

You can install AQUAMIS by installing the Bioconda package, by installing the Docker container or by cloning this repository and installing all dependencies with conda.

AQUAMIS relies on the conda package manager for all dependencies.
Please set up conda on your system as explained [here](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html).

AQUAMIS requires the package manager [mamba](https://anaconda.org/conda-forge/mamba) to resolve all software dependencies.
Install it via `conda install mamba` to your base environment first or via the provided AQUAMIS installation script [<path_to_aquamis>/scripts/aquamis_setup.sh](/scripts/aquamis_setup.sh)).

The installation of reference databases from the BfR file server or other suitable sources is mandatory (see chapter [_From Source_](#from-source)). 

#### Placeholders for Paths in this Manual

| Placeholder              | Path                                                                                                                                            |
|--------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------|
| `<path_to_conda>`        | is the conda installation folder, type `conda info --base` to retrieve its absolute path, typically `~/anaconda3` or `~/miniconda3`             |
| `<path_to_envs>`         | is the folder that holds your conda environments, typically `<path_to_conda>/envs`                                                              |
| `<path_to_installation>` | is the parent folder of the AQUAMIS repository                                                                                                  |
| `<path_to_aquamis>`      | is the base folder of the AQUAMIS repository, i.e. `<path_to_installation>/AQUAMIS`                                                             |
| `<path_to_databases>`    | is the parent folder of your databases, by default, AQUAMIS uses `<path_to_aquamis>/reference_db`, but you are free to choose a custom location |
| `<path_to_data>`         | is the working directory for an AQUAMIS analysis typically containing a subfolder `<path_to_data>/fastq` with your fastq read files             |

### 🐍 From Bioconda

AQUAMIS requires the package manager `mamba` to resolve all dependencies.

```shell
mamba create -n aquamis -c conda-forge -c bioconda aquamis
```

Please complement the Bioconda installation with reference databases of your choice or via the [setup script](/scripts/aquamis_setup.sh) (see previous chapter _From Source_).
This setup script is also available from within the Bioconda installation after `conda activate aquamis` or,
for direct execution, from the Bioconda installation path `<path_to_envs>/aquamis/opt/aquamis/scripts/aquamis_setup.sh`.

### 💻 From Source

To install the latest stable version of AQUAMIS, please clone the git repository on your system.

```shell
cd <path_to_installation>
git clone https://gitlab.com/bfr_bioinformatics/AQUAMIS.git
```

Next, please execute the [setup script](/scripts/aquamis_setup.sh) with the appropriate options:

```shell
bash <path_to_aquamis>/scripts/aquamis_setup.sh --help
```

to ...
1. install the conda dependency manager `mamba`
2. create the [conda](Manual Conda Environment Setup) environment `aquamis`
3. complete the conda environment setup with [Quast modules](QUAST modules - BUSCO)
4. install external [databases](Database Setup) to the default folder `<path_to_aquamis>/reference_db` and
5. download [test data](Test data) to run a proficiency test.

#### Manual Conda Environment Setup

Alternatively, please initialize a conda base environment containing `snakemake` and `mamba` (mamba is faster in resolving dependencies), then:

```shell
mamba env create -f <path_to_aquamis>/envs/aquamis.yaml
```

This creates an environment named `aquamis` containing all [dependencies](/envs/aquamis.yaml).
It is found under `<path_to_conda>/envs/aquamis`.

For custom database paths, please see the chapter [Database setup](Database Setup).

### 🐋 From Docker

*Prerequisite:*
Install the Docker engine for your favourite operating system (e.g. [Ubuntu Linux](https://docs.docker.com/engine/install/ubuntu/)). Check for co-installation of the plugins `docker-compose` and `docker-buildx` (the default on main installation routes).

Download the _latest_ version of AQUAMIS from [Docker Hub](https://hub.docker.com/r/bfrbioinformatics/aquamis/) and note down the Docker Image ID on your system  (hereafter referred as $docker_image_id) with the shell commands:

```shell
docker pull bfrbioinformatics/aquamis:latest
docker images bfrbioinformatics/aquamis:latest -q
```

or with the `docker-compose` strategy, execute the remote installation script which will create an `./AQUAMIS` subfolder in your current working directory:

```shell
cd <path_to_installation>
bash <(curl -s https://gitlab.com/bfr_bioinformatics/AQUAMIS/-/raw/master/docker/get_aquamis_docker.sh)
```

and start-up the containers:

```shell
export LOCAL_USER_ID=$(id -u $USER)
export AQUAMIS_HOST_PATH=<path_to_data>
docker compose --file <path_to_aquamis>/docker/docker-compose.yaml up --detach
```

To process data and write results, Docker needs a *volume mapping* from a host directory containing your sequence data (`<path_to_data>`) to the Docker container (`/AQUAMIS/analysis`).
Your sample list (`samples.tsv`) needs to be located within `<path_to_data>` and contain relative paths (or fixed paths from the container perspective) to your NGS reads in the same or another child directory. 
You may generate a Docker-compatible sample list in your host directory (`<path_to_data>/samples.tsv`) by executing the `create_sampleSheet.sh` from the container with the following terminal commands:

```
host:<path_to_data>$ ls fastq/
sample1_R1.fastq   sample1_R2.fastq   sample2_R1.fastq   sample2_R2.fastq
```

```shell
docker exec --user $LOCAL_USER_ID aquamis_app  \
  /AQUAMIS/scripts/create_sampleSheet.sh  \
  --mode ncbi  \
  --fastxDir /AQUAMIS/analysis/fastq  \
  --outDir /AQUAMIS/analysis
```

With the following command, AQUAMIS is started within the Docker container and will process any options appended:

```shell
docker exec --user $LOCAL_USER_ID aquamis_app  \
  micromamba run -n aquamis  \
  /AQUAMIS/aquamis.py  \
  --docker <path_to_data>  \
  --working_directory /AQUAMIS/analysis  \
  --sample_list /AQUAMIS/analysis/samples.tsv  \
  --<any_other_AQUAMIS_options>
```

**Notes on Docker images:**
The Docker image _latest_ represents the latest build from the Gitlab repository.
It requires additional reference databases (also provided via the [setup script](/scripts/aquamis_setup.sh)) as well as a set of test data files (fastq) for validation purposes via Docker volume binds.
These volume binds are provided by the docker-compose setup.

**Notes on Docker usage:**
The container path `/AQUAMIS/analysis` is fixed and may not be altered.
Any subdirectories of `<path_to_data>` will be available as subdirectories under `/AQUAMIS/analysis/`.
Our container is able to write results with the Linux user and group ID of your choice (`UID` and `GID`, respectively) to blend into your host file permission setup.
With the environment variable `LOCAL_USER_ID=$(id -u $USER)` the UID of the currently executing user is inherited, change it according to your needs.
The absolute host path mapped to the container has to be provided as the `aquamis.py` argument `--docker <path_to_data>`, too.
It is used for correcting file paths in the result JSON files of each sample to match the host perspective.

For reference, please consult the `/AQUAMIS/scripts/aquamis_wrapper.sh` module `run_docker_compose()`.
This module may need to be uncommented/activated for execution in the function `run_modules()`.
Advanced users may copy the wrapper script to their host directory (prerequisite: `docker compose up`, see described above) for a more convenient execution.
Adapt `aquamis_wrapper.sh` and the run sheet `aquamis_runs.tsv` according to your needs:

```shell
docker cp aquamis_app:/AQUAMIS/scripts/aquamis_wrapper.sh  ./
docker cp aquamis_app:/AQUAMIS/scripts/helper_functions.sh ./
docker cp aquamis_app:/AQUAMIS/scripts/aquamis_runs.tsv    ./
```


## 🚀 Usage

### Execution

To run AQUAMIS, source the conda environment `aquamis` and call the wrapper script:

```shell
conda activate aquamis
python3 aquamis.py --help
```

<details><summary>help file</summary>
<p>

```text
usage: aquamis.py [-h] [--sample_list SAMPLE_LIST]
                  [--working_directory WORKING_DIRECTORY]
                  [--snakefile SNAKEFILE] [--run_name RUN_NAME]
                  [--docker DOCKER] [--qc_thresholds QC_THRESHOLDS]
                  [--json_schema JSON_SCHEMA] [--json_filter JSON_FILTER]
                  [--min_trimmed_length MIN_TRIMMED_LENGTH] [--mashdb MASHDB]
                  [--mash_kmersize MASH_KMERSIZE]
                  [--mash_sketchsize MASH_SKETCHSIZE]
                  [--mash_protocol MASH_PROTOCOL] [--kraken2db KRAKEN2DB]
                  [--taxlevel_qc TAXLEVEL_QC] [--read_length READ_LENGTH]
                  [--taxonkit_db TAXONKIT_DB]
                  [--confindr_database CONFINDR_DATABASE]
                  [--shovill_tmpdir SHOVILL_TMPDIR]
                  [--shovill_ram SHOVILL_RAM] [--shovill_depth SHOVILL_DEPTH]
                  [--assembler ASSEMBLER] [--shovill_kmers SHOVILL_KMERS]
                  [--shovill_extraopts SHOVILL_EXTRAOPTS]
                  [--shovill_modules SHOVILL_MODULES]
                  [--shovill_output_options SHOVILL_OUTPUT_OPTIONS]
                  [--mlst_scheme MLST_SCHEME]
                  [--quast_min_contig QUAST_MIN_CONTIG]
                  [--quast_min_identity QUAST_MIN_IDENTITY] [--use_conda]
                  [--conda_frontend] [--condaprefix CONDAPREFIX]
                  [--threads THREADS] [--threads_sample THREADS_SAMPLE]
                  [--thread_factor THREAD_FACTOR] [--batch BATCH]
                  [--force FORCE] [--forceall] [--no_assembly] [--single_end]
                  [--iontorrent] [--ephemeral] [--remove_temp]
                  [--rule_directives RULE_DIRECTIVES] [--profile PROFILE]
                  [--fix_fails] [--dryrun] [--unlock] [--logdir LOGDIR]
                  [--version]

optional arguments:
  -h, --help            show this help message and exit
  --sample_list SAMPLE_LIST, -l SAMPLE_LIST
                        [REQUIRED] List of samples to analyze, as a three
                        column tsv file with columns sample and fastq paths.
                        Can be generated with provided script
                        create_sampleSheet.sh
  --working_directory WORKING_DIRECTORY, -d WORKING_DIRECTORY
                        [REQUIRED] Working directory where results are saved
  --snakefile SNAKEFILE, -s SNAKEFILE
                        Path to Snakefile of AQUAMIS; default: <AQUAMIS>/Snakefile
  --run_name RUN_NAME, -r RUN_NAME
                        Name of the sequencing run for all samples in the
                        sample list, e.g.
                        "210401_M02387_0709_000000000-HBXX6", or a self-chosen
                        analysis title
  --docker DOCKER       Mapped volume path of the host system
  --qc_thresholds QC_THRESHOLDS
                        Definition of thresholds in JSON file;
                        default: <AQUAMIS>/resources/AQUAMIS_thresholds.json
  --json_schema JSON_SCHEMA
                        JSON schema used for validation;
                        default: <AQUAMIS>/resources/AQUAMIS_schema_v20220619.json
  --json_filter JSON_FILTER
                        Definition of thresholds in JSON file;
                        default: <AQUAMIS>/thresholds/AQUAMIS_schema_filter_v20220619.json
  --min_trimmed_length MIN_TRIMMED_LENGTH
                        Minimum length of a read to keep; default: 15
  --mashdb MASHDB       Path to reference mash database;
                        default: <AQUAMIS>/reference_db/mash/mashDB.msh
  --mash_kmersize MASH_KMERSIZE
                        kmer size for mash, must match size of database;
                        default: 21
  --mash_sketchsize MASH_SKETCHSIZE
                        sketch size for mash, must match size of database;
                        default: 1000
  --mash_protocol MASH_PROTOCOL
                        Transfer protocol for reference retrieval, choose
                        between https or ftp; default: "https"
  --kraken2db KRAKEN2DB
                        Path to kraken2 database; default: <AQUAMIS>/reference_db/kraken2
  --taxlevel_qc TAXLEVEL_QC
                        Taxonomic level for kraken2 classification quality
                        control. Choose S for species or G for genus; default:
                        "G"
  --read_length READ_LENGTH
                        Read length to be used in bracken abundance
                        estimation; default: 150
  --taxonkit_db TAXONKIT_DB
                        Path to taxonkit_db; default: <AQUAMIS>/reference_db/taxonkit
  --confindr_database CONFINDR_DATABASE
                        Path to confindr databases; default: <AQUAMIS>/reference_db/confindr
  --shovill_tmpdir SHOVILL_TMPDIR
                        Fast temporary directory; default: /tmp/shovill_$USER
  --shovill_ram SHOVILL_RAM
                        Limit amount of RAM (in GB, integer) provided to
                        shovill; default: 16
  --shovill_depth SHOVILL_DEPTH
                        Sub-sample --R1/--R2 to this depth. Disable with
                        --depth 0; default: 100
  --assembler ASSEMBLER
                        Assembler to use in shovill, choose from megahit
                        velvet skesa spades; default: spades
  --shovill_kmers SHOVILL_KMERS
                        K-mers to use <blank=AUTO>; default: ""
  --shovill_extraopts SHOVILL_EXTRAOPTS
                        Extra assembler options in quotes and equal sign
                        notation e.g. spades: --shovill_extraopts="--
                        iontorrent"; default: ""
  --shovill_modules SHOVILL_MODULES
                        Module options for shovill; choose from --noreadcorr
                        --trim --nostitch --nocorr; default: --noreadcorr;
                        Note: add choices as string in quotation marks
  --shovill_output_options SHOVILL_OUTPUT_OPTIONS
                        Extra options for shovill; default: ""
  --mlst_scheme MLST_SCHEME
                        Extra option for MLST; default: ""
  --quast_min_contig QUAST_MIN_CONTIG
                        Extra option for QUAST; default: 500
  --quast_min_identity QUAST_MIN_IDENTITY
                        Extra option for QUAST; default: 80
  --use_conda           Utilize the Snakemake "--useconda" option, i.e. Smk
                        rules require execution with a specific conda env
  --conda_frontend      Do not use mamba but conda as frontend to create
                        individual conda environments
  --condaprefix CONDAPREFIX, -c CONDAPREFIX
                        Path of default conda environment, enables recycling
                        of built environments; default: <AQUAMIS>/conda_env
  --threads THREADS, -t THREADS
                        Number of Threads/Cores to use. This overrides the
                        "<AQUAMIS>/profiles" settings
  --threads_sample THREADS_SAMPLE, -p THREADS_SAMPLE
                        Number of Threads to use per sample in multi-threaded
                        rules; default: 1
  --thread_factor THREAD_FACTOR
                        Factor to increase threads_sample for shovill and
                        quast; default: 3
  --batch BATCH, -b BATCH
                        Compute sample list in batches. Please state a
                        fraction string, e.g. 1/3
  --force FORCE, -f FORCE
                        Snakemake force. Force recalculation of output (rule
                        or file) specified here
  --forceall            Snakemake force. Force recalculation of all steps
  --no_assembly         Snakemake All-Rule: Only trimming and kraken analysis
  --single_end          Use only single-end reads as input
  --iontorrent          Use single-end Ion Torrent reads as input
  --ephemeral           Snakemake All-Rule: Remove all temporary data except
                        result JSONs and Reports
  --remove_temp         Remove large temporary files. May lead to slower re-
                        runs but saves disk space
  --rule_directives RULE_DIRECTIVES
                        Process DAG with rule grouping and/or rule
                        prioritization via Snakemake rule directives YAML;
                        default: <AQUAMIS>/profiles/smk_directives.yaml
                        equals no directives
  --profile PROFILE     (A) Full path or (B) directory name under
                        "<AQUAMIS>/profiles" of a Snakemake config.yaml with
                        Snakemake parameters, e.g. available CPUs and RAM.
                        Default: "bfr.hpc"
  --fix_fails           Re-run Snakemake after failure removing failed samples
  --dryrun, -n          Snakemake dryrun. Only calculate graph without
                        executing anything
  --unlock              Unlock a Snakemake execution folder if it had been
                        interrupted
  --logdir LOGDIR       Path to directory where .snakemake output is to be
                        saved
  --version, -V         Print program version
```

</p>
</details>

For example:

```shell
<path_to_aquamis>/aquamis.py -l <path_to_data>/samples.tsv -d <path_to_data> -s <path_to_aquamis>/Snakefile
```

You can also run Snakemake directly:

```shell
snakemake --keep-going --rerun-incomplete --cores --configfile <path_to_data>/config_aquamis.yaml --snakefile <path_to_aquamis>/Snakefile
```

A convenient batch execution is provided by the script [aquamis_wrapper.sh](scripts/aquamis_wrapper.sh) together with [aquamis_runs.tsv](scripts/aquamis_runs.tsv).

```bash
bash <path_to_aquamis>/scripts/aquamis_wrapper.sh -n
```

It is intended for experienced users who may change settings in section "## (1) Hardcoded Parameters".
Variables are assigned repeatedly to present available options and allow quick setting changes by changing the order of assignment to the intended behaviour (last assignment will be used).     

### Configuration 

AQUAMIS is built to be used routinely.
To ensure a maximum comparability of the results, a default `config_aquamis.yaml` file is generated when calling the `aquamis.py` wrapper script.
The wrapper itself only allows configuring basic functionalities.
The config_aquamis.yaml can be initialized by starting AQUAMIS with the dry-run flag -n .
Then, you can alter it to configure AQUAMIS in more detail.

## 📑 Results

AQUAMIS will provide you with an interactive, browser-based report, showing the most important measures of your data on the first sight. 
All tables in the report can be sorted and filtered.
*Short Summary Table* shows the key values for a quick estimation of the success of your sequencing experiment and the assembly. 
*Detailed Assembly Table* is giving many additional measures.
*Thresholds* is a copy of the applied threshold definition file.
In addition to the tables, many measures are provided as graphical feedback.
*Plots per Run* and *Plots per Sample* are generated for one complete sequencing experiment and each show measures on one specific dataset, respectively. 


### JSON output

In addition, all results are stored in JSON format in the subfolders `/json/pre_assembly`, `/json/post_assembly` and `/json/post_qc` of your current working directory `<path_to_data>`.
The content of `/json/pre_assembly` files is a subset of `/json/post_assembly` and combines trimming, contamination assessment and read-based taxonomic classification results prior to the assembly stage.
`/json/pre_assembly` represents the final digest when assembly is omitted by enforcing the Snakemake rule *all_trimming_only*, whereas `/json/post_qc` represents the final digest of the full pipeline including the quality assessment.
Each JSON file is named after its corresponding sample and has the following high-level structure:

```text
.
├── sample
│   ├── run_metadata
│   │   ├── pipeline
│   │   ├── timestamp
│   │   ├── sample_description
│   │   │   ├── sample
│   │   │   ├── path
│   │   │   └── hash
│   │   └── pipeline_metadata
│   │       ├── stage
│   │       ├── host_system
│   │       ├── config
│   │       ├── software
│   │       └── database_information
│   ├── summary
│   └── qc_assessment
└── pipelines
    ├── fastp
    ├── confindr
    ├── kraken2
    │   ├── read_based
    │   └── contig_based
    ├── shovill
    ├── samstats
    ├── mlst
    ├── mash
    ├── quast
    ├── busco
    └── aquamis
```

The node ...
* `sample/analysis` holds metadata on the sample fastq data paths, times of analyses, version info, database hashes and analysis parameters of each performed AQUAMIS call.
* `sample/summary` combines selected results of all modules, representing the **Detailed Assembly Table** and is also available as a single line per sample in the `<path_to_data>/reports/summary_report.tsv`.  
* `sample/qc_assessment` holds QC evaluations based on the thresholds defined in the file `AQUAMIS_thresholds.json`. For reference, a copy of the latter definition file is available for queries in the tab **Thresholds** of the assembly report.
* `pipelines/` stores the detailed results of each bioinformatic module/tool in a *full take* approach.  

For easy data mining of multiple sample JSON files in `R`, please follow the methods used in the markdown cells `Import Sample JSONs and Deserialize` and `read_data` of `<path_to_aquamis>/scripts/write_report.Rmd` using the R packages `jsonlite`, `rrapply` and `purrr`.

## 🛢 Database Setup

To complete the Conda environment and DB setup, archives are provided on the BfR file server with two installation options.

### (1) via the AQUAMIS setup script:

```shell
bash <path_to_aquamis>/scripts/aquamis_setup.sh --help
bash <path_to_aquamis>/scripts/aquamis_setup.sh --busco
```

### (2) or manually:

#### ConFindr database

The ConFindr installation already provides databases for _Listeria_, _Salmonella_ and _E. coli_.
Additional databases for _Campylobacter_, _Bacillus_, _Brucella_, _Staphyloccus_ can be found here:

```shell
cd <path_to_databases>   # free to choose
wget --output-document confindr_db.tar.gz https://gitlab.bfr.berlin/bfr_bioinformatics/aquamis_databases/-/raw/main/confindr_db.tar.gz
tar -xzvf confindr_db.tar.gz -C <path_to_databases>
```

Specify the path `<path_to_databases>/confindr` in the `--confindr_database` flag.

You may also consider using the species agnostic rMLST database described [here](https://olc-bioinformatics.github.io/ConFindr/install/#downloading-confindr-databases).

#### Kraken2 and bracken database

We propose using the latest minikraken2 and associated bracken database, see [here](https://benlangmead.github.io/aws-indexes/k2) for details
Alternatively you can download a legacy version:

```shell
cd <path_to_databases>   # free to choose
wget --output-document minikraken2.tgz https://gitlab.bfr.berlin/bfr_bioinformatics/aquamis_databases/-/raw/main/minikraken2.tar.gz
tar -zxvf minikraken2.tgz
```

Specify the path `<path_to_databases>/minikraken2` in the `--kraken2db` flag.

For later identification of the database used in an analysis, we calculated SHA256 hashes of various published TAR archives and the k-mer database within (`hash.k2d`). These can be reviewed in the JSON `<path_to_aquamis>/resources/kraken2_db_hashes.json`.

#### Taxonomy database

```shell
cd <path_to_databases> && mkdir <path_to_databases>/taxonkit   # free to choose
wget --output-document taxdump.tar.gz https://gitlab.bfr.berlin/bfr_bioinformatics/aquamis_databases/-/raw/main/taxdump.tar.gz  #  54MB or ftp://ftp.ncbi.nih.gov/pub/taxonomy/taxdump.tar.gz
tar -xzvf taxdump.tar.gz -C <path_to_databases>/taxonkit/
```

Specify the path `<path_to_databases>/taxonkit` in the `--taxonkit_db` flag.

#### MASH database

```shell
cd <path_to_databases> && mkdir <path_to_databases>/mash  # free to choose
wget --output-document mashDB.tar.gz https://gitlab.bfr.berlin/bfr_bioinformatics/aquamis_databases/-/raw/main/mashDB.tar.gz
tar -xzvf mashDB.tar.gz -C <path_to_databases>/mash/
```

Specify the path `<path_to_databases>/mash` in the `--mashdb` flag.

#### QUAST modules - BUSCO

To detect the path of your Quast environment and associated Python library path, you may type:

```shell
find <path_to_envs>/aquamis -name quast
```

```shell
cd <path_to_envs>/aquamis/lib/python3.7/site-packages/quast_libs/busco/   # exact path depends on conda installation
wget --output-document bacteria.tar.gz https://gitlab.bfr.berlin/bfr_bioinformatics/aquamis_databases/-/raw/main/bacteria.tar.gz
tar -xzvf bacteria.tar.gz
```

#### QUAST modules - Augustus

Augustus is an additional dependency to Quast v5 that should be downloaded and installed automatically.
In case there is a network issue, please install it manually by typing:

```shell
cd <path_to_envs>/aquamis/lib/python3.7/site-packages/quast_libs   # exact path depends on conda installation
wget -O augustus.tar.gz https://gitlab.bfr.berlin/bfr_bioinformatics/aquamis_databases/-/raw/main/augustus.tar.gz
tar -xzvf augustus.tar.gz
```

## 🧬 Test Data

Test data is provided as an archive of fastq files on the BfR file server with two download options.

(1) via the AQUAMIS setup script:

```shell
bash <path_to_aquamis>/scripts/aquamis_setup.sh --help
bash <path_to_aquamis>/scripts/aquamis_setup.sh --test_data
```

(2) or manually:

```shell
wget --output-document test_data.tar.gz https://gitlab.bfr.berlin/bfr_bioinformatics/aquamis_databases/-/raw/main/test_data.tar.gz
tar -xzvf test_data.tar.gz -C <path_to_data>/
cd <path_to_data>
<path_to_aquamis>/scripts/create_sampleSheet.sh --help
```

You can find the AQUAMIS report for this test data set [here](https://bfr_bioinformatics.gitlab.io/AQUAMIS/report_test_data/assembly_report.html).

A use case report with failed QC samples can be found [here](https://bfr_bioinformatics.gitlab.io/AQUAMIS/report_fail_data/assembly_report.html).

## Running AQUAMIS in Batches

```
bash scripts/aquamis_wrapper.sh -n         # dryrun of the batch script with helpful environment information
bash scripts/aquamis_wrapper.sh --dryrun   # dryrun of the snakemake workflows, this already creates the directories
bash scripts/aquamis_wrapper.sh --auto     # non-interactive testing of all modules
```

The wrapper script along with the [run table aquamis_runs.tsv](scripts/aquamis_runs.tsv) allows to run single or consecutive rounds of AQUAMIS.
Non-dryrun script executions along with detailed environment information will be logged to scripts/aquamis_wrapper.log.

1. **Manual Mode**: the variable `dir_data` in the `define_paths()` function of `aquamis_wrapper.sh` _is set_.
The module selector `run_modules()` will be called only once.
2. **Batch Mode**: the variable `dir_data` in the `define_paths()` function of `aquamis_wrapper.sh` _is not set_.
The module selector `run_modules()` will be called on every run (every line not starting with a #) in the table `chewieSnake_runs.tsv`. 

## ⚖️ License

AQUAMIS is distributed under the terms of the BSD 3-Clause License.

## ✉️ Contact

Please consult the [AQUAMIS project website](https://gitlab.com/bfr_bioinformatics/AQUAMIS) for questions.

If this does not help, please feel free to consult:
* Carlus Deneke <Carlus.Deneke (at) bfr.bund.de> or
* Holger Brendebach <Holger.Brendebach (at) bfr.bund.de>
