Sketch Guide
Written by Brian Bushnell
Last updated March 16, 2017


Overview

BBMap’s Sketch tools use a technique called MinHash to rapidly compare large sequences.  The result is similar to BLAST – a list of hits from a query sequence to various reference sequences, sorted by similarity – but the mechanisms are very different.  Sketch is designed to be extremely fast (potentially thousands of times faster than BLAST) while having a very low disk and memory footprint, but gives more approximate results and does not produce alignments.  The sensitivity is typically lower compared to alignment when seeking remote homology, though default parameters are adequate to identify the species of a raw PacBio library (~85% identity to reference), and detect homology between full bacterial genomes with 76% identity.  The Sketch tools support both nucleotide and amino acid sequences.


Theory

Sketch creation works like this:

1) Gather all length-K kmers in a sequence.
2) Apply a hash function to them.
3) Keep the N smallest hash codes and call this set a "sketch".

A hash code is a number generated from a sequence using a hash function.  For example, consider the genome ACGTTA.  This could be broken into 3 overlapping tetramers: ACGT, CGTT, GTTA.  A hash function might transform ACGT -> 27, CGTT -> 97, and GTTA -> 42.  In this case, if you were generating a sketch of fixed size 2, the two lowest (27 and 42) would be kept, and the other discarded.  In practice, a kmer length of 31 is typically used, genomes tend to contain millions of kmers, and sketches use a fixed size of typically 10,000.

Once a reference database such as RefSeq is converted to sketches, unknown query sequence such as a bacterial assembly can be compared to it by converting the sequence into a sketch, and comparing that sketch to all reference sketches.  The result is the kmer identity, or the approximate fraction of 31-mers the query shares with each reference sequence.  This strongly correlates with sequence identity, and thus can be used as an approximation for relatedness.

For two sequences A and B with size N and M, comparing them using alignment traditionally requires O(N*M) operations.  However, the sketches of A and B each have fixed size S (where S<<N and S<<M).  Sketch comparison is time linear to sketch size – comparing the sketches of A and B requires only O(S) operations.  Since S is a constant, unaffected by input size, this can be simplified to O(1).   Furthermore, when comparing a query sketch to a huge set of reference sketches, it is not necessary to directly compare it to all of them – only those with which it shares kmers, which can be rapidly calculated by indexing the sketches.  This allows reducing the reference sketch set  to only the subset of reference sketches related to the query in O(S*R) operations, where S is sketch size and R is the number of related sketches (which share a kmer with the query).  Again, since S is a constant, this is simplified to O(R); and in practice, R is small so the time needed to look up each kmer (a memory random access) dominates the time needed to iterate through all elements containing that kmer (memory sequential access), so this reference sketch set reduction takes an expected O(1) time.  The actual comparison to the members of the reduced set is still O(R).

In essence, this means that whether you have a database of 3k organisms of size 300kbp each and a 30kbp query, or 2m organisms of size 200Mbp each and a 2Gbp query, the expected amount of time needed to compare the query to the database is only affected by the number of organisms related to the query.  So if the tiny database had more organisms related to the query than the huge database, the huge database would be faster than the tiny database.  A corollary is that adding unrelated sequences to the reference does not affect query time.


Installation

Download and unzip the BBMap package from https://sourceforge.net/projects/bbmap/
Depends on Java 1.6 or higher.  The shellscripts require bash, but are just for convenience.


Usage

The Sketch tools consist of 4 programs – SketchMaker, CompareSketch, SendSketch, and TaxServer, accessed through their respective shellscripts sketch.sh, comparesketch.sh, sendsketch.sh, and taxserver.sh.  It is possible to sketch and compare two random fasta files with the command “comparesketch.sh in=x.fa ref=y.fa”, but in general, usage would follow this pattern:

First, sketch a reference database, either creating one sketch per sequence:

   sketch.sh in=refseq.fa.gz out=refseq#.sketch files=31 mode=sequence

Or one sketch per taxid (aggregated at some level, like subspecies):

   sketch.sh in=refseq.fa.gz out=refseq#.sketch files=31 mode=taxa tree=tree.taxtree.gz gi=gitable.int1d.gz taxlevel=subspecies

This will create thousands of reference sketches, packed into 31 files.  Making multiple files is optional but allows faster loading.  tree.taxtree.gz and gitable.int1d.gz are created using NCBI’s taxdump files; please see TaxonomyGuide.txt for details, or the “Taxonomy Files” section below.  They are necessary to group sequences by taxid by parsing the sequence headers (in taxa mode), but are not necessary when sketching in per-sequence mode.  When parsing headers for taxonomy, Sketch tools currently support NCBI’s naming format or Silva’s naming format.  Accession numbers are also supported.

Once the reference has been sketched, you can compare things to it using CompareSketch:

   comparesketch.sh in=assembly.fa refseq*.sketch

This will list the reference sketches that best match the assembly.  Alternatively, the query can be processed on a per-sequence basis, giving lists of top hits for each contig:

   comparesketch.sh in=assembly.fa refseq*.sketch mode=sequence

Each of these will load all the reference sketches and compare the query sketch(s) to them.  With a large reference set and only a single query, the time will be dominated by loading the reference.  To circumvent this, it is possible to maintain the reference set in memory in a server, accessible via http queries: 

   taxserver.sh port=1234 tree.taxtree.gz gi=gitable.int1d.gz refseq*sketch &

This loads the RefSeq sketches into memory, and listens on port 1234.  To query them, you would use SendSketch:

   sendsketch.sh in=assembly.fa address=http://localhost:1234/sketch

There is a persistent instance of TaxServer running at JGI at https://taxonomy.jgi-psf.org/sketch, and if no address is specified, that will be used.  JGI’s instance currently has nt sketches loaded.


Examples

This is an example of using SendSketch to sketch chromosome 4 of Aspergillus nidulans FGSC A4 and send it to JGI’s TaxServer.  The TaxServer compares it to nt, and reports the top hits.

   sendsketch.sh in=chrIV.fa

Loaded 1 sketches in    0.219 seconds.
Results for ChrIV_A_nidulans_FGSC_A4:
WKID 99.79%     KID 9.49%       matches 949     compared 951    taxID 227321    gSize 43269215  Aspergillus nidulans FGSC A4
WKID 13.33%     KID 1.92%       matches 6       compared 45     taxID 1688608   gSize 15544     Synthetic plasmid pFC333
WKID 13.04%     KID 1.90%       matches 6       compared 46     taxID 1688605   gSize 15734     Synthetic plasmid pFC330
WKID 12.50%     KID 1.92%       matches 6       compared 48     taxID 1688607   gSize 15531     Synthetic plasmid pFC332
WKID 12.00%     KID 1.82%       matches 6       compared 50     taxID 1688606   gSize 16364     Synthetic plasmid pFC331
Total Time:     0.563 seconds.

In this case, the top hits are Aspergillus nidulans FGSC A4 and a few synthetic plasmids.  The most important columns are KID and WKID, which stand for Kmer IDentity and Weighted Kmer IDentity.  The kmer identity is simply the fraction of sketch kmers that are shared – in this case, 9.49%, because only a single chromosome was used as the query, which constitutes around 9.5% of the full genome.    WKID calculates the identity compensating for the difference in genomic sizes, resulting in 99.79%.  This is not 100% because the strains are slightly different.  You can read this as “The smaller sequence comprises 9.49% of the larger sequence, and the larger sequence contains 99.79% of the smaller sequence” – in kmer-space.  Their sketches share 949 of the 10000 kmers.


Taxonomy Files

tree.taxtree.gz and gitable.int1d.gz are created from NCBI's tax dump, like this:

wget ftp://ftp.ncbi.nih.gov/pub/taxonomy/taxdmp.zip
wget ftp://ftp.ncbi.nih.gov/pub/taxonomy/gi_taxid_nucl.dmp.gz
wget ftp://ftp.ncbi.nih.gov/pub/taxonomy/gi_taxid_prot.dmp.gz
wget ftp://ftp.ncbi.nih.gov/pub/taxonomy/accession2taxid/*.gz

unzip taxdmp.zip
taxtree.sh names.dmp nodes.dmp tree.taxtree.gz
gitable.sh gi_taxid_nucl.dmp.gz,gi_taxid_prot.dmp.gz gitable.int1d.gz

