BBTools changelog and todo list.

V35.
35.00
Changed Gene.toChromosome to return an int rather than a byte.
Changed gitable.int2d name to gitable.int1d since it is a 1D array.
Added taxa support for ArrayListSet.
Added % support for output in Reformat.  Requested by Alex Copeland.
Added gitable.sh script for generating gitable.int1d.gz tax translator.
35.01
Fixed BBDuk crash when K>31 and stats output was enabled.  Noted by Alex Spunde.
Fixed repair.sh failure on fint flag.
Fixed SplitPairsAndSingles not working on interleaved input anymore.
Split Tadpole's mincount flag into mincountseed and mincountextend (mcs and mce).
Added rcomp flag to BBMap.  Requested by Bryce Foster.
Added merge flag to KmerCountExact.
35.02
maq flag now accepts 2 arguments: maq=Q,B.  If second argument is specified, only the initial B bases will be used to calculate the quality.
Added minprob and maq flags to Tadpole and KmerCountExact.
Fixed memory detection in calcmem.sh not working when ulimit=unlimited.  Thanks for the debugging help from Jason S!
Added some getters to KmerForest and KmerNode.
Enabled Tadpole kmer harvesting from victim buffer.
Greatly accelerated Tadpole by allowing threads to compete for tables, rather than using fixed allocation.
Accelerated Tadpole by increasing default number of tables per thread.
35.03
Wrote Shred and shred.sh.
BBMap can now output mapping stats to a file with the statsfile= flag.  Requested by Vasanth.
35.04
Integrated extension into BBMerge (extend= flag).
BBNorm now does ecc after deciding whether to discard a read, not before.
35.05
Fixed FilterByCoverage ignoring minCoverage if pre-normalization covstats not given.
35.06
Added BBMap lengthtag.  Requested by Esther Singer. 
35.07
Fixed rbb flag in BBNorm not working (conflated with parser flag).
Integrated a shellscript modification that allows shellscripts to be symlinked and still find the correct classpath.  Thanks Elmar Pruess!
35.08
Fixed rcompmate flag; it was triggering an assertion error.
Added BBMap showprogress2 flag.
Got rid of ReadInputStream.preferBlocks and associated methods.
Simplified how Reformat works with in1 vs ffin1, and sam files.
Fixed bug in which Reformat was dropping header lines.  Reported by Gloria F.
Fixed bug in BBMergeOverlapper pfilter for reads of different length.
Fixed bug in BBMergeOverlapper for reads of different length.
35.09
Removed BBMerge hist2 and hist3 which were redundant; added showhiststats flag.
Added BBMerge prealloc and prefilter flags.
Removed some old BBMerge functionality (outinsert, perfectonly, etc).
Changed extend to extend1 and extend2.
Completely rewrote BBMerge's code path to break it into small modular functions.
Memory allocation exceptions in HashArray are now handled gracefully.
BBMerge now uses tadpole for kfilter.
BBMerge can now extend before or after merge attempts.
Tadpole can now do error correction via pincer.
Tadpole can now do error correction via tail also.
Added genome size esimation to KmerCountExact (via CallPeaks).  This will be printed in the peaks output header.
Fixed BBMap slowdown caused by rescue in LMP libraries.  Thanks Marc Strous and Xiaoli Dong (Metawatt team) for helping me track it down!
35.10
Removed a debugging line from Tadpole that made it creash when extending reads.
35.11
Removed a debug assertion from SamReadInputStream.  Found by Kurt LaButti.
Improved descriptions in kmercountexact.sh.
Centralized memory statistics printing in Shared.
Separated Tadpoles load phase (KmerTableSet) from build phase (Tadpole).
Added catch blocks for memory exceptions when reading objects from disk.
Added catch blocks for memory exceptions when indexing.
Added catch blocks for memory exceptions in ChromosomeArrays and CoverageArrays.
RandomReads now correctly outputs names in fasta format.
RandomReads now has simple names without custom BBMap coordinates.
KmerCountExact now uses KmerTableSet.
Parsing is more robust for Tadpole, KmerCountExact, and KmerTableSet.
Coverage estimate (based on first peak) now in KmerCountExact.  Requested by Vasanth.
Added ihist PercentOfPairs header line.
Added trim support to KmerTableSet.
Added triangle filter for smoothing histograms before peak calling.  Vastly improves result.  Requested by Alex Copeland.
35.12
Updated shellscripts to have consistent formatting, and fixed various typos.
Reimplemented outinsert for BBMerge.  Requested by Matt Nolan.
35.13
Wrote Tadpole.explore.
Removed debugging parameters (rid, pos) from Tadpole/KmerTableSet ownership functions.
Fixed massive performance problem in KmerArray - victim buffer was being searched for nonexistent kmers.
Wrote function to clear and regenerate tables after shaving.
Shaving now seems to work correctly.
Reduced mcs (minclustersize) in Dedupe from 2 to 1, to match the documentation.
Added Shaver bubble-removal and improved statistics tracking.
Added Tadpole markBadBases (mbb) flag for turning bases covered by low-count kmers into N.
Added Tadpole mode=correct/ecc for correction without extension. 
Tadpole now uses in/out when in extend/ecc mode and ine/oute are not specified. 
35.14
Added iterative seeding with decreasing depth to Tadpole, via contigpasses and contigpassmult flags.
Added Tadpole mdo (markdeltaonly) flag; default true.
Tadpole can now do error marking (mbb) without error correction (ecc).
Tadpole ownership is now automatic.
Added driver.FilterLines and filterlines.sh for filtering text lines.
Verified that an issue with transcriptome mapping is due to an incorrect transcriptome rather than a bug.
Fixed bug in which BBMap subfilter passed sites with no cigar string.  Noted by lankage (SeqAnswers).
mapq of reads with primary site filtered out is now very low (under 4).
BBDuk can now stop after X outm or outu bases.  Requested by R. Westerman.
35.15
Fixed minor bug in Seal in which unmatched reads were not being incremented, causing unmatched read rate to be displayed as 0.
Fixed a bug in parseKMG for decimal values.
BBMerge now supports error-correction with Tadpole.
BBMerge now supports iterative extension.
BBMerge will now always output the original read sequence for reads that don't get merged, rather than the extended or error-corrected sequence.
BBMerge minor output formatting bugs fixed.
BBMerge now calls Tadpole rather than Tadpole_old.
Wrote a shellscript for TaxTree.
Wrote Postfilter and postfilter.sh, a wrapper for BBMap and FilterByCoverage to postfilter SPAdes assemblies.
35.16
Fixed a bug in FilterByCoverage that was filtering everything if cov0 was not specified.
Found and fixed some small bugs in Tadpole, such as not add the very last base of a contig.
Fixed non-determinism in Tadpole by looking for hidden back branches and using extension return codes.
Created ukmer package and Tadpole2, which supports unlimited kmer lengths.  Tadpole2 will be automatically called by Tadpole if K>31.
35.17
Made Tadpole an abstract superclass of Tadpole1 and Tadpole2, with massive reduction in duplicate code.
BBMerge now supports unlimited kmer lengths.
Made AbstractKmerTableSet superclass of KmerTableSet and KmerTableSetU.
KmerCountExact now supports unlimited kmer lengths.
35.18
Made Shaver and abstract superclass for Shaver1 and Shaver2.
KmerCountExact now supports shave and rinse operations.
Prefilter now works optimally with K>31, thanks to new hash routine.
Fixed KmerCountExact not writing peaks without khist set.
35.19
Fixed a crash in read extension with K>31.
35.20
Re-added lines for unmerged read to BBMerge outinsert stream.  Requested by Matt Nolan.
Added some new header lines to KmerCountExact peaks output - ploidy, het rate, repeat content, etc.
35.21
Fixed Tadpole contig coverage estimation.
Added Tadpole mincoverage flag.
Fixed crash in ReadWrite when attempting to create filenames containing the pipe symbol.
Fixed an invalid assertion in HashArrayHybrid resize().
Added IntList.contains().
Fixed a tricky bug with Seal qhdist not looking for matches with substitutions if it first found a match without substitutions.  Noted by sdriscoll.
Dedupe for some reason had interleaved name detection disabled.  This is now enabled.  Noted by Bede Constantinides.
Fixed new BBMerge crash bug with outinsert.  Noted by Matt Nolan.
35.22
Added truncateheadersymbol flag to filterbyname.
Added some postfilter flags with defaults optimized to increase speed.
Fixed Seal qhdist again; I had forgotten to sort a list.  Noted by sdriscoll.
Enabled pigz by default in BBNorm.
Made some improvements to peak-calling.
35.23
Fixed a Tadpole2 assertion error when error-correcting with K>31 and variable-length reads.
Added minconsecutivebases flag to Reformat/BBDuk.
Added locking and lock testing to HashBuffer; unclear whether speed increased.
35.24
Added BBDuk maskfullycovered flag.
Added SummarizeSealStats ignoresametaxa and ignoresamebarcode flags.
Wrote ReduceSilva and reducesilva.sh for shrinking Silva database by removing entries with redundant taxonomy.
35.25
Added SummarizeSealStats ignoresamelocation flag.
35.26
Fixed ignoresamelocation pulling from incorrect field.
Documented mlf flag.  Requested by Alex Copeland.
Added kmg support to minlength and maxlength.  Requested by Bill A.
Fixed a bug in BBMap when handling subfilter on multimapped reads.  Noted by vout.
Improved BBMap fixXY() function.
Fixed major bug in BBMerge; outu read2 was reverse-complemented.
Added a function to soft-clip reads with a long indel anchored by very few bases.
Added ftm, ftl, ftr, ftr2 flags to BBMerge.
Added qtrim2 flag to BBMerge (trim on overlap failure).
Fixed implementation of shave and rinse to properly handle backward branches.
Fixed shave mindepth at 1 instead of variable.
35.27
Fixed a crash bug in BBMap tip indel clipping during fast mode.
35.28
Checksites now verifies correct site ordering.
ensureMatchStringOnPrimary now ensures correct ss ordering if it changes anything.
ensureMatchStringsOnSiteScores now ensures correct ss ordering if it changes anything.
These changes resolved an assertion crash bug.
35.29
Fixed an assertion bug in tip indel clipping.  Noted by Bryce Foster.
Fixed a couple places where clipped bases were not counted when calculating match position.
Fixed another bug with fast match strings related to clipping tip indels.
35.30
Fixed another bug related to clipping tip indels and resorting.  Noted by Bryce Foster and Matt Nolan.
Clipped tip indels are now replaced with matches or mismatches.
Fixed a missing else in SamLine.
35.31
Fixed a bug in toLocalAlignment() when a read has zero matches to reference.
35.32
Fixed an instance where alignments exceeding window size yielded ss with inconsistent lengths.
Improved calculation of amount of padding needed when alignments exceed window.
35.33
Removed kmersamplerate and readsamplerate from bloom package to simplify code.
Corrected handling of minprob in bloom package.
Added Tadpole minprobprefilter and minprobmain flags.
KmerCountExact now disables minProbMain when prefilter is enabled.
Tadpole can now do multipass prefiltering.
Tadpole can prefilter for an automatic number of passes.
Tadpole now supports 1-bit final-pass prefiltering.
Fixed a bug in fixLimitsXY() - only Y needs adjustment, not X.
Fixed a bug in generateMatchString in which sorting was not redone when results changed. 
Added functionality to Bloom prefilter to allow arbitrary cutoffs, rather than just using the filter's max value.
35.34
Fixed a compile error due to Bloom filter changes.
35.35
Added MergeBigelow for combining custom Bigelow text files.
Updated Shred to add equal flag, to shred reads into equal lengths rather than a fixed length.
Tracked down a few bugs regarding ss score-setting order.
Temporarily disabled CHECKSITES and SiteScore.setScore() assertions, which are mainly of interest for efficiency, not correctness.
35.36
Added mlf flag to BBDuk.
Fixed qhdist crash with values over 1 (params were reversed).
Stripped qfin/qfout support from rqcfilter since nobody will ever use it.
Made files of the common kmers found in ribosomes (/global/projectb/sandbox/gaag/bbtools/ribo) using ReduceSilva, Dedupe, and KmerCountExact.
Added ribo filtering to RQCFilter.
35.37
Fixed a ribo filtering flag for RQCFilter.
Added RQCFilter ribodb, ribohdist, riboedist flags.
Added RQCFilter extend flag (allows BBMerge read extension).
Fixed path in file-list (directory was being prepended).
35.38
Improved Tadpole help info.
Added Pileup coverage standard deviation calculation.  Requested by Bill A.
Fixed one last (?) assertion error in BBMap.  Reported by Vasanth and Shijie.
35.39
Postfilter now unloads Data after mapping.
Added trim flag to postfilter and filterbycoverage.
35.40
Fixed a null pointer in Read.validate().
Fixed read extension in Tadpole when K<=31; the wrong method was called, causing a crash.  Noted by Westerman.
Added Tadpole contig trimming flag.
Fixed colossal BBMerge bug - read 2 was being merged as a reverse complement.  Not sure when that started...
35.41
Added spaceslash flag to RandomReads to allow space to be omitted from read names prior to slash pairnum.  Requested by Rob Egan.
35.42
Slightly altered Tadpole1 to allow condensed assembly of kmer sets; added flag ibo (ignore bad owner).
Made KmerCompressor and kcompress.sh.  Generates a concise fasta representation of the set of kmers occuring at least N times.
35.43
Fixed crash in BBDuk wheen using MinKmerFraction (mkf) flag on single-ended reads.
Added fuse flag to KmerCompressor.
Fixed a crash in BBMapPacBio versions, caused by not percolating over a change in normal BBMap.  Noted by Teshome.
35.44
KmerCountExact khist was overflowing if there were more than 2 billion kmers of a given depth.  Converted counts to long array.
Wrote AbstractRemoveThread for removing kmers with counts outside of a certain range.
Added mincr and maxcr (min count to retain and max count to retain) flags to Tadpole.
Fixed incorrect haploid_fold_coverage in peaks.txt.  Noted by Kurt.
Fixed KmerCountExact not writing peaks file if no khist was specified.
Fixed Tadpole differentiation between in/out and ine/oute.  Now only in and out are needed.
Added Reads and Bases columns to Dedupe output.  Requested by Esther S.
35.45
Added driver.CountSharedLines and countsharedlines.sh.  Requested by Esther S.
35.46
Added smoothing control flags for KmerCountExact.
Caught invalid values of K in BBMap.
Added some additional header lines in peaks output.
35.47
Fixed BBMap incorrect NM tags for reads with soft-clipping.  Noted by Rob Egan.
35.48
Disabled second parameter being automatically interpreted as an output file when = is not specified, in most cases.  This is ambiguous as the second parameter might be a file for input read 2.
Fixed a new bug in newly fixed NM tag gen ^^;.  Also noted by Rob Egan.
Identity calculations no longer penalize regions skipped as introns if the intronlen flag is set.  Suggested by Rob Egan.
35.49
Clarified error messages for reads failing barcode filter.
Added cutprimers flag to include flanking primer sequence.
BBMerge trimq can now be an array for multiple attempts.
Multithreaded memory allocation for bloom filters; moderate speed increase.
Added mingc/maxgc to BBDuk and BBDuk2.
Added BBDuk mcf (min covered fraction) flag.
35.50
Added KmerCompressor max flag.
Clarified crossblock help regarding input file lists.
Enclosed all iterations of Dedupe overlapLists with a null check.
KmerCompressor is not deterministic when multithreaded (kmers may be used more than once); reduced buildthreads to 1.
Added LogLog class for cardinality estimation, and loglog.sh.
Enabled loglog flag for Reformat and BBDuk.
35.51
Added X bit to bamscript generated by BBMap.
Loglog can now accept multiple files.
Changed settings of removehuman in rqcfilter to be faster (requires 2 hits now).
Fixed a null pointer exception in BBMerge with quality.
Fixed a bug with BBMerge ecco flag being ignored.
Upgraded Seal to allow requiring full containments of ref sequences.  Requested by Ernst O.
35.52
Fixed issue with "ignorebadquality" flag being ignored in some cases.  Noted by Alicia C.
35.53
Added mouse to RQCFilter.
Switched RQCFilter and RemoveHuman to k=14 for a 4x speedup.
Modified rqcfilter.sh to allow mouse, cat, dog, and human removal concurrently on 40GB nodes.
Disabled test for too-high quality scores because it was annoying when dealing with PacBio reads.
35.54
Added TadpoleWrapper and tadwrapper.sh, which runs Tadpole multiple times with different kmer lengths and recommends the best length.  Requested by Alex Copeland.
Added normandcorrectwrapper.sh, which runs BBNorm then Tadpole.  Requested by Stephan Trong.
Added clear() operation to KmerTableSets.
35.55
Changed Character.isAlphabetic() calls to Character.isLetter().
Modified CountBarcodes to add more flags (for dual barcodes).
RQCFilter pigz and unpigz now default to true.
Moved parsing of threads and recalpairnum from parseCommon to parseCommonStatic.
Increased sensitivity of ribo removal (96.6% to 98.8%) by using a larger kmer set.
Adjusted BBMap default per-thread memory usage estimate after a crash.  Noted by Matt Nolan.
35.57
Fixed a change to removehuman.sh default memory allocation.
35.58
Fixed SynthMDA handling of minlen flag.
35.59
Added sam 1.4 -> 1.3 support to reformat, via sam=1.3 flag.
Added RQCFilter filterqhdist flag.  Requested by Adam Rivers.
Slightly reduced default mininsert in BBDuk from 50 to 40.
Added some additional comments to BBDuk.
Added # support for BBMap output files.  Requested by Adrian Pelin.
Fixed rqcfilter.sh not grab enough memory on slot-scheduled Mendel nodes...  hopefully.
Added name flag to FuseSequence.
35.60
Data.clear() now also clears scaffold information in BBSplitter.
Added removemicrobes flag to RQCFilter.
Added removehuman2.sh for aggressive human contaminant removal versus an unmasked reference.  Requested by Alicia Clum.
Modified RQCFilter to support unmasked mouse, cat, dog, and human references.  Requested by Alicia Clum.
Allowed entropyfilter bbduk flag instead of just entropy.
Unpigz is now used in certain cases where it was prevented before, like reading lists of names in filterbyname.
35.61
Corrected some names in RQCFilter file-list.txt.
35.62
Split writeReproduceHeader off from writeReproduceFile.
Added BBTools version and RQCFilter command to RQCFilter reproduce.sh log.
Added humanpath, catpath, dogpath, mousepath flags to RQCFilter and clarified them in the documentation.
Improved documentation of bbduk2.sh.
Fixed BBMergeOverlapper.c to match java version.
35.63
Fixed some BBMergeOverlapper.c syntax errors.
35.64
Fixed more BBMergeOverlapper.c syntax errors.
35.65
Fixed a BBMergeOverlapper.c runtime errors in quality-free mode.
Finally working again!
35.66
Changed testQuality() to assign ASCII-64 to the specific case of N bases with quality B.
Added preliminary support for dsrc compression.  However, the program does not appear to work correctly in Windows.
Added header output (.header extension).  Requested by Matt Nolan.
Added coverage calculation ignoring deletions.
AssemblyStats (stats.sh) now has fastq support; requested by several people.
Added file type and extension documentation as readme_filetypes.txt.
Removed obsolete changelogs for BBDuk and Reformat.
35.67
Added "_part" suffix before the part number to names of automatic-split fasta reads.  This fixes a problem with underscore-number-named sequences in BBEst.  Noted by Kurt L.
Fixed a corner-case in filterbycoverage's handling of trimmed reads that drop below the length cutoff.  Noted by Stephan Trong.
35.68
Wrote KmerComparator and KmerComparator2 for comparing reads by pivot kmers.
Wrote KmerSort for sorting reads by pivots.
Wrote KmerSplit for binning reads by pivots.
Wrote Clump class for storing ordered overlapping reads.
Wrote Clumpify to wrap KmerSplit and KmerSort.
Wrote ClumpList to turn a list of clumped reads into a list of clumps.
Added preliminary consensus operations to Clump and KmerSort.
Added KmerReduce to produce the set of pivot kmers.
Fixed an out-of-bounds error in CutPrimers.  Noted by vmikk (SeqAnswers).
Moved UnicodeToAscii (which did not work), TableLoaderLockFreeU, and TableReaderU to z_old, since they cause compilation problems.  Noted by Martin M.
Fixed prefilter onepass mode causing a crash.
Added clump package.
Added kmer-count restrictions to Clump pivot selection.  Not clear whether it is useful.
Added local maximum capability to KmerComparator.
Fixed BBNorm to work with kmers>31, for normalization (not error-correction yet).  Not fully tested, though.  Noted by Kurt L.
KmerTableSet read loading now does read validation per thread.  This allows better multithreaded scaling.
BBDuk and Seal also now do validation per thread.
35.69
Multithreaded kmer table dumping by KmerCountExact and Tadpole; over 4x speedup.
35.70
Tadpole now does validation per-thread when error-correcting.  Slight speed increase.
Fixed a bug in KmerCountExact in which prefilter did not work with K>31, due to using key() instead of xor().  Noted by Kurt L.
35.71
Added A_SampleMT, with full line-by-line comments.
Improved A_Sample's comments.
Added kmer histogram generation to rqcfilter (khist flag).
Reorganized rqcfilter.sh documentation.
Added a_sample_mt.sh.
Fixed documentation in shuffle.sh.
Running any program with -version, -help, etc now prints a useful message.
35.72
LogLog now retains last cardinality estimate in a static field.
RQCFilter now chooses BBNorm or KmerCountExact for the khist depending on the estimated cardinality.
Kmer histograms now have a header by default.
35.73
HashBufferU now only tries to acquire a lock every 16th time, like HashBuffer.
Removed some checks for the literal string null.
Fixed some else-if fallthroughs where else was missing.
Addressed some compiler warnings in kmer, ukmer packages.
Wrote FilterByTaxa for filtering of sequences labelled with their taxonomy (gi number or ncbi taxID).
Wrote PrintTaxonomy.
Wrote taxonomy.sh and filterbytaxa.sh.
35.74
Added peaks output to rqcfilter.
Made TaxFilter class and revised FilterByTaxa to use it.
Added FilterByTaxa support to RQCFilter for microbial decontamination.
Fixed a bug in which empty files had their format misdetected.
Fixed a couple array-out-of-bounds errors with unicode characters in genetic sequence.  They are now converted to N.
35.75
Enabled worker thread read validation in A_SampleMT.
Wrote SplitByTaxa and splitbytaxa.sh.
35.76
Changed documentation structure.  There is now changelog.txt, readme.txt, UsageGuide.txt, and ToolDescriptions.txt.
Fixed IntList resize overflow bug.  Noted by jazz710 (SeqAnswers).
Removed unicode2ascii.sh since it does not work.
Wrote FungalRelease and fungalrelease.sh.  Requested by Kurt and Jasmyn.
35.77
BBDuk can now call CalcTrueQuality to generate matrices if given a sam file.
Added scaffold name remapping legend to FungalRelease.  Requested by Kurt and Jasmyn.
Wrote BBDukGuide.
Wrote BBMergeGuide.
Wrote TadpoleGuide.
35.78
Fixed infinite recursion when setting threadcount.  Found by Matt Nolan.
35.79
Changed BBNorm defaults to target=100 min=5.
Wrote Reformat guide.
Wrote Seal guide.
Wrote Taxonomy guide.
Added Tools.startsWith(byte[], String)
Revised GiToNcbi and TaxTree functions to allow gi_ as well as gi|, to avoid pipe symbol.
35.79
Standardized syntax of gitable and taxtree flags, and added "auto" option.
35.80
Added FilterBySequence and filterbysequence.sh.
Wrote PreprocessingGuide.
Wrote DedupeGuide.
Wrote BBNormGuide.
Removed bbmap20.sh, bbnorm20.sh, bbsplit20.sh, and khist20.sh since they can now have memory set explicitly.
35.81
Unified shellscipts between private and public release - module load commands now only run if NERSC_HOST==genepool, and "-l" removed from /bin/bash header.
JNI mode is now enabled by default if NERSC_HOST==genepool.
35.82
Fixed a double-print of BBMap version number.
Updated projectb pre-deploy version of BBTools compiled jni code and disabled auto-NJI-enable when NERSC_HOST==genepool.
Wrote guides for A_Sample, BBMask, Stats, CalcUniqueness, Repair, SplitNextera, Clumpify, and AddAdapters.
Wrote BBMapGuide.
35.83
Added "banns" to RandomReads.
35.84
Fixed an unnecessary assertion for negative values of pairedScore in Tools.removeLowQualitySitesUnpaired2.  Noted by Jason S.
Fixed a bug in SamLine.makeMdTag for handling deletions called off the end of a scaffold.  Noted by Jason S.
35.85
Fixed a bug in which BBMap was verifying the presence of the wrong input file.  Noted by Adrian P.



TODO: Add minprob to LogLog.

TODO: Heejung encountered a random null-pointer exception in ByteFile2.run() at "list[loc]=s;".  However, I manually examined the code and this state appears to be impossible.  Perhaps it is a JVM bug?
TODO: Autoset bits and prefilter for khist based on cardinality.

TODO: KmerSet prefilter=1 onepass does not work (prefilter=2 onepass does work).

TODO: Validate BBNorm results with k>31.
TODO: Add summary of how many reads got removed to BBDuk. (Hemant).
TODO: Add Tadpole ability to screen reads with kmers only occurring at most N times, or having errors/Ns after correction. (Torben).

TODO: Program that can demux a sequence file into multiple sequence files randomly.
TODO: SynthMDA with a short reference outputs lots of reads with Ns (Alex Copeland).
TODO: Parser.parse should go at the end, not beginning, of parse blocks for all programs.
TODO: Tadpole should keep nodes with only outward branches.
TODO: Print kmer coverage information after Tadpole assemblies  (Alex Copeland).
***TODO: Replace QuadHeap with a heap of longs.  The current implementation is very slow on NUMA machines.
**TODO: Compare Seal performance and correctness with countvector flag.  One may be faster for large numbers of ref sequences.
TODO: Seal mcf flag.
TODO: Represent covariant depth as a vector with 1.0 for max depth for binning.
*TODO: Allow kcompress direct set subtraction and intersection.
*TODO: Add outu support to filterbyname
*TODO: Speed up BBMap indexing.

TODO: Print information about which reference sequences hit which locations in which reads, for Seal.
TODO: Second extra base for BBDuk edit distance...?
TODO: Thoroughly vet the assertions in CHECKSITES and SiteScore.setScore() to ensure they will do not incur false positive error messages.
TODO: BBMap shave and rinse are reducing contig length at level 2.
TODO: bbcountunique should use a longer K and have an offset rather than just looking at the first K bases.
TODO: Pincer could handle arbitrary problems - indels, error bursts, etc.
TODO: Tail can handle bursts if it simply continues until X bases concur.
TODO: Port pincer/tail over to BBNorm.
TODO: Use entropy to determine how many bases to extend past errors.

TODO: BBMap is not handling pairing when ambig=all.  Pairing should be done at a SS level.  (Elmar P). 
TODO: Tadpole multipass prefilter, and auto prefilter passes.
TODO: BBMap MPI mode.
TODO: Seal needs behavior with qhdist to be toggleable between searching or not searching for mutant kmers if a nonmutant kmer is found.
TODO: BBDuk with hdist should reprocess the reference multiple times, first with hdist=0, then hdist=1, etc.  That will improve specificity.
*TODO: qtrim=r trimq=6 (or even 3) improves BBMerge rate for 2x250, 422 insert library.  
TODO: BBMerge - Track number of errors detected/corrected and error locations.
TODO: Use small heap to reorder HashArray1D to optimize it.
TODO: Dump kmers to text by way when max size is exceeded, then reload by way and re-dump low count kmers.

TODO: Tadpole degenerate contig output.
TODO: Locked/managed HashArray expansion.
TODO: Fractional (1/4) way allocation per build thread.
*TODO: extendToRight should return an exit code, not just true or false.  May not need to be released.
*TODO: Tadpole - first, classify all kmers as junction or non-junction (via ownership).
***TODO: Always verify that left max is yields prev kmer/evicted base.  If not, that is a hidden branch.
*TODO: Allocation schedule for HashArrays.
*TODO: Optional synchronized resize on final schedule slot to minimize memory use.

TODO: MS state for MSA, always, for M1 state.
TODO: extin and extout flags for BBMap.
TODO: FastaReadInputStream asserts false for headers with no sequence.

TODO: Speed up shaving (exploration) where possible.

TODO: Seal kmer rank promotion with 1D arrays.
TODO: Partition program, round-robin with equal number of bp or equal number of sequences per output.
TODO: msa.sh should accept a file instead of literal.
TODO: BBMap bed format (Alex C).
TODO: BBMap fix for crash in filterbyname on sam file - SamLine 1490, assert(start_<=stop_).
TODO: Reformat lhist and readlength.sh should have equivalent information. "I prefer readlength.sh info"
TODO: Tadpole/KCE double-lock and double-buffer with LongLists for loading.
TODO: xmx=auto or percentage
TODO: reformat: multithread?
TODO: bbduk: move read validation to per-thread
TODO: (write scaffolder)
TODO: (write polishing/consensus tool)
TODO: (write breaker)


V34.
34.00
Fixed a bug in BandedAlignerConcrete related to width being allowed to be even.
34.01
IdentityMatrix is now much faster for ghigh-identity sequences, and allows the 'width' flag to increase speed.
Updated FilterReadsByName to allow "names=<read filename>", supporting fastq, fasta, and sam.  So, one file will be filtered according to the names of reads in a second file.  "names=<file>" where the file is just a list of names is still supported.
34.02
Fixed a couple errors in ConcurrentReadInputStreamD.
Added fetching of a dummy list from "empty" for crisD, both master and slave.
Added A_SampleD, which uses crisD.  It now works correctly for master.
Renamed various ConcurrentReadStreamInterface classes.
Added an abstract superclass for all ConcurrentReadInputStreams, which extends Thread.  Now, cris can be started directly without making a new thread.
Changed all instances of wrapping cirs in a thread to just use start directly.  These are mostly commented with "//4567" to find if something was missed (like starting the cris twice).
Increased cris stability by removing "returnList(ListNum, boolean)" and replacing it with "returnList(long, boolean)".  Lists may no longer be recycled.
34.03
Added scaffoldstats to BBQC and RQCFilter fileList logs.  Requested by Bryce F.
Fixed a strange deadlock in Dedupe/ConcurrentCollectionReadInputStream caused by making CRIS a Thread subclass.  This will still occur if CRIS goes back to being a Thread.  Noted by Shoudan.
34.04
Removed hitCount tracking from Seal.
"qtrim=<integer>" is now allowed for all classes using Parser.parseTrim().
Parser.parseZip, parseInterleaved, parseQuality, parseTrim, parseFasta, and parseCommonStatic were integrated into most classes; reduced code size by almost 200kb.
Parser.parseTrim got some extra functionality, like maxNs.
Made an abstract superclass for KmerCount* classes, allowing removal of some code.
Removed all KmerCount.countFasta() methods; they must now use a CRIS.
Retired ErrorCorrectMT (superceded ny KmerNormalize).
Fixed bug in BBDuk, Seal, and ReformatReads - when quality trimming and force-trimming, count of trimmed reads could go over 100%.  Now these counts are independent.  Noted by ysnapus (SeqAnswers).
Removed "minscaf" and "mincontig" flags from Parser.parseFasta() because they were conflated.
Determined cause of Kurt's error message in Dedupe - lower-case letters can trigger a failure.
Dedupe now defaults to "tuc=t" (all input is made upper-case).
Moved CRIS factory from CGRIS to CRIS.
Copied cc2-cc5 to /global/projectb/sandbox/gaag/TestData/SingleCell/SimMockCommunity/plate*/.  These are simulated cross-contaminated single cell plates.
Removed conflated "qual" flag from RandomReads; "q" should be used instead to set all read quality values to a single number.
Fixed conflated "renamebymapping" flag in RenameReads.
"tbr" flag is conflated in KmerNormalize; adjusted so that it now controls both "tossBadReads" (reads with errors) and "tossBrokenReads" (reads with the wrong number of quality scores).
Conflated "gzip" flag in ChromArrayMaker/FastaToChromArrays changed to "gz".
Handled conflated "ziplevel" flag in AbstractMapper.
Conflated "fakequality" flag resolved by moving from BBMap to Parser and renaming "fakefastaquality"/"ffq".
Added hdist2 and edist2 to BBDuk.  These allow independently specifying hdist/edist for full-length kmers and short kmers when using mink.
Added trimhdist2 to RQCFilter/BBQC.
*** Added path and mapref flags to RQCFilter/BBQC; they can now map to an arbitrary genome instead of just human.
Added Shared.USE_MPI field (parsed by Parser.parseCommonStatic; "mpi" or "usempi").
Added Shared.MPI_RANK field (should be set automatically).
Added Shared.MPI_KEEP_ALL field.  This controls whether CRISD objects retain all reads, or just some of them.
CRIS now automatically returns a CRISD when USE_MPI is enabled, as a slave or master depending on whether rank==0.
ListNum is now Serializable.
CRISD now transmits ListNum objects rather than ArrayLists, so that the number is preserved.
Added Maxns flag to reformat.
Fixed BBQC and RQCFilter's unnecessary addition of "usejni" to BBMap phase, since it is now already parsed by parseCommonStatic.
BBQC now defaults to normalization and ecc off, but can be enabled with the "ecc" and "norm" flags, and supports cecc flag.
Added notes on compiling JNI version suggested by sdriscoll.
34.05
Commented out a reference to ErrorCorrectMT in MateReadsMT.
34.06
FindPrimers (msa.sh) now accepts multiple queries (primers) and will use the best-matching of them.
Added a BBMap flag to disable score penalty due to ambiguous alignments ("penalizeambiguous" or "pambig").  Requested by Matthias.
Fixed failure to start CRIS in A_SampleD.
Fixed some incorrect division in CRISD.
Added MPI_NUM_RANKS to Shared.  This is parsed by parser via e.g. "mpi=4".
Added BBMap flags subfilter, insfilter, delfilter, inslenfilter, dellenfilter, indelfilter, editfilter.  These function similarly to idfilter.  Requested by sdriscoll.
34.07
Dedupe now automatically calls Dedupe2 if more than 2 affixes are requested.
Added "subset" (sst) and "subsetcount" (sstc) flags to Dedupe.
Added "printLengthInEdges" (ple) flag to Dedupe.
34.08
Finished Dedupe subset processing for graph file generation.
34.09
Fixed bug where 'k' was not added to filename in RQCFilter.  Noted by Vasanth.
34.10
Documented "ordered" and "trd" flags for BBDuk/Seal.
Added crismpi flag to allow switching between crisd and crismpi.
Added shared.mpi package, containing MPIWrapper and ConcurrentReadInputStreamMPI.
34.11
Added detection of read length, interleaving, and quality coding to FileFormat objects, but these fields are not currently read.
FileFormat.main() now outputs read length, if in fastq format.
Reformat will now allow sam -> sam conversion; not useful in practice, but maybe useful in testing.
Added flag "mpikeepall", default true.
Fixed deadlock when mpikeepall=false.  Noted by Jon Rood.
34.12
Added 'auto' option to gcbins and idbins flags.  Requested by Seung-jin.
Added dedupe "addpairnum" flag to control whether ".1" is appended to numbered graph nodes.
Added real quality to qhist plot, when mhist is being generated.
Moved maxns and maq to AFTER quality trimming in RQCFilter and BBDuk.
Added "ftm" (forcetrimmodulo) flag to BBDuk/Reformat/RQCFilter/BBQC.  Default 5 for RQCFilter/BBQC, 0 otherwise.
34.13
Fixed a missing "else" in RQCFilter/BBQC.  Noted by Kurt LaButti.
34.14
Added .size() to ListNum.
CrisD gained "unicast" method.  Also, unicast and listen now have mandatory toRank parameter.
Made CrisD MPI methods protected rather than private, so they can be overridden.
Refactored RTextOutputStream3.
34.15
Added Shared.LOW_MEMORY:
Disables multithreaded index gen.
Disables multithreaded ReadWrite writeObjectInThread method.
Disables ByteFile2.
For some reason it does not really seem to reduce memory consumption...
Added BBMap qfin1 and qfin2 flags.
Updated BBMap to use more modern input stream initialization.
Added mapnt.sh for mapping to nt on a 120g node.
34.16
Changed RQCFilter "t" to mean "trimmed"; "k" was removed.
Added parser noheadersequences (nhs) flag for sam files with millions of ref sequences.
Documented "ambig" flag in Seal.
Fixed issue where Shared.READ_BUFFER_NUM_BUFFERS was not getting changed with THREADS was changed.  Now both are private and get set together.
Verified that mapnt.sh works on 120G nodes.
34.17
RTextOutputStream3 renamed to ConcurrentReadOutputStream.
ReadStreamByteWriter refactored to be cleaner.
Merged MPI dev branch into master.
34.18
Moved Seal's maxns/maq to after trimming.
Added chastity filter to bbduk and reformat (reads containing " 1:Y:" or " 2:Y:").  Requested by Lynn A.
Dedupe outd stream now produces correctly interleaved reads.  Requested by Lynn A.
Replaced Dedupe TextStreamWriters with ByteStreamWriters, for read output.
34.19
Added parseCommon() to BBDuk, allowing samplerate flag.
34.20
FASTA_WRAP moved to Shared.
Numeric qual output is now wrapped after the same number of bases as fasta output.
"Low quality discards:" line is now triggered by chastity filter.
SPLIT_READS and MIN_READ_LEN are now disabled when processing reference in BBDuk/Seal.
Seal gained parseCommon and parseQuality.
34.21
Fixed MIN_READ_LEN bug (set to 0; should have been set to 1)
34.22
Added qfin (qual file) flags to BBDuk/Seal.
Applied BBDuk restrictleft and restrictright to filtering and masking; before, it was only valid for trimming.
Added calcCigarBases.
Required includeHardClip parameter for all calls to calcCigarLength(), start(), or stop().
Fixed bug in pileup caused by hard-clipped reads.  Noted by Casey B.
34.23
DecontaminateByNormalization was excluding contigs with length under 50bp, which caused an assertion error.
Fixed a crash in BBDuk2 when not using a reference.  Noted by Dave O.
Added entropy filter to BBDuk/BBDuk2.  Set "entropy=X" where X is 0 to 1 to filter reads with less entropy than X.
34.24
Added maxreads flag to readlength.sh.
Fixed bug in BBMap - when directly outputting coverage, secondary alignments were never being used.
BBMap now uses the "ambig" and "secondary" flags to determine whether to include secondary site coverage.  Specifically, "ambig=all" will use secondary sites, while other modes will not unless "secondary=t".  In other words, use of secondary sites in coverage will be exactly the same as use of them in a sam output file.  Removed "uscov=t                 Include secondary alignments when calculating coverage." from shellscript.
Fixed minid trumping minratio when both were specified.  Now, the last one specified will be used.
Added pileup support for reads with asterisks instead of bases, as long as they have a cigar string.  Also sped up calculation of read stop position.
Cigar string 'M' symbols are now converted to match string 'N' symbols if there is no reference.
34.25
BBMerge initialization order bug fixed; it was preventing jni from being used with the "loose" or "vloose" flags.  Noted by sarvidsson (SeqAnswers).
34.26
Fixed semiperfect mode allowing non-semiperfect rescued alignments.  Noted by Dave O.
Fixed ReadStats columns header for qhist when mhist was also generated.
Fixed an inequality in BBMergeOverlapper that favored shorter overlaps with an equal number of mismatches, in some cases.  Had no impact on a normal 1M read benchmark except when margin=0, where it tripled the false-positive rate.
34.27
Enabled verbose mode in BBMergeOverlapper.
34.28
Added "align2." to sam header command line of BBMap.
Fixed bug in BBMap that could cause "=" to be printed for "rnext" even when pairs were on different scaffolds.  Noted by rkanwar (SeqAnswers).
34.30
Reformat can now produce indelhist from sam files prior to v1.4.
Fixed a crash bug in BBMap caused by an improper assertion.  Noted by Rob Egan.
34.31
BBDuk/Seal now recognize "scafstats" flag as equivalent to "stats".
Seal now defaults to 5 stats columns (includes #bp).
Wrote BBTool_ST, and abstract superclass for singlethreaded BBTools.
Clarified documentation of "trimq=X" as meaning "regions with average quality under X will be trimmed".
Fixed major bug in RQCFilter/BBQC:  "forcetrimmod" was being set to same value as "ktrim".  Noted by Brian Foster.
34.32
Changed the way BBMerge handles qualities to make it 40% faster (in java mode).  Reduced size of jni matrix accordingly.
Fixed lack of readgroup tags for unmapped reads in sam format.  Noted by Rahul (SeqAnswers).
Ensured Read.CHANGE_QUALITY affects both lower (<0) and upper (>41) values.
34.33
Pushed BBMergeOverlapper.c to commit.
34.34
Documented trimfragadapter and removehuman in RQCFilter.
Added Parser flag for Shared.READ_BUFFER_LENGTH (readbufferlength).
Added Parser flag for Shared.READ_BUFFER_MAX_DATA (readbufferdata).
Added Parser flag for Shared.READ_BUFFER_NUM_BUFFERS (readbuffers).
RQCFilter now accepts multiple references for decontamination by mapping.
Added FuseSequence (the first BBTool_ST subclass) and fuse.sh, for gluing contigs together with Ns.
Reformatted many scripts' help info to remove echo statements.
Fixed bugs in stats and countgc; they were not including undefined bases when printing the length in gcformat=1 and gcformat=4.
Replaced all instances of .bases.length with .length(), to prevent null pointer exceptions (for example in sam lines with no bases).
Added cat and dog flags to rqcfilter.
Changed defaults of BBMask to reduce amount masked in cat and dog to ~1% of genome.  This still masks all of the coincidental low-complexity hits from fungi.
Determined that dog is contaminated with fungus, particularly chr7 and chr13.
34.35
Fixed a bug in which data was retained from the prior index when indexing a second fasta file in nodisk mode.
34.36
Disabled an assertion in BBMerge that the input is paired; it crashes if the input file is empty.
34.37
NSLOTS is now ignored if at least 16, to account for new 20-core nodes. 
ReadWrite.getOutputStream now creates the directory structure if it does not already exist.  Problem discovered by Brian Foster.
BBQC and RQCFilter now strip directory names before writing temp files.
BBDuk now correctly reports number of reads quality-filtered.
Added "unmappedonly" flag to Reformat.
RQCFilter now defaults to using TMPDIR.
34.38
BBMap now prints reads/second correctly.  Before, it actually displayed pairs/second with paired data.
Added maxq flag to BBMerge, which allows quality values over 41 where reads overlap.  Requested by Eric J.
Changed CoveragePileup from TextFiles to ByteFiles; increased read speed by 3.65x.
Changed CoveragePileup from TextStreamWriters to ByteStreamWriters; increased write speed by 1.46x.
Fixed a bug in BBQC/RQCFilter: paired input and interleaved output was getting its paired status lost.  Noted by Simon P.
Reformat, when in "mappedonly" or "unmappedonly" mode, now excludes reads with no bases or secondary alignments.
34.39
Human contaminant removal is now optional in BBQC.
34.40
ConcurrentReadOutputStream made abstract superclass.
Added ConcurrentGenericReadInputStream, the default implementation.
Added ConurrentReadOutputStreamD, distributed template.
Merged some duplicate methods in MPIWrapper/ConcurrentReadInputStreamMPI.
34.41
Added some features to CoveragePileup, FilterByCoverage, and DecontaminateByNormalization to quantify low-coverage regions on otherwise high-coverage contigs.
Added parser fastadump flag to toggle dumping of kmers as fasta vs 2-columns.
Fixed a couple bugs in RQCFilter which mixed up names of stats files for trimming and filtering.
RQCFilter will now map to cat, dog, and human together with BBSplit if all three are specified, and produce "refstats.txt".
BBDuk/Seal now support ambiguous IUPAC codes in reference sequences.
34.42
ByteFile now returns empty lines as byte[0] instead of null.  This allows processing of fastq files with 0-length reads.  Noted by lankage (SeqAnswers).
Fixed a bug in FastaToChromArrays2 - blank lines in fasta files were interpreted as breaks between sequences.  Noted by Alex Spunde.
Fixed "unmappedonly" flag in reformat.sh - it was providing inverted output.  Noted by Kristin T.
34.43
Improved MD tag generation.  Reference Ns were not being counted, and unnecessary zeros were appearing between adjacent substitutions.  Noted by Jason S.
expectedErrors() and averageQuality() both now require a boolean parameter, includeUndefined.
Fixed a bug in BBQC's output directory - primary output was going to scratch.  Noted by Simon P.
Added path to BBSplit's help menu.  Noted by Ed K.
34.44
TranslateSixFrames now can accept AA input and produce NT output.
Merged dev branch into master.
Enabled CrosMPI to be created when CRIS_MPI is set to true.
BBDuk and Seal now use MPI streams correctly for reading the reference (when MPI is enabled).
Added truseq_rna.fa.gz to resources.
34.45
Added BBDuk skipr1/skipr2 flags.  Requested by Stephanie H.
Fixed a null pointer in ConcurrentReadOutputStreamD.
34.46
Added SamLine.parseFlagOnly(byte[]) for rapid classification of sam lines.
Revised SplitSamFile and added splitsam.sh to the public distribution.  It's now fast (~540MB/s).
Added a table of contents to /resources/.
34.47
Fixed bug parens around in FindTipDeletions; it was sometimes running when it should have been disabled.
Added swap flag to Reformat, for substituting one base for another (as in bisulfite treatment).
Added underscore flag to Reformat.
Fixed threads flag in BBDuk; it was getting parsed in 2 places and never set.
Added qhdist/qhdist2 flags to BBDuk/Seal for mutating query kmers.  Suggested by sdriscoll (SeqAnswers).
Corrected mkh flag in Seal.  Noted by Vasanth S.
FastaReadInputStream now has a mandatory amino field in constructor.
34.48
Bitsets and coverage arrays can now both be disabled in pileup.
Reorganized buffer lengths in BBIndexPacBio to reduce memory usage and support long (6000bp) reads with shorter kmers, down to 9bp.
Added small rna adapter path to BBQC/RQCFilter.
Fixed FilterReadsByName processing of sam files; bug found by Marissa Miller.
Accelerated and reduced memory usage of FilterReadsByName; moved name parsing over to Tools.
Added ReadStreamWriter.USE_ATTACHED_SAMLINE.
34.49
Fixed qin/qout flags in many classes; they were being ignored.  Noted by Jason H.
Added Nextera LMP adapter sequences.
34.50
Added AssemblyStats format=7.  Requested by Andrew Tritt.
34.51
Added physical (aka fragment) coverage flag to Pileup and BBMap.
Added rpkm/fpkm output to Piluep and BBMap.  Requested by Vasanth.
Changed Seal FPKM calcluation; it was dividing by number of mapped reads rather than number of mapped fragments.
34.52
Added SmallKmerFrequency and commonkmers.sh. Requested by Bill A.
Fixed a bug in ReadStreamByteWriter; "attachment" mode was printing a period instead of newline.
34.53
Added graphical display of GC level to gchist.  The gcplot flag works with all programs that use gchist.  Requested by Kecia D.
Added reparse to BBTool_ST.  This allows parsing of subclass fields which are otherwise overwritten by their defaults.
Added count output to SmallKmerFrequency.
34.54
Added cumulative column to gchist.  This is also enabled by the gcplot flag.  Requested by Seung-Jin.
Added BBMap normcov and normcovo flags.  Requested by Vasanth.
Added support for out= to stats and statswrapper.  Requested by Brian Foster.
Fixed a bug in which stdout was being closed by closing a PrintWriter that wrapped it.
Disabled a message about read pairing for sam input.
Finished DedupeByMapping and created dedupebymapping.sh.
34.55
Fixed a bug in BBMap's coverage flags; normcovo was called normcovOverall.  Noted by Matt Nolan.
34.56
Fixed qin flag being ignored by BBMap.  Noted by Adrian P.
Removed obsolete classes ReformatFasta and ReverseComplement (both handled by ReformatReads now).
34.57
Added BBMap timetag flag and thist output.
Fixed bug in AssemblyStats GC output.  Noted by Jasmyn P.
Added format=0 to stats (no output).
34.58
Version bump.
34.59
BBSplit now supports # operator in filenames.  Requested by Vicente G.
BBMap now prevents cross-scaffold alignments if any output file is sam or bam, not just the primary one.
Reformat now has a primaryonly flag to prevent output of secondary alignments.
Added KillSwitch class.  This will kill the process after X seconds with under Y CPU utilization.  It is invoked by the command line argument "monitor" for post programs.
34.60
BBDuk can now remove reads with less than X% of any single base. Requested by Alicia C.
Added reformat 'filterbits' and 'requiredbits' flags.
Removed obsolete colorspace-specific fields Read.expectedErrors and Read.mapLength.
Wrote SplitNexteraLMP.java and splitnextera.sh.
Added Read.subRead(from, to).
Added BBMap call to ReadStats.checkFiles() to force a crash before running, rather than after running, if there are problems.
Changed nextera_LMP_linker.fa.gz to a double linker after examining real data.
Modified BBTool_ST for greater flexibility with additional IO streams.
Wrote MultiStateAligner9XFlat.java.  For testing.  A flatter, faster MSA.
34.61
Completely removed all support for Solid colorspace.
Removed ChromosomeArrayCompressed.
Removed MultiStateAligner9fs.
Removed FastaStream, QualStream, FastqReadInputStream_old.
Renamed FastaQualReadInputStream3 to FastaQualReadInputStream.
Added kmer.TableLoaderLockfree.  This unifies the load portion of BBDuk and Seal for filling AbstractKmerTables.
Added kmer.TableReader.  This makes it easy to read data from kmer tables.
Added error message for KmerCountExact if k<1 or k>31.
Added warning to BBDuk if no kmers are loaded but a kmer operation is specified.
Added kmask flag to BBDuk.sh help and clarified ktrim flag.
Timer now automatically self-starts upon creation.
Added NexteraLMP support to rqcfilter.
Added versions of Illumina contaminant files without Nextera adapter junctions.
34.62
Fixed null pointer in BBDukF.  Found by Alicia C.
34.63
Created tax package for processing NCBI taxonomy data.
Added IntList.getUniqueCount()
Added TaxNode and TaxTree, for accumulating taxa counts.
Added GiToNcbi, for translating gi numbers to taxa ids.
Added RenameGiToNcbi, for renaming sequences (e.g. nt) with their taxa id.
Added SortByTaxa, for sorting sequences based on taxonomy for better compression.
TaxTree and GiToNcbi now support serialized input; much smaller.
Integrated taxonomy support into Seal.
Seal now uses 9 ways, and uses pigz when loading the reference.
Added ftr2 (forcetrimright2) flag, which allows trimming a fixed number of bases on the rightmost end.
Added BBMap parsing logic to prevent bad vad values of maxsites and maxsites2.
Fixed BBTools failure to find primes.txt.gz if there is a space in the classpath (Matt Kearse).
All calls to average quality now require a max number of bases to process.
Added maqb flag (min average quality bases); maq calculation will be restricted to that many leading bases.  (Shoudan)
Retired FilterReads (superceded by Reformat and BBNorm).
Added reformat "tossjunk", "fixjunk", and "aminoin" flags.
34.64
Fixed off-by-one error in forcetrimright2.
34.65
Made gi2taxid.sh, which calls RenameGiToNcbi.
RenameGiToNcbi updated to split input into valid and invalid output, where invalid gets anything with no taxid.
Added FileFormat.isFasta(String) method.
Improved SortByTaxa.  Now does preorder traversal of tree, and supports dummy nodes, fusing, and promotion.
Added sortbytaxa.sh.
BBSplit ref= can now point to directories.  Requested by Manuel K.
34.66
Fixed an uncaught overflow in ByteBuilder.expand().
Added SortByTaxa max fusion length.
Added barcode filtering to Reformat and BBDuk.
Sped up chastity filtering and allowed it to process reads with / before read number.
Added chastity filtering and barcode filtering to RQCFilter.
34.67
Fixed BBDuk double-counting chastity/barcode-filtered reads.
Fixed BBDuk overcounting of reads that were trimmed by overlap.
BBMap nzo flag now affects refstats and scafstats in addition to covstats (Vasanth).
Added BBMap sortstats flag.
Added BBMap rebuild flag (Vasanth).
34.68
Added qtrim=window flag (Alicia).
Added slashspace flag to Reformat (disable space when adding /1 and /2 to read names).
Added clearzone to Seal (Vasanth).
Added stoptag to Reformat.
Added boundstag to Parser (Shoudan).
34.69
Slight fix for samline inbounds detection.
Fixed a corrupt Truseq RNA adapter sequence.
Added Truseq RNA adapters to adapters.fa.
34.70
Added BBMerge useratio mode (enabled by default in vloose mode).
Added BBMerge adapter processing.
Added BBDuk kmask=lowercase.
34.71
Added BBMerge uloose and vstrict modes.
Added BBMerge requireratiomatch, ratiominoverlapreduction, and ratiooffset flags.
34.72
Adjusted BBMerge uloose settings.
34.73
Added RandomReads path flag.
Increased BBMerge defaults to -Xmx1000m and readbufferlen=400 to improve scaling.
baseToComplementExtended array now maps lowercase letters to lowercase letters.
Changed BBMerge to use floating-point probabilities.
Fixed missing quote mark in bbsplit.sh.  Noted by Manuel K.
34.74
Added Reformat quantize flag (Alicia Clum).
34.75
RandomReads ignored q=x in perfect mode, and perfect flag did not work without a number.
Added Reformat skipreads flag.
Added DualCris, for dual input files of unequal length.
Repair.sh now works if r1 and r2 files are unequal length.
Added flags for BBMerge normalmode and ratiomode.
Made BBMerge default to ratiomode-only for default and loose stringencies.
BBMerge efilter now enabled by default in all modes.
Improved BBMerge efilter (now occurs after both merge modes).
Updated BBMerge efilter to examine only the trailing X bases depending on mininsert setting.
Made normalmode and ratiomode more independent, so requireratiomatch flag is more efficient.
Added ratiomode settings for strict, vstrict and ustrict.  Ustrict has requireratiomatch enabled.
Increased readlength.sh default memory and reduced number of reads in buffer.
Added BBMerge ordered flag; disabled by default.
34.76
Accelerated BBMerge by using a buffer for quality values translated to probabilities.
Added early exits before mininsert0 and minoverlap, and added mininsert0 flag.
Added ecc mode, for correction only and no merging.
Tested removal of runtime division; speed unchanged.
34.77
Added BBMerge pfilter flag; discards overlaps with low probability mismatches.
Fixed BBMergeOverlapper.expectedMismatches() and probability().  Both were considering the wrong bases.
Increased mateByOverlapRatioJava speed with altBadlimt.  Slightly increases false-positive rate.
Added reformat itn flag (convert iupac symbols to N).
Simplified mateByOverlap and mateByOverlapRatio - removed no-quality loop.
BBMerge uloose settings tweaked; no longer uses normalmode.
34.78
Fixed BBDuk compatibility with new BBMergeOverlapper float array requirement.
34.79
Integrated new BBMergeOverlapper ratiomode call into BBDuk.  Uses settings similar to strict mode.
Adjusted BBMerge defaults some more.
34.80
Added outu flag to demuxbyname.
Added overlapWithoutQuality (owq) and overlapUsingQuality (ouq) flags to BBMerge; default is ouq=f owq=t.
Made a quality-free loop in BBMergeOverlapper now that quality is disabled by default; 20% faster.
Removed minoi flag and some legacy fields from Overlapper that dealt with mapping information.
34.81
Split read verification into a different function outside of constructor.
Added a mode for worker threads to verify reads, instead of at construction time.
Added input file check to BBMerge.
Greatly increased speed of overlapper with findBestRatio function.
34.82
Minor changes to BBMerge entropy calculations.
Added BBMerge static function errorCorrect().
Changed fast from normalmode to ratiomode.
Added ecc flag to BBDuk and Seal.
34.83
Fixed DualCris and SplitPairsAndSingles (repair.sh).  Crash bug found by GenoMax.
34.84
Added overlap flag to BBNorm to regulate whether overlapping is used for error correction.
BBMerge now has static functions mergeableFraction() and makeInsertHistogram().
Added ecc flag to kmercountexact.
Added normbins flag to BBMap and fixed normcovo flag.
34.85
Finished ArrayListSet.
Created MultiCros and a functional reference implementation in main().
Upgraded Seal handling of refnames mode; now all sequences for a ref file get the same ID (in that mode).
Added BBSplit-style output for Seal refstats.  Requested by Vasanth.
Removed outsingle from Seal.
Added multiple output streams to Seal with the pattern flag; acts like basename in BBSplit.
DemuxByName now supports arbitrary output streams without needing a name list.
Moved directory parsing from BBSplit to Tools.getFileOrFiles().
34.86
Fixed text string for output stats of FilterByName.  Noted by Alex C.
Tiny change to BBMergeOverlapper regarding Ns.
Added mkf (minkmerfraction) flag to Seal.  Requested by Vasanth S.
34.87
Added /ref/qual/ directory for recalibration matrices.
Added recalibrate() function to CalcTrueQuality.
recalibrate (recal) is now a parser flag and has been enabled for BBMerge, Reformat, and BBDuk.
Added fixheader flag to parser.  Requested by Shijie.
Changed parseInt to parseLong for matrix loading.
Added qb123 matrix.
Added estimate by max observed error rate rather than average.
34.89
Added unicode2ascii.sh for fixing files with strange symbols.
34.90
Fixed FastaReadInputStream's ability to process extended ascii characters (128-255).
Fixed BBDuk, Reformat, and Seal sometimes ignoring the ftr2 flag.
Added adapter sequence detection to BBMerge (outa flag).
Made CalcTrueQuality multithreaded.
Fixed ROOT_QUALITY not getting set with the path flag.
Added path flag to BBDuk and BBMerge.
Added notags flag to BBMap.
34.91
CalcTrueQuality now tracks paired reads independently.
BBDuk and various programs no longer deadlock waiting for sam header to be read.
Added BBMerge iupacton (itn) flag.
Seal can now write sam output from sam input.
BBDuk can write sam output from sam input, but only for quality recalibration, not other operations.
Made RemapQuality to test recalibration.
Internal 2-pass recalibration is now working.
Changed $CMD to eval $CMD in all shellscripts, which allows escaping spaces in filenames using backslash.  Thanks Jon!
Added qchist (qualityCountHistogram).  Gives counts of bases with each quality score.
Fixed BBDuk not testing to see if qahist was set.
Changed CalcTrueQuality default observationcutoff to 100, because higher settings cause odd-looking graphs.
34.92
Added RenameReads prefixonly flag.
Added driver.SummarizeCoverage to summarize cross-contamination scafstats files.
Changed calcmem.sh to work correctly even if ulimit fails.  Noted by Tomas B.  Thanks to vladr (stackoverflow)!
Changed bbduk.sh/bbduk2.sh default ram to 1400m from 2000m so that they should work on 32-bit MacOS systems without setting the -Xmx flag.
Removed some redundancy from TextFile.
DecontaminateByNormalization now named CrossBlock.
Added Read.uToT and parser utot flag for converting uracil to thymine in reads.
BBMap now converts U to T when generating the reference, and all degenerate bases to N.
Changed SummarizeCoverage to be memory-efficient with large files (e.g. coverage vs nt).
Removed colorspace from ChromosomeArray.
Fixed a bug in which cigar strings were sometimes not printed for secondary alignments or when using filters.  Noted by Jason S.
Added qb12 matrix to CalcTrueQuality.
Added support for adjusting read quality score limits beyond 2~41 with the mincalledquality and maxcalledquality flags.
Recalibration matrices may be extended above Q41 with the recalqmax flag, for processing consensus reads.
34.93
Fixed version flag in parser; it was being ignored if there were arguments.
Fixed calcmem.sh behavior on a Mac (or other system without /proc/mem) when ulimit=unlimited.
Fixed issue where BBTool_ST subclasses were having params overridden by defaults.
ReadStats now correctly tracks read1 and read2 for qhist using sam input, instead of lumping them together.
Fixed CalcTrueQuality's recalibration of sam input; it was applying the read1 profile to both reads.  Finally works perfectly.
Added BBSplit force-rebuild logic.
Coverage histogram now goes to 1 million in 32bit mode, instead of 64k.
Added pjet to rqcfilter/bbqc.
Fixed a division-by-zero bug in ReadStats.  Noted by Seung-Jin.
Removed confusing readme line stating that BBMap is free for noncommercial use.  This is true, but it is also free for commercial use.
Added stdev to covstats output (as long as arrays are enabled).
Added driver.SummarizeSealStats and summarizeseal.sh for analyzing cross-contamination results using Seal stats output.
Added summarizescafstats.sh script for driver.SummarizeCoverage.
Added jgi.FilterReadsWithSubs it select only those reads with substitution errors for bases in a specified quality range.
Added phix_adapters.fa.gz to /resources and updated contents.
Added qahist average deviation header line.
34.94
BBMap now produces an error message if indexing fastq files rather than just crashing.
Added config file support to all BBTools via the config= flag.
Added some sample config files and a config file readme.
Moved bloom filter and count-min sketch data structures to bloom package.
34.95
Added trd flag to reformat.sh help.  Noted missing by Esther Singer.
Added merge support to SplitNexteraLMP.  Currently unknown which is better, merge=t or f.
Added better output names to RQCFilter.  Requested by Bryce F.
Added kmer ownership to AbstractKmerTable.
KmerLink is now an AbstractKmerTable subclass.
Wrote Tadpole.
Fixed bug when reading empty fasta files.  Noted by Matt K.
Added SamLine.countTrailingClip, and modified countLeadingClip.  Now both have soft/hard clipping toggles.
Removed static SamLine.SUBTRACT_LEADING_SOFT_CLIP and replaced with required parameter.
Reduced default initial CoverageArray size from 16m to 500.
Added mincov, maxcov, and delcov flags to BBMask.
Updated BBMask readme.
Split IntList/LongList toString method into SetView and ListView.
Added Pileup toggle for including soft-clipped bases; default false.
Made Tadpole shell script.
Fixed Seal stats header indicating over 100% matched sequences with ambig=all.  Noted by Esther Singer.
AbstractKmerTable classes now correctly return -1 instead of 0 if a key is not present.
Added AbstractKmerTable.clearOwner() to clean up ownership trails of abandoned contigs.
Fixed various bugs in Tadpole.
Added substring and case flags to FilterByName.  Requested by Esther Singer.
Tadpole now works correctly multithreaded.
34.96
Fixed a bug in FastaReadInputStream not shutting down subprocesses when done.
Added LMP insert-size detection mode to Tadpole.
Added ByteBuilder.appendKmer(kmer, k).
Added Tadpole read-extension mode.
Tadpole now builds contigs from kmer seeds rather than contig seeds.  Slower but more consistent.
Tadpole is now a complete assembler.
34.97
Added length, coverage, and GC to Tadpole contig names.
Added directional substrings for filterbyname.  Requested by Esther Singer.
Disabled module lines in reformat.sh.  Noted by Xiaoli D.
Renamed summarizecoverage to summarizescafstats.
Added ambig and kfilter flags to CrossBlock.  Requested by Ken H.
34.98
BBDuk/BBDuk2 default maxrskip set to 1 (disabled), to reduce confusion.
Fixed Seal generating ArrayListSets even when pattern output was not specified.
Fixed Seal bug classifying both read1 and read2 as matched when only one matched in kpt=f mode (this IS the intended behavior in default kpt=t mode).  Noted by Alex Spunde.
Fixed string compare bug in FilterReadsByName making substring=header and substring=name fail.  Noted by Esther Singer.
34.99
Fixed BBMap not printing coverage statistics when machineout=t.  Noted by Vasanth Singan.
Added minlen flag to filterbyname.
Changed Dedupe primary structure from HashMap to LinkedHashMap to (somewhat) preserve input order.



TODO: BBDuk crashes with K>31 (Alex Spunde).
TODO: Memory autodetection does not work on Amazon.
TODO: BBMap machineout to file (Vasanth).
TODO: chrombits and CHROMS_PER_BLOCK may be obsolete and ready to remove.
TODO: out=stdout.bam does not work.
TODO: Include deletions toggle for Pileup.
TODO: Soft-clipping coverage flag.
TODO: Add match/cigar/SamLine trimming to TrimRead.
TODO: Write Hollow.
TODO: Multithread splitnextera.
TODO: config flag in Parser
TODO: Normalize CalcTrueQuality on 50% GC by tracking GC rates (etc) observed in reads.
TODO: Make Recalibrate class and recalibrate.sh to automate everything.
TODO: Track quality-score accuracy per base location.
TODO: Track quality-score accuracy per base letter.
TODO: Tool to extract reads mapped to a specific locus.
TODO: Make it easy to test a decontam tool on the synth datasets.
TODO: Map unknowns in 48-sample-plate.
TODO: BBMerge return codes. -1 no solution, -2 ambig, -3 too long (short overlap), -4 too short.
TODO: Seal speed and mkf flags should work together.
TODO: Apply Seal refnames upgrade to taxonomy handling, if not already done.
TODO: BBNorm histout with 1pass/ecc does not seem to generate anything.
TODO: randomreads does not name reads by origin in fasta format.
TODO: Hamming distance for demuxbyname.
TODO: BBMerge has a lower merge rate when r1 and r2 are different lengths. (simple fix - swap them [done]).
TODO: MultiCros wrapper and hash-based multi-listnum object.
TODO: Reformat should be able to trim mapped sam files (Aldo J).
TODO: Mask bases overlapping from Dedupe graph (Shoudan).
TODO: Seal split capability, or BBSplit for short sequences (Manuel K).
TODO: RQCFilter - dynamically switch between $TMPDIR or /dev/shm depending on input size and available disk space.
TODO: BBMerge - trim adapters for unmerged reads (?)
TODO: Fungal pipeline: FindErrors?
*TODO: BBMap calls calcCorrectness even when data is not synthetic.
TODO: BBMap File containing all reads/pairs that are not completely contained within a single contig. (Shoudan)
TODO: BBDuk/Seal - enable tracking of kmers by reference file rather than reference sequence.
TODO: Batch setting for BBDuk to operate on multiple files and auto-name output.
TODO: Get data from Chris B, count mismatched pairs, send to E.
TODO: Stats does not accurately estimate BBMap RAM usage for K=15.
TODO: Accelerate maxindel=0 mode for BBMap by banning MSA usage.
*TODO: BBMerge poster.
TODO: Redo DedupeByMapping so that it can handle sorted input using a heap.
TODO: MSA Flat - remove states to increase speed.
TODO: Dedupe does not work with sam input. (Lynn A.)
TODO: Change all instances of "remove bases with quality below minq" to "...trimq" in shellscripts.
TODO: Parse extra part of sam lines into a byte array (optionally).
*TODO: Dedupe crash on input in C:\temp\dd1\bad.fa (Shoudan).
TODO: Tile-based statistics and filtering for BBMap, BBDuk, etc.
TODO: Pileup could calculate ref/nonref coverage.
TODO: Stats needs a fastq mode.
TODO: Marcel wants a program to essentially sort reads and remove duplicates that are at least X identity.
TODO: Move parsing of "threads" to parseCommonStatic and adjust all relevant classes.
TODO: Add 'remap' from Reformat flag to BBMap.
*TODO: BBMerge won't go below 17bp in normal mode or 26bp in loose mode, regardless of minoi flag.
TODO: BBMerge dynamic mode - test to determine best overlap limits.
TODO: Bed output of masked regions by BBMask, or regions with Ns.
TODO: Bed output of regions with coverage abover or below X (Bob Bower).
TODO: Document append in shellscripts.
TODO: Genbank format parser (Sam D).  Looks confusing.
TODO: Decontam should break at (or N-mask) low-coverage areas rather than discarding the whole contig.
TODO: BED support for pileup.  And make Pileup faster by ignoring irrelevent sam fields.
TODO: CrossMask.  Accept set of files; for each, mask using BBDuk with all others as ref.
TODO: Study bisulfite data on BBMap.  Possibly use multiple reference copies with different transforms (C->T, A-G, both, neither).  
TODO: Shellscripts are not able to handle paths containing spaces.
TODO: Add mininsert flag to BBMap.  And maybe maxinsert.
TODO: Parse MD tag when available.
TODO: CC rates for all 3 platforms in one chart; ignore R1/R2 differences.
*TODO: Dedupe loses reads when using paired data and run multithreaded.
TODO: document nhs flag.
TODO: Filter cross-contam plates with only depth and length, test cc rates.
TODO: Fix dedupe crash when minclustersize=1.
TODO: Clarify or fix what minid does in Dedupe.
TODO: Add ribosomal filtering to rqc.
TODO: Update BandedAlignerJNI for quicker width reset.
TODO: Optional penalty when seq ends before ref in banded.
TODO: Make sure AddAdapters is adding them correctly, i.e., reverse-complemented (or not).
TODO: Make list of proposed higher stringency adapter trimming changes and send to Vasanth/Erika.
TODO: Retire ErrorCorrect, and move the functionality over to another class.
TODO: Implement ErrorCorrectBulk in KmerNormalize.  It is used in MateReadsMT.
TODO: BBMerge should allow optional inline error-correction for reads that fail to merge, and revert if they still fail.
TODO: Retire KmerCount7MT (non-atomic version).
*TODO: It appears that timeslip is being correctly applied by fillLimited (etc), but not by calcDelScore() or calcAffineScore().
TODO: Dedupe should warn if lowercase letters are present. (Kurt)


v33.
Added "usemodulo" flag to BBMap.  Allows throwing away 80% of reference kmers to save memory.  Slight reduction in sensitivity.  Requested by Rob Egan.
Moved GetReads back to jgi package and fixed shellscript.
Fixed rare crash when using "local" mode on paired-end data on highly-repetitive genomes (Creinhardtii).  Found by Vasanth S.
Improved "usemodulo" mode - it was biased against minus-strand hits.  Now, it keeps kmers where (kmer%5==rkmer%5).  Result is virtually no reduction in sensitivity (zero in error-free reads, and less than 0.01% in reads with 8% error).
BBMap will now discard reads shorter than "minlen".
Added "idhistbins" or "idbins" flag to BBMap; allows setting the number of bins used in the idhist.
Rescaled BBMap's MAPQ to be lower.  It is now 0 for unmapped, 1-3 for ambiguous, and roughly 4-45 otherwise, with higher values allowed for longer reads.
Added a much flatter MSA version, "MultiStateAligner9Flat", requested by JJ Chai.
Fixed SNR output formatting.
Added "forcesectionname" flag; fasta reads will always get an "_1" at the end, even if they are not broken into multiple pieces. (requested by Shoudan)
Changed "fastareadlen" suffixes to only be appended when read is > maxlen rather than >=
Reorganized SamLine and created SamHeader class.
Modified CountBarcodes to append sub distance from expected barcodes and 'valid' for valid barcodes.
Fixed null pointer exception related to "qhist", "aqhist", and "qahist".  Noted by Harald (seqanswers).
Fixed issue of readlength.sh breaking up reads when processing fasta files without a fasta extension.
Updated BBDuk documentation.
Added "maxlength" and qahist support to BBDuk.
Added "minoverlap" and "mininsert" to BBDuk.
Added "maxlength" to BBMerge.
Created countbarcodes.sh
Added edit distance column to CountBarcodes output.
Added raw mapping score tag, YS:i:, controlled by "scoretag" flag and disabled by default.
Added 'cq' (changequality) flag to reformat.  Default: true.
Fixed mhist being generated from sam files.
Added readgroup support; a readgroup field "xx" can be specified with the flag "rgxx=value".
Updated 'usemodulo' flag to use (kmer%9==0 || rkmer%9==0).  Requiring the remainders to be equal unevenly affected palindromes and thus even kmer lengths.
Updated RemoveHuman to use 'usemodulo' flag and reduced RAM allotment from 23g to 10g.  Updated index location of HG19 masked.
Added "idfilter" to BBMap.
Made BandedAligner abstract superclass and created BandedAlignerConcrete for the Java implementation, and BandedAlignerJNI for the C version.
Made file extension detection more robust against capitalization.
Added outsingle to BBDuk.
Replaced FastaToChromArrays with ChromArrayMaker.  Now, indexing can be done from fastq files instead of just fasta.
Fixed MAJOR bug in which reference was split up into pieces (as of 33.12).
Reverted to old version of reference loader (as of 33.13) as there was still a bug (skipping every other scaffold).
BBDuk (and BBDuk2) now better support kmer masking!  Every occurance of a kmer is individually masked.
Added parseQuality (qin, qout, etc) to Dedupe.
Changed Dedupe default cluster stats cutoff to 2 (from 10), min cluster size to 2, and by default these values are linked.
Added 'outbest' to Dedupe, writing the representative read per cluster (regardless of 'pbr' flag).  This is mainly for 16s clustering.
Fixed sorting of depths in pileup.sh.  Noted by Alicia Clum.
Fixed 'outbest' of Dedupe (was writing to wrong stream).
Slightly accelerated read trimming.
Added read/base count tracking to ConcurrentReadStreamInterface.
Added display of exact number of input and output bases and reads to reformat.sh (requested by Seung-Jin).
Fixed capital letters changing to lower-case in output filenames when using the "basename" flag with BBSplit.  Noted by Shoudan Liang.
Added Tools.condenseStrict(array).
Fixed fast/slow flags with BBSplit.  Noted by Shoudan Liang.
Added 3-frames option to TranslateSixFrames by adding the flag "frames=3".  Requested by Anne M.
TranslateSixFrames now defaults to fasta format when the file extension is unclear.
Added "estherfilter.sh" for filtering blastall queries.
Added option of getting an input stream from a process with null file argument.
Wrote FastaToChromArrays2 based on ByteFile/ByteBuilder for slightly better indexing speed and lower memory use.
Modified ChromosomeArray to work with ByteBuilder.
Fixed reformat displaying wrong number of input reads when run interleaved (due to recent changes).
Added minratio, maxindel, minhits, and fast flags to BBQC, for controlling BBMap.
Fixed "assert(false)" statement accidentally left in SamPileup from testing.  Noted by Brian Foster.
Added kfilter and local flags to BBQC.
Fixed "bs" (bamscript) flag with BBSplit.  Previously, it did not include the per-reference output streams.
Added Jonathan Rood's C code and JNI class for Dedupe.
Modified dedupe shellscripts to allow JNI code.
BBSplit was not outputting any reads when reference files had uppercase letters (as a result of the recent case-sensitivity change).  This has been fixed.  Noted by Shoudan Liang.
BBMap can now output fastq files with reads renamed to indicate mapping location, using the flags "rbm" and "don" (renamebymapping and deleteoldname).
FastaQualInputStream replaced by FastaQualInputStream3.  At least 2.5x faster, and correctly reads input in which fasta and qual lines are wrapped at different lengths.  Bug noted by Kurt LaButti.
Added bqhist, which allows box plots of read quality-per-base-location.
Fixed a slowdown when making quality histograms due to recalculating probability rather than using cached value.
Default sam format is now 1.4.
RemoveHuman/BBQC/RQCFilter now default to minhits=1 because 'usemodulo' reduces the number of valid keys.
Programs no longer default to outputting to stdout when "out=" is not specified because it's annoying.  To write to stdout set "out=stdout.fq" (for example).
AssemblyStats now counts IUPAC and invalid characters seperately.  X and N now denote gaps between contigs, but no other symbols do.  The code was also cleaned somewhat.  The output formatting changed slightly.
Preliminarily integrated Jon Rood's JNI versions of BandedAligner and MultiStateAligner into both Java code and shellscripts to test Genepool deployment.
C code is now in /jni/ folder, at same level as /resources/ and /docs/.
Clarified documentation of BBMap, BBSplit, and BBWrap to differentiate some parameters.  For example, "refstats" only works with BBSplit.
Added LW and RW (whisker values) columns to bqhist output, set at the 2nd and 98th percentiles.  Requested by Seung-Jin Sul.
BBQC will now compress intermediate files to level 2 instead of level 4, to save time.
Fixed incompatibility of dot graph output and other output in Dedupe.
Reverted to default "minhits=2" for RemoveHuman, because minhits=1 took 5x as long.
Added median, mean, and stdev to gchist.  Requested by Seung-Jin.
Added obqhist (overall base quality histogram).  Requested by Seung-Jin.
Fixed various places, such as BBDuk, where the "int=true" flag caused references to be loaded interleaved.  Noted by Jessica Jarett.
Added some parser flags to allow dynamically enabling verbose mode and assertions specifically for certain classes. 
Fixed a bug in BBMap that made secondary alignments sometimes not get cigar strings.
Added "addprefix" mode to rename reads, which simply prepends a prefix to the existing name.
Clarified documentation of different histogram outputs in shellscripts.
Ported BBMapThread changes over to BBMap variants.
Restructured SamPileup and renamed it to CoveragePileup.  Now supports Read objects (instead of just SamLines).
Integrated CoveragePileup with BBMap and documented new flags.
CoveragePileup: Added a concise coverage output, stranded coverage, and read-start-only coverage.
Removed an obsolete Java classes and some shellscripts.
Increased robustness of BBDuk's detection invalid file arguments, and clarified the error messages.  Noted by Scott D.
Fixed a problem with interleaving not being forced on fasta input.
Paired output files will now force BBDuk input to be treated as interleaved.
BBDuk now tracks statistics on which reference sequences were trimmed or masked - previously, it just tracked what was filtered.
Reverse-complemented Nextera adapters and added them to official release (/resources/nextera.fa.gz).
Added Illumina adapter sequence legal disclaimer to /docs/Legal_Illumina.txt
Implemented GC calculation from index, for generating coverage stats while mapping.
Tracked down strangeness with BBDuk.  It is possible for "rcomp=f" to slightly reduce sensitivity when "mm=t" using an even kmer length, due to asymmetry.  This appears to be correct.
Merged in revised JNI Dedupe version that should be working correctly.  Verified that it returns same answer as non-JNI version.  Tests indicate roughly triple speed, when working with PacBio reads of insert.
BBMap JNI version now seems roughly 30% faster than Java version.
Added insert size quartiles to BBMap and BBMerge.  Requested by Alex Copeland.
Fixed rare bug related to SiteScore.fixXY(), caused by aligning reads with insufficient padding, fixing the tips, but not changing the start/stop positions.  Found by Brian Foster.
Fixed a race condition in TextStreamWriter that could randomly cause a deadlock in numerous different programs.  Found by Shoudan Liang.
Added "maxsites2" flag to allow over 800 alignments for a given read.
Fixed bounds of kmer masking in BBDuk; they were off by 2 (too big).
Fixed unintended debug print line.  Noted by Shoudan Liang.
Updated RandomReadInputStream to work with the newer RandomReads3 class.
ConcurrentGenericReadInputStream now supports RandomReadInputStream3 as a producer.
Fixed kmer dumping from CountKmersExact.
Fixed length of vector created in BBMergeOverlapper (4->5).  Noted by Jon Rood.
Changed default kmer length in BBDuk to 27 so that the 'maskmiddle' base will be in the middle for both forward and reverse kmers.
"pairlen" flag accidentally deleted from BBMap; restored.  Noted by HGV (seqanswers).
BBMerge now has a JNI version from Jonathan Rood - 60% faster than pure Java.  Requires compiling the C code; details are in /jni/README.txt.
Wrapped BBMerge JNI initializer in a conditional, so it will not try to load unless "usejni" is specified.  
Added "parseCommonStatic" to BBMerge and BBDuk (to allow JNI flag parsing).
Commented out "module load" and "module unload" statements in public version.
Added 'printlastbin' or 'plb' flag to countunique to produce a final bin smaller than binsize.  Suggested for use in cumulative mode.  Requested by Andrew Tritt.
Added support for bzip2 and pbzip2 compression and decompression.  The programs must be installed to use bz2 format.
Elminated use of "sh" when launching subprocesses.  This also allows pigz compression support in Windows.
Files were not being closed after "testInterleaved()".  Fixed.
Improved error messages when improper quality values are observed.
Updated hard-coded adapter path to include Nextera adapters.  This affects BBQC and RQCFilter.
Improved file format detection.  Now FileFormat (testformat.sh) will print a warning when the contents and extension don't match, and it can differentiate between sam and fastq.  Problem noted by Vasanth Singan.
Fixed issue where "scafstats" output was printing inflated numbers with chimeric paired reads, or pairs with only one mapped read.  Noted by HGV (seqanswers).
Closed stream after reading in FileFormat.
Unrolled, debranched, and removed assertion function calls from BBMerge inner loop.
Fixed a bug in which findTipDeletions was not changing the bounds of the gap array.
Added getters and setters for SiteScores that enforce gap correctness.
Improved GapTools to test for and fix non-ascending points.
Forced use of setters in TranslateColorspaceRead, AbstractMapThread, and BBIndex* classes; this caught some inconsistencies that should increase stability and correctness.
Enabled jni-mode alignment by default for BBQC and removehuman.
Added a BBMap output line indicating how many reads survived for use with, e.g., removehuman.  Requested by Brian Foster.
Added messages to BBQC to indicate which phase is executing.  Requested by Brian Foster.
SiteScore start and stop are exclusively set by methods now.  Fixed a bug with local flag noted by Vasanth Singan.
Added MaximumSpanningTree generation to Dedupe (mst flag).
Merged in faster BBMerge overlapper JNI version; now 90% faster than Java with fastq and 70% faster with fasta.
Improved Dedupe's support for paired reads: fixed an assertion, and added "in1" and "in2".
Fixed a assertion involving semiperfect alignments of repetitive reads, that go out of the alignment window.  Found by Alicia Clum.
Fixed idhist mean calculation.  Added mode, median, stdev, both by read count and base count.
Better documented ConcurrentReadStreamInterface.
Fixed a crash in CoveragePileup when using 32-bit mode.
Fixed a couple instances in which the first two arguments being unrecognized would not be noticed.
Fixed a bug in pileup causing coverage fraction to be reported incorrectly, if arrays were not being used.  Noted by Vasanth Singan.
Fixed a twocolumn mode in pileup; it was generating no output.
Added additional parse flags to pileup, such as "stats" and "outcov".
Added additional output fields to coverage stats - total number of covered bases, and number of reads mapped to plus and minus strands.
CountKmersExact: Added preallocation (faster, less memory) and a one-pass-mode for the prefilter (faster, but nondeterministic).
Replaced most instances of "Long.parseLong" with "Tools.parseKMG" to support kilo, mega, and giga abbreviated suffixes.
Added jgi.PhylipToFasta and phylip2fasta.sh, for converting interleaved phylip files to fasta.  Requested by Esther Singer.
v33.58
Began listing point-version numbers in this readme.
Added jgi.A_Sample2, an simpler template for a concurrent pipe-filter stage.
Added jgi.MakeChimeras, a tool for making chimeric PacBio reads from input non-chimeric reads.  Also, makechimeras.sh.  Requested by Esther Singer.
Added support for normalized binning to CoveragePileup.  Requested by Vasanth Singan.
v33.59
Fixed pileup's normalized scaling when dealing with 0-coverage scaffolds.
v33.60
Added driver.FilterReadsByName.java and filterbyname.sh.  Allows inclusion or exclusion of reads by name.
Added midpad flag to RandomReads (allows defining inter-scaffold padding).
v33.61
Added ConcurrentReadInputStreamD, prototype for MPI-version of input stream.
Made Read and all classes that might be attached to reads Serializable.
Added DemuxByName and demuxbyname.sh which allows a single file to be split into multiple files based on read names.
v33.62
Added FilterByCoverage and filterbycoverage.sh to filter assemblies based on contig coverage stats (from Pileup).
Added CovStatsLine, an object representation of Pileup's coverage stats.
Added '#' symbol to coverage stats header.
v33.63
Fixed path in filterbycoverage.sh
v33.64
Added custom scripts driver.MergeCoverageOTU and mergeOTUs.sh for Esther.
Added DecontaminateByNormalization, for automating SAG plate decontamination.
Fixed legacy code that set KmerNormalize to use 8 threads in some cases.
Added "fixquality" for capping quality scores at 41. Requested by Bryce Foster.
Added fasta output to kmercountexact. Requested by Alex Copeland.
Added kmer histogram to kmercountexact (2-column and 3-column). Requested by Alex Copeland.
Added multiple memory-related and output formatting flags to kmercountexact.
Made KmerNode a subclass of AbstractKmerTable.
Improved Data's "unloadall" to also clear scaffold-related data.
Removed obsolete class CoverageArray1.
v33.65
Reduced preallocated memory in kmercountexact to avoid a crash on high memory machines.  Also reduced total number of threads.
v33.66
"CountKmersExact.java" renamed to "KmerCountExact.java".
kmercountexact now writes histogram and kmer dump simultaneously in seperate threads.
kmercountexact.sh now specifies both -Xms and -Xmx.
CountKmersExact will no longer run out of memory if -Xms is not specified; instead, it will preallocate a smaller table.
v33.67
Messed with MDA amp in RandomReads a bit.
Added parser "ztd" ("zipthreaddivisor") flag.  Defaults to 2 for removehuman.sh.
Added BBMerge flags "maq" (minaveragequality) and "mee" (mmaxexpectederrors).  Reads violating these will not be attempted to merge.
Added BBMerge "efilter" flag, to allow disabling of the efilter.  Efilter bans merges of reads that have more than the expected number of errors, based on quality scores.
Closed A_Sample2 I/O streams after completion.  Noted by Jon Rood.
Created SynthMDA, a program to make a synthetic MDA'd single cell genome.  This genome would be used as a reference for RandomReads.
Added Reformat "vpair" or (verifypairing) flag, which allows validation of pair names.  Before, it was just interleaved reads.
Pair name validation will now accept identical names, if the "ain" (allowidenticalnames) flag is set.
Updated reformat.sh, repair.sh, bbsplitpairs.sh with new flags.
Removed FastaReadInputStream_old.java.
Added "forcelength" flag to MakeChimeras.
v33.68
Added "ihist" flag to rqcfilter, default "ihist.txt".  Unless this is set to null, BBMerge will run to generate the insert size histogram after filtering completes.
AbstractKmerTable preallocation is now multithreaded.  Unfortunately, this did not result in a speedup.
Added ByteBuilder-related methods to certain Read output formats.
Added ByteStreamWriter.  This is a threaded writer with low overhead, and is substantially faster than TextStreamWriter (perhaps 2x speed).
Fixed a bug in KmerNode (traversing wrong branch during dump).
All AbstractKmerTable subclasses now dump kmers using bsw/ByteBuilder instead of tsw/StringBuilder.
Added ForceTrimLeft/ForceTrimRight flags to Dedupe (requested by Bryce/Seung-Jin).
v33.69
FilterByCoverage (and thus DecontaminatebyNormalization) now produce a log file indicating which contigs were removed.
FilterByCoverage and DecontaminatebyNormalization can now optionally process coverage before and after normalization, and not remove contigs unless the coverage changes by at least some ratio (default 2).  Enable with "mapraw" and optionally "minratio" flag.
Added ihist to file-list.txt.  TODO:  Verify success.
Reads longer than 200bp are now detected as ASCII-33 regardless of their quality values.  This helps with handling PacBio CCS/ROI data.
Added support in FixPairsAndSingles (repair.sh) for reads with names that do not contain whitespace, but still end with "/1" and "/2".
Added qout flag to RandomReads3.
Refactored TextStreamWriter to be more like ByteStreamWriter.
Added gcformat 0 (no base content info printed) to AssemblyStats2 (stats.sh).
v33.70
Updated RQCFilter and BBQC to bring them closer together and improve some of their defaults.  RQCFilter now has more parameters such as k for filtering and trimming.
RQCFilter now correctly produces the insert size histogram.
v33.71
Fixed a bug in Dedupe preventing overlap detection when 'absorb match' and 'absorb containment' were both disabled.  Noted by Shoudan Liang.
Optimized synthetic MDA procedure.
v33.72
Fixed a bug in SynthMDA.java.  Further tweaked parameters.
Added synthmda.sh.
v33.73
Further tweaked SynthMDA defaults to better match some real data sent to me by Shoudan and Alex.
Fixed a bug in BBDuk's mask mode in which all bases in a masked read were assigned quality 0.  Noted by luc (SeqAnswers).
Fixed a small error in KmerCountExact's preallocation calculation.
Added preallocation to BBDuk/BBDuk2.  Not recommended for BBDuk2 because the tables may need unequal sizes.
Added "restrictleft" and "restrictright" flags to BBDuk (not BBDuk2).  These allow only looking for kmer matches in the leftmost or rightmost X bases.  Requested by lankage (SeqAnswers).
v33.74
Added jgi.Shuffle.java to input a read set and output it in random order.  It can also sort by various things (coordinates, sequence, name, and numericID).
Added CallPeaks, which can call peaks from a histogram.  Requested by Kurt LaButti.
Integrated peak calling into BBNorm and KmerCountExact.
BBNorm now has a "histogramcolumns" flag, so it can produce Jellyfish-compatible output.
Added callpeaks.sh.
v33.75
CallPeaks now calls by raw kmer count rather than unique kmer count.  This better detects higher-order peaks.
Finished CrossContaminate.java and added crosscontaminate.sh.
Added "header" and "headerpound" to pileup.sh, to control header presence and whether they start with "#".
Added "prefix" flag to SynthMDA and RandomReads3, to better track origin of reads during cross-contamination trials.
RQCFilter and BBQC now parse 'usejni' flag; rqcfilter.sh and bbqc.sh default to this being enabled.
Added "uselowerdepth" flag to BBNorm (default true).  Allows normalization by depth of higher or lower read.  Set to false by DecontaminateByNormalization.
v33.76
Fixed a bug in synthmda.sh command line.
Fixed build number not being parsed by SynthMDA.
Added some error handling to CrossContaminate, so it shouldn't hang as a result of missing files.
v33.77
SynthMDA now nullifies reference in memory prior to generating reads.
Parser was not correctly setting the number of compression threads when exactly 1 was requested.
Shuffle is now multithreaded, and CrossContaminate defaults to shufflethreads=3.
Shuffle now removes reads as they are printed, reducing memory usage.
Created shellscript templates for generating and assembling full plates of synth MDA data, and ran successfully.
*SamLine was fixed when generating pnext from clipped reads.  Still needs work; pos1 and pos2 need to be recalculated considering clipping.
BBDuk now tracks #contaminant bases as well as #contaminant reads per scaffold for stats.  Additional flag "columns=5" enables this output.
BBDuk stats are now sorted by #bases, not #reads.
BBDuk counting arrays changed from int to long to handle potential overflow.
v33.78
Modified DemuxByName to handle affixes of variable length (though it's less efficient with multiple lengths).
v33.79
Changed the way "pos" and "pnext" are calculated for paired reads to be consistent.  Bug had been noted with soft-clipped reads by Rob Egan.
Changed LOCAL_ALIGN_TIP_LENGTH from 8 to 1.  Previously, soft-clipping would only occur if at least 8 bases would be clipped; not sure why I did that.
Changed the way "tlen" is calculated to compensate for clipping.
v33.80
Changed default decontaminate minratio from2 to 0 (disabling it) because of false negatives.
Changed default decontaminate mincov from 4 to 5 due to a false negative.
Changed default decontaminate kfilter from 63 to 55 to better reflect Spades defaults.
Fixed a bug in filterbycoverage which was outputting contaminant contigs instead of clean contigs.
Added outd (outdirty) flag to FilterByCoverage.
v33.81
Changed decontaminate normalization target from 100 to 50, and minlength from 0 to 500.
Changed decontaminate minc and minp flags from int to float.
v33.82
Changed cross contaminate probability root from 2 to 3 (increasing amount of lower-level contamination).
Fixed a crash bug in sam file generation caused by the change in the way pos was calculated.
v33.83
Added aecc=f, cecc=f, minprob=0.5, depthpercentile=0.8 flags to DecontaminateByNormalization.  Defaults are as listed.
Dropped mindepth to 3 and maxdepth to target; target default changed to 20.
Changed the way mindepth is handled in normalization; now it is based on the depth of the higher read.
v33.84
Added BBNorm prebits flag for setting prefilter cell size (default 2).
Added Decontaminate filterbits and prefilterbits flags, default 32 and 4.  4 was chosen because MDA data has high error kmer counts.
v33.85
Fixed parsing of decontaminate minc and minp (parsed as ints; should have been floats)
Changed default minc to 3.5.
Change default ratio to 1.2.
v33.86
Changed decontaminate default dp to 0.75.
Changed decontaminate default prebits to 2.
Changed decontaminate default minr (min reads) to 20.  Some tiny (~500bp) low-coverage contigs were getting through.
Changed decontaminate mindepth to 2.
Decontaminate results now prints extra columns for read counts and pre-norm coverage.
v33.87
Added "covminscaf" flag to BBMap and Pileup, to supress output of really short contigs.  Default 0.
Changed CrossContaminate coverage distribution from cubic to geometric.
v33.88
Shuffle removing reads caused incredible slowness; it should have set reads to null.  Fixed.
v33.89
Added HashArrayA, HashForestA, KmerNodeA and updated AbstractKmerTable to allow sets of values per kmer.
Refactored all AbstractKmerTable subclasses.
Added scaffold length tracking to BBDuk (for RPKM).
Added RPKM output to BBDuk (enable with "rpkm" flag).
BBDuk now unloads kmers after finishing processing reads.
v33.90
BBDuk counter arrays are now local per-thread, to prevent cache-thrashing.
Added IntList.toString()
Created Seal class, based on BBDuk with values stored in arrays.
Adjusted auto skip settings of BBDuk (increased size threshold for longer skips).
Added BBDuk skip flag (controls minskip and maxskip).
Fixed a bug in DemuxByName/DecontaminateByNormalization/CrossContaminate: attempt to read directories as files.
v33.91
Fixed a bug in BBDuk related to clearing data too early.  Noted by Brian Foster.
v33.92
Added per-reference-file stats counting to BBDuk/Seal, and "refstats" flag.
Added returnList(boolean) to ConcurrentReadStreamInterface.
Removed an extra listen() call from ConcurrentReadInputStreamD.
Documented "addname" flag for stats.sh.
Implemented restrictleft and restrictright for BBDuk2.
Added "nzo" flag for BBDuk/Seal.
Added sdriscoll's reformatted shellscript help for BBDuk and BBMap.  Thanks!
Added more documentation to bbmap.sh (usequality flag).
Added maq (minaveragequality) flag to BBMap, at request of sdriscoll.
Added rename flag to BBDuk/Seal - renames reads based on what sequences they matched.
Added userefnames flag BBDuk/Seal - the names of reference files are used, rather than scaffold IDs.

v33.93
maxindel flag now allows KMG suffix.
Added "speed" flag to BBDuk/Seal.
Added read processing time to BBDuk/Seal output.
BBDuk "fbm" (findbestmatch) mode is now much faster, using variable rather than fixed-length counters.
Fixed BBDuk2 not working when using the "ref" flag rather than "filterref".
Changed AbstractKmerTable subclass names to *1D and *2D.
Made KmerNode a superclass of KmerNode1D and KmerNode2D and eliminated redundant methods.
Eliminated 2D version of HashForest; it now works with 1D and 2D nodes.
Made HashArray a superclass of HashArray1D and HashArray2D.
Created HashArrayHybrid.
Added slow debugging methods to AbstractKmerTable classes, to verify that values were present after being added.
Fixed bug in KmerNode1D; was never changing its value on 'set'.  Probably only affected Seal.  Seal 1D now appears to produce identical output for prealloc and non-prealloc.
Finished debugging KmerNode2D, KmerForest, HashArray2D, HashArrayHybrid, and Seal.
Added "fbm" and "fum" to Seal.
Seal now defaults to 7 ways.
Adjusted Seal's memory preallocation.
Added -Xms flag to BBMergeGapped BBNorm shellscripts.
v33.94
Added -Xms flag to BBDuk and Seal.
Added qskip flag to BBDuk and Seal (for skipping query kmers).
v33.95
Seal now defaults to HashArrayHybrid rather than HashArrayArray2D
v33.96
Fixed a slowdown in Seal and BBDuk caused by sorting list of ID hits.
v33.97
Wrote driver.CorrelateIdentity and matrixtocolumns.sh for identity correlations between 16S and V4.
Wrote jgi.IdentityMatrix and idmatrix.sh for all-to-all alignment.
Added BandedAligner.alignQuadruple() to check all orientations.
BandedAligner now does not clear the full arrays, only the used portion, which can vary depending on read length.
v33.98
No change - build failure.
v33.99
Changed BandedAligner.PenalizeOffCenter().  Indels were getting double-penalized when they led to length mismatches between query and ref.
Added AlignDouble(), but it looks like AlignQuadruple is the only viable method for calculating full identity when the sequences do not start or stop at the same place.
Added test method to ReadStats to ensure the files are safe to write (ReadStats.testFiles()).
Fixed a bug bqhist output giving read 1 and read 2 same values. Noted by Shoudan/Bryce
Fixed a bug in BBDuk initialization when no kmer input supplied. Noted by Bill A.
Fixed a bug in BBDuk/Seal giving a spurious warning.
Detected race condition in ByteFile2 triggered by closing early.  Not very important.
Added jni path flags to BBDuk shellscript command line.
Wrote FindPrimers and msa.sh to locate primer sites.  Uses MultiStateAligner; outputs in sam format.
Wrote CutPrimers and cutprimers.sh to cut regions flanked by mapped primer locations from sequences, e.g. V4.

TODO: Plot correlation of V4 and 16s.
TODO: Add length into edges of Dedupe output. (Ted)
TODO: Benchmark Seal.  Speed seems inconsistent.
TODO: Locking version of Seal.
TODO: HashArray resize - grow fast up to a limit, then resize to exactly the max allowable.
TODO: Alicia BBMap PacBio slowdown (try an older version...)
TODO: BBMerge rename mode with insert sizes.
TODO: Dump info about Seal kmer copy histogram.
TODO: rpkm for pileup / BBMap.
TODO: Dedupe crash bug. (Kurt)
TODO: CallPeaks minwidth should be a subsumption threshold, not creation threshold.
TODO: CallPeaks should not subsume peaks with valleys in between that are very low.
*TODO: Make TextStreamWriter an abstract superclass.
TODO: BBDuk split mode
TODO: Add option for BBMap to convert U to T. (Asaf Levy)
TODO: Add dedupe support for graphing containments and matches.
TODO: Log normalization.
TODO: Prefilterpasses (prepasses)
TODO: Test forcing msa.scoreNoIndels to always run bidirectionally.
TODO: Message for BBNorm indicating pairing (this is nontrivial)
TODO: Average quality for pileup.sh
TODO: Fix ChromArrayMaker which may skip every other scaffold (for now I have reverted to old, correct version). ***Possibly fixed by disabling interleaving; TODO: Test.
TODO: Consider changing ConcurrentGenericReadInputStream to put read/base statistics into incrementGenerated(), or at least in a function.
TODO: BBSplit produces alignments to the wrong reference in the output for a specific reference. (Shoudan)
TODO: Change the way Ns are handled in cigar strings, both input and output.
TODO: Add #clipped reads/bases to BBMap output.
TODO: Add method for counting number of clipped bases in a read and unclipped length.
TODO: Orientation statistics for BBMap ihist.
TODO: Clarify documentation of 'reads' flag to note that it means reads OR pairs.
TODO: bs flag does not work with BBWrap (Shoudan).
TODO: Fasta input tries to sometimes keep reading from the file when a limited number of reads is specified.  Gives error message but output is fine.
TODO: 'saa' flag sometimes does not work (Shoudan).
TODO: Kmer transition probabilities for binning.
TODO: One coverage file per scaffold; abort if over X scaffolds. (Andrew Tritt)
TODO: Enable JNI by default for BBMap and Dedupe on Genepool.
TODO: Disable cigar string generation when dumping coverage only (?).  This will disable stats, though.
TODO: Pipethread spawned when decompressing from standard in with an external process.
TODO: FileFormat should test interleaving and quality individually on files rather than relying on a static field.
TODO: Refstats (BBSplit) still reports inflated rates for pairs that don't map to the same reference.  This behavior is difficult to change because it is conflated with BBSPlit's output streams.


v32.
Revised all shellscripts to better detect memory in Linux.  This should massively increase reliability and ease of use.
Added append flag.  Allows appending to output files instead of overwriting.
Append flag now should work with BBWrap, with sam files, and with gzipped files.
All statistics are now stored in longs, rather than ints.
Added statistics tracking of # bases as well as # reads.  Updated human-readable output to show 4 columns.
Split bbmerge into gapped (split kmer) and ungapped (overlap only) versions.  bbmerge.sh calls the ungapped version.
Added "qahist" to bbmap - match/sub/ins/del histogram by quality score.
Fixed "pairlen" flag; it was only being used if greater than the default. (Noted by Harald on seqanswers)
Added insert size median and standard deviation to output stats.  The 'ihist=' flag must be set to enable this, otherwise the data won't be tracked. (Requested by Harald on seqanswers)
Fixed bug in which non-ACGTN IUPAC symbols were not being converted to N. (Noted by Leanne on seqanswers)
Changed shellscripts from DOS to Unix EOL encoding.
Added support for "-h" and "--help" in shellscripts (before it was just in java files).
Created Dedupe2 - faster, and supports 1-cluster-per-file output.
Created Dedupe3 - supports more than 2 affix tables.  Uses slightly more memory.
BBMap now generates "sort" shellscripts even if the output is in bam format.
pileup.sh now prints a coverage summary to standard out.
Added 'split' flag to BBMask.
Fixed bug in randomreads allowing paired reads to come from 'nearby' scaffolds.
Documented randomreads.sh.
Added gaussian insert size distribution to randomreads.
Fixed a bug in calcmem.sh that prevented requesting memory that Linux considered 'cached'.
TODO: Penalize score of sites with errors near read tips, and long deletions.
Added "Median_fold" column to pileup.  You need to set 'bitset=
Changed default quality-filtering mode to average probability rather than average quality score.
Default number of threads now takes the environment variable NSLOTS into consideration.  However, because Mendel nodes have hpyerthreading enabled, if NSLOTS>8 and (# processors)==NSLOTS*2, then #processors will be used instead.  So it is still recommended that you set threads manually if you don't have exclusive access to a node.
Fixed bbmerge, which was crashing on fasta input.
Fixed gaussian insert size distribution in randomreads (it was causing a crash).
Enabled unpigz support in Windows (decompression only).
TODO:  BBNorm needs in1/in2/out1/out2 support.
Added mingc and maxgc to reformat.
Added 'passes' flag to BBQC and reduced default passes to 1 if normalization is disabled.
Swapped FileFormat's method signature "allowFileRead" and "allowSubprocess" parms for some functions, as they were inconsistent.  This may have unknown effects.
TODO: unclear if fasta files are currently checked for interleaving.  Method added to "FASTQ".
TODO: FileFormat should perhaps test for quality format and interleaving.
Fixed reversed variables in "machineout" stats for %mapped and %unambiguous.  Found by Michael Barton.
Added "testformat.sh".
Fixed dedupe "csf" output to work even when no other outputs specified.
Fixed dedupe erroneous assumption that "bandwidth" had not been custom-specified.
Changed MakeLengthHistogram (readlength.sh) default behavior to place reads in lower bins rather than closest bins.  Toggle with "round" flag.
Added "repair" flag to SplitPairsAndSingles.  Created "repair.sh".
Fixed a bug in which tabs were not allowed in fasta headers.
Improved BBMerge: default minqo 7->8, made margin a parameter, added 'strict' macro that reduces false positive rate.
Added "samestrand" flag to RandomReads.
Fixed a dedupe bug with "pto" and paired reads; read2 was not getting a UnitID.
Fixed a bug in which the BBMap stats for insertion rate was sometimes higher than the true value.
Fixed bugs in BBMerge; increased speed slightly.
Created grademerge.sh to grade merged reads.
Added 'variance' flag to randomreads; used to make qualities less uniform between reads.
BBDuk now has overwrite=true by default.
calcmem.sh now sets -Xmx and -Xms from each other if only one was specified.

Fixed bug with "ambig=all" and "stoptag" flags being used together.  Found by WhatSoEver (seqanswers).
Added 'findbestmatch'/'fbm' flag to BBDuk; reports the reference sequence sharing the greatest number of kmers with the read.
Shellscripts no longer try to calculate memory before displaying help (noted by Kjiersten Fagnan).
-ea and -da are now valid parameters for all shellscripts.
Improved documentation of Dedupe.
Added "loose" and "vloose" modes to BBMerge.
Added novel-kmer-filtering to BBMerge - bans merged reads that create a novel kmer.  Does not seem to help.
Added entropy-detection to BBMerge - minimum allowed overlap is determined by entropy rather than a constant.  Moderate improvement.
Fixed bug causing "repair.sh" script to not work.  Noted by SES (seqanswers).
Added "fast" mode to BBMerge.
Fixed a rounding problem in RandomReads that caused gaussian distribution to have 2x frequency of intended reads at exactly insert size of double read length.
Added exponential decay insert size distribution to RandomReads, for use in LMP libraries.
TODO: Track different paired read orientation rates (innie, outie, same direction, etc) with BBMap.
Added sssr (secondarysitescoreratio) and ssao (secondarysiteasambiguousonly) flags.  Response to WhatSoEver (seqanswers).
Ambiguously-mapped reads that print a primary site now print a minimum of 1 secondary site, and all sites with the same score as the top secondary site.
Improved error message for paired reads with unequal number of read 1 vs read 2.  Response to Salvatore (seqanswers).
Updated bbcountunique.sh help message.
Changed AddAdapters default to "arc=f" (no reverse-complement adapters).  Added "addpaired" flag (adds adapter to same location of both reads).
Added BBDuk/BBDuk2 "tbo" (trimbyoverlap) flag.  Vastly reduces false-negatives with no increase in false-positives.
Adding "fragadapter" flag to RandomReads.  Also added ability to handle multiple different adapters for both read 1 and read 2.  Adapters are added to paired reads with insert size shorter than read length.
Added "ordered" flag to BBDuk/BBDuk2.
Added "tpe" (trimpairsevenly) flag to BBDuk/BBDuk2.  This works in conjunction with kmer-trimming to the right.  Slightly decreases false negatives and doubles false positives.
Updated rqcfilter and bbqc with 'tbo' and 'tpe' flags.
TODO: Migrate RQCFilter to BBDuk2.
Improved addadapters to better handle reads annotated by renamereads.
BBMap's fillLimited routine is now affected by 'sssr' flag, if secondary sites are enabled.  This will make things slightly slower when secondary sites are enabled, if sssr uses a low value (default is 0.95).
statswrapper now allows comma-delimited files.
Added standard deviation to BBMerge (requested by Bryce F).
Added "tbo" (trimbyoverlap) flag to BBMerge, as an alternative to joining.
Updated help for 'ambig' in bbmap.sh to remove the obsolete information that 'ambig=all' did not support sam output.
Updated BBMapSkimmer and its shellscript to default to 'ambig=all', which is its intended mode.
BBDuk no longer defaults to "out=stdout.fq" because that was incredibly annoying.  Now it defaults to "out=null".
Changed BBDuk default mink from 4 to 6.
Changed BBDuk, Reformat, SplitPairsAndSingles default trimq from 4 to 6.
Added "ftr"/"ftl" flags to BBDuk.
Added "bbmapskimmer" to the list of options parsed by BBWrap.  (Noted by JJ Chai)
Corrected documentation of idtag and stoptag - both default to false, not true. (Noted by JJ Chai)
Added "mappedonly" flag to reformat. (Requested by Kristen T)
Added "rmn" (requirematchingnames) flag to Dedupe.  Requested by Alex Copeland.
Added ehist, indelhist, idhist, gchist, lhist flags to BBMap, BBDuk, and Reformat.
Added removesmartbell.sh wrapper for pacbio.RemoveAdapters2.
Fixed instance in KmerCoverage where input stream was being started twice.  Noted by Alicia Clum.
Added "ngn" (NumberGraphNodes) flag to dedupe; default true.  Allows toggling of labelling graph nodes with read number or read name.
"slow" flag now disables a heuristic that skipped mapping reads containing only kmers that are highly overrepresented in the reference.  Problem noted by Shoudan Liang.
Added MergeBarcodes and mergebarcodes.sh
Identity is now calculated neutrally by default.
Added "qin" and "qout" documentation to bbnorm shellscripts. Noted by muol (seqanswers).
Changed qhist to ouput additional columns - both linear averages and logrithmic averages.
Added mode to BBMerge output.
Added mode, min, max, median, and standard deviation to ReadLength output.  The mode and std dev are affected by bin size, so will only be exactly correct when bin size is 1.
Added "nzo" (nonzeroonly) flag to ReadLength.
Created "A_Sample", a template for programs that input reads, perform some function, and output reads.
BBNorm now works correctly with dual input and output files.  Noted by Olaf (seqanswers).
Added mode to BBMap insert size statistics.
Added CorrelateBarcodes and filterbarcodes.sh, for analyzing and filtering reads by barcode quality.
Added "aqhist" (average quality histogram) to ReadStats - can be used by BBMap, BBDuk, Reformat.


v31.
TODO:  Change pipethreads to redirects (where possible), and hash pipethreads by process, not by filename.
TODO:  Improve scoring function by using gembal distribution and/or accounting for read length.
TextStreamWriter was improperly testing for output format 'other'.  Noted by Brian Foster.
Fixed bug for read stream 2 in RTextOutputStream3.  Found by Brian Foster.
Fixed bug in MateReadsMT creating an unwanted read stream 2.  Found by Brian Foster.
TrimRead.testOptimal() mode added, and made default when quality trimming is performed; old mode can be used with 'otf=f' flag.
Fixed a couple cases where output file format was set to "ordered" even though the process was singlethreaded; this had caused an out-of-memory crash noted by Bill A.
Changed shellscripts of MapPacBio classes to remove "interleaved=false" term.
Reduced Shared.READ_BUFFER_LENGTH from 500 to 200 and Shared.READ_BUFFER_MAX_DATA from 1m to 500k, to reduce ram usage of buffers.
Noticed small bug in trimming; somehow a read had a 'T' with quality 0, which triggered assertion error.  I disabled the assertion but I'm not sure how it happened.
Fixed bug in which pigz was not used to decompress fasta files.
All program message information now defaults to stderr.
Added "ignorebadquality" (ibq) flag for reads with out-of-range quality.
TODO: mask by information content
Added "mtl"/"mintrimlength" flag (default 60).  Reads will not be trimmed shorter than that.
Made 'tuc' (to uppercase) default to true for bbmap, to prevent assertion errors.  Reads MUST be uppercase to match reference.
Added new tool, BBMask.
Reads and SamLines can now be created with null bases.  
SamLines to Read is now faster, skipping colorspace check.
Added deprecated 'SOH' symbol support to FastaInputStream.  This will be replaced with a '>'.  Needed to process NCBI's NT database.
Added "sampad" or "sp" flag to BBMask, to allow masking beyond bounds of mapped reads.
TODO: %reads with ins, del, splice
TODO: #bases mapped/unmapped, avg read length mapped/unmapped
Dedupe now tracks and prints scaffolds that were duplicates with "outd=".  (request by Andrew Tritt)
Updated all shellscripts to support the -h and --help flags. (suggested by Westerman)
RAM detection is now skipped if user supplies -Xmx flag, preventing a false warning. (noted by Westerman)
Created AddAdapters.java.  Capable of adding adapter sequence to a fastq file, and grading the trimmed file for correctness.
Removed some debug code from FileFormat causing a crash on "stdin" with no extension.  Noted by Matt Nolan.
Added BBWrap and bbwrap.sh.  Wraps BBMap to allow multiple input/output files without reloading the reference.
Added support for breaking long fastq reads into shorter reads (maxlength and minlength flags).  Requested by James Han.
Added Pileup support for residual bins smaller than binsize. Flag "ksb", "keepshortbins".  Requested by Kurt LaButti.
Fixed support for breaking long reads; was failing on the last read in the set.  Noted by James Han.
Improved accuracy slightly by better detecting when padding is needed.
Improved verbose output from MSA.
Created TranslateSixFrames, first step toward amino acid mapping.
Improved RandomReads ability to simulate PacBio error profile.
Fixed crash when using BBSplit in PacBio mode. (Noted by Esther Singer)
May have improved ability to read relatively-pathed files if "." is not in $PATH. (nope, seems not)
Fixed crash when using "usequality=f" flag with fasta input reads. (Noted by Esther Singer)
Corrected behaviour of minlength with regards to trimming; it was not always working correctly.
Added "bhist" (base composition histogram) flag.

v30.
Disabled compression/decompression subprocesses when total system threads allowed is less than 3.
Fixed assertion error in calcCorrectness in which SiteScores are not necessarily sorted if AMBIGUOUS_RANDOM=true.  Noted by Brian Foster.
Fixed bug in toLocalAlignment with respect to considering XY as insertions, not subs.  
TODO: XY should be standardized as substitutions.
Added scarf input support. Requested by Alex Copeland.
TODO: Allow sam input with interleaved flag.
TODO: Make pigz a module dependency or script load.
Fixed bug with nodisk mode dropping the name of the first scaffold of every 500MB chunk after the first.  Noted by Vasanth Singan.
Overhaul of I/O channel creation.  Sequence files are now initialized with a FileFormat object which contains information about the format, permission to overwrite, etc.
Increased limit of number of index threads in Windows in nodisk mode (since disk fragmentation is no longer relevant).
Renamed Read.list to sites; added Read.topSite() and Read.numSites(); replaced many instances of things like "r.sites!=null && !r.sites.isEmpty()"
Refactored to put Read and all read-streaming I/O classes in 'stream' package.
Moved kmer hashing and indexing classes to kmer package.
Moved Variation, subclasses, and related classes to var package.
Moved FastaToChrom and ChromToFasta to dna package.
Moved pacbio error correction classes to pacbio package.
Removed stack, stats, primes, and other packages; prefixed all unused pacakges with z_.
TODO: Sites failing Data.isSingleScaffold() test should be clipped, not discarded.
RandomReads3 no longer adds /1 and /2 to paired fastq read names by default (can be enabled with 'addpairnum' flag).
Added "inserttag" flag; adds the insert size to sam output.
Fixed insert size histogram anomaly.  There was a blip at insert==(read1.length+read2.length) because the algorithm used to calculate insert size was different for reads that overlap and reads that don't overlap.
Skimmer now defaults to cigar=true.
Added maxindel1 and maxindel2 (or maxindelsum) flags.
Removed OUTER_DIST_MULT2 because it caused assertion errors when different from OUTER_DIST_MULT; changed OUTER_DIST_MULT from 15 to 14.
Added shellscript for skimmer, bbmapskimmer.sh
TODO:  Document above changes to parameters.



v29.
New version since major refactoring.
Added FRACTION_GENOME_TO_EXCLUDE flag (fgte).  Setting this lower increases sensitivity at expense of speed.  Range is 0-1 and default is around 0.03.
Added setFractionGenometoExclude() to Skimmer index.
LMP librares were not being paired correctly.  Now "rcs=f" may be used to ignore orientation when pairing.  Noted by Kurt LaButti.
Allocating memory to alignment score matrices caused uncaught out-of-memory error on low-memory machines, resulting in a hang.  This is now caught and results in an exit.  Noted by Alicia Clum.
GPINT machines are now detected and restricted to 4 threads max.  This helps prevent out-of-memory errors with PacBio mode.
Fixed sam output bug in which an unmapped read would get pnext of 0 rather than 1 when its mate mapped off the beginning of a scaffold.  Noted by Rob Egan.
Added memory test prior to allocating mapping threads.  Thread count will be reduced if there is not enough memory.  This is to address the issue noted by James Han, in which the PacBio versions would crash after running out of memory on low-memory nodes.
TODO:  Detect and prevent low-memory crashes while loading the index by aborting.
Fixed assertion error caused by strictmaxindel mode (noted by James Han).
Added flag "trd" (trimreaddescriptions) which truncates read names at the first whitespace.
Added "usequality/uq" flag to turn on/off usage of quality information when mapping.  Requested by Rob Egan.
Added "keepbadkeys/kbk" flag to prevent discarding of keys due to low quality.  Requested by Rob Egan.
Fixed crash with very long reads and very small kmers due to exceeding length of various kmer array buffers.
Avg Initial Sites and etc no longer printed for read 2 data.
TODO:  Support for selecting long-mate-pair orientation has been requested by Alex C.
Fixed possible bug in read trimming when the entire read was below the quality threshold.
Fixed trim mode bug: "trim=both" was only trimming the right side.  "qtrim" is also now an alias for "trim".
Fixed bug in ConcurrentGenericReadInputStream causing an incorrect assertion error for input in paired files and read sampling.  Found by Alex Copeland.
Added insert size histogram: ihist=<file>
Added "machineout" flag for machine-readable output stats.
TODO: reads_B1_100000x150bp_0S_0I_0D_0U_0N_interleaved.fq.gz (ecoli) has 0% rescued for read1 and 0.7% rescued for read 2.  After swapping r1 and r2, .664% of r2 is rescued and .001% of r1 is rescued.  Why are they not symmetric?
Added 'slow' flag to bbmap for increased accuracy.  Still in progress.
Added MultiStateAligner11ts to MSA minIdToMinRatio().
Changed the way files are tested for permission to write (moved to Tools).
Fixed various places in which version string was parsed as an integer.
Added test for "help" and "version" flags.
Fixed bug in testing for file existence; noted by Bryce Foster.
Fixed issue with scaffold names not being trimmed on whitespace boundaries when 'trd=t'.  Noted by Rob Egan.
Added pigz (parallel gzip) support, at suggestion of Rob Egan.
Improved support for subprocesses and pipethreads; they are now automatically killed when not needed, even if the I/O stream is not finished.  This allows gunzip/unpigz when a file is being partially read.
Added shellscript test for the hostname 'gpint'; in that case, memory will be capped at 4G per process.
Changed the way cris/ros are shut down.  All must now go through ReadWrite.closeStreams()
TODO:  Force rtis and tsw to go through that too.
TODO:  Add "Job.fname" field.
Made output threads kill processes also.
Modified TrimRead to require minlength parameter.
Fixed a bug with gathering statistics in BBMapPacBioSkimmer (found by Matt Scholz).
Fixed a bug in which reads with match string containing X/Y were not eligible to be semiperfect (Found by Brian Foster).
Fixed a bug related to improving the prior fix; I had inverted an == operator (Found by Brian Foster).
Added SiteScore.fixXY(), a fast method to fix reads that go out-of-bounds during alignment.  Unfinished; score needs to be altered as a result.
Added "pairsonly" or "po" flag.  Enabling it will treat unpaired reads as unmapped, so they will be sent to 'outu' instead of 'outm'.  Suggested by James Han and Alex Copeland.
Added shellscript support for java -Xmx flag (Suggested by James Han).
Changed behavior: with 'quickmatch' enabled secondary sites will now get cigar strings (mostly, not all of them).
"fast" flag now enables quickmatch (50% speedup in e.coli with low-identity reads).  Very minor effect on accuracy.
Fixed bug with overflowing gref due GREFLIMIT2_CUSHION padding.  Found by Alicia Clum.
Fixed bug in which writing the index would use pigz rather than native gzip, allowing reads from scaffolds.txt.gz before the (buffered) writing finished.  Rare race condition.  Found by Brian Foster.
Fixed stdout.fa.gz writing uncompressed via ReadStreamWriter.
Added "allowSubprocess" flag to all constructors of TextFile and TextStreamWriter, and made TextFile 'tryAllExtensions' flag the last param.
allowSubprocess currently defaults to true for ByteFiles and ReadInput/Output Streams.
TODO: TextFile and TextStreamWriter (and maybe others?) may ignore ReadWrite.killProcess().
TODO: RTextOutputStream3 - make allowSubprocess a parameter
TODO: Assert that first symbol of reference fasta is '>' to help detect corrupt fastas.
Improved TextStreamWriter, TextFile, and all ReadStream classes usage of ReadWrite's InputStream/OutputStream creation/destruction methods.
All InputStream and OutputStream creation/destruction now has an allowSubprocesses flag.
Added verbose output to all ReadWrite methods.
Fixed bug in which realigned SiteScores were not given a new perfect/semiperfect status.  Noted by Brian Foster and Will Andreopoulos.


v28.
New version because the new I/O system seems to be stable now.
Re-enabled bam input/output (via samtools subprocess).  Lowered shellscript memory from 85% to 84% to provide space for samtools.
Added "-l" to "#!/bin/bash" at top.  This may make it less likely for the environment to be messed up.  Thanks to Alex Boyd for the tip.
Addressed potential bug in start/stop index padding calculation for scaffolds that began or ended with non-ACGT bases.
Made superclass for Index.
Made superclass for BBMap.
Removed around 5000 lines of code as a result of dereplication into superclasses.
Added MultiStateAligner11ts, which uses arrays for affine transform instead of if blocks.  Changing insertions gave a ~5% speedup; subs gave an immeasurably small speedup.
Found bug in calculation of insert penalties during mapping.  Fixing this bug increases speed but decreases accuracy, so it was modified toward a compromise.


v27.
Added command line to sam file header.
Added "msa=" flag.  You can specify which msa to use by entering the classname.
Added initial banded mode.  Specify "bandwidth=X" or "bandwidthratio=X" accelerate alignment.
Cleaned up argument parsing a bit.
Improved nodisk mode; now does not use the disk at all for indexing.  BBSplitter still uses the disk.
Added "fast" flag, which changes some paramters to make mapping go faster, with slightly lower sensitivity.
Improved error handling; corrupt input files should be more likely to crash with an error message and less likely to hang.  Noted by Alex Copeland.
Improved SAM input, particularly coordinates and cigar-string parsing; this should now be correct but requires an indexed reference.  Of course this information is irrelevant for mapping so this parsing is turned off by default for bbmap.
Increased maximum read speed with ByteFile2, by using 2 threads per file.  May be useful in input-speed limited scenarios, as when reading compressed input on a node with many cores.  Also accelerates sam input.
TODO:  Consider moving THREADS to Shared.
Updated match/cigar flag syntax.
Updated shellscript documentation.
Changed ByteFile2 from array lists to arrays; should reduce overhead.
TODO:  Increase speed of sam input.
TODO:  Increase speed of output, for all formats.
TODO:  Finish ReadStreamWriter.addStringList(), which allows formatting to be done in the host.
In progress:  Moving all MapThread fields to abstract class.
MapThread now passes reverse-complemented bases to functions to prevent replication of this array.
Fixed very rare bug when a non-semiperfect site becomes semiperfect after realignment, but subsequently is no longer highest-ranked.
strictmaxindel can now be assigned a number (e.g. stricmaxindel=5).
If a fasta read is broken into pieces, now all pieces will recieve the _# suffix in their name.  Previously, the first piece was exempt.
TODO:  Consider changing SamLine.rname to a String and seq, qual to byte[].
Changed SamLine.seq, qual to byte[].  Now stored in original read order and only reversed for minus strand during I/O.
Added sortscaffolds flag (requested by Vasanth Singan).
Fixed XS tag bug; in some cases read 2 was getting opposite flag (noted by Vasanth Singan).
Fixed bug when reading sam files without qualities (noted by Brian Foster).
Fixed bug where absent cigar strings were printed as "null" instead of "*" as a result of recent changes to sam I/O (noted by Vasanth Singan).
Found error when a read goes off the beginning of a block.  Ref padding seems to be absent, because Ns were replaced by random sequence.  Cause is unknown; cannot replicate.
Fixed Block.getHitList(int, int).
Changed calcAffineScore() to require base array for information when throwing exceptions.
Changed generated bamscript to unload samtools module before loading samtools/0.1.19.
sam file idflag and stopflag are both now faster, particularly for perfect mappings.  But both default to off because they are still slow nonetheless.
Fixed bug in BBIndex in which a site was considered perfect because all bases matched the reference, but some of the bases were N.  Canonically, reads with Ns can never be perfect even if the ref has Ns in the same locations.
Fixed above bug again because it was not fully fixed:  CHECKSITES was allowing a read to be classified as perfect even if it contained an N.
Increased sam read speed by ~2x; 30MB/s to 66MB/s
Increased sam write speed from ~18MB/s to ~32MB/s on my 4-core computer (during mapping), with mapping at peak 42MB/s with out=null.  Standalone (no mapping) sam output seems to run at 51MB/s but it's hard to tell.
Increased fasta write from 118MB/s to 140 MB/s
Increased fastq write from 70MB/s to 100MB/s
Increased fastq read from 120MB/s (I think) to 296MB/s (663 megabytes/sec!) with 2 threads or 166MB/s with 1 thread
Some of these speed increases come from writing byte[] into char[] buffer held in a ThreadLocal, instead of turning them into Strings or appending them byte-by-byte.
All of these speed optimizations caused a few I/O bugs that temporarily affected some users between Oct 1 and Oct 4, 2013.  Sorry!
Flipped XS tag from + to - or vice versa.  I seem to have misinterpreted the Cufflinks documentation (noted by Vasanth Singan).
Fixed bug in which (as a result of speed optimizations) reads outside scaffold boundaries, in sam 1.3 format, were not getting clipped (Noted by Brian Foster).
Changed default behavior of all shellscripts to run with -Xmx4g if maximum memory cannot be detected (typically, because ulimit=infinity).  Was 31.  Unfortunately things will break either way.
Fixed off-by-1 error in sam TLEN calculation; also simplified it to give sign based on leftmost POS and always give a plus and minus even when POS is equal.
Added sam NH tag (when ambig=all).
Disabled sam XM tag because the bowtie documentation and output do not make any sense.
Changed sam MD and NM tags to account for 'N' symbol in cigar strings.
Made sam SM tag score compatible with mapping score.
Fixed bug in SamLine when cigar=f (null pointer when parsing match string). (Found by Vasanth Singan)
Fixed bug in BBMapThread* when local=true and ambiguous=toss (null pointer to read.list). (Found by Alexander Spunde)
Changed synthetic read naming and parsing (parsecustom flag) to use " /1" and " /2" at the end of paired read names. (Requested by Kurt LaButti)
Increased fastq write to 200MB/s (590 megabytes/s)
Increased fasta write to 212MB/s (624 megabytes/s measured by fastq input)
Increased sam write to 167MB/s (492 megabytes/s measured by fastq input)
Increased bread write to 196MB/s (579 megabytes/s measured by fastq input)
bf2 (multithreaded input) is now enabled by default on systems with >4 cores, or in ReformatReads always.
Fixed RTextOutputStream3.finishedSuccessfully() returning false when output was in 2 files.
Changed output streams to unbuffered.  No notable speed increase.
Fixed bug in ByteFile2 in which reads would be recycled when end of file was hit (found by Brian Foster, Bryce Foster, and Kecia Duffy).


v26.
Fixed crash from consecutive newlines in ByteFile.
Made SiteScore clonable/copyable.
Removed @RG line from headers.  It implies that reads should be annotated with addition fields based on the RG line information.
Changed sam flags (at advice of Joel Martin).  Now single-ended reads will never have flags 0x2, 0x40, or 0x80 set.
Added correct insert size average to output stats, in place of old inner distance and mapping length.
Fixed crash when detecting length of SamLines with no cigar string.  (Found by Shayna Stein)
Added flag "keepnames" which keeps the read names unchanged when writing in sam format.  Normally, a trailing "/1", "/2", " 1", or " 2" are stripped off, and if read 2's name differs from read 1's name, read 1's name is used for both.  This is to remain spec-compliant with the sam format.  However, in some cases (such as grading synthetic reads tagged with the correct mapping location) it is useful to retain the original name of each read.
Added local alignment option, "local".  Translates global alignments into a local alignments using the same affine transform (and soft-clips ends).
Changed killbadpairs default to false.  Now by default improperly paired reads are allowed.
Merged TranslateColorspaceRead versions into a single class.
Added interleaved input and output for bread format.  May be useful for error correction pipeline.
TODO:  Mode where reads are mapped to multiple scaffolds, but are mapped at most one time per scaffold.  I.e., remove all but top site per scaffold (and forbid self-mapping).
Fixed yet another instance of negative coordinates appearing in an unmapped read, which the new version of samtools can't handle.
Fixed bug in counting ambiguous reads; was improperly including in statistics reads that were ambiguous but had a score lower than minratio.
Fixed rare crash found related to realignment of reads with ambiguous mappings (found by Rob Egan).
Unified many of the differences between the MapThread variants, and added a new self-checking function (checkTopSite) to ensure a Read is self-consistent.
Added some bitflag fetch functions to SamLine and fixed 'pairedOnSameChrom()' which was not handling the '=' symbol.
TODO: Make GENERATE_BASE_SCORES_FROM_QUALITY a parameter, default false in BBMapPacBio and true elsewhere. (I verified this should work fine)
TODO: Make GENERATE_KEY_SCORES_FROM_QUALITY a parameter, default true (probably even in BBMapPacBio). (I verified this should work fine)
Updated LongM (merged with LongM from Dedupe).
Fixed bug in SamLine in which clipped leading indels were not considered, causing potential negative coordinates.  (Found by Brian Foster)
TODO: Match strings like NNNNNNDDDDDNNNNNmmmmmmmmmmmmmmmmm...mmmmmmm should never exist in the first place.  Why did that happen?
Added "strictmaxindel" flag (default: strictmaxindel=f).  Attempts to kill mappings in which there is a single indel event longer than the "maxindel" setting.  Requested by James Han.
TODO: Ensure strictmaxindel works in all situations, including rescued paired ends and recursively regenerated padded match strings.
TODO: Redo msa to be strictly subtractive.  Start with score=100*bases, then use e.g. 0 for match, -1 for del, -370 for sub, -100 for N, etc.  No need for negative values.
Changed TIMEBITS in MultiStateAligner9PacBio from 10 to 9 to address a score underflow assertion error found by Alicia Clum.  The underflow occuerd around length 5240; new limit should be around 10480.
TODO: Alicia found an error of exceeding gref bounds.
Fixed race condition in TextStreamWriter.
Improved functionality of splitter.  Now you can index once and map subsequently using "basename" without specifying "ref=" every single time.
"Reads Used" in output now dispays the number of reads used.  Before, for paired reads, it would display the number of pairs (half as many).
Added bases used to reads used at Kurt's request.
Improved bam script generation.  Now correctly sets samtools memory based on detected memory, and warns user that crashes may be memory-related.
Fixed an obsolete assertion in SamLine found by Alicia.
Added XS tag option ("xstag=t") for Cufflinks; the need for this was noted by requested by Vasanth Singan.
Added 'N' cigar operation for deletions longer than X bases (intronlen=X).  Also needed by Cufflinks.
Secondary alignments now get "*" for bases and qualities, as recommended by the SAM spec.  This saves space, but may cause problems when converting sam into other formats.
Fixed bug that caused interleaved=true to override in2.  Now if you set in and in2, interleaved input will be disabled.  (noted by Andrew Tritt).
Fixed some low-level bugs in I/O streams.  When shutting down streams I was waiting until !Thread.isAlive() rather than Thread.getState()==Thread.State.TERMINATED, which caused a race condition (since a thread is not alive before it starts execution).
Added debugging file with random name written to /ref/ directory.  This should help debugging if somewhere deep in a pipeline multiple processes try to index at the same location simultaneously.  Suggested by Bryce Foster.
Fixed log file generation causing a crash if the /ref/ directory did not exist, found by Vasanth Singan.  Also logging is now disabled by default but enabled if you set "log=t".
Input sequence data will now translate '.' and '-' to 'N' automatically, as some fasta databases appear to use '.' instead of 'N'.  (Thanks to Kecia Duffy and James Han)
Added capability to convert lowercase reads to upper case (crash on lowercase noted by Vasanth Singan).


v25.
Increased BBMapPacBio max read length to 6000, and BBMapPacBioSkimmer to 4000.
Fixed bugs in padding calculations during match string generation.
Improved some assertion error output.
Added flag "maxsites" for max alignments to print.
Added match field to sitescore.
Made untrim() affect sitescores as well.
Decreased read array buffer from 500 to 20 in MapPacBio.
TODO:  stitcher for super long reads.
TODO:  wrapper for split reference mapping and merging.
Improved fillAndScoreLimited to return additional information.
Added flag "secondary" to print secondary alignments.  Does not yet ensure that all secondary alignments will get cigar strings, but most do.
Added flag "quickmatch" to generate match strings for SiteScores during slow align.  Speeds up the overall process somewhat (at least on my PC; have not tested it on cluster).
Improved pruning during slow align by dynamically increasing msa limit.
Addressed a bug in which reads sometimes have additional sites aligned to the same coordinates as the primary site.  The bug can still occur (typically during match generation or as a result of padding), but is detected and corrected during runtime.
Tracked down and fixed a bug relating to negative coordinates in sam output for unmapped reads paired with reads mapped off the beginning of a scaffold, with help from Rob Egan.
Disabled frowny-face warning message which had caused some confusion.
TODO:  Add verification of match strings on site scores.
Made superclass for MSA.  This will allow merging of redundant code over the various BBMap versions.
Fixed a crash-hang out-of-memory error caused by initialization order.  Now crashes cleanly and terminates.  Found by James Han.
Fixed bug in output related to detecting cigar string length under sam 1.4 specification (found by Rob Egan).
Added flag "killbadpairs"/"kbp".
Added flag "fakequality" for fasta.
Permanently fixed bugs related to unexpected short match strings caused by error messages.
Increased speed of dynamic program phase when dealing with lots of Ns.
TODO:  In-line generation of short match string when printing a read, rather than mutating the read. (mutation is now temporary)
Added flag, "stoptag".  Allows generation of SAM tag YS:i:<read stop location>
Added flag, "idtag".  Allows generation of SAM tag YI:f:<percent identity>

v24.
Fixed bug that slightly reduced accuracy for reads with exactly 1 mismatch.  They were always skipping slow align, sometimes preventing ambiguous reads from being detected.
Increased speed of MakeRocCurve (for automatic grading of sam files from synthetic reads).  Had used 1 pass per quality level; now it uses only 1 pass total.
Increased accuracy of processing reads and contigs with ambiguous bases (in mapping phase).
Adjusted clearzones to use gradient functions and asymptotes rather than step functions.  Reduces false positives and increases true positives, especially near the old step cutoffs.
Fixed trimSitesBelowCutoff assertion that failed for paired reads.
Added single scaffold toggle to RandomReads.  Default 'singlescaffold=true'; forces reads to come from a single scaffold).  This can cause non-termination if no scaffolds are long enough, and may bias against shorter scaffolds.
Added min scaffold overlap to RandomReads.  Default 'overlap=1'; forces reads to overlap a scaffold at least this much.  This can cause non-termination if no scaffolds are long enough, and may bias against shorter scaffolds.
Fixed setPerfect().  Previously, reads with 'N' overlapping 'N' in the reference could be considered perfect matches, but no reads containing 'N' should ever be considered a perfect mapping to anything.
Formalized definition of semiperfect to require read having no ambiguous bases, and fixed "isSemiperfect()" function accordingly.
Shortened and clarified executable names.
Fixed soft-clipped read start position calculation (mainly relevant to grading).
Prevented reads from being double-counted when grading, when a program gives multiple primary alignments for a read.
Fixed a bug in splitter initialization.
Added "ambiguous2".  Reads that map to multiple references can now be written to distinct files (prefixed by "AMBIGUOUS_") or thrown away, independantly of whether they are ambiguous in the normal sense (which includes ambiguous within a single reference).
Added statistics tracking per reference and per scaffold.  Enable with "scafstats=<file>" or "refstats=<file>".
"ambiguous" may now be shortened to "ambig" on the command line.
"true" and "false" may now be shortened to t, 1, or f, 0.  If omitted entirely, "true" is assumed; e.g. "overwrite" is equivalent to "overwrite=true".
Added stderr as a vaild output destination specified from the command line.
BBSplitter now has a flag, "mapmode"; can be set to normal, accurate, pacbio, or pacbioskimmer.
Fixed issue where stuff was being written to stdout instead of stderr and ended up in SAM files (found by Brian Foster).
TODO: Add secondary alignments.
TODO: Unlimited length reads.
TODO: Protein mapping.
TODO: Soft clipping in both bbmap and GradeSamFile.  Should universally adjust coords by soft-clip amount when reported in SAM format.
Fixed assertion error concerning reads containing Ns marked as perfect, when aligned to reference Ns (found by Rob Egan).
Fixed potential null-pointer error in "showprogress" flag.

v23.
Created BBSplitter wrapper for BBMap that allows merging any number references together and splitting the output into different streams.
Added support for ambiguous=random with paired reads (before it was limited to unpaired).
TODO: Iterative anchored alignment for very long reads, with a full master gref.
TODO: untrim=c/m/s/n/r
TODO: mode=vfast/veryfast: k=14 minratio=0.8 minhits=2 maxindel=20
TODO: mode=slow/accurate: BBMapi
TODO: mode=pacbio: BBMapPacBio k=12
TODO: mode=rnaseq
TODO: Put untrim in caclStatistics section
TODO: Test with MEGAN.
Finished new random read generator.  Much faster, and solves coordinate problem with multiple indels.
Improved error message on read parsing failures.
TODO: Insert size histogram
TODO: "outp=", output for reads that mapped paired
TODO: "outs=", output for reads that mapped singly
Corrected assertion in "isSingleScaffold()"
Fixed a rare bug preventing recursive realignment when ambiguous=random (found by Brian Foster)
Added samversion/samv flag.  Set to 1.3 for cigar strings with 'M' or 1.4 for cigar strings with '=' and 'X'.  Default is 1.3.
Added enforcement of thread limit when indexing.
Added internal autodetection of gpint machines.  Set default threadcount for gpints at 2.
Improved ability to map with maxindel=0
Added XM:i:<N> optional SAM flag because some programs seem to demand it.  Like all extra flags, this is omitted if the read is not mapped.  Otherwise, it is set to 1 for unambiguously mapped reads, and 2 or more for ambiguously mapped reads.  The number can range as high as the total number of equal-scoring sites, but this is not guaranteed unless the "ambiguous=random" flag is used.
Fixed bug in autodetection of paired ends, found by Rob Egan.



v22.
Added match histogram support.
Added quality histogram support.
Added interleaving support to random read generator.
Added ability to disable pair rescue ("rescue=false" flag), which can speed things up in some cases.
Disabled dynamic-programming slow alignment phase when no indels are allowed.
Accelerated rescue in perfect and semiperfect mode.
Vastly accelerated paired mapping against references with a very low expected mapping rate.
Fixed crash in rescue caused by reads without quality strings (e.g. paired fasta files). (found by Brian Foster)


v21.
If reference specified is same as already-processed reference, the old index will not be deleted.
Added BBMap memory usage estimator to assembly statistics tool:  java -Xmx120m jgi.AssemblyStats2 <fasta file> k=<kmer size for BBMap>
Added support for multiple output read streams: all reads (set by out=), mapped reads (set by outm=), and unmapped reads (set by outu=).  They can be in different formats and any combination can be used at once.  You can set pair output to secondary files with out2, outm2, and outu2.
Changed definition of "out=".  You can no longer specify split output streams implicitly by using a "#" in the filename; it must be explicit.  the "#" wildcard is still allowed for input streams.
Fixed a bug with sam input not working. (found by Brian Foster)
Added additional interleaved autodetection pattern for reads named "xxxxx 1:xxxx" and "xxxxx 2:xxxx"
Fixed a bug with soft-clipped deletions causing an incorrect cigar length. (found by Brian Foster)
Fixed a bug with parsing of negative numbers in byte arrays.
TODO: Found a new situation in which poly-N reads preferentially map to poly-N reference (probably tip search?)
Fixed a bug in which paired reads occasionally are incorrectly considered non-semiperfect. (found by Brian Foster)
Added more assertion tests for perfection/imperfection status.
Added blacklist support.  This allows selection of output stream based on the name of the scaffold to which a read maps.
Created Blacklist class, allowing creation of blacklists and whitelists.
Added outb (aka outblacklist) and outb2 streams, to output reads that mapped to blacklisted scaffolds. 
Added flag "outputblacklisted=<true/false>" which contols whether blacklisted reads are printed to the "out=" stream.  Default is true.
Added support for streaming references. e.g. "cat ref1.fa ref2.fa | java BBMap ref=stdin.fa"
Updated and reorganized this readme.
Removed a dependency on Java 7 libraries (so that the code runs in Java 6).
Added per-read error rate histogram.  Enable with qhist=<filename>
TODO: generate standard deviation.
Added per-base-position M/S/D/I/N rate tracking.  Enable with mhist=<filename>
Added quality trimming.  Reads may be trimmed prior to mapping, and optionally untrimmed after mapping, so that no data is lost.  Trimmed bases are reported as soft-clipped in this case.
Trimming will extend until at least 2 consecutive bases have a quality greater than trimq (default 5).
Added flags:  trim=<left/right/both/false>, trimq=<5>, untrim=<true/false>
TODO:  Correct insert size in realtime for trim length.
TODO:  Consider adding a TrimRead pointer to reads, rather than using obj.
TODO:  Consider extending match string as 'M' rather than 'C' as long as clipped bases match.
Found and made safe some instances where reads could be trimmed to less than kmer length.
Found and fixed instance where rescue was attempted for length-zero reads.
Fixed an instance where perfect reads were not marked perfect (while making match string).


v20.1 (not differentiated from v20 since the differences are minor)
Fixed a minor, longstanding bug that prevented minus-strand alignment of rads that only had a single valid key (due to low complexity or low quality).
Increased accuracy of perfectmode and semiperfectmode, by allowing mapping of reads with only one valid key, without loss of speed.  They still don't quite match normal mode since they use fewer keys.
Added detection of and error messages for reads that are too long to map. 
Improved shell script usage information.


v20.
Made all MapThreads subclasses of MapThread, eliminating duplicate code.
Any exception thrown by a MapThread will now be detected, allowing the process to complete normally without hanging.
Exceptions (e.g. OutOfMemory) when loading reference genome are now detected, typically causing a crash exit instead of a hang.
Exceptions (e.g. OutOfMemory) when generating index are now detected, causing a crash exit instead of a hang.
Exceptions in output stream (RTextOutputStream) subthreads are now detected, throwing an exception.
Added support for soft clipping.  All reads that go off the ends of scaffolds will be soft-clipped when output to SAM format. (The necessity of this was noted by Rob Egan, as negative scaffold indices can cause software such as samtools to crash)


v19.
Added support for leading FASTA comments (denoted by semicolon).
Fixed potential problem in FASTA read input stream with very long reads.
Recognizes additional FASTA file extensions: .seq, .fna, .ffn, .frn, .fsa, .fas
Disabled gzip subprocesses to circumvent a bug in UGE: Forking can cause a program to be terminated.  Gzip is still supported.
Slightly reduced memory allocation in shellscript.
Ported "Analyze Index" improvement over to all versions (except v5).
Added flags: fastaminread, showprogress
Fixed problem noted by Rob Egan in which paired-end reads containing mostly 'N' could be rescued by aligning to the poly-N section off the end of a contig.
Fixed: Synthetic read headers were being improperly parsed by new FASTQ input stream.
Made a new, faster, more correct version of "isSemiperfect".
Added "semiperfect" test for reads changed during findDeletions.
Identified locations in "scoreNoIndels" where call 'N' == ref 'N' is considered a match.  Does not seem to cause problems.
Noted that SAM flag 0x40 and 0x80 definitions differ from my usage.


v18.
Fastq read input speed doubled.
Fasta read input speed increased 50%.
Increased speed of "Analyze Index" by a factor of 3+ (just for BBMap so far; have not yet ported change over to other versions).
Fixed an array out-of-bounds bug found by Alicia Clum.
Added bam output option (relies on Samtools being installed).
Allows gzip subprocesses, which can sometimes improve gzipping and gunzipping speed over Java's implementation (will be used automatically if gzip is installed).  This can be disabled with with the flags "usegzip=false" and "usegunzip=false".
Started a 32-bit mode which allows 4GB per block instead of 2GB, for a slight memory savings (not finished yet).
Added nondeterministic random read sampling option.
Added flags: minscaf, startpad, stoppad, samplerate, sampleseed, kfilter, usegzip, usegunzip


v17.
Changed the way error rate statistics are displayed.  All now use match string length as denominator.
Identified error in random read generator regarding multiple insertions.  It will be hard to fix but does not matter much.
Found out-of-bounds error when filling gref.  Fixed (but maybe not everywhere...).
Added random mapping for ambiguous reads.
Changed index from 2d array to single array (saves a lot of memory).
Increased speed by ~10%.
Improved index generation and loading speed (typically more than doubled).
Changed chrom format to gzipped.
Added "nodisk" flag; index is not written to disk.
Fixed a rare out-of-bounds error.
Increased speed of perfect read mapping.
Fixed rare human PAR bug.


v16.  Changes since last version:
Supports unlimited number of unscaffolded contigs.
Supports piping in and out.  Set "out=stdout.sam" and "in=stdin.fq" to pipe in a fastq file and pipe out a sam file (other extensions are also supported).
Ambiguously named files (without proper extensions) will be autodetected as fasta or fastq (though I suggest not relying on that).
Added additional flags (described in parameters section): minapproxhits, padding, tipsearch, maxindel.
minapproxhits has a huge impact on speed.  Going from 1 to 2 will typically at least double the speed (on a large genome) at some cost to accuracy.


v15.  Changes since last version:
Contig names are retained for output.
SAM header @SQ tags fixed.
SAM header @PG tag added.
An out-of-bounds error was fixed.
An error related to short match strings was found and possibly handled.
All versions now give full statistics related to %matches, %substitutions, %deletions, and %insertions (unless match string generation is disabled).
Increased speed and accuracy for tiny (<20MB) genomes.
Added dynamic detection of scaffold sizes to better partition index, reducing memory in some cases.
Added command-line specification of kmer length.
Added more command line flags and described them in this readme.
Allowed overwriting of existing indices, for ease of use (only when overwrite=true).  For efficiency you should still only specify "ref=" the first time you map to a particular reference, and just specify the build number subsequently.
