Auspice Version 2.0
===================

What's New
----------

-  A new JSON format for datasets, which condenses metadata and tree data into a single JSON and allows for expanded functionality in v2.
-  A whole host of UI updates to improve Auspice's visualisations, including pie charts, better rendering of dates, and more easily accessible metadata.

Read on for rationale behind this update and a detailed explanation of the changes.

|auspice-v2-gif| *Switching between colorings in Auspice v2*

Backstory
---------

Auspice started as the `nextstrain.org <https://nextstrain.org>`__ visualisation tool, in fact for a long time nextstrain.org *was* Auspice. Slowly, we turned Auspice into a stand-alone tool and started expanding its capacity. This resulted in many v1.x releases (we made it all the way to v1.39.0), and things had a habit of changing a little too frequently, with not enough focus on documentation and communication.

So in a way, this is the first proper Auspice release! There are a few big and visible changes, including using pie charts to represent discrete variables on a map, but a lot of the changes were done to make things easier behind the scenes. There are also a bunch of really cool features such as the ability to define custom servers, change the aesthetics of the Auspice client, display multiple trees and write narratives which were technically part of Auspice v1 but undocumented. This documentation website is now up to date and should allow you to understand how to use Auspice to its full potential (`contributions are always welcome! <https://github.com/nextstrain/auspice/blob/-/DEV_DOCS.md>`__)

In general, we hope that this documentation and the help messages of the various ``auspice`` subcommands are vastly improved. The major changes which Auspice v2 brings are detailed below 👇 If you have any comments please `contact us <https://nextstrain.org/contact>`__ or say hi on `twitter <https://twitter.com/nextstrain>`__.

I don't want to upgrade - how can I continue to use Auspice v1?
---------------------------------------------------------------

While we don't think there is any reason not to upgrade, version 1 will always be available via npm:

.. code:: bash

   npm install --global auspice@version1

Pie charts to represent discrete variables on a map
---------------------------------------------------

We used to use colour blending to represent all the different traits present at a given location. This worked pretty well for continuous traits (e.g. the dates of isolates in each country), but performed poorly for discrete traits (e.g. the different flu clades present in each country). We now use pie charts for discrete traits which give a much nicer summary of the data:

|pie-charts| *Notice the difference? Auspice v1 (left) & v2 (right)*

New dataset JSON format
-----------------------

For about a year we've been seeing the limitations of the "v1" meta + tree JSONs (the dataset files behind Auspice). Additionally the format of the input JSONs to Auspice changed a bit throughout the lifetime of Auspice v1 - fine when it's an internal tool, but not so great when other people start to use it!

We've settled on a new single "v2" JSON, containing pretty similar information to the v1 JSONs but in a format which allows us to expand the functionality of Auspice.

   Don't panic - ``auspice view`` and `nextstrain.org <https://nextstrain.org>`__ will automatically convert the "v1" JSONs into the new format for you so all of the old datasets will continue to work with Auspice v2.

+-----------------+--------------------------------------------------------------------------------------------------+------------------+----------------------------------------------------+
| file            | schema                                                                                           | auspice versions | description                                        |
+=================+==================================================================================================+==================+====================================================+
| "tree" JSON     | `Link <https://github.com/nextstrain/augur/blob/master/augur/data/schema-export-v1-tree.json>`__ | v1, v2           | Decorated phylogenetic tree                        |
+-----------------+--------------------------------------------------------------------------------------------------+------------------+----------------------------------------------------+
| "meta" JSON     | `Link <https://github.com/nextstrain/augur/blob/master/augur/data/schema-export-v1-meta.json>`__ | v1, v2           | The "metadata" associated with a phylogenetic tree |
+-----------------+--------------------------------------------------------------------------------------------------+------------------+----------------------------------------------------+
| "v2" JSON       | `Link <https://github.com/nextstrain/augur/blob/master/augur/data/schema-export-v2.json>`__      | v2               | The single input format required for Auspice v2    |
+-----------------+--------------------------------------------------------------------------------------------------+------------------+----------------------------------------------------+

If you use Augur to construct your JSONs, then the next version of Augur (v6) will give you the option to produce "v1" or "v2" JSONs. If you have specific queries, the JSON schemas (above) should help you out, but here is a list of the main changes (with links to PRs where relevent, if you're *really* keen!):

Strings parsed unchanged
^^^^^^^^^^^^^^^^^^^^^^^^

For historical reasons, we used to "prettify" certain strings in Auspice - e.g. "usa" became "USA", "new_zealand" became "New Zealand". We dug ourselves into a bit of a hole here, as you can imagine the list of exceptions kept growing and growing. Auspice v2 now generally parses strings as they appear in the JSON, allowing you full control over how things appear. (For backwards compatability, we've kept the "prettifying" function in the v1-v2 conversion logic so the look of v1 JSONs shouldn't change.)

Both metadata and tree data in a single JSON
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The most visible change is probably the reduction of two JSONs (``_meta.json`` + ``_tree.json``) into a single JSON. This single JSON has three required (and no optional) keys:

.. code:: json

   {
     "version": "schema version",
     "meta": {},
     "tree": {}
   }

Gene / Genome definitions are now in GFF format.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

v1 Auspice JSONs used 0-based starts for the gene positions and ``1``/``-1`` for which strand the gene was on. We now use one-based start & end coordinates, following GFF format, and a "+" or "-" to denote the strand. These definitions are now under the ``genome_annotations`` property (formerly ``annotations``). See `PR #770 <https://github.com/nextstrain/auspice/pull/770>`__.

Changes to how node data is stored
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Each tree node (internal & terminal) can now contain the following properties \* ``name`` (required) -- formerly this was ``strain`` \* ``node_attrs`` -- attributes associated with the node (sequence, date, location) as opposed to changes from one node to another. \* Node attributes can now be objects and contain confidence information if available. \* A ``hidden`` node attribute can control Auspice's display of the node \* Author information is now contained under the ``author`` key, and the ``author_info`` dictionary is no longer present in the JSON. \* ``branch_attrs`` -- attributes associated with the branch from the parent node to this node, such as branch lengths, mutations, support values \* ``branch_attrs.mutations`` -- both AA & nucleotide mutations are now defined in the same object. \* ``children`` (unchanged)

Colorings, geographic resolutions, and defaults
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The ``colorings`` property (formerly ``color_options``) is now an array of objects, the properties of which are easier to understand. This guarantees the ordering appears in Auspice as you define it in the JSON. See `PR #748 <https://github.com/nextstrain/auspice/pull/748>`__.

The ``geo_resolutions`` property (formerly ``geo``) is similarly an array of objects.

The ``display_defaults`` property (formerly ``defaults``) now contains keys which are snake_case instead of camelCase.

Multiple maintainers
^^^^^^^^^^^^^^^^^^^^

The maintainer, displayed in the Auspice footer, was previously limited to a single string value and corresponding URL. We now allow multiple maintainers, each with their own (optional) URL.

Continous, Categorical, Ordinal, and Boolean Color Scales
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Traits with the "boolean" colour type which will use a pre-defined yellow & blue colour scale. Currently "continous" and "categorical" scales both use the same colour scale. Note that "discrete" types from v1 JSONs will be interpreted as "categorical".

More information in tree info boxes
-----------------------------------

We've made more things available when you hover over the tree or click on a tree tip. For instance, v1 would use the aa-nt toggle in the entropy panel to decide which mutations to display, and it was frustrating to have to scroll down to switch the toggle just to see what nucleotide mutations were on a branch! We now show both.

|more-tree-info| *Auspice v1 (left) & v2 (right). v2 shows more information on both tree hover (upper panel) & when clicking on tips (lower panel).*

Display of second trees
-----------------------

Auspice has had the ability to display two trees side-by-side for a while now (and `finally documented <../advanced-functionality/second-trees.md>`__). If you wanted to, say, compare influenza HA & NA trees, the URL used to look like "flu/seasonal/h3n2/ha:na/2y". This turned out to be problematic when coming up with suitable candidates for potential second-trees, and also made it impossible to compare, for instance, "ha/2y" with "na/3y"

We now use a more verbose syntax to define the display of multiple trees, specifying the entire pathname for both datasets. The above example is now "flu/seasonal/h3n2/ha/2y:flu/seasonal/h3n2/na/2y". Any available datasets can be compared using this URL syntax, even if the result is rather nonsensical. The old syntax will continue to work and will automatically correct to the new syntax (and show you a warning).

P.S. The list of available second trees, which is displayed in the sidebar, is now handed to Auspice by the :ref:`getAvailable API request <server-api-charon-getavailable>`.

Display better dates on the tree axis
-------------------------------------

Internally, we use decimal dates (e.g. 2012.3 is around the start of may) so that's what we displayed on the tree. It turns out this is pretty hard to interpret when looking at small timespans! We now (a) show dates on the tree's x-axis using months & days, depending on the timespan displayed, and (b) try to use more informative grid spacings. These help with the interpretation of trees over smaller time scales. See `PR #804 <https://github.com/nextstrain/auspice/pull/804>`__.

|time-labels| *Above: Auspice v1's decimal labels were somewhat hard to interprete. Below: v2 displays calendar dates as appropriate, and uses more intelligent grid spacing.*

Map "reset zoom" button zooms to include all demes
--------------------------------------------------

There's now a button at the top-right of the map which will trigger the map to reset the zoom. See `PR #802 <https://github.com/nextstrain/auspice/pull/802>`__.

Consistent colouring of missing data in the tree
------------------------------------------------

If your analysis produces results in ``-`` (gaps), ``X`` (unknown residue) or ``N`` (unknown nucelotide) then we now colour these grey, making it much easier to see when data is missing. See `PR #799 <https://github.com/nextstrain/auspice/pull/799>`__.

|base-colours| *Same analysis, different colour schemes, different interpretation.*

Removal of Twitter & Google Analytics
-------------------------------------

These were a holdover from the early days when `nextstrain.org <https://nextstrain.org>`__ and Auspice were the same thing. We've now removed all calls to Twitter, and made Google Analytics opt in. See :doc:`requests made from the client <../customise-client/requests>` for details on exactly what requests are made and how to opt-in to Google Analytics if you desire.

Improvements in the entropy panel
---------------------------------

We improved the usability of the entropy (genomic diversity) panel, as well as fixing a few hidden bugs -- see `PR #771 <https://github.com/nextstrain/auspice/pull/771>`__. For instance, you can now see which codon a nucleotide codes for (and vice-versa).

.. figure:: ../assets/v2-entropy.gif
   :alt: entropy

   entropy

Auspice responds to server redirects for datasets
-------------------------------------------------

This allows custom servers (`nextstrain.org <https://nextstrain.org>`__, for instance!) to smoothly inform Auspice that, e.g., a ``getDataset`` request to "/flu" (which doesn't actually exist) should be "/flu/seasonal/h3n2/ha/3y". See `PR #778 <https://github.com/nextstrain/auspice/pull/778>`__.

Importing (server) code from Auspice
------------------------------------

Auspice now makes a few helper commands available for those who are writing custom Auspice servers. See :ref:`these docs <server-api-importing-code-from-auspice>` for more info.

New Auspice subcommand: ``auspice convert``
-------------------------------------------

This is a utility command to convert between dataset formats. Currently, it only converts "Auspice v1" JSONs into "Auspice v2" JSONs, using the same code that is :ref:`programatically importable <server-api-convertfromv1>`.

Right now, ``auspice view`` will automatically convert "v1" JSONs into "v2" JSONs, so there's no need to do this yourself.

Ability to show a "build" source URL in the sidebar
---------------------------------------------------

Auspice used to contain some hard-coded logic which was used by nextstrain to display a link to the GitHub repo behind community URLs. We have now generalised this, and the :ref:`getAvailable API request <server-api-charon-getavailable>` can define a ``buildUrl`` property for each dataset which auspice will display in the sidebar.

``auspice view`` uses a custom Auspice client if present
--------------------------------------------------------

It's possible to use ``auspice build`` to :doc:`build a custom auspice client <../customise-client/overview>`. If this has been done, then running ``auspice view`` will serve it -- before you had to run ``auspice view --customBuild``. This streamlines generating custom auspice bundles and serving them locally.

.. |auspice-v2-gif| image:: ../assets/v2-pie-charts.gif
.. |pie-charts| image:: ../assets/v2-pie-charts.png
.. |more-tree-info| image:: ../assets/v2-tree-info.png
.. |time-labels| image:: ../assets/v2-time-labels.png
.. |base-colours| image:: ../assets/v2-base-colours.png
