=======================================
Upgrading Anaconda Enterprise Notebooks
=======================================

Summary
=======

IMPORTANT: These instructions are for upgrading Anaconda Enterprise Notebooks (AEN) to the 
current version 4.1.2 from 4.1.1 ONLY. Each version must be upgraded iteratively from the 
previous version. Do not skip versions.

Upgrade instructions to previous versions:

- :doc:`AEN 4.1.1 upgrade instructions </ae-notebooks/4.1.1/install/installation_update>`
- :doc:`AEN 4.1.0 upgrade instructions </ae-notebooks/4.1.0/install/installation_update>`
- :doc:`AEN 4.0.0 upgrade instructions </ae-notebooks/4.0/install/installation_update>`

For upgrades from versions before those listed, please contact your
enterprise support representative for help.

NOTE: Named Service Account functionality is available with Anaconda Enterprise
Notebooks 4.0.0+ for new installations only. It is not available for
upgraded installations. Contact your enterprise support representative 
for more information.

An upgrade to the Anaconda Enterprise Notebooks Platform requires all
instances of the three service categories to be upgraded individually:

-  *AEN Server*
-  *AEN Gateway*
-  *AEN Compute*

An installation of Anaconda Enterprise Notebooks typically consists
of several *Compute* instances running on independent *Compute Nodes*,
and multiple *Gateways* if there is more than one *Data Center*
tied into AEN. More details can be found in the :doc:`Overview <index>`.

The upgrade process requires that all AEN service instances be stopped, 
upgraded, and then restarted. These instructions describe how to perform 
this process.

Note: Any commands that call for the ``root`` user can also be done
using ``sudo``.

If you encounter any difficulty during the process, please refer to the
:doc:`Troubleshooting Guide <../admin/troubleshooting>` which
provides guidance on:

-  processes
-  configuration files
-  log files
-  ports

In the event you are unable to resolve an installation or upgrade
problem, please contact your enterprise support representative.

.. _aen-installation-update-preflight:

Pre-Flight Check
================

IMPORTANT: DO NOT PROCEED WITHOUT a tested backup of your installation 
before starting the upgrade.

IMPORTANT: TERMINATE ALL AEN APPS AND STOP ALL PROJECTS before starting 
the upgrade process.

Before upgrading each service on each host, perform the following operation:

#. Suspend the services on each of the nodes with the following commands::

       sudo service wakari-server stop
       sudo service wakari-gateway stop
       sudo service wakari-compute stop

#. Set the AEN Functional ID (NFI) and AEN Functional Group (NFG), which are 
   described in :doc:`the installation instructions <index>`.
   
   The NFI is the username of the AEN Service Account which is used to run all 
   AEN services and is also the username of the AEN Admin account.
   
   The NFG is the Linux group that includes the AEN Service Account, so all 
   files and directories that have the owner NFI also have the group NFG.
   
   When upgrading AEN, set the NFI and NFG to the NFI and NFG of the current 
   installation.
   
   The default NFI is "wakari", and "aen_admin" or any other name may be used 
   instead.
   
   Set the NFI and NFG with these commands, replacing "wakari" if you are
   using another name::
   
        export AEN_SRVC_ACCT="wakari"
        export AEN_SRVC_GRP="wakari"

AEN Server
==========

NOTE: If you are using LDAP-based authentication, back up the configuration file:

``/opt/wakari/wakari-server/etc/wakari/wk-server-config.json``

After the *Server* has been upgraded, copy that file back into the same location 
as before the upgrade.

The following operations are all done on the *Server* host:

#. Stop the ElasticSearch service::

       sudo service elasticsearch stop

#. Remove any previous index::

       sudo rm -rf /var/lib/elasticsearch/*

   NOTE: You can choose to keep the old index but if you detect any issues with
   the search capabilities after the upgrade, you will need to perform the following
   steps to start with a clean index::

       sudo service wakari-server stop
       sudo service elasticsearch stop
       sudo rm -rf /var/lib/elasticsearch/*
       sudo service elasticsearch start
       sudo service wakari-server start

#. Upgrade the *Server*::

       pushd /tmp
       curl -O http://j.mp/aen-server-update-4-1-2

       sudo -E -u $AEN_SRVC_ACCT /opt/wakari/miniconda/bin/conda install \
                        -p /opt/wakari/wakari-server          \
                        --file aen-server-update-4-1-2

       sudo -E -u $AEN_SRVC_ACCT /opt/wakari/miniconda/bin/conda install \
                        -p /opt/wakari/wakari-server          \
                        --no-deps                             \
                        wakari-enterprise-server-conf-update=2.0.3
       popd

#. Start ElasticSearch::

       sudo service elasticsearch start

   Or, if you do *not* want to use the search features, edit your
   server's ``config.json``::

       /opt/wakari/wakari-server/etc/wakari/config.json

   adding the line item: ``"SEARCH_ENABLED": false``.

#. Start the *Server*::

       sudo service wakari-server start

#. Check that the *Server* is running properly::

       sudo service wakari-server status

#. Finally, use your web browser to connect to AEN Server, using the
   correct protocol (*http* or *https*), hostname, and port number.

AEN Gateway
===========

Repeat the following operations on each *Gateway* host.

#. Upgrade the *Gateway*::

       pushd /tmp
       curl -O http://j.mp/aen-gateway-update-4-1-2

       sudo -E -u $AEN_SRVC_ACCT /opt/wakari/miniconda/bin/conda install \
                        -p /opt/wakari/wakari-gateway         \
                        --file aen-gateway-update-4-1-2

       sudo -E -u $AEN_SRVC_ACCT /opt/wakari/miniconda/bin/conda install \
                        -p /opt/wakari/wakari-gateway         \
                        --no-deps                             \
                        wakari-enterprise-gateway-conf-update=2.0.3
       popd

#. Start the *Gateway*::

    sudo service wakari-gateway start

#. Check that the *Gateway* is running properly::

    sudo service wakari-gateway status

#. Finally, use your web browser to connect to the *Gateway*, using the
   correct protocol (*http* or *https*), hostname, and port number.

AEN Compute
===========

Repeat the following operations on each host where an AEN
Compute service is running.

The processes killed with ``killall`` are run by the ``$AEN_SRVC_ACCT`` user, so
they can be killed as root with ``sudo killall`` or killed as the
``$AEN_SRVC_ACCT`` user with ``sudo -u $AEN_SRVC_ACCT killall``. These example
commands show the ``sudo killall`` option.

#. Check for any ``wakari-indexer`` processes running::

       ps aux | grep wakari-indexer

   NOTE: If you stopped all the projects, you will not see any wakari-indexer
   processes running.
   
#. Terminate any remaining ``wakari-indexer`` processes::

       sudo killall wakari-indexer

#. Check for any AEN apps (gateone, workbench, viewer, terminal, notebook)
   processes running::

        ps aux | grep wk-app-gateone
        ps aux | grep wk-app-workbench
        ps aux | grep wk-app-viewer
        ps aux | grep wk-app-terminal
        ps aux | grep jupyter-notebook

   NOTE: If you stopped all the projects, you will not see any AEN app
   processes running.
   
#. Terminate any remaining AEN apps processes that you find::
   
        sudo killall wk-app-gateone
        sudo killall wk-app-workbench
        sudo killall wk-app-viewer
        sudo killall wk-app-terminal
        sudo killall jupyter-notebook

#. Verify the contents of ``/opt/wakari/anaconda/.condarc``. Modify it 
   to contain the following entries (and possibly others if you 
   customized the ``.condarc`` file). Be sure to remove the 
   ``auto_update_conda: false`` because we are going to upgrade conda in the 
   following steps.
   
   NOTE: Modify the file as the ``AEN_SRVC_ACCT`` user (or be sure to keep the same ownership).

   ::

       channels:
         - https://conda.anaconda.org/t/<TOKEN>/anaconda-nb-extensions
         - r
         - https://conda.anaconda.org/wakari
         - defaults

       create_default_packages:
         - anaconda-client
         - ipykernel

   NOTE: Contact your enterprise support representative to get your ``<TOKEN>`` for
   the Anaconda channel referenced above.
   
   NOTE: Replace ``<TOKEN>`` with the actual token from your 
   enterprise support representative.

#. Upgrade *conda* in the root environment::

        sudo -E -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda install \
                        -p /opt/wakari/anaconda conda=4.3.14

   After the successful conda upgrade, modify the ``/opt/wakari/anaconda/.condarc`` file
   and re-add the line ``auto_update_conda: false`` removed in the previous step.

#. Upgrade *Anaconda* in the root environment::

        pushd /tmp
        curl -O http://j.mp/aen-anaconda-update-4-1-2

        sudo -E -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda install \
                        -p /opt/wakari/anaconda              \
                        --file aen-anaconda-update-4-1-2

        popd

#. Initialize the root environment to prime the package cache::

        sudo -E -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda create \
                        -p /opt/wakari/testenv \
                        --clone root

   and test the offline cloning step with::
   
        sudo -E -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda create \
                        -p /opt/wakari/testenvoffline \
                        --clone root --offline

   finally remove those test environments with::
   
        sudo rm -rf /opt/wakari/testenv
        sudo rm -rf /opt/wakari/testenvoffline

#. Upgrade each *Compute* service::

       pushd /tmp
       curl -O http://j.mp/aen-compute-update-4-1-2
       
       sudo -E -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda install \
                        -p /opt/wakari/wakari-compute        \
                        --file aen-compute-update-4-1-2

       sudo -E -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda install \
                        --no-deps                            \
                        -p /opt/wakari/wakari-compute        \
                        wakari-enterprise-compute-conf-update=2.0.4
       popd

   NOTE: From AEN 4.1.0 and above, ``wakari-app-gateone`` is deprecated and no
   longer supported.

   NOTE: When upgrading the wakari-compute environment you may see `ImportError`
   warnings with some nbextensions. As long as the `Validating` message is
   `OK`, the `ImportError` warnings are harmless (just a consequence of the
   post-link presence on those packages).

#. Run the upgrade_412.sh script to make your 4.1.1 projects compatible with the 4.1.2 codebase::

        pushd /tmp
        curl -o upgrade_412.sh http://j.mp/upgrade_412
        chmod a+x upgrade_412.sh
        ./upgrade_412.sh
        popd

#. Install necessary dependencies:

   Note: Skip this step if you already have these dependencies installed from
   previous installations.

   ::
   
        sudo yum groupinstall "X Window System" -y
        sudo yum install git -y

   Note: If you don't want to install the whole X Window System, you need to
   install, at least, the following packages to have R plotting support::

        sudo yum install -y libXrender libXext libXdmcp libSM libICE libXt \
        dejavu-sans-fonts dejavu-serif-fonts dejavu-fonts-common \
        fontpackages-filesystem

#. Start the *Compute* service::

        sudo service wakari-compute start

#. Verify the *Compute* service is running properly::

        sudo service wakari-compute status
        
#. Repeat the upgrade procedure for all the Compute Nodes in your Data Center. 
        
Post-Flight Check
=================

#. You can now start the projects and start using AEN apps.

#. If you have a :ref:`customized default
   environment <aen-default-project-env>`, you may choose to upgrade it
   depending on the needs of your users.

NOTE: Upgrading the ``default`` environment at
``/opt/wakari/anaconda/envs/default`` does NOT automatically upgrade the
``default`` environment in the users **pre-existing** projects. For
pre-existing projects, the upgrade (if it is requested) should be done on a
per-user basis.

#. If you did not stop all your projects before upgrading, then the first time 
   you start an app you will see an error page requesting you to restart the app. 
   Restart the app to complete the upgrade.
   
#. If you still see old apps or the old IPython logo in the notebook app, reload 
   the page to reset the browser cache.

.. toctree::
   :hidden:

   scripts/project_permissions_py
   scripts/repair_team_members_py
