=======================================
Upgrading Anaconda Enterprise Notebooks
=======================================

Summary
=======

These instructions are for upgrading Anaconda Enterprise Notebooks to the 
current version 4.1 from 4.0 ONLY. 

For help upgrading from a version older than 4.0, see the :doc:`AEN 4.0 upgrade instructions </ae-notebooks/4.0/install/installation_update>`, and contact your enterprise support representative for help.

NOTE: The Named Service Account functionality is available with Anaconda Enterprise
Notebooks 4.0.0 and above for new installations only, and is not available for upgraded installations.

An upgrade to the Anaconda Enterprise Notebooks Platform requires all
instances of the three service categories to be upgraded individually:

-  *AEN Server*
-  *AEN Gateway*
-  *AEN Compute*

A typical installation of Anaconda Enterprise Notebooks likely consists
of several *Compute* instances running on independent *Compute Nodes*,
and possibly multiple *Gateways* if there is more than one *Data Center*
tied into Anaconda Enterprise Notebooks. More details can be found in
the :doc:`Overview <index>`.

The upgrade process requires that all Anaconda Enterprise Notebooks
service instances be stopped, upgraded, and then restarted. These
instructions describe how to perform this process.

Note: Any commands that call for the ``root`` user can also be done
using ``sudo``.

If you encounter any difficulty during the process, please refer to the
:doc:`Troubleshooting Guide <../admin/troubleshooting>` which
provides guidance on:

-  processes
-  configuration files
-  log files
-  ports

In the event you are unable to resolve an installation or upgrade
problem please contact your enterprise support representative.

.. _aen-installation-update-preflight:

Pre-Flight Check
================

IMPORTANT: You must have a tested backup of your installation before starting the upgrade.

IMPORTANT: We recommend that you terminate all AEN apps and stop all projects
before starting the upgrade process.

Before upgrading each service on each host, you should perform the
following operation:

#. Suspend the services on each of the nodes with the following commands::

       sudo service wakari-server stop
       sudo service wakari-gateway stop
       sudo service wakari-compute stop

#. Set the AEN Functional ID (NFI) and AEN Functional Group (NFG), which are 
   described in :doc:`the installation instructions <installation>`.
   
   The NFI is the username of the AEN Service Account which is used to run all 
   AEN services and is also the username of the AEN Admin account.
   
   The NFG is the Linux group that includes the AEN Service Account, so all 
   files and directories that have the owner NFI also have the group NFG.
   
   When upgrading AEN, set the NFI and NFG to the NFI and NFG of the current 
   installation.
   
   The default NFI is "wakari", and "aen_admin" or any other name may be used 
   instead.
   
   Set the NFI and NFG with these commands, replacing "aen_admin" if you are
   using another name::
   
        export AEN_SRVC_ACCT="aen_admin"
        export AEN_SRVC_GRP="aen_admin"

AEN Server
==========

**NOTE:** If you are using LDAP-based authentication, back up
the configuration file:

``/opt/wakari/wakari-server/etc/wakari/wk-server-config.json``

After the *Server* has been upgraded, copy that file back into its location from before the upgrade.

The following operations are all done on the *Server* host.

#. If you are taking advantage of the search capabilities provided by
   ElasticSearch and a Java Runtime Environment (JRE 8):

   Stop the ElasticSearch service::

       sudo service elasticsearch stop

   Remove any previous index::

       sudo rm -rf /var/lib/elasticsearch/*

**NOTE:** You can choose to keep the old index but if you detect any issues with
the search capabilities after the upgrade, you will need to perform the following
steps to start with a clean index::

    sudo service wakari-server stop
    sudo service elasticsearch stop
    sudo rm -rf /var/lib/elasticsearch/*
    sudo service elasticsearch start
    sudo service wakari-server start

#. Upgrade the *Server*::

       pushd /tmp
       wget http://j.mp/aen-server-update-latest

       sudo -u $AEN_SRVC_ACCT /opt/wakari/miniconda/bin/conda install \
                        -p /opt/wakari/wakari-server          \
                        --file aen-server-update-latest

       sudo -u $AEN_SRVC_ACCT /opt/wakari/miniconda/bin/conda install \
                        -p /opt/wakari/wakari-server          \
                        --no-deps                             \
                        wakari-enterprise-server-conf-update
       popd

#. Start ElasticSearch::

       sudo service elasticsearch start

   Or, if you do *not* want to use the search features, edit your
   server's ``config.json``::

       /opt/wakari/wakari-server/etc/wakari/config.json

   adding the line item: ``"SEARCH_ENABLED": false``.

#. Start the *Server*::

       sudo service wakari-server start

#. Check that the *Server* is running properly::

       sudo service wakari-server status

#. Finally, use your web browser to connect to AEN Server, using the
   correct protocol (*http* or *https*), hostname, and port number.

AEN Gateway
===========

The following operations are all done on **each** *Gateway* host.

#. Upgrade the *Gateway*::

       pushd /tmp
       wget http://j.mp/aen-gateway-update-latest

       sudo -u $AEN_SRVC_ACCT /opt/wakari/miniconda/bin/conda install \
                        -p /opt/wakari/wakari-gateway         \
                        --file aen-gateway-update-latest

       sudo -u $AEN_SRVC_ACCT /opt/wakari/miniconda/bin/conda install \
                        -p /opt/wakari/wakari-gateway         \
                        --no-deps                             \
                        wakari-enterprise-gateway-conf-update
       popd

#. Start the *Gateway*::

    sudo service wakari-gateway start

#. Check that the *Gateway* is running properly::

    sudo service wakari-gateway status

#. Finally, use your web browser to connect to the *Gateway*, using the
   correct protocol (*http* or *https*), hostname, and port number.

AEN Compute
===========

The following operations are all done on **each** host where an AEN
Compute service is running.

#. Check for any ``wakari-indexer`` processes running::

       ps aux | grep wakari-indexer

   **NOTE:** If you stopped all the projects, you should not see any wakari-indexer
   processes running.
   
   Terminate any remaining ``wakari-indexer`` processes::

       killall wakari-indexer

#. Check for any AEN apps (gateone, workbench, viewer, terminal, notebook)
   processes running::

        ps aux | grep wk-app-gateone
        ps aux | grep wk-app-workbench
        ps aux | grep wk-app-viewer
        ps aux | grep wk-app-terminal
        ps aux | grep notebookwrapper

   **NOTE:** If you stopped all the projects, you should not see any AEN app
   processes running.
   
   Terminate any remaining AEN apps processes that you find::
   
        killall wk-app-gateone
        killall wk-app-workbench
        killall wk-app-viewer
        killall wk-app-terminal
        killall notebookwrapper

#. Verify the contents of ``/opt/wakari/anaconda/.condarc``. Modify it properly
   to contain the following entries (and possibly others if you customized the .condarc file).
   
   **NOTE:** Please modify the file as the AEN_SRVC_ACCT user (or be sure to keep the same ownership).

   ::

       channels:
         - https://conda.anaconda.org/t/``<TOKEN>``/anaconda-nb-extensions
         - r
         - https://conda.anaconda.org/wakari
         - defaults

       create_default_packages:
         - anaconda-client
         - ipykernel
       auto_update_conda: false

   **NOTE:** Contact our sales representatives to get your ``<TOKEN>`` for
   the Anaconda channel referenced above.

#. Remove any pinning to specific conda versions from Anaconda Enterprise 
   Notebooks 4.0.0::
   
       sudo rm /opt/wakari/anaconda/conda-meta/pinned

#. As root, modify the ownership of the package cache::

       chown -R $AEN_SRVC_ACCT:$AEN_SRVC_GRP /opt/wakari/anaconda/pkgs

#. Upgrade the *Compute* service.

   **NOTE:** From AEN 4.1.0 onward, `wakari-app-gateone` is deprecated and no
   longer supported, so we remove that package with conda here.

   ::

       pushd /tmp
       wget http://j.mp/aen-compute-update-latest
       wget http://j.mp/aen-anaconda-update-latest
       
       sudo -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda install \
                        -p /opt/wakari/wakari-compute        \
                        --file aen-compute-update-latest

       sudo -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda remove  \
                        -p /opt/wakari/wakari-compute        \
                        wakari-app-gateone

       sudo -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda install \
                        --no-deps                            \
                        -p /opt/wakari/wakari-compute        \
                        wakari-enterprise-compute-conf-update

       sudo -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda install \
                        -p /opt/wakari/anaconda conda=4.2.9

       sudo -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda remove  \
                        -p /opt/wakari/anaconda              \
                        spyder ipython-we pandoc keyring     \
                        ecdsa util-linux sphinx_rtd_theme    \
                        mklfft numbapro numbapro_cudalib     \
                        ncurses nlopt paramiko               \
                        wakari-client conda-manager conda-api\
                        conda_analysis_package

       sudo -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda install \
                        -p /opt/wakari/anaconda              \
                        --file aen-anaconda-update-latest
       popd

   **NOTE:** When upgrading the wakari-compute environment nbextensions you
   may see `ImportError` warnings. As long as the `Validating` message is
   `OK`, the `ImportError` warnings are harmless.
   
   **NOTE:** If the package `ncurses` produces a `CondaOSError`, remove this file::
   
        sudo rm /opt/wakari/anaconda/lib/terminfo

   After removing the `terminfo` file, run the last conda install command
   again to complete the anaconda upgrade.

#. Initialize the root environment to prime the package cache::

       sudo -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda create \
                        -p /opt/wakari/testenv \
                        --clone root

   and test the offline cloning step with::
   
        sudo -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda create \
                         -p /opt/wakari/testenvoffline \
                         --clone root --offline

   finally remove those test environments with::
   
        sudo rm -rf /opt/wakari/testenv
        sudo rm -rf /opt/wakari/testenvoffline

#. Run the `upgrade_410` script to make your old projects compatible
   with the 4.1.0 codebase::
   
        pushd /tmp
        wget http://j.mp/upgrade_410
        chmod a+x upgrade_410
        ./upgrade_410
        popd

#. Start the *Compute* service::

        sudo service wakari-compute start

#. Verify the *Compute* service is running properly::

       sudo service wakari-compute status

Post-Flight Check
=================

#. You can now start the projects and start using the AEN apps.

#. If you have a :doc:`customized default
   environment <../user/projects>`, you may choose to upgrade it
   depending on the needs of your users.

**NOTE:** Upgrading the ``default`` environment at
``/opt/wakari/anaconda/envs/default`` does not automatically upgrade the
``default`` environment in the users **pre-existing** projects. For
pre-existing projects the upgrade (if it is requested) should be done on a
per-user basis.

Project Permissions
===================

Since version 0.10.0 Anaconda Enterprise Notebook uses POSIX.1e access control lists
to manage shared content, rather than plain POSIX user groups. If you
are upgrading from an AEN version earlier than 0.5.0, please execute the
:doc:`project\_permissions.py <scripts/project_permissions_py>` script on
every Compute node::

    /opt/wakari/wakari-compute/bin/python project_permissions.py /projects

**NOTE:** In some deployments, the project directory may be different
from ``/projects``.

Then, on the AEN *Server* host, run
:doc:`repair\_team\_members.py <scripts/repair_team_members_py>`::

    /opt/wakari/wakari-server/bin/python repair_team_members.py

These scripts will read all the *Project* and *Team* information from
the AEN database, and re-add individuals to all the *Project* on all
*Compute* nodes.

.. toctree::
   :hidden:

   scripts/project_permissions_py
   scripts/repair_team_members_py
   scripts/upgrade_400_sh
