User Guide
==========

.. contents::
   :local:
   :depth: 2

Overview
--------

Typing the ``adam`` command shows the different subcommands that can be used:

.. code-block:: none

    $ adam
    Usage: adam [OPTIONS] COMMAND [ARGS]...

    Options:
      --version                       Show the version and exit.
      -l, --log-level [info|debug|error]
                                      Logging level  [default: debug]
      -h, --help                      Show this message and exit.

    Commands:
      cmd                   Execute a command on the nodes
      describe              Describe an Adam cluster or profile
      ec2                   EC2 options
      enterprise-notebooks  Anaconda Enterprise Notebooks options
      info                  Display Adam version, system, and license information
      list                  List Adam clusters and profiles
      provision             Provision options
      remove                Remove a cluster
      repository            Anaconda Repository options
      salt                  Execute a Salt module
      scale                 Anaconda Scale options
      ssh                   SSH to one of the nodes (0-based index)
      up                    Create a cluster from a profile

You can view more information about a subcommand using the ``-h`` or ``--help``
option, for example, ``adam up --help``.

More detail on the available subcommands is shown in the sections below.

Manage Anaconda on the Cluster
------------------------------

Using Anaconda Scale, you can create and manage conda packages, including Python
and R, in multiple conda environments across your nodes. Refer to the
`Anaconda Scale documentation <https://docs.continuum.io/anaconda-scale>`_
for more information about installing and managing Anaconda across a cluster.

Install and Manage Platform Components
--------------------------------------

* ``adam enterprise-notebooks``
* ``adam repository``
* ``adam scale``

For more information on installing and managing platform components, refer to
the :doc:`getting-started` documentation.

Remote Commands
---------------

You can execute remote commands on all of the cluster nodes using ``adam cmd``:

.. code-block:: none

    $ adam cmd --help
    Usage: adam cmd [OPTIONS] COMMAND

    Options:
      -n, --cluster TEXT  Cluster name  [required]
      -t, --target TEXT   Target nodes  [default: *]
      -h, --help          Show this message and exit.

For example, to run the ``date`` command on all nodes:

.. code-block:: bash

    $ adam cmd -n cluster date
    192.168.1.1: Wed Jun 29 03:08:10 UTC 2016
    192.168.1.2: Wed Jun 29 03:08:10 UTC 2016
    192.168.1.3: Wed Jun 29 03:08:10 UTC 2016
    192.168.1.4: Wed Jun 29 03:08:10 UTC 2016

SSH to Nodes
------------

You can SSH to the cluster nodes using ``adam ssh``:

.. code-block:: none

    $ adam ssh -n cluster
    Last login: Wed Jun 29 03:08:49 2016 from 172.18.0.1
    [root@192.168.1.1 ~]#

By default, you will connect to the head node. You can optionally include the
node to SSH to, where node 0 is the head node, node 1 is the first compute node,
and so on:

.. code-block:: none

    $ adam ssh -n cluster 2
    Last login: Wed Jun 29 03:10:14 2016 from 172.18.0.1
    [root@192.168.1.3 ~]#

Version and License Information
-------------------------------

You can view version and license information using ``adam info``:

.. code-block:: none

    $ adam info
    Adam version: 4.1.0
    Plugins:
      adam_enterprise_notebooks: 4.1.0
      adam_repository: 4.1.0
      adam_notebook: 4.1.0
      adam_dask: 4.1.0

    Platform: Darwin-15.6.0-x86_64
    Darwin Kernel Version 15.6.0: Thu Jun 23 18:25:34 PDT 2016; root:xnu-3248.60.10~1/RELEASE_X86_64
    Python version: 3.5.1 |Continuum Analytics, Inc.| (default, Dec  7 2015, 11:24:55)
    [GCC 4.2.1 (Apple Inc. build 5577)]
    Processor: i386
    Byte-ordering: little

    License information:
    Number of nodes currently in use: 14
    Number of licensed nodes: 16
    Number of managed clusters: 2

    Valid platform component licenses:
    1: Anaconda Repository Enterprise. End date: 2017-05-24
    2: Anaconda Cluster. End date: 2017-05-24
    3: Wakari Enterprise. End date: 2017-05-24
    4: mkl-optimizations. End date: 2017-05-24
    5: iopro. End date: 2017-05-24
    6: accelerate. End date: 2017-05-24

List Cluster Information
------------------------

You can view a list of running clusters using ``adam list``:

.. code-block:: none

    $ adam list
    cluster
    second-cluster
    third-cluster

You can view a list of cluster profiles using ``adam list profiles``:

.. code-block:: none

    $ adam list profiles
    profile
    second-profile
    third-profile

You can view more detailed information about a cluster or profile using
``adam describe``. For example, to view information about the cluster named
``cluster``:

.. code-block:: none

    $ adam describe -n cluster
    conda:
      anaconda_hash: md5=7097150146dd3b83c805223663ebffcc
      anaconda_url: https://repo.continuum.io/miniconda/Miniconda2-4.3.21-Linux-x86_64.sh#adam-ci
      enabled: true
      rootdir: /opt/continuum
    name: cluster
    nodes:
    - host: ci_head_1
      keypair: ~/.ssh/my_private_key
      port: 22
      username: root
    - host: ci_compute_1
      keypair: ~/.ssh/my_private_key
      port: 22
      username: root

Creating, Provisioning, and Removing Clusters
---------------------------------------------

You can create, provision, and remove clusters using ``adam up``,
``adam provision``, and ``adam remove``.

To create a new cluster named ``cluster`` from a profile named ``profile``:

.. code-block:: none

    $ adam up -n cluster profile

To (re)provision an existing cluster named ``cluster``:

.. code-block:: none

    $ adam provision -n cluster

To remove the cluster definition file for an existing bare-metal cluster named
``cluster``:

.. code-block:: none

    $ adam remove -n cluster
    Are you sure you want to remove the cluster cluster? [y/N]: y

To destroy an existing cloud-based cluster named ``cluster`` and terminate the
corresponding instances:

.. code-block:: none

    $ adam ec2 -n cluster destory
    Are you sure you want to destroy the cluster cluster? [y/N]: y

Adding Nodes to an Existing Cluster
-----------------------------------

The following steps describe how to add additional nodes to an existing cluster
in Anaconda Adam.

To add additional nodes to a cluster managed by Anaconda Adam, you can modify
an existing cluster profile by specifying additional node(s). For example, if
you want to add the following nodes:

- ``node10.company.com``
- ``node11.company.com``
- ``node12.company.com``

to a cluster profile located at ``~/.continuum/adam/profile.d/profile.yaml``
with the following settings:

.. code-block:: yaml

    name: profile
    provider: bare

    bare:
      username: centos
      # password: password  # Optional
      port: 22
      keypair: ~/.ssh/my-private-key
      nodes:
      - host: node1.company.com
      - host: node2.company.com
      - host: node3.company.com
      - host: node4.company.com

Assuming the same authentication mode and credentials for the additional nodes,
you can add a ``host`` line for each node to the cluster profile:

.. code-block:: yaml

    name: profile
    provider: bare

    bare:
      username: centos
      # password: password  # Optional
      port: 22
      keypair: ~/.ssh/my-private-key
      nodes:
      - host: node1.company.com
      - host: node2.company.com
      - host: node3.company.com
      - host: node4.company.com
      - host: node10.company.com
      - host: node11.company.com
      - host: node12.company.com

Then, you can re-run the ``adam up`` command to reprovision the cluster:

.. code-block:: none

    $ adam up -n cluster profile

The ``adam up`` command will prompt to overwrite the cluster definition file
located at ``~/.continuum/adam/cluster.d/cluster.yaml`` to include the newly
defined nodes. During this reprovision process, Anaconda Adam will **not**
reinstall on the previously defined nodes.

Additional Provision Commands (Advanced)
----------------------------------------

By default, ``adam provision`` will install check SSH connectivity, create a
default user, install Salt, sync formulas, and provision enabled plugins and
security settings that are defined in the cluster profile.

If you only need to run a portion of the ``adam provision`` process, you can
view the available subcommands:

.. code-block:: none

    $ adam provision --help
    Usage: adam provision [OPTIONS] COMMAND [ARGS]...

      Execute all the provisioning steps for a cluster

      See subcommands help for more info. Usage: adam provision -n

    Options:
      -n, --cluster TEXT    Cluster name  [required]
      --flush-iptables      Flush iptables  [default: False]
      --selinux-permissive  Set SELinux to permissive  [default: False]
      --selinux-context     Set SELinux contexts  [default: False]
      --plugins             Install enabled plugins on provision  [default: False]
      -h, --help            Show this message and exit.

    Commands:
      anaconda-user       Create the default anaconda user in all the nodes
      check-ssh           Check SSH connection to the nodes
      flush-iptables      Flush iptables
      plugins             Provision enabled plugins
      roles               Set roles on the cluster nodes
      salt                Provision Salt
      security-settings   Execute the security settings
      selinux-context     Set SELinux contexts
      selinux-permissive  Set SELinux to permissive
      sync                Sync formulas and settings

For example, to check SSH connectivity to all of the nodes in an existing
cluster:

.. code-block:: none

    $ adam provision -n cluster check-ssh
    Checking SSH connection to nodes
    2016-10-03 22:10:49,327 - adam.models - DEBUG - Checking ssh connection for 54.88.20.164
    2016-10-03 22:10:50,905 - adam.models - DEBUG - SSH connection to 54.88.20.164: OK
    2016-10-03 22:10:51,008 - adam.models - DEBUG - Checking ssh connection for 52.90.168.130
    2016-10-03 22:10:51,996 - adam.models - DEBUG - SSH connection to 52.90.168.130: OK
    +---------------+---------------+
    | Node IP       | SSH Available |
    +---------------+---------------+
    | 54.88.20.164  | True          |
    | 52.90.168.130 | True          |
    +---------------+---------------+
    | All nodes     | True          |
    +---------------+---------------+

Execute Salt Commands (Advanced)
--------------------------------

You can execute Salt modules using ``adam salt``.

For example, to execute the ``test.ping`` Salt module:

.. code-block:: none

    $ adam salt -n cluster '*' test.ping
    192.168.1.1: true
    192.168.1.2: true
    192.168.1.3: true
    192.168.1.4: true
