{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# HPO with dask-ml and cuml\n",
    "\n",
    "## Introduction\n",
    "\n",
    "&emsp; &emsp; &emsp; [Hyperparameter optimization](https://cloud.google.com/ai-platform/training/docs/hyperparameter-tuning-overview) is the task of picking the values for the hyperparameters of the model that provide the optimal results for the problem, as measured on a specific test dataset. This is often a crucial step and can help boost the model accuracy when done correctly. Cross-validation is often used to more accurately estimate the performance of the models in the search process. Cross-validation is the method of splitting the training set into complementary subsets and performing training on one of the subsets, then predicting the models performance on the other. This is a potential indication of how the model will generalise to data it has not seen before.\n",
    "\n",
    "Despite its theoretical importance, HPO has been difficult to implement in practical applications because of the resources needed to run so many distinct training jobs.\n",
    "\n",
    "The two approaches that we will be exploring in this notebook are :\n",
    "\n",
    "\n",
    "#### 1. GridSearch\n",
    "\n",
    "&emsp; &emsp; &emsp; As the name suggests, the \"search\" is done over each possible combination in a grid of parameters that the user provides. The user must manually define this grid.. For each parameter that needs to be tuned, a set of values are given and the final grid search is performed with tuple having one element from each set, thus resulting in a Catersian Product of the elements.\n",
    "\n",
    "&emsp; &emsp; &emsp;For example, assume we want to perform HPO on XGBoost. For simplicity lets tune only `n_estimators` and `max_depth`\n",
    "\n",
    "&emsp; &emsp; &emsp;`n_estimators: [50, 100, 150]`\n",
    "\n",
    "&emsp; &emsp; &emsp;`max_depth: [6, 7, ,8]`\n",
    "    \n",
    "&emsp; &emsp; &emsp; The grid search will take place over |n_estimators| x |max_depth| which is 3 x 3 = 9. As you have probably guessed, the grid size grows rapidly as the number of parameters and their search space increases.\n",
    "\n",
    "#### 2. RandomSearch\n",
    "\n",
    "\n",
    "&emsp; &emsp; &emsp; [Random Search](http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf) replaces the exhaustive nature of the search from before with a random selection of parameters over the specified space. This method can outperform GridSearch in cases where the number of parameters affecting the model's performance is small (low-dimension optimization problems). Since this does not pick every tuple from the cartesian product, it tends to yield results faster, and the performance can be comparable to that of the Grid Search approach. It's worth keeping in mind that the random nature of this search means, the results with each run might differ.\n",
    "\n",
    "Some of the other methods used for HPO include:\n",
    "\n",
    "1. Bayesian Optimization\n",
    "\n",
    "2. Gradient-based Optimization\n",
    "\n",
    "3. Evolutionary Optimization\n",
    "\n",
    "To learn more about HPO, some papers are linked to at the end of the notebook for further reading.\n",
    "\n",
    "Now that we have a basic understanding of what HPO is, let's discuss what we wish to achieve with this demo. The aim of this notebook is to show the importance of hyper parameter optimisation and the performance of dask-ml GPU for xgboost and cuML-RF and compare the performance with the dask-ml CPU version. We'll highlight the improvements in model performance and the run-time for Grid and Random searches in the CPU and GPU versions.\n",
    "\n",
    "We make use of the [Higgs dataset](https://archive.ics.uci.edu/ml/datasets/HIGGS). It solves a binary classification problem to distinguish between signal processes that produce Higgs bosons from those that do not. There are a total of 28 features, first 21 are measured and the last 7 are functions of these. It has 11 Million entries.\n",
    "\n",
    "Let's get started!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore') # Reduce number of messages/warnings displayed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "import numpy as np\n",
    "import cupy as cp\n",
    "import pandas as pd\n",
    "\n",
    "import cudf\n",
    "import cuml\n",
    "import rmm\n",
    "import xgboost as xgb\n",
    "\n",
    "import sklearn.model_selection as sk\n",
    "import dask_ml.model_selection as dcv\n",
    "from dask.distributed import Client, wait\n",
    "from dask_cuda import LocalCUDACluster\n",
    "\n",
    "from sklearn import datasets\n",
    "from sklearn.metrics import make_scorer\n",
    "from sklearn.metrics import accuracy_score as sk_acc\n",
    "\n",
    "from cuml.neighbors import KNeighborsClassifier\n",
    "from cuml.preprocessing.model_selection import train_test_split\n",
    "from cuml.metrics.accuracy import accuracy_score\n",
    "\n",
    "import os\n",
    "from urllib.request import urlretrieve\n",
    "import gzip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Spinning up a CUDA Cluster\n",
    "\n",
    "We start a local cluster and keep it ready for running distributed tasks with dask.\n",
    "\n",
    "\n",
    "[LocalCUDACluster](https://github.com/rapidsai/dask-cuda) launches one Dask worker for each GPU in the current systems. It's developed as a part of the RAPIDS project.\n",
    "Learn More:\n",
    "- [Setting up Dask](https://docs.dask.org/en/latest/setup.html)\n",
    "- [Dask Client](https://distributed.dask.org/en/latest/client.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table style=\"border: 2px solid white;\">\n",
       "<tr>\n",
       "<td style=\"vertical-align: top; border: 0px solid white\">\n",
       "<h3 style=\"text-align: left;\">Client</h3>\n",
       "<ul style=\"text-align: left; list-style: none; margin: 0; padding: 0;\">\n",
       "  <li><b>Scheduler: </b>tcp://127.0.0.1:45309</li>\n",
       "  <li><b>Dashboard: </b><a href='http://127.0.0.1:8005/status' target='_blank'>http://127.0.0.1:8005/status</a>\n",
       "</ul>\n",
       "</td>\n",
       "<td style=\"vertical-align: top; border: 0px solid white\">\n",
       "<h3 style=\"text-align: left;\">Cluster</h3>\n",
       "<ul style=\"text-align: left; list-style:none; margin: 0; padding: 0;\">\n",
       "  <li><b>Workers: </b>4</li>\n",
       "  <li><b>Cores: </b>4</li>\n",
       "  <li><b>Memory: </b>404.27 GB</li>\n",
       "</ul>\n",
       "</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<Client: 'tcp://127.0.0.1:45309' processes=4 threads=4, memory=404.27 GB>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cluster = LocalCUDACluster(dashboard_address=\"127.0.0.1:8005\")\n",
    "client = Client(cluster)\n",
    "\n",
    "client"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Preparation\n",
    "\n",
    "We download the Higgs dataset and decompress the file for reading it in as input. We also assign appropriate column names and dtypes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def download_higgs(compressed_filepath, decompressed_filepath):\n",
    "    higgs_url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/00280/HIGGS.csv.gz'\n",
    "    if not os.path.isfile(compressed_filepath):\n",
    "        urlretrieve(higgs_url, compressed_filepath)\n",
    "    if not os.path.isfile(decompressed_filepath):\n",
    "        cf = gzip.GzipFile(compressed_filepath)\n",
    "        with open(decompressed_filepath, 'wb') as df:\n",
    "            df.write(cf.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dir = '/home/hyperopt/data/'\n",
    "if not os.path.exists(data_dir):\n",
    "    print('creating data directory')\n",
    "    os.system('mkdir /home/data/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "compressed_filepath = os.path.join(data_dir, 'HIGGS.csv.gz') # Set this as path for gzipped Higgs data file, if you already have\n",
    "decompressed_filepath = os.path.join(data_dir, 'HIGGS.csv') # Set this as path for decompressed Higgs data file, if you already have\n",
    "\n",
    "# Uncomment this line to download the dataset.\n",
    "# download_higgs(compressed_filepath, decompressed_filepath)\n",
    "\n",
    "col_names = ['label'] + [\"col-{}\".format(i) for i in range(2, 30)] # Assign column names\n",
    "dtypes_ls = ['int32'] + ['float32' for _ in range(2, 30)] # Assign dtypes to each column\n",
    "input_data = cudf.read_csv(decompressed_filepath, names=col_names, dtype=dtypes_ls)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "labels = input_data.label.reset_index().drop(['index'], axis=1)\n",
    "for col in labels.columns:\n",
    "    labels[col] = labels[col].astype('float32')\n",
    "data = input_data.drop(['label'], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "11000000"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "from contextlib import contextmanager\n",
    "# Helping time blocks of code\n",
    "@contextmanager\n",
    "def timed(txt):\n",
    "    t0 = time.time()\n",
    "    yield\n",
    "    t1 = time.time()\n",
    "    print(\"%32s time:  %8.5f\" % (txt, t1 - t0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Fraction\n",
    "\n",
    "We will make use of `data_fraction` variable that takes value between `(0, 1.0]` to specify the fraction of data to make use. The CPU version crashes when it makes use of larger portion of the dataset, so we will not run the CPU version for a fraction more than `0.1`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_fraction = 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1100000\n"
     ]
    }
   ],
   "source": [
    "N_ROWS = int(len(data) * data_fraction)\n",
    "\n",
    "# Define some default values to make use of across the notebook for a fair comparison\n",
    "N_FOLDS = 5\n",
    "N_ESTIMATORS = 100\n",
    "MAX_DEPTH = 5\n",
    "N_ITER = 100\n",
    "print(N_ROWS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = data[:N_ROWS]\n",
    "labels = labels[:N_ROWS]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Splitting Data\n",
    "\n",
    "We split the data randomnly into train and test sets using the [cuml train_test_split](https://docs.rapids.ai/api/cuml/stable/api.html#cuml.preprocessing.model_selection.train_test_split) and create CPU versions of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(data,\n",
    "                                                    labels,\n",
    "                                                    test_size=0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_cpu = X_train.to_pandas()\n",
    "y_cpu = y_train.label.to_numpy()\n",
    "\n",
    "\n",
    "X_test_cpu = X_test.to_pandas()\n",
    "y_test_cpu = y_test.label.to_numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup Custom cuML scorers\n",
    "\n",
    "The search functions (such as GridSearchCV) for scikit-learn and dask-ml expect the metric functions (such as accuracy_score) to match the “scorer” API. This can be achieved using the scikit-learn's [make_scorer](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.make_scorer.html) function.\n",
    "\n",
    "We will generate a `cuml_scorer` with the cuML `accuracy_score` function.  You'll also notice an `accuracy_score_wrapper` which primarily converts the y label into a `float32` type. This is because some cuML models only accept this type for now and in order to make it compatible, we perform this conversion.\n",
    "\n",
    "We also create helper functions for performing HPO in 4 different modes: \n",
    "1. `gpu-grid`: Perform GPU based GridSearchCV\n",
    "2. `gpu-random`: Perform GPU based RandomizedSearchCV\n",
    "3. `cpu-grid`: Perform CPU based GridSearchCV\n",
    "4. `cpu-random`: Perform CPU based RandomizedSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def accuracy_score_wrapper(y, y_hat): \n",
    "    \"\"\"\n",
    "        A wrapper function to convert labels to float32, \n",
    "        and pass it to accuracy_score.\n",
    "        \n",
    "        Params:\n",
    "        - y: The y labels that need to be converted\n",
    "        - y_hat: The predictions made by the model\n",
    "    \"\"\"\n",
    "    y = y.astype(\"float32\") # cuML RandomForest needs the y labels to be float32\n",
    "    return accuracy_score(y, y_hat, convert_dtype=True)\n",
    "\n",
    "accuracy_wrapper_scorer = make_scorer(accuracy_score_wrapper)\n",
    "cuml_accuracy_scorer = make_scorer(accuracy_score, convert_dtype=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def do_HPO(model, gridsearch_params, scorer, X, y, mode='gpu-Grid', n_iter=10):\n",
    "    \"\"\"\n",
    "        Perform HPO based on the mode specified\n",
    "        \n",
    "        mode: default gpu-Grid. The possible options are:\n",
    "        1. gpu-grid: Perform GPU based GridSearchCV\n",
    "        2. gpu-random: Perform GPU based RandomizedSearchCV\n",
    "        3. cpu-grid: Perform CPU based GridSearchCV\n",
    "        4. cpu-random: Perform CPU based RandomizedSearchCV\n",
    "        \n",
    "        n_iter: specified with Random option for number of parameter settings sampled\n",
    "        \n",
    "        Returns the best estimator and the results of the search\n",
    "    \"\"\"\n",
    "    if mode == 'cpu-grid':\n",
    "        print(\"cpu-grid selected\")\n",
    "        clf = dcv.GridSearchCV(model,\n",
    "                              gridsearch_params,\n",
    "                              cv=N_FOLDS,\n",
    "                              scoring=scorer)\n",
    "    elif mode == 'gpu-grid':\n",
    "        print(\"gpu-grid selected\")\n",
    "        clf = dcv.GridSearchCV(model,\n",
    "                               gridsearch_params,\n",
    "                               cv=N_FOLDS,\n",
    "                               scoring=scorer)\n",
    "    elif mode == 'gpu-random':\n",
    "        print(\"gpu-random selected\")\n",
    "        clf = dcv.RandomizedSearchCV(model,\n",
    "                               gridsearch_params,\n",
    "                               cv=N_FOLDS,\n",
    "                               scoring=scorer,\n",
    "                               n_iter=n_iter)\n",
    "    elif mode == 'cpu-random':\n",
    "        print(\"cpu-random selected\")\n",
    "        clf = dcv.RandomizedSearchCV(model,\n",
    "                               gridsearch_params,\n",
    "                               cv=N_FOLDS,\n",
    "                               scoring=scorer,\n",
    "                               n_iter=n_iter)\n",
    "    else:\n",
    "        print(\"Unknown Option, please choose one of [gpu-grid, gpu-random, cpu-grid, cpu-random]\")\n",
    "        return None, None\n",
    "    res = clf.fit(X, y)\n",
    "    print(\"Best clf and score {} {}\\n---\\n\".format(res.best_estimator_, res.best_score_))\n",
    "    return res.best_estimator_, res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_acc(model, X_train, y_train, X_test, y_test, mode_str=\"Default\"):\n",
    "    \"\"\"\n",
    "        Trains a model on the train data provided, and prints the accuracy of the trained model.\n",
    "        mode_str: User specifies what model it is to print the value\n",
    "    \"\"\"\n",
    "    y_pred = model.fit(X_train, y_train).predict(X_test)\n",
    "    score = accuracy_score(y_pred, y_test.astype('float32'), convert_dtype=True)\n",
    "    \n",
    "    print(\"{} model accuracy: {}\".format(mode_str, score))\n",
    "                                         "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(880000, 28)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Launch HPO\n",
    "\n",
    "We will first see the model's performances without the gridsearch and then compare it with the performance after searching.\n",
    "\n",
    "### XGBoost\n",
    "\n",
    "To perform the Hyperparameter Optimization, we make use of the sklearn version of the [XGBClassifier](https://xgboost.readthedocs.io/en/latest/python/python_api.html#module-xgboost.sklearn).We're making use of this version to make it compatible and easily comparable to the scikit-learn version. The model takes a set of parameters that can be found in the documentation. We're primarily interested in the `max_depth`, `learning_rate`, `min_child_weight`, `reg_alpha` and `num_round` as these affect the performance of XGBoost the most.\n",
    "\n",
    "Read more about what these parameters are useful for [here](https://xgboost.readthedocs.io/en/latest/parameter.html)\n",
    "\n",
    "We'll first perform the GPU version and then compare it with how long CPU version took to run. We will run only the `dask-ml` GridSearch and RandomSearch in this notebook.\n",
    "\n",
    "\n",
    "#### Default Performance\n",
    "\n",
    "We first use the model with it's default parameters and see the accuracy of the model. In this case, it is 71%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Default model accuracy: 0.7132499814033508\n"
     ]
    }
   ],
   "source": [
    "model_gpu_xgb_ = xgb.XGBClassifier(tree_method='gpu_hist')\n",
    "\n",
    "print_acc(model_gpu_xgb_, X_train, y_cpu, X_test, y_test_cpu)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Parameter Distributions\n",
    "\n",
    "The way we define the grid to perform the search is by including ranges of parameters that need to be used for the search. In this example we make use of [np.arange](https://docs.scipy.org/doc/numpy/reference/generated/numpy.arange.html) which returns an ndarray of even spaced values, [np.logspace](https://docs.scipy.org/doc/numpy/reference/generated/numpy.logspace.html#numpy.logspace) returns a specified number of ssamples that are equally spaced on the log scale. We can also specify as lists, NumPy arrays or make use of any random variate sample that gives a sample when called. SciPy provides various functions for this too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For xgb_model\n",
    "model_gpu_xgb = xgb.XGBClassifier(tree_method='gpu_hist')\n",
    "\n",
    "# More range \n",
    "params_xgb = {\n",
    "    \"max_depth\": np.arange(start=3, stop = 15, step = 3), # Default = 6\n",
    "    \"alpha\" : np.logspace(-3, -1, 5), # default = 0\n",
    "    \"learning_rate\": [0.05, 0.1, 0.15], #default = 0.3\n",
    "    \"min_child_weight\" : np.arange(start=2, stop=10, step=3), # default = 1\n",
    "    \"n_estimators\": [100, 200, 1000]\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### RandomizedSearchCV\n",
    "\n",
    "We'll now try [RandomizedSearchCV](https://dask-ml.readthedocs.io/en/latest/modules/generated/dask_ml.model_selection.RandomizedSearchCV.html).\n",
    "`n_iter` specifies the number of parameters points theat the search needs to perform. Here we will search `N_ITER` (defined earlier) points for the best performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gpu-random selected\n",
      "Best clf and score XGBClassifier(alpha=0.001, base_score=0.5, booster='gbtree',\n",
      "              colsample_bylevel=1, colsample_bynode=1, colsample_bytree=1,\n",
      "              gamma=0, gpu_id=-1, learning_rate=0.05, max_delta_step=0,\n",
      "              max_depth=12, min_child_weight=2, missing=nan, n_estimators=1000,\n",
      "              n_jobs=1, objective='binary:logistic', random_state=0,\n",
      "              reg_alpha=0, reg_lambda=1, scale_pos_weight=1, subsample=1,\n",
      "              tree_method='gpu_hist', verbose=1) 0.7502772559810091\n",
      "---\n",
      "\n",
      "                  XGB-gpu-random time:  2124.28109\n",
      "Searched over 100 parameters\n"
     ]
    }
   ],
   "source": [
    "mode = \"gpu-random\"\n",
    "\n",
    "with timed(\"XGB-\"+mode):\n",
    "    res, results = do_HPO(model_gpu_xgb,\n",
    "                                   params_xgb,\n",
    "                                   cuml_accuracy_scorer,\n",
    "                                   X_train,\n",
    "                                   y_cpu,\n",
    "                                   mode=mode,\n",
    "                                   n_iter=N_ITER)\n",
    "print(\"Searched over {} parameters\".format(len(results.cv_results_['mean_test_score'])))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gpu-random model accuracy: 0.7528954744338989\n"
     ]
    }
   ],
   "source": [
    "print_acc(res, X_train, y_cpu, X_test, y_test_cpu, mode_str=mode)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gpu-grid selected\n",
      "Best clf and score XGBClassifier(alpha=0.1, base_score=0.5, booster='gbtree', colsample_bylevel=1,\n",
      "              colsample_bynode=1, colsample_bytree=1, gamma=0, gpu_id=-1,\n",
      "              learning_rate=0.05, max_delta_step=0, max_depth=12,\n",
      "              min_child_weight=5, missing=nan, n_estimators=1000, n_jobs=1,\n",
      "              objective='binary:logistic', random_state=0, reg_alpha=0,\n",
      "              reg_lambda=1, scale_pos_weight=1, subsample=1,\n",
      "              tree_method='gpu_hist', verbose=1) 0.7508500106381422\n",
      "---\n",
      "\n",
      "                    XGB-gpu-grid time:  11053.56941\n",
      "Searched over 540 parameters\n"
     ]
    }
   ],
   "source": [
    "mode = \"gpu-grid\"\n",
    "\n",
    "with timed(\"XGB-\"+mode):\n",
    "    res, results = do_HPO(model_gpu_xgb,\n",
    "                                   params_xgb,\n",
    "                                   cuml_accuracy_scorer,\n",
    "                                   X_train,\n",
    "                                   y_cpu,\n",
    "                                   mode=mode)\n",
    "print(\"Searched over {} parameters\".format(len(results.cv_results_['mean_test_score'])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gpu-grid model accuracy: 0.7526272535324097\n"
     ]
    }
   ],
   "source": [
    "print_acc(res, X_train, y_cpu, X_test, y_test_cpu, mode_str=mode)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Improved performance\n",
    "\n",
    "There's a 4% improvement in the performance.\n",
    "\n",
    "We notice that performing grid search and random search yields similar performance improvements. We will stick to performing Random Search for the rest of the notebook with the other classifiers with the assumption that there will not be a major difference in performance if the ranges are large enough. \n",
    "\n",
    "In order to compare the running time, we will also run the CPU version of this later in the notebook to see what timing improvements look like."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing the Search\n",
    "\n",
    "Let's plot some graphs to get an understanding how the parameters affect the accuracy. The code for these plots are included in `cuml/experimental/hyperopt_utils/plotting_utils.py`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Mean/Std of test scores\n",
    "\n",
    "We fix all parameters except one for each of these graphs and plot the effect the parameter has on the mean test score with the error bar indicating the standard deviation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "from cuml.experimental.hyperopt_utils import plotting_utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 34900 characters]\n",
      "text/plain": [
       "<Figure size 1440x360 with 5 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plotting_utils.plot_search_results(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Heatmaps \n",
    "   - Between parameter pairs (we can do a combination of all possible pairs, but only one are shown in this notebook) \n",
    "   - This gives a visual representation of how the pair affect the test score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 19476 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "df_gridsearch = pd.DataFrame(results.cv_results_)\n",
    "plotting_utils.plot_heatmap(df_gridsearch, \"param_max_depth\", \"param_n_estimators\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CPU version\n",
    "\n",
    "Now, running the model in CPU version to notice the difference in performance in terms of time. The main difference you would notice is that the `tree_method` is set to `hist` instead of `gpu_hist`. The interface remains the same and we can even make use of the same parameters that we defined earlier (in fact, this is necessary for a fair comparison). \n",
    "\n",
    "Note: Remember the `data_fraction` flag from earlier and we will ensure we only run the CPU version if we are using less than 1% of the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cpu-random selected\n",
      "Best clf and score XGBClassifier(alpha=0.03162277660168379, base_score=0.5, booster='gbtree',\n",
      "              colsample_bylevel=1, colsample_bynode=1, colsample_bytree=1,\n",
      "              gamma=0, gpu_id=-1, learning_rate=0.05, max_delta_step=0,\n",
      "              max_depth=12, min_child_weight=8, missing=nan, n_estimators=1000,\n",
      "              n_jobs=1, objective='binary:logistic', random_state=0,\n",
      "              reg_alpha=0, reg_lambda=1, scale_pos_weight=1, subsample=1,\n",
      "              tree_method='hist', verbose=1) 0.7507897727272728\n",
      "---\n",
      "\n",
      "                  XGB-cpu-random time:  27485.06421\n",
      "Searched over 100 parameters\n",
      "cpu-random model accuracy: 0.7510272860527039\n"
     ]
    }
   ],
   "source": [
    "if data_fraction <= 0.1:\n",
    "    model_cpu_xgb = xgb.XGBClassifier(tree_method='hist')\n",
    "\n",
    "    mode = \"cpu-random\"\n",
    "    with timed(\"XGB-\" + mode):\n",
    "        res, results = do_HPO(model_cpu_xgb,\n",
    "                   params_xgb,\n",
    "                   'accuracy',\n",
    "                   X_cpu,\n",
    "                   y_cpu,\n",
    "                   mode=mode,\n",
    "                   n_iter=N_ITER)\n",
    "    \n",
    "    print(\"Searched over {} parameters\".format(len(results.cv_results_['mean_test_score'])))\n",
    "    \n",
    "    print_acc(res , X_cpu, y_cpu, X_test_cpu, y_test_cpu,\n",
    "              mode_str=mode)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember the GPU version of RandomizedSearch ran in 2124 seconds.\n",
    "\n",
    "And the CPU version took 27485 seconds over the same number of parameters.\n",
    "\n",
    "#### We notice a <b> 13x speed-up </b>. \n",
    "\n",
    "For the rest of the notebook, we will run just the GPU version. You are free to swap out of the mode to CPU options to see what happens with different models.\n",
    "\n",
    "#### Running CPU Modes\n",
    "\n",
    "Each cell that calls this `do_HPO` function has a `mode` variable defining what task we want it to perform. \n",
    "\n",
    "1. For CPU experiments, use `cpu-random` or `cpu-grid` depending of which experiment you wish to run. \n",
    "\n",
    "2. If you make use of `cpu-random`, specify `n_iter` for number of parameter combinations to consider. It defaults to just 10.\n",
    "\n",
    "3. For CPU modes, the scorer we use is the default 'accuracy' metric.\n",
    "\n",
    "<b> DO NOT</b> use `cuml_scorer` or the `accuracy_wrapper_scorer` both of these are used for the GPU options.\n",
    "\n",
    "#### Runtime observations\n",
    "These are the observed runtimes for different runs with different combinations. All of these are run on 4 Tesla T4 GPUs 16GB RAM. You will notice that some of the combinations do not have a CPU runtime comparison, this was intentionally done to avoid the redundancy in experiments and prevent the kernel crashing.\n",
    "\n",
    "| data_fraction | parameters | GPU Time |CPU Time\n",
    "|---------------|------------| ------|---|\n",
    "|0.01|100|258.98545|1131.57301|\n",
    "| |180|380.23315|2079.6394|\n",
    "||240|616.53354|-|\n",
    "|0.1|100|705.42218|9832.40570|\n",
    "||270|1998.64815|-|\n",
    "|0.5|250|4909.64941|-|\n",
    "|0.7|180|5696.12868|-|\n",
    "\n",
    "We are searching over a small space because we want to compare with the CPU version, it is recommended that you change the following values to observe changes in performance:\n",
    "\n",
    "1. `data_fraction` >= 0.5\n",
    "2. Increase parameter ranges to search over 1000 parameters and `N_ITER` >= 400"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### KNNClassifier\n",
    "\n",
    "We are going to now perform HPO with [cuml-KNNClassifier](https://docs.rapids.ai/api/cuml/stable/api.html#cuml.neighbors.KNeighborsClassifier). We will again note the default performance and notice what HPO does to improve this.\n",
    "\n",
    "We will run GridSearch for KNN because we are tuning just one parameter, which is `n_neighbors`, this defines the number of neighbors to consider for the classification.\n",
    "\n",
    "We plot a simple 2D plot to observe the effect of neighbors on accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Default accuracy 0.625322699546814\n"
     ]
    }
   ],
   "source": [
    "# KNN-Classifier\n",
    "model_knn_ = KNeighborsClassifier(n_neighbors=5)\n",
    "\n",
    "model_knn_.fit(X_train, y_train)\n",
    "print(\"Default accuracy {}\".format(accuracy_score(model_knn_.predict(X_test), y_test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gpu-grid selected\n",
      "Best clf and score KNeighborsClassifier(weights='uniform') 0.6498602271080017\n",
      "---\n",
      "\n",
      "                    KNN-gpu-grid time:  351.95597\n"
     ]
    }
   ],
   "source": [
    "model_knn = KNeighborsClassifier(n_neighbors=5)\n",
    "\n",
    "ks = [i for i in range(1, 40)]\n",
    "params_knn = {'n_neighbors': ks\n",
    "             }\n",
    "\n",
    "mode = \"gpu-grid\"\n",
    "with timed(\"KNN-\"+mode):\n",
    "    res, results = do_HPO(model_knn,\n",
    "               params_knn,\n",
    "               cuml_accuracy_scorer,\n",
    "               X_train,\n",
    "               y_cpu.astype('int32'),\n",
    "                mode=mode)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gpu-grid accuracy 0.6510545611381531\n"
     ]
    }
   ],
   "source": [
    "res.fit(X_train, y_train)\n",
    "print(\"{} accuracy {}\".format(mode, accuracy_score(res.predict(X_test), y_test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame(results.cv_results_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7fbffde94048>"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 22864 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import seaborn as sns; sns.set()\n",
    "import matplotlib.pyplot as plt\n",
    "sns.lineplot(x=\"param_n_neighbors\", y=\"mean_test_score\", data=df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We notice a 3% increase in the accuracy. \n",
    "\n",
    "\n",
    "## Conclusion and Next Steps\n",
    "\n",
    "We notice improvements in the performance for a really basic version of the GridSearch and RandomizedSearch on just 10% of the data. Generally, the more data we use, the better the model performs, so you are encouraged to try for larger data and broader range of parameters.\n",
    "\n",
    "This experiment can also be repeated with different classifiers and different ranges of parameters to notice how HPO can help improve the performance metric. In this example, we have chosen a basic metric - accuracy, but you can use more interesting metrics that help in determining the usefulness of a model. You can even send a list of parameters to the scoring function. This makes HPO really powerful, and it can add a significant boost to the model that we generate.\n",
    "\n",
    "\n",
    "#### Further Reading\n",
    "\n",
    "- [The 5 Classification Evaluation Metrics You Must Know](https://towardsdatascience.com/the-5-classification-evaluation-metrics-you-must-know-aa97784ff226)\n",
    "- [11 Important Model Evaluation Metrics for Machine Learning Everyone should know](https://www.analyticsvidhya.com/blog/2019/08/11-important-model-evaluation-error-metrics/)\n",
    "- [Algorithms for Hyper-Parameter Optimisation](http://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf)\n",
    "- [Forward and Reverse Gradient-Based Hyperparameter Optimization](http://proceedings.mlr.press/v70/franceschi17a/franceschi17a-supp.pdf)\n",
    "- [Practical Bayesian Optimization of Machine\n",
    "Learning Algorithms](http://papers.nips.cc/paper/4522-practical-bayesian-optimization-of-machine-learning-algorithms.pdf)\n",
    "- [Random Search for Hyper-Parameter Optimization](http://jmlr.csail.mit.edu/papers/volume13/bergstra12a/bergstra12a.pdf)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
