12 - Use Automated Machine Learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Use Automated Machine Learning\n",
    "\n",
    "There are many kinds of machine learning algorithm that you can use to train a model, and sometimes it's not easy to determine the most effective algorithm for your particular data and prediction requirements. Additionally, you can significantly affect the predictive performance of a model by preprocessing the training data, using techniques such as normalization, missing feature imputation, and others. In your quest to find the *best* model for your requirements, you may need to try many combinations of algorithms and preprocessing transformations; which takes a lot of time and compute resources.\n",
    "\n",
    "Azure Machine Learning enables you to automate the comparison of models trained using different algorithms and preprocessing options. You can use the visual interface in [Azure Machine Learning studio](https://ml/azure.com) or the SDK to leverage this capability. The SDK gives you greater control over the settings for the automated machine learning experiment, but the visual interface is easier to use.\n",
    "\n",
    "## Before you start\n",
    "\n",
    "In addition to the latest version of the **azureml-sdk** and **azureml-widgets** packages, you'll need the **azureml-train-automl** package to run the code in this notebook. Run the cell below to verify that it is installed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip show azureml-train-automl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Connect to your workspace\n",
    "\n",
    "With the required SDK packages installed, now you're ready to connect to your workspace.\n",
    "\n",
    "> **Note**: If you haven't already established an authenticated session with your Azure subscription, you'll be prompted to authenticate by clicking a link, entering an authentication code, and signing into Azure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import azureml.core\n",
    "from azureml.core import Workspace\n",
    "\n",
    "# Load the workspace from the saved config file\n",
    "ws = Workspace.from_config()\n",
    "print('Ready to use Azure ML {} to work with {}'.format(azureml.core.VERSION, ws.name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare data\n",
    "\n",
    "You don't need to create a training script for automated machine learning, but you do need to load the training data. In this case, you'll use a dataset containing details of diabetes patients, and then split this into two datasets: one for training, and another for model validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Dataset\n",
    "\n",
    "default_ds = ws.get_default_datastore()\n",
    "\n",
    "if 'diabetes dataset' not in ws.datasets:\n",
    "    default_ds.upload_files(files=['./data/diabetes.csv', './data/diabetes2.csv'], # Upload the diabetes csv files in /data\n",
    "                        target_path='diabetes-data/', # Put it in a folder path in the datastore\n",
    "                        overwrite=True, # Replace existing files of the same name\n",
    "                        show_progress=True)\n",
    "\n",
    "    #Create a tabular dataset from the path on the datastore (this may take a short while)\n",
    "    tab_data_set = Dataset.Tabular.from_delimited_files(path=(default_ds, 'diabetes-data/*.csv'))\n",
    "\n",
    "    # Register the tabular dataset\n",
    "    try:\n",
    "        tab_data_set = tab_data_set.register(workspace=ws, \n",
    "                                name='diabetes dataset',\n",
    "                                description='diabetes data',\n",
    "                                tags = {'format':'CSV'},\n",
    "                                create_new_version=True)\n",
    "        print('Dataset registered.')\n",
    "    except Exception as ex:\n",
    "        print(ex)\n",
    "else:\n",
    "    print('Dataset already registered.')\n",
    "\n",
    "\n",
    "# Split the dataset into training and validation subsets\n",
    "diabetes_ds = ws.datasets.get(\"diabetes dataset\")\n",
    "train_ds, test_ds = diabetes_ds.random_split(percentage=0.7, seed=123)\n",
    "print(\"Data ready!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare compute\n",
    "\n",
    "One of the benefits of cloud compute is that it scales on-demand, enabling you to provision enough compute resources to process multiple child-runs of an automated machine learning experiment in parallel.\n",
    "\n",
    "Use the following code to specify an Azure Machine Learning compute cluster (it will be created if it doesn't already exist)..\n",
    "\n",
    "> **Important**: Change *your-compute-cluster* to the name of your compute cluster in the code below before running it! Cluster names must be globally unique names between 2 to 16 characters in length. Valid characters are letters, digits, and the - character."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.compute import ComputeTarget, AmlCompute\n",
    "from azureml.core.compute_target import ComputeTargetException\n",
    "\n",
    "cluster_name = \"your-compute-cluster\"\n",
    "\n",
    "try:\n",
    "    # Check for existing compute target\n",
    "    training_cluster = ComputeTarget(workspace=ws, name=cluster_name)\n",
    "    print('Found existing cluster, use it.')\n",
    "except ComputeTargetException:\n",
    "    # If it doesn't already exist, create it\n",
    "    try:\n",
    "        compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_DS11_V2', max_nodes=2)\n",
    "        training_cluster = ComputeTarget.create(ws, cluster_name, compute_config)\n",
    "        training_cluster.wait_for_completion(show_output=True)\n",
    "    except Exception as ex:\n",
    "        print(ex)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Note**: Compute instances and clusters are based on standard Azure virtual machine images. For this exercise, the *Standard_DS11_v2* image is recommended to achieve the optimal balance of cost and performance. If your subscription has a quota that does not include this image, choose an alternative image; but bear in mind that a larger image may incur higher cost and a smaller image may not be sufficient to complete the tasks. Alternatively, ask your Azure administrator to extend your quota.\n",
    "\n",
    "## Configure automated machine learning\n",
    "\n",
    "Now you're ready to configure the automated machine learning experiment.\n",
    "\n",
    "One of the most important configuration settings is the metric by which model performance should be evaluated. You can retrieve a list of the metrics that are calculated by automated machine learning for a particular type of model task (classification or regression) like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import azureml.train.automl.utilities as automl_utils\n",
    "\n",
    "for metric in automl_utils.get_primary_metrics('classification'):\n",
    "    print(metric)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Having decided the metric you want to optimize (in this example, *AUC_weighted*), you can configure the automated machine learning run. To do this, you'll need an AutoML configuration that specifies the target metric as well as options like the data to use, how many combinations to try, and so on.\n",
    "\n",
    "> **Note**: In this example, you'll restrict the experiment to 4 iterations to reduce the amount of time taken. In reality, you'd likely try many more iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.train.automl import AutoMLConfig\n",
    "\n",
    "automl_config = AutoMLConfig(name='Automated ML Experiment',\n",
    "                             task='classification',\n",
    "                             compute_target=training_cluster,\n",
    "                             training_data = train_ds,\n",
    "                             validation_data = test_ds,\n",
    "                             label_column_name='Diabetic',\n",
    "                             iterations=4,\n",
    "                             primary_metric = 'AUC_weighted',\n",
    "                             max_concurrent_iterations=2,\n",
    "                             featurization='auto'\n",
    "                             )\n",
    "\n",
    "print(\"Ready for Auto ML run.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run an automated machine learning experiment\n",
    "\n",
    "OK, you're ready to go. Let's run the automated machine learning experiment.\n",
    "\n",
    "> **Note**: This may take some time!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from azureml.core.experiment import Experiment\n",
    "from azureml.widgets import RunDetails\n",
    "\n",
    "print('Submitting Auto ML experiment...')\n",
    "automl_experiment = Experiment(ws, 'mslearn-diabetes-automl-sdk')\n",
    "automl_run = automl_experiment.submit(automl_config)\n",
    "RunDetails(automl_run).show()\n",
    "automl_run.wait_for_completion(show_output=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## View child run details\n",
    "\n",
    "When the experiment has completed, view the output in the widget, and click the run that produced the best result to see its details.\n",
    "\n",
    "Then click the link to view the experiment details in Azure Machine Learning studio and view the overall experiment details before viewing the details for the individual run that produced the best result. There's lots of information here about the performance of the model generated.\n",
    "\n",
    "You can also retrieve all of the child runs and view their metrics using the SDK:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for run in automl_run.get_children():\n",
    "    print('Run ID', run.id)\n",
    "    for metric in run.get_metrics():\n",
    "        print('\\t', run.get_metrics(metric))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get the best run\n",
    "\n",
    "You can retrieve the best-performing run, and view its details like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_run, fitted_model = automl_run.get_output()\n",
    "print(best_run)\n",
    "print('\\nBest Model Definition:')\n",
    "print(fitted_model)\n",
    "print('\\nBest Run Transformations:')\n",
    "for step in fitted_model.named_steps:\n",
    "    print(step)\n",
    "print('\\nBest Run Metrics:')\n",
    "best_run_metrics = best_run.get_metrics()\n",
    "for metric_name in best_run_metrics:\n",
    "    metric = best_run_metrics[metric_name]\n",
    "    print(metric_name, metric)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, having found the best performing model, you can register it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Model\n",
    "\n",
    "# Register model\n",
    "best_run.register_model(model_path='outputs/model.pkl', model_name='diabetes_model',\n",
    "                        tags={'Training context':'Auto ML'},\n",
    "                        properties={'AUC': best_run_metrics['AUC_weighted'], 'Accuracy': best_run_metrics['accuracy']})\n",
    "\n",
    "# List registered models\n",
    "for model in Model.list(ws):\n",
    "    print(model.name, 'version:', model.version)\n",
    "    for tag_name in model.tags:\n",
    "        tag = model.tags[tag_name]\n",
    "        print ('\\t',tag_name, ':', tag)\n",
    "    for prop_name in model.properties:\n",
    "        prop = model.properties[prop_name]\n",
    "        print ('\\t',prop_name, ':', prop)\n",
    "    print('\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **More Information**: For more information Automated Machine Learning, see the [Azure ML documentation](https://docs.microsoft.com/azure/machine-learning/how-to-configure-auto-train)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6 - AzureML",
   "language": "python",
   "name": "python3-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}