IndiaFlood Navigator_ Detailed Risk Analysis and Forecasting - P4 Snap Random Forest Classifier - Notebook.ipynb

{"metadata": {"kernelspec": {"name": "python3", "display_name": "Python 3.11", "language": "python"}, "language_info": {"name": "python", "version": "3.11.9", "mimetype": "text/x-python", "codemirror_mode": {"name": "ipython", "version": 3}, "pygments_lexer": "ipython3", "nbconvert_exporter": "python", "file_extension": ".py"}}, "nbformat_minor": 4, "nbformat": 4, "cells": [{"cell_type": "markdown", "source": "![svg image](data:image/svg+xml,%3Csvg%20version%3D%221.1%22%20id%3D%22Layer_1%22%20xmlns%3D%22http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg%22%20xmlns%3Axlink%3D%22http%3A%2F%2Fwww.w3.org%2F1999%2Fxlink%22%20x%3D%220px%22%20y%3D%220px%22%0A%09%20viewBox%3D%220%200%201796%20100%22%20style%3D%22enable-background%3Anew%200%200%201796%20100%3B%22%20xml%3Aspace%3D%22preserve%22%3E%0A%3Cstyle%20type%3D%22text%2Fcss%22%3E%0A%09.st0%7Bfill-rule%3Aevenodd%3Bclip-rule%3Aevenodd%3Bfill%3Aurl%28%23SVGID_1_%29%3B%7D%0A%09.st1%7Bclip-path%3Aurl%28%23SVGID_00000178915391776407424850000007341318225791887491_%29%3B%7D%0A%09.st2%7Bfill%3Aurl%28%23SVGID_00000072959074018647863330000000566853838299270530_%29%3B%7D%0A%09.st3%7Bclip-path%3Aurl%28%23SVGID_00000088832798360344817880000014420072439252153242_%29%3B%7D%0A%09.st4%7Bfill%3Aurl%28%23SVGID_00000115487599576311285860000000799671087605023907_%29%3B%7D%0A%09.st5%7Bfill%3Anone%3Bstroke%3A%23FFFFFF%3Bstroke-width%3A2%3Bstroke-miterlimit%3A10%3B%7D%0A%09.st6%7Bfill%3Anone%3Bstroke%3A%23FFFFFF%3Bstroke-width%3A1.5%3Bstroke-miterlimit%3A10%3B%7D%0A%09.st7%7Bopacity%3A0.2%3Bfill%3Aurl%28%23SVGID_00000045580851177316760270000016912490179557547437_%29%3Benable-background%3Anew%20%20%20%20%3B%7D%0A%09.st8%7Bfill%3A%23FFFFFF%3B%7D%0A%09.st9%7Bfont-family%3A%27IBMPlexSans-Medium%27%3B%7D%0A%09.st10%7Bfont-size%3A32px%3B%7D%0A%09.st11%7Bfont-family%3A%27IBMPlexSans%27%3B%7D%0A%09.st12%7Bfill%3A%233D3D3D%3B%7D%0A%09.st13%7Bfill%3A%23939598%3B%7D%0A%3C%2Fstyle%3E%0A%3Crect%20width%3D%221796%22%20height%3D%22100%22%2F%3E%0A%3ClinearGradient%20id%3D%22SVGID_1_%22%20gradientUnits%3D%22userSpaceOnUse%22%20x1%3D%2242.8625%22%20y1%3D%221027.998%22%20x2%3D%2279.71%22%20y2%3D%221027.998%22%20gradientTransform%3D%22matrix%281%200%200%201%200%20-978%29%22%3E%0A%09%3Cstop%20%20offset%3D%220%22%20style%3D%22stop-color%3A%23AB74FF%22%2F%3E%0A%09%3Cstop%20%20offset%3D%220.21%22%20style%3D%22stop-color%3A%23866AFF%22%2F%3E%0A%09%3Cstop%20%20offset%3D%220.75%22%20style%3D%22stop-color%3A%232A4FFF%22%2F%3E%0A%09%3Cstop%20%20offset%3D%221%22%20style%3D%22stop-color%3A%230645FF%22%2F%3E%0A%3C%2FlinearGradient%3E%0A%3Cpath%20class%3D%22st0%22%20d%3D%22M52.4%2C45.9c0-2.3%2C1.8-4.1%2C4.1-4.1s4.1%2C1.8%2C4.1%2C4.1S58.8%2C50%2C56.5%2C50l0%2C0c-2.2%2C0.1-4-1.7-4.1-3.9%0A%09C52.4%2C46%2C52.4%2C46%2C52.4%2C45.9z%20M77.5%2C52.5c-0.8-1.1-1.4-2.3-1.9-3.5c1.2-4.5%2C0.7-8.6-1.8-11.9c-2.9-3.8-8.2-6-14.5-6.1%0A%09c-4.5-0.1-8.8%2C1.7-12%2C4.8c-3%2C3-4.6%2C7.2-4.5%2C11.5c-0.1%2C2.9%2C0.9%2C5.8%2C2.7%2C8.1c0.8%2C0.8%2C1.3%2C1.9%2C1.4%2C3v4.5c-0.8%2C0.5-1.4%2C1.3-1.4%2C2.3%0A%09c0.2%2C1.5%2C1.5%2C2.6%2C3%2C2.4c1.2-0.2%2C2.2-1.1%2C2.4-2.4c0-1-0.5-1.9-1.4-2.3v-4.5c0-2-1-3.3-1.9-4.6c-1.5-1.9-2.2-4.2-2.1-6.5%0A%09c0-3.5%2C1.4-6.9%2C3.8-9.4c2.7-2.7%2C6.3-4.1%2C10-4.1c5.5%2C0%2C9.8%2C1.9%2C12.1%2C5c2%2C2.8%2C2.5%2C6.3%2C1.4%2C9.6c-0.4%2C1.2%2C0.6%2C2.7%2C2.3%2C5.6%0A%09c0.6%2C0.9%2C1.2%2C1.9%2C1.6%2C2.9c-0.9%2C0.7-2%2C1.2-3.1%2C1.5c-0.5%2C0.4-0.7%2C0.9-0.8%2C1.5V65c0%2C0.4-0.1%2C0.8-0.4%2C1.1c-0.3%2C0.2-0.7%2C0.3-1.1%2C0.3%0A%09c-1.6-0.3-3.4-0.7-5.2-1.1v-4.8c0.8-0.5%2C1.4-1.4%2C1.4-2.3c0-1.5-1.2-2.7-2.7-2.7s-2.7%2C1.2-2.7%2C2.7c0%2C1%2C0.5%2C1.9%2C1.4%2C2.3v4.1%0A%09c-0.4-0.1-0.7-0.1-1.1-0.3c-4.5-1.1-4.5-2.6-4.5-3.4v-8.3c3.2-0.7%2C5.4-3.5%2C5.5-6.7c-0.1-3.8-3.3-6.7-7.1-6.6c-3.6%2C0.1-6.4%2C3-6.6%2C6.6%0A%09c0%2C3.2%2C2.3%2C6%2C5.5%2C6.7v8.3c0%2C2%2C0.7%2C4.6%2C6.6%2C6.1c3%2C0.8%2C6%2C1.5%2C9.1%2C1.9c0.3%2C0%2C0.6%2C0.1%2C0.8%2C0.1c1%2C0%2C1.9-0.3%2C2.6-1%0A%09c0.9-0.8%2C1.4-1.9%2C1.4-3.1v-4.5c2-0.8%2C4.1-2%2C4.1-3.7C79.7%2C55.9%2C79%2C54.6%2C77.5%2C52.5z%22%2F%3E%0A%3Cg%3E%0A%09%3Cg%3E%0A%09%09%3Cdefs%3E%0A%09%09%09%3Cpath%20id%3D%22SVGID_00000103988023533243019020000015870306665051389114_%22%20d%3D%22M52.4%2C45.9c0-2.3%2C1.8-4.1%2C4.1-4.1s4.1%2C1.8%2C4.1%2C4.1%0A%09%09%09%09S58.8%2C50%2C56.5%2C50l0%2C0c-2.2%2C0.1-4-1.7-4.1-3.9C52.4%2C46%2C52.4%2C46%2C52.4%2C45.9z%20M77.5%2C52.5c-0.8-1.1-1.4-2.3-1.9-3.5%0A%09%09%09%09c1.2-4.5%2C0.7-8.6-1.8-11.9c-2.9-3.8-8.2-6-14.5-6.1c-4.5-0.1-8.8%2C1.7-12%2C4.8c-3%2C3-4.6%2C7.2-4.5%2C11.5c-0.1%2C2.9%2C0.9%2C5.8%2C2.7%2C8.1%0A%09%09%09%09c0.8%2C0.8%2C1.3%2C1.9%2C1.4%2C3v4.5c-0.8%2C0.5-1.4%2C1.3-1.4%2C2.3c0.2%2C1.5%2C1.5%2C2.6%2C3%2C2.4c1.2-0.2%2C2.2-1.1%2C2.4-2.4c0-1-0.5-1.9-1.4-2.3v-4.5%0A%09%09%09%09c0-2-1-3.3-1.9-4.6c-1.5-1.9-2.2-4.2-2.1-6.5c0-3.5%2C1.4-6.9%2C3.8-9.4c2.7-2.7%2C6.3-4.1%2C10-4.1c5.5%2C0%2C9.8%2C1.9%2C12.1%2C5%0A%09%09%09%09c2%2C2.8%2C2.5%2C6.3%2C1.4%2C9.6c-0.4%2C1.2%2C0.6%2C2.7%2C2.3%2C5.6c0.6%2C0.9%2C1.2%2C1.9%2C1.6%2C2.9c-0.9%2C0.7-2%2C1.2-3.1%2C1.5c-0.5%2C0.4-0.7%2C0.9-0.8%2C1.5V65%0A%09%09%09%09c0%2C0.4-0.1%2C0.8-0.4%2C1.1c-0.3%2C0.2-0.7%2C0.3-1.1%2C0.3c-1.6-0.3-3.4-0.7-5.2-1.1v-4.8c0.8-0.5%2C1.4-1.4%2C1.4-2.3c0-1.5-1.2-2.7-2.7-2.7%0A%09%09%09%09s-2.7%2C1.2-2.7%2C2.7c0%2C1%2C0.5%2C1.9%2C1.4%2C2.3v4.1c-0.4-0.1-0.7-0.1-1.1-0.3c-4.5-1.1-4.5-2.6-4.5-3.4v-8.3c3.2-0.7%2C5.4-3.5%2C5.5-6.7%0A%09%09%09%09c-0.1-3.8-3.3-6.7-7.1-6.6c-3.6%2C0.1-6.4%2C3-6.6%2C6.6c0%2C3.2%2C2.3%2C6%2C5.5%2C6.7v8.3c0%2C2%2C0.7%2C4.6%2C6.6%2C6.1c3%2C0.8%2C6%2C1.5%2C9.1%2C1.9%0A%09%09%09%09c0.3%2C0%2C0.6%2C0.1%2C0.8%2C0.1c1%2C0%2C1.9-0.3%2C2.6-1c0.9-0.8%2C1.4-1.9%2C1.4-3.1v-4.5c2-0.8%2C4.1-2%2C4.1-3.7C79.7%2C55.9%2C79%2C54.6%2C77.5%2C52.5z%22%2F%3E%0A%09%09%3C%2Fdefs%3E%0A%09%09%3CclipPath%20id%3D%22SVGID_00000163075222211376611200000003176213376109670824_%22%3E%0A%09%09%09%3Cuse%20xlink%3Ahref%3D%22%23SVGID_00000103988023533243019020000015870306665051389114_%22%20%20style%3D%22overflow%3Avisible%3B%22%2F%3E%0A%09%09%3C%2FclipPath%3E%0A%09%09%3Cg%20style%3D%22clip-path%3Aurl%28%23SVGID_00000163075222211376611200000003176213376109670824_%29%3B%22%3E%0A%09%09%09%0A%09%09%09%09%3ClinearGradient%20id%3D%22SVGID_00000111891139058751766020000017047336746994893708_%22%20gradientUnits%3D%22userSpaceOnUse%22%20x1%3D%22-1341.4%22%20y1%3D%22805.5%22%20x2%3D%22449.7%22%20y2%3D%22805.5%22%20gradientTransform%3D%22matrix%281%200%200%201%200%20-978%29%22%3E%0A%09%09%09%09%3Cstop%20%20offset%3D%220%22%20style%3D%22stop-color%3A%23AB74FF%22%2F%3E%0A%09%09%09%09%3Cstop%20%20offset%3D%220.21%22%20style%3D%22stop-color%3A%23866AFF%22%2F%3E%0A%09%09%09%09%3Cstop%20%20offset%3D%220.75%22%20style%3D%22stop-color%3A%232A4FFF%22%2F%3E%0A%09%09%09%09%3Cstop%20%20offset%3D%221%22%20style%3D%22stop-color%3A%230645FF%22%2F%3E%0A%09%09%09%3C%2FlinearGradient%3E%0A%09%09%09%0A%09%09%09%09%3Crect%20x%3D%22-1341.4%22%20y%3D%22-876.9%22%20style%3D%22fill%3Aurl%28%23SVGID_00000111891139058751766020000017047336746994893708_%29%3B%22%20width%3D%221791.1%22%20height%3D%221408.8%22%2F%3E%0A%09%09%09%3Cg%3E%0A%09%09%09%09%3Cg%3E%0A%09%09%09%09%09%3Cdefs%3E%0A%09%09%09%09%09%09%0A%09%09%09%09%09%09%09%3Crect%20id%3D%22SVGID_00000103231572956238640970000011034158940870041505_%22%20x%3D%22-1341.4%22%20y%3D%22-876.9%22%20width%3D%221791.1%22%20height%3D%221408.8%22%2F%3E%0A%09%09%09%09%09%3C%2Fdefs%3E%0A%09%09%09%09%09%3CclipPath%20id%3D%22SVGID_00000076562667476129641040000017034118924293963927_%22%3E%0A%09%09%09%09%09%09%3Cuse%20xlink%3Ahref%3D%22%23SVGID_00000103231572956238640970000011034158940870041505_%22%20%20style%3D%22overflow%3Avisible%3B%22%2F%3E%0A%09%09%09%09%09%3C%2FclipPath%3E%0A%09%09%09%09%09%3Cg%20style%3D%22clip-path%3Aurl%28%23SVGID_00000076562667476129641040000017034118924293963927_%29%3B%22%3E%0A%09%09%09%09%09%09%0A%09%09%09%09%09%09%09%3ClinearGradient%20id%3D%22SVGID_00000002353593499038243250000000421055474946009746_%22%20gradientUnits%3D%22userSpaceOnUse%22%20x1%3D%2236%22%20y1%3D%221028.05%22%20x2%3D%2286.5%22%20y2%3D%221028.05%22%20gradientTransform%3D%22matrix%281%200%200%201%200%20-978%29%22%3E%0A%09%09%09%09%09%09%09%3Cstop%20%20offset%3D%220%22%20style%3D%22stop-color%3A%23AB74FF%22%2F%3E%0A%09%09%09%09%09%09%09%3Cstop%20%20offset%3D%220.21%22%20style%3D%22stop-color%3A%23866AFF%22%2F%3E%0A%09%09%09%09%09%09%09%3Cstop%20%20offset%3D%220.75%22%20style%3D%22stop-color%3A%232A4FFF%22%2F%3E%0A%09%09%09%09%09%09%09%3Cstop%20%20offset%3D%221%22%20style%3D%22stop-color%3A%230645FF%22%2F%3E%0A%09%09%09%09%09%09%3C%2FlinearGradient%3E%0A%09%09%09%09%09%09%0A%09%09%09%09%09%09%09%3Crect%20x%3D%2236%22%20y%3D%2224.1%22%20style%3D%22fill%3Aurl%28%23SVGID_00000002353593499038243250000000421055474946009746_%29%3B%22%20width%3D%2250.5%22%20height%3D%2251.9%22%2F%3E%0A%09%09%09%09%09%3C%2Fg%3E%0A%09%09%09%09%3C%2Fg%3E%0A%09%09%09%3C%2Fg%3E%0A%09%09%3C%2Fg%3E%0A%09%3C%2Fg%3E%0A%3C%2Fg%3E%0A%3Ccircle%20class%3D%22st5%22%20cx%3D%2256.5%22%20cy%3D%2245.9%22%20r%3D%225.4%22%2F%3E%0A%3Ccircle%20class%3D%22st6%22%20cx%3D%2248.3%22%20cy%3D%2265%22%20r%3D%221.6%22%2F%3E%0A%3Ccircle%20class%3D%22st6%22%20cx%3D%2264.8%22%20cy%3D%2258.2%22%20r%3D%221.6%22%2F%3E%0A%3ClinearGradient%20id%3D%22SVGID_00000124860236446931535290000008328801365185342639_%22%20gradientUnits%3D%22userSpaceOnUse%22%20x1%3D%22773.8%22%20y1%3D%221028%22%20x2%3D%221796%22%20y2%3D%221028%22%20gradientTransform%3D%22matrix%281%200%200%201%200%20-978%29%22%3E%0A%09%3Cstop%20%20offset%3D%220%22%20style%3D%22stop-color%3A%23161616%22%2F%3E%0A%09%3Cstop%20%20offset%3D%220.52%22%20style%3D%22stop-color%3A%23AB74FF%22%2F%3E%0A%09%3Cstop%20%20offset%3D%220.62%22%20style%3D%22stop-color%3A%23866AFF%22%2F%3E%0A%09%3Cstop%20%20offset%3D%220.88%22%20style%3D%22stop-color%3A%232A4FFF%22%2F%3E%0A%09%3Cstop%20%20offset%3D%221%22%20style%3D%22stop-color%3A%230645FF%22%2F%3E%0A%3C%2FlinearGradient%3E%0A%3Crect%20x%3D%22773.8%22%20style%3D%22opacity%3A0.2%3Bfill%3Aurl%28%23SVGID_00000124860236446931535290000008328801365185342639_%29%3Benable-background%3Anew%20%20%20%20%3B%22%20width%3D%221022.2%22%20height%3D%22100%22%2F%3E%0A%3Ctext%20transform%3D%22matrix%281%200%200%201%201498.5952%2059.46%29%22%20class%3D%22st8%20st9%20st10%22%3EPipeline%20notebook%3C%2Ftext%3E%0A%3Ctext%20transform%3D%22matrix%281%200%200%201%20101.02%2059.33%29%22%20class%3D%22st8%20st11%20st10%22%3EAutoAI%3C%2Ftext%3E%0A%3Crect%20x%3D%22231.1%22%20y%3D%2234%22%20class%3D%22st12%22%20width%3D%221%22%20height%3D%2232%22%2F%3E%0A%3Ctext%20transform%3D%22matrix%281%200%200%201%20256.29%2059.66%29%22%20class%3D%22st13%20st11%20st10%22%3EPart%20of%20IBM%20Watson%C2%AE%20Studio%3C%2Ftext%3E%0A%3C%2Fsvg%3E)\n# Pipeline 4 Notebook - AutoAI Notebook v2.0.3\n\nConsider these tips for working with an auto-generated notebook:\n- Notebook code generated using AutoAI will execute successfully. If you modify the notebook, we cannot guarantee it will run successfully.\n- This pipeline is optimized for the original data set. The pipeline might fail or produce sub-optimal results if used with different data.  If you want to use a different data set, consider retraining the AutoAI experiment to generate a new pipeline. For more information, see <a href=\"https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/autoai-notebook.html\">Cloud Platform</a>. \n- Before modifying the pipeline or trying to re-fit the pipeline, consider that the code converts dataframes to numpy arrays before fitting the pipeline (a current restriction of the preprocessor pipeline).\n", "metadata": {}}, {"cell_type": "markdown", "source": "<a id=\"content\"></a>\n## Notebook content\n\nThis notebook contains a Scikit-learn representation of AutoAI pipeline. This notebook introduces commands for retrieving data, training the model, and testing the model. \n\nSome familiarity with Python is helpful. This notebook uses Python 3.11 and scikit-learn 1.3.", "metadata": {}}, {"cell_type": "markdown", "source": "## Notebook goals\n\n-  Scikit-learn pipeline definition\n-  Pipeline training \n-  Pipeline evaluation\n\n## Contents\n\nThis notebook contains the following parts:\n\n**[Setup](#setup)**<br>\n&nbsp;&nbsp;[Package installation](#install)<br>\n&nbsp;&nbsp;[AutoAI experiment metadata](#variables_definition)<br>\n&nbsp;&nbsp;[Watson Machine Learning connection](#connection)<br>\n**[Pipeline inspection](#inspection)** <br>\n&nbsp;&nbsp;[Read training data](#read)<br>\n&nbsp;&nbsp;[Train and test data split](#split)<br>\n&nbsp;&nbsp;[Create pipeline](#preview_model_to_python_code)<br>\n&nbsp;&nbsp;[Train pipeline model](#train)<br>\n&nbsp;&nbsp;[Test pipeline model](#test_model)<br>\n**[Store the model](#saving)**<br>\n**[Summary and next steps](#summary_and_next_steps)**<br>\n**[Copyrights](#copyrights)**", "metadata": {"pycharm": {"name": "#%% md\n"}}}, {"cell_type": "markdown", "source": "<a id=\"setup\"></a>\n# Setup", "metadata": {}}, {"cell_type": "markdown", "source": "<a id=\"install\"></a>\n## Package installation\nBefore you use the sample code in this notebook, install the following packages:\n - ibm-watsonx-ai,\n - autoai-libs,\n - scikit-learn,\n - snapml\n", "metadata": {}}, {"cell_type": "code", "source": "!pip install ibm-watsonx-ai==1.0.* | tail -n 1\n!pip install autoai-libs~=2.0 | tail -n 1\n!pip install scikit-learn==1.3.* | tail -n 1\n!pip install -U lale~=0.8.0 | tail -n 1\n!pip install snapml==1.14.* | tail -n 1", "metadata": {"execution": {"iopub.execute_input": "2020-10-12T14:00:45.009458Z", "iopub.status.busy": "2020-10-12T14:00:45.007968Z", "iopub.status.idle": "2020-10-12T14:00:46.037702Z", "shell.execute_reply": "2020-10-12T14:00:46.038270Z"}, "pycharm": {"name": "#%%\n"}, "scrolled": true, "id": "a4f5b549-917c-41bc-a897-a2fbf5240473"}, "outputs": [], "execution_count": null}, {"cell_type": "markdown", "source": "Filter warnings for this notebook.", "metadata": {}}, {"cell_type": "code", "source": "import warnings\n\nwarnings.filterwarnings('ignore')", "metadata": {"id": "b8365b09-e756-42c0-9ab2-1ba0f1e7dace"}, "outputs": [], "execution_count": null}, {"cell_type": "markdown", "source": "<a id=\"variables_definition\"></a>\n## AutoAI experiment metadata\nThe following cell contains the training data connection details.  \n**Note**: The connection might contain authorization credentials, so be careful when sharing the notebook.", "metadata": {}}, {"cell_type": "code", "source": "from ibm_watsonx_ai.helpers import DataConnection\nfrom ibm_watsonx_ai.helpers import ContainerLocation\n\ntraining_data_references = [\n    DataConnection(\n        data_asset_id='652aaac5-5866-475f-b526-7222de928f20'\n    ),\n]\ntraining_result_reference = DataConnection(\n    location=ContainerLocation(\n        path='auto_ml/0f8891f2-a13a-4f71-8de7-7c1f426d03fa/wml_data/b8f5d853-ee1e-4652-b775-31c442c13b38/data/automl',\n        model_location='auto_ml/0f8891f2-a13a-4f71-8de7-7c1f426d03fa/wml_data/b8f5d853-ee1e-4652-b775-31c442c13b38/data/automl/model.zip',\n        training_status='auto_ml/0f8891f2-a13a-4f71-8de7-7c1f426d03fa/wml_data/b8f5d853-ee1e-4652-b775-31c442c13b38/training-status.json'\n    )\n)", "metadata": {"execution": {"iopub.execute_input": "2020-10-12T14:00:49.797633Z", "iopub.status.busy": "2020-10-12T14:00:49.796778Z", "iopub.status.idle": "2020-10-12T14:00:57.182715Z", "shell.execute_reply": "2020-10-12T14:00:57.183132Z"}, "pycharm": {"is_executing": true}, "id": "2f7b185c-b213-4d89-b854-a260b086580e"}, "outputs": [], "execution_count": null}, {"cell_type": "markdown", "source": "The following cell contains input parameters provided to run the AutoAI experiment in Watson Studio.", "metadata": {}}, {"cell_type": "code", "source": "experiment_metadata = dict(\n    prediction_type='binary',\n    prediction_column='Flood Occurred',\n    holdout_size=0.1,\n    scoring='accuracy',\n    csv_separator=',',\n    random_state=33,\n    max_number_of_estimators=2,\n    training_data_references=training_data_references,\n    training_result_reference=training_result_reference,\n    deployment_url='https://us-south.ml.cloud.ibm.com',\n    project_id='20de206e-8205-4443-b8b4-fc2fec8523d4',\n    positive_label=1,\n    drop_duplicates=True,\n    include_batched_ensemble_estimators=[],\n    feature_selector_mode='auto'\n)", "metadata": {"execution": {"iopub.execute_input": "2020-10-12T14:00:57.187305Z", "iopub.status.busy": "2020-10-12T14:00:57.186602Z", "iopub.status.idle": "2020-10-12T14:00:57.188392Z", "shell.execute_reply": "2020-10-12T14:00:57.188878Z"}, "pycharm": {"name": "#%%\n"}, "id": "fa1f8c2d-6442-4e20-bb2a-d49f7ee20d0c"}, "outputs": [], "execution_count": null}, {"cell_type": "markdown", "source": "## Set `n_jobs` parameter to the number of available CPUs", "metadata": {}}, {"cell_type": "code", "source": "import os, ast\nCPU_NUMBER = 4\nif 'RUNTIME_HARDWARE_SPEC' in os.environ:\n    CPU_NUMBER = int(ast.literal_eval(os.environ['RUNTIME_HARDWARE_SPEC'])['num_cpu'])", "metadata": {"id": "337a9ce7-caeb-4d73-bc9c-a6242217091d"}, "outputs": [], "execution_count": null}, {"cell_type": "markdown", "source": "<a id=\"connection\"></a>\n## Watson Machine Learning connection\n\nThis cell defines the credentials required to work with the Watson Machine Learning service.\n\n**Action**: Provide the IBM Cloud apikey, For details, see [documentation](https://cloud.ibm.com/docs/account?topic=account-userapikey).", "metadata": {}}, {"cell_type": "code", "source": "api_key = 'PUT_YOUR_APIKEY_HERE'", "metadata": {"id": "d00529e9-6c30-4fa3-9d5c-eff976c4d987"}, "outputs": [], "execution_count": null}, {"cell_type": "code", "source": "from ibm_watsonx_ai import Credentials\n\ncredentials = Credentials(\n    api_key=api_key,\n    url=experiment_metadata['deployment_url']\n)", "metadata": {"id": "cc5debc2-5e43-4717-95de-2d0d2e97cd1f"}, "outputs": [], "execution_count": null}, {"cell_type": "code", "source": "from ibm_watsonx_ai import APIClient\n\nclient = APIClient(credentials)\n\nif 'space_id' in experiment_metadata:\n    client.set.default_space(experiment_metadata['space_id'])\nelse:\n    client.set.default_project(experiment_metadata['project_id'])\n    \ntraining_data_references[0].set_client(client)", "metadata": {"id": "939de93b-11ad-470b-8f62-69bffc460af8"}, "outputs": [], "execution_count": null}, {"cell_type": "markdown", "source": "<a id=\"inspection\"></a>\n# Pipeline inspection", "metadata": {}}, {"cell_type": "markdown", "source": "<a id=\"read\"></a>\n## Read training data\n\nRetrieve training dataset from AutoAI experiment as pandas DataFrame.\n\n**Note**: If reading data results in an error, provide data as Pandas DataFrame object, for example, reading .CSV file with `pandas.read_csv()`. \n\nIt may be necessary to use methods for initial data pre-processing like: e.g. `DataFrame.dropna()`, `DataFrame.drop_duplicates()`, `DataFrame.sample()`.\n", "metadata": {}}, {"cell_type": "code", "source": "X_train, X_test, y_train, y_test = training_data_references[0].read(experiment_metadata=experiment_metadata, with_holdout_split=True, use_flight=True)", "metadata": {"execution": {"iopub.execute_input": "2020-10-12T14:01:16.076169Z", "iopub.status.busy": "2020-10-12T14:01:16.075589Z", "iopub.status.idle": "2020-10-12T14:01:19.190233Z", "shell.execute_reply": "2020-10-12T14:01:19.190807Z"}, "pycharm": {"is_executing": true, "name": "#%%\n"}, "id": "4df027bb-40ec-4118-832e-bd0d5c53842d"}, "outputs": [], "execution_count": null}, {"cell_type": "markdown", "source": "<a id=\"preview_model_to_python_code\"></a>\n## Create pipeline\nIn the next cell, you can find the Scikit-learn definition of the selected AutoAI pipeline.", "metadata": {"pycharm": {"name": "#%% md\n"}}}, {"cell_type": "markdown", "source": "#### Import statements.", "metadata": {"pycharm": {"name": "#%% md\n"}}}, {"cell_type": "code", "source": "from autoai_libs.transformers.exportable import NumpyColumnSelector\nfrom autoai_libs.transformers.exportable import CompressStrings\nfrom autoai_libs.transformers.exportable import NumpyReplaceMissingValues\nfrom autoai_libs.transformers.exportable import NumpyReplaceUnknownValues\nfrom autoai_libs.transformers.exportable import boolean2float\nfrom autoai_libs.transformers.exportable import CatImputer\nfrom autoai_libs.transformers.exportable import CatEncoder\nimport numpy as np\nfrom autoai_libs.transformers.exportable import float32_transform\nfrom sklearn.pipeline import make_pipeline\nfrom autoai_libs.transformers.exportable import FloatStr2Float\nfrom autoai_libs.transformers.exportable import NumImputer\nfrom autoai_libs.transformers.exportable import OptStandardScaler\nfrom sklearn.pipeline import make_union\nfrom autoai_libs.transformers.exportable import NumpyPermuteArray\nfrom autoai_libs.cognito.transforms.transform_utils import TA1\nimport autoai_libs.utils.fc_methods\nfrom autoai_libs.cognito.transforms.transform_utils import FS1\nfrom autoai_libs.cognito.transforms.transform_utils import TGen\nimport autoai_libs.cognito.transforms.transform_extras\nfrom snapml import SnapRandomForestClassifier", "metadata": {"pycharm": {"is_executing": true, "name": "#%%\n"}, "id": "8c354f5c-b30a-4b50-af26-c2795af58328"}, "outputs": [], "execution_count": null}, {"cell_type": "markdown", "source": "#### Pre-processing & Estimator.", "metadata": {}}, {"cell_type": "code", "source": "numpy_column_selector_0 = NumpyColumnSelector(columns=[8, 9, 11, 12])\ncompress_strings = CompressStrings(\n    compress_type=\"hash\",\n    dtypes_list=[\"char_str\", \"char_str\", \"float_int_num\", \"float_int_num\"],\n    missing_values_reference_list=[\"\", \"-\", \"?\", float(\"nan\")],\n    misslist_list=[[], [], [], []],\n)\nnumpy_replace_missing_values_0 = NumpyReplaceMissingValues(\n    filling_values=float(\"nan\"), missing_values=[]\n)\nnumpy_replace_unknown_values = NumpyReplaceUnknownValues(\n    filling_values=float(\"nan\"),\n    filling_values_list=[float(\"nan\"), float(\"nan\"), 100001, 100001],\n    missing_values_reference_list=[\"\", \"-\", \"?\", float(\"nan\")],\n)\ncat_imputer = CatImputer(\n    missing_values=float(\"nan\"),\n    sklearn_version_family=\"1\",\n    strategy=\"most_frequent\",\n)\ncat_encoder = CatEncoder(\n    dtype=np.float64,\n    handle_unknown=\"error\",\n    sklearn_version_family=\"1\",\n    encoding=\"ordinal\",\n    categories=\"auto\",\n)\npipeline_0 = make_pipeline(\n    numpy_column_selector_0,\n    compress_strings,\n    numpy_replace_missing_values_0,\n    numpy_replace_unknown_values,\n    boolean2float(),\n    cat_imputer,\n    cat_encoder,\n    float32_transform(),\n)\nnumpy_column_selector_1 = NumpyColumnSelector(\n    columns=[0, 1, 2, 3, 4, 5, 6, 7, 10]\n)\nfloat_str2_float = FloatStr2Float(\n    dtypes_list=[\n        \"float_num\", \"float_num\", \"float_num\", \"float_num\", \"float_num\",\n        \"float_num\", \"float_num\", \"float_num\", \"float_num\",\n    ],\n    missing_values_reference_list=[],\n)\nnumpy_replace_missing_values_1 = NumpyReplaceMissingValues(\n    filling_values=float(\"nan\"), missing_values=[]\n)\nnum_imputer = NumImputer(missing_values=float(\"nan\"), strategy=\"median\")\nopt_standard_scaler = OptStandardScaler(use_scaler_flag=False)\npipeline_1 = make_pipeline(\n    numpy_column_selector_1,\n    float_str2_float,\n    numpy_replace_missing_values_1,\n    num_imputer,\n    opt_standard_scaler,\n    float32_transform(),\n)\nunion = make_union(pipeline_0, pipeline_1)\nnumpy_permute_array = NumpyPermuteArray(\n    axis=0, permutation_indices=[8, 9, 11, 12, 0, 1, 2, 3, 4, 5, 6, 7, 10]\n)\nta1 = TA1(\n    fun=np.rint,\n    name=\"round\",\n    datatypes=[\"numeric\"],\n    feat_constraints=[autoai_libs.utils.fc_methods.is_not_categorical],\n    col_names=[\n        \"Latitude\", \"Longitude\", \"Rainfall (mm)\", \"Temperature (\u00b0C)\",\n        \"Humidity (%)\", \"River Discharge (m\u00b3/s)\", \"Water Level (m)\",\n        \"Elevation (m)\", \"Land Cover\", \"Soil Type\", \"Population Density\",\n        \"Infrastructure\", \"Historical Floods\",\n    ],\n    col_dtypes=[\n        np.dtype(\"float32\"), np.dtype(\"float32\"), np.dtype(\"float32\"),\n        np.dtype(\"float32\"), np.dtype(\"float32\"), np.dtype(\"float32\"),\n        np.dtype(\"float32\"), np.dtype(\"float32\"), np.dtype(\"float32\"),\n        np.dtype(\"float32\"), np.dtype(\"float32\"), np.dtype(\"float32\"),\n        np.dtype(\"float32\"),\n    ],\n)\nfs1_0 = FS1(\n    cols_ids_must_keep=range(0, 13),\n    additional_col_count_to_keep=12,\n    ptype=\"classification\",\n)\nt_gen = TGen(\n    fun=autoai_libs.cognito.transforms.transform_extras.NXOR,\n    name=\"nxor\",\n    arg_count=2,\n    datatypes_list=[[\"numeric\"], [\"numeric\"]],\n    feat_constraints_list=[\n        [autoai_libs.utils.fc_methods.is_not_categorical],\n        [autoai_libs.utils.fc_methods.is_not_categorical],\n    ],\n    col_names=[\n        \"Latitude\", \"Longitude\", \"Rainfall (mm)\", \"Temperature (\u00b0C)\",\n        \"Humidity (%)\", \"River Discharge (m\u00b3/s)\", \"Water Level (m)\",\n        \"Elevation (m)\", \"Land Cover\", \"Soil Type\", \"Population Density\",\n        \"Infrastructure\", \"Historical Floods\", \"round(Latitude)\",\n        \"round(Longitude)\", \"round(Rainfall (mm))\", \"round(Temperature (\u00b0C))\",\n        \"round(Humidity (%))\", \"round(River Discharge (m\u00b3/s))\",\n        \"round(Water Level (m))\", \"round(Elevation (m))\",\n        \"round(Population Density)\",\n    ],\n    col_dtypes=[\n        np.dtype(\"float32\"), np.dtype(\"float32\"), np.dtype(\"float32\"),\n        np.dtype(\"float32\"), np.dtype(\"float32\"), np.dtype(\"float32\"),\n        np.dtype(\"float32\"), np.dtype(\"float32\"), np.dtype(\"float32\"),\n        np.dtype(\"float32\"), np.dtype(\"float32\"), np.dtype(\"float32\"),\n        np.dtype(\"float32\"), np.dtype(\"float32\"), np.dtype(\"float32\"),\n        np.dtype(\"float32\"), np.dtype(\"float32\"), np.dtype(\"float32\"),\n        np.dtype(\"float32\"), np.dtype(\"float32\"), np.dtype(\"float32\"),\n        np.dtype(\"float32\"),\n    ],\n)\nfs1_1 = FS1(\n    cols_ids_must_keep=range(0, 13),\n    additional_col_count_to_keep=12,\n    ptype=\"classification\",\n)\nsnap_random_forest_classifier = SnapRandomForestClassifier(\n    compress_trees=True,\n    gpu_ids=np.array([0], dtype=np.uint32),\n    max_depth=3,\n    max_features=0.18797982711615588,\n    n_estimators=61,\n    n_jobs=CPU_NUMBER,\n    random_state=33,\n)\n", "metadata": {"pycharm": {"is_executing": true, "name": "#%%\n"}, "id": "21426b1c-7e2f-4412-849c-7e9b48571a41"}, "outputs": [], "execution_count": null}, {"cell_type": "markdown", "source": "#### Pipeline.", "metadata": {}}, {"cell_type": "code", "source": "pipeline = make_pipeline(\n    union,\n    numpy_permute_array,\n    ta1,\n    fs1_0,\n    t_gen,\n    fs1_1,\n    snap_random_forest_classifier,\n)", "metadata": {"pycharm": {"is_executing": true, "name": "#%%\n"}, "id": "896c1b69-f822-49fc-a667-39f14fda37c7"}, "outputs": [], "execution_count": null}, {"cell_type": "markdown", "source": "<a id=\"train\"></a>\n## Train pipeline model\n", "metadata": {}}, {"cell_type": "markdown", "source": "### Define scorer from the optimization metric\nThis cell constructs the cell scorer based on the experiment metadata.", "metadata": {"pycharm": {"name": "#%% md\n"}}}, {"cell_type": "code", "source": "from sklearn.metrics import get_scorer\n\nscorer = get_scorer(experiment_metadata['scoring'])", "metadata": {"pycharm": {"is_executing": true, "name": "#%%\n"}, "id": "b81bd989-c3d7-4f38-812c-e2eda22e9c53"}, "outputs": [], "execution_count": null}, {"cell_type": "markdown", "source": "<a id=\"test_model\"></a>\n### Fit pipeline model\nIn this cell, the pipeline is fitted.", "metadata": {"pycharm": {"name": "#%% md\n"}}}, {"cell_type": "code", "source": "pipeline.fit(X_train.values, y_train.values.ravel());", "metadata": {"execution": {"iopub.execute_input": "2020-10-12T14:01:19.291734Z", "iopub.status.busy": "2020-10-12T14:01:19.244735Z", "iopub.status.idle": "2020-10-12T14:01:19.338461Z", "shell.execute_reply": "2020-10-12T14:01:19.338958Z"}, "pycharm": {"is_executing": true, "name": "#%%\n"}, "scrolled": true, "id": "8aa8f95e-0234-456b-9dea-446f39c7dfc0"}, "outputs": [], "execution_count": null}, {"cell_type": "markdown", "source": "<a id=\"test_model\"></a>\n## Test pipeline model", "metadata": {}}, {"cell_type": "markdown", "source": "Score the fitted pipeline with the generated scorer using the holdout dataset.", "metadata": {"pycharm": {"name": "#%% md\n"}}}, {"cell_type": "code", "source": "score = scorer(pipeline, X_test.values, y_test.values)\nprint(score)", "metadata": {"execution": {"iopub.execute_input": "2020-10-12T14:02:03.910267Z", "iopub.status.busy": "2020-10-12T14:02:03.909710Z", "iopub.status.idle": "2020-10-12T14:02:03.914154Z", "shell.execute_reply": "2020-10-12T14:02:03.914727Z"}, "pycharm": {"is_executing": true, "name": "#%%\n"}, "id": "c6e5b99d-f92b-4ba2-a323-5c08980f1368"}, "outputs": [], "execution_count": null}, {"cell_type": "code", "source": "pipeline.predict(X_test.values[:5])", "metadata": {"pycharm": {"name": "#%%\n"}, "id": "a177354b-8245-4d80-ba63-83696b1d97e7"}, "outputs": [], "execution_count": null}, {"cell_type": "markdown", "source": "<a id=\"saving\"></a>\n## Store the model\n\nIn this section you will learn how to store the trained model.", "metadata": {}}, {"cell_type": "code", "source": "model_metadata = {\n    client.repository.ModelMetaNames.NAME: 'P4 - Pretrained AutoAI pipeline'\n}\n\nstored_model_details = client.repository.store_model(model=pipeline, meta_props=model_metadata, experiment_metadata=experiment_metadata)", "metadata": {"id": "85b87466-611b-457f-8301-8542a3f99376"}, "outputs": [], "execution_count": null}, {"cell_type": "markdown", "source": "Inspect the stored model details.", "metadata": {}}, {"cell_type": "code", "source": "stored_model_details", "metadata": {"id": "092ec816-0943-4dd5-bfa4-0ca255880bcf"}, "outputs": [], "execution_count": null}, {"cell_type": "markdown", "source": "<a id=\"deployment\"></a>\n## Create online deployment", "metadata": {"jupyter": {"outputs_hidden": false}, "collapsed": false}}, {"cell_type": "markdown", "source": "You can use the commands below to promote the model to space and create online deployment (web service).\n\n<a id=\"working_spaces\"></a>\n### Working with spaces\n\nIn this section you will specify a deployment space for organizing the assets for deploying and scoring the model. If you do not have an existing space, you can use [Deployment Spaces Dashboard](/ml-runtime/spaces?context=icp4data) to create a new space, following these steps:\n\n- Click **New Deployment Space**.\n- Create an empty space.\n- Select Cloud Object Storage.\n- Select Watson Machine Learning instance and press **Create**.\n- Copy `space_id` and paste it below.\n\n**Tip**: You can also use the API to prepare the space for your work. Learn more [here](https://github.com/IBM/watson-machine-learning-samples/blob/master/cloud/notebooks/python_sdk/instance-management/Space%20management.ipynb).\n\n**Action**: Assign or update space ID below.", "metadata": {"jupyter": {"outputs_hidden": false}, "collapsed": false}}, {"cell_type": "raw", "source": "space_id = \"PUT_YOUR_SPACE_ID_HERE\"\n\nmodel_id = client.spaces.promote(asset_id=stored_model_details[\"metadata\"][\"id\"], source_project_id=experiment_metadata[\"project_id\"], target_space_id=space_id)", "metadata": {"jupyter": {"outputs_hidden": false}, "collapsed": false}}, {"cell_type": "markdown", "source": "#### Prepare online deployment", "metadata": {"jupyter": {"outputs_hidden": false}, "collapsed": false}}, {"cell_type": "raw", "source": "client.set.default_space(space_id)\n\ndeploy_meta = {\n        client.deployments.ConfigurationMetaNames.NAME: \"Incrementally trained AutoAI pipeline\",\n        client.deployments.ConfigurationMetaNames.ONLINE: {},\n    }\n\ndeployment_details = client.deployments.create(artifact_uid=model_id, meta_props=deploy_meta)\ndeployment_id = client.deployments.get_id(deployment_details)", "metadata": {"jupyter": {"outputs_hidden": false}, "collapsed": false}}, {"cell_type": "markdown", "source": "#### Test online deployment", "metadata": {"jupyter": {"outputs_hidden": false}, "collapsed": false}}, {"cell_type": "raw", "source": "import pandas as pd\n\nscoring_payload = {\n    \"input_data\": [{\n        'values': pd.DataFrame(X_test[:5])\n    }]\n}\n\nclient.deployments.score(deployment_id, scoring_payload)", "metadata": {"jupyter": {"outputs_hidden": false}, "collapsed": false}}, {"cell_type": "markdown", "source": "<a id=\"cleanup\"></a>\n### Deleting deployment\nYou can delete the existing deployment by calling the `client.deployments.delete(deployment_id)` command.\nTo list the existing web services, use `client.deployments.list()`.", "metadata": {"jupyter": {"outputs_hidden": false}, "collapsed": false}}, {"cell_type": "markdown", "source": "<a id=\"summary_and_next_steps\"></a>\n# Summary and next steps\nYou successfully completed this notebook!\nYou learned how to use AutoAI pipeline definition to train the model.\nCheck out our [Online Documentation](https://www.ibm.com/cloud/watson-studio/autoai) for more samples, tutorials, documentation, how-tos, and blog posts.", "metadata": {}}, {"cell_type": "markdown", "source": "<a id=\"copyrights\"></a>\n### Copyrights\n\nLicensed Materials - Copyright \u00a9 2024 IBM. This notebook and its source code are released under the terms of the ILAN License. Use, duplication disclosure restricted by GSA ADP Schedule Contract with IBM Corp.\n\n**Note:** The auto-generated notebooks are subject to the International License Agreement for Non-Warranted Programs (or equivalent) and License Information document for Watson Studio Auto-generated Notebook (License Terms), such agreements located in the link below. Specifically, the Source Components and Sample Materials clause included in the License Information document for Watson Studio Auto-generated Notebook applies to the auto-generated notebooks.  \n\nBy downloading, copying, accessing, or otherwise using the materials, you agree to the <a href=\"https://www14.software.ibm.com/cgi-bin/weblap/lap.pl?li_formnum=L-AMCU-BYC7LF\">License Terms</a>\n\n___", "metadata": {}}]}