From 91855ad7262059450a09efce7faab74461b30922 Mon Sep 17 00:00:00 2001 From: data hound Date: Sat, 23 Jan 2021 13:45:16 +0530 Subject: [PATCH 1/3] Fix typo in DataTransformers.ipynb Section 2 - MultiInputTransformer and not MultiOutputTransformer --- .pre-commit-config.yaml | 24 +- notebooks/DataTransformers.ipynb | 2738 +++++++++++++++--------------- 2 files changed, 1381 insertions(+), 1381 deletions(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index e80f5374e..c8552eb7f 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -1,12 +1,12 @@ -repos: - - repo: https://github.com/psf/black - rev: 19.10b0 - hooks: - - id: black - - - repo: https://github.com/timothycrosley/isort - rev: 5.3.2 - hooks: - - id: isort - additional_dependencies: [toml] - exclude: ^.*/?setup\.py$ +repos: + - repo: https://github.com/psf/black + rev: 19.10b0 + hooks: + - id: black + + - repo: https://github.com/timothycrosley/isort + rev: 5.3.2 + hooks: + - id: isort + additional_dependencies: [toml] + exclude: ^.*/?setup\.py$ diff --git a/notebooks/DataTransformers.ipynb b/notebooks/DataTransformers.ipynb index 8221ee37a..c4b07a7ec 100644 --- a/notebooks/DataTransformers.ipynb +++ b/notebooks/DataTransformers.ipynb @@ -1,1388 +1,1388 @@ { - "nbformat": 4, - "nbformat_minor": 0, - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.8.5-final" - }, + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "z22BE9uhvoxO" + }, + "source": [ + "# Data Transformers" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "hapoJed-voxP" + }, + "source": [ + "Keras support many types of input and output data formats, including:\n", + "\n", + "* Multiple inputs\n", + "* Multiple outputs\n", + "* Higher-dimensional tensors\n", + "\n", + "In this notebook, we explore how to reconcile this functionality with the sklearn ecosystem via SciKeras Data Transformer interface.\n", + "\n", + "\n", + "\n", + "
\n", + "\n", + " Run in Google Colab \n", + "\n", + "View source on GitHub
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "iT-ibpi7voxQ" + }, + "source": [ + "### Table of contents" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ekJWKPFMvoxR" + }, + "source": [ + "* [Data transformer interface](#Data-transformer-interface)\n", + " * [get_metadata method](#get_metadata-method)\n", + "* [1. Multiple outputs](#1-1-multiple-outputs)\n", + " * [1.1 Define Keras Model](#1-1-define-keras-model)\n", + " * [1.2 Define output transformer](1-2-define-output-transformer)\n", + " * [1.3 Test classifier](#1-3-test-classifier)\n", + "* [2. Multiple inputs](#2-multiple-inputs)\n", + " * [2.1 Define Keras Model](#2-1-define-keras-model)\n", + " * [2.2 Define input transformer](#2-2-define-input-transformer)\n", + " * [2.3 Test regressor](#2-3-test-regressor)\n", + "* [3. Multidimensional inputs with MNIST dataset](#3-multidimensional-inputs-with-MNIST-dataset)\n", + " * [3.1 Define Keras Model](#3-1-define-keras-model)\n", + " * [3.2 Define transformer](#3-2-define-transformer)\n", + " * [3.3 Test classifier](#3-3-test-classifier)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "6avb3GBQDQyG" + }, + "source": [ + "Install SciKeras" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "qCcyTjVkvoxR" + }, + "outputs": [], + "source": [ + "!python -m pip install scikeras" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "EZveNcetDQyL" + }, + "source": [ + "Silence TensorFlow warnings to keep output succint." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "id": "ekNmO_GPDQyL" + }, + "outputs": [], + "source": [ + "import warnings\n", + "from tensorflow import get_logger\n", + "get_logger().setLevel('ERROR')\n", + "warnings.filterwarnings(\"ignore\", message=\"Setting the random state for TF\")" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": { + "id": "Sf4j-x4DvoxV" + }, + "outputs": [], + "source": [ + "import numpy as np\n", + "from scikeras.wrappers import KerasClassifier, KerasRegressor\n", + "from tensorflow import keras" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "hCuOBH8AvoxX" + }, + "source": [ + "## Data transformer interface" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "i3fAUKBUvoxY" + }, + "source": [ + "SciKeras enables advanced Keras use cases by providing an interface to convert sklearn compliant data to whatever format your Keras model requires within SciKeras, right before passing said data to the Keras model.\n", + "\n", + "This interface is implemented in the form of two sklearn transformers, one for the features (`X`) and one for the target (`y`). SciKeras loads these transformers via the `target_encoder` and `feature_encoder` methods.\n", + "\n", + "By default, SciKeras implements `target_encoder` for both KerasClassifier and KerasRegressor to facilitate common types of tasks in sklearn. The default implementations are `scikeras.utils.transformers.ClassifierLabelEncoder` and `scikeras.utils.transformers.RegressorTargetEncoder` for KerasClassifier and KerasRegressor respectively. Information on the types of tasks that these default transformers are able to perform can be found in the [SciKeras docs](https://scikeras.readthedocs.io/en/latest/advanced.html#data-transformers).\n", + "\n", + "Below is an outline of the inner workings of the data transfomer interfaces to help understand when they are called:" + ] + }, + { + "cell_type": "code", + "execution_count": 39, + "metadata": { + "id": "QM74xeoe-1S-" + }, + "outputs": [], + "source": [ + "if False: # avoid executing pseudocode\n", + " from scikeras.utils.transformers import (\n", + " ClassifierLabelEncoder,\n", + " RegressorTargetEncoder,\n", + " )\n", + "\n", + "\n", + " class BaseWrapper:\n", + " def fit(self, X, y):\n", + " self.target_encoder_ = self.target_encoder\n", + " self.feature_encoder_ = self.feature_encoder\n", + " y = self.target_encoder_.fit_transform(y)\n", + " X = self.feature_encoder_.fit_transform(X)\n", + " self.model_.fit(X, y)\n", + " return self\n", + " \n", + " def predict(self, X):\n", + " X = self.feature_encoder_.transform(X)\n", + " y_pred = self.model_.predict(X)\n", + " return self.target_encoder_.inverse_transform(y_pred)\n", + "\n", + " class KerasClassifier(BaseWrapper):\n", + "\n", + " @property\n", + " def target_encoder(self):\n", + " return ClassifierLabelEncoder(loss=self.loss)\n", + " \n", + " def predict_proba(self, X):\n", + " X = self.feature_encoder_.transform(X)\n", + " y_pred = self.model_.predict(X)\n", + " return self.target_encoder_.inverse_transform(y_pred, return_proba=True)\n", + "\n", + "\n", + " class KerasRegressor(BaseWrapper):\n", + "\n", + " @property\n", + " def target_encoder(self):\n", + " return RegressorTargetEncoder()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Yg_0PtqhwNSo" + }, + "source": [ + "To substitute your own data transformation routine, you must subclass the wrappers and override one of the encoder defining functions. You will have access to all attributes of the wrappers, and you can pass these to your transformer, like we do above with `loss`." + ] + }, + { + "cell_type": "code", + "execution_count": 40, + "metadata": { + "id": "Lb8uZq_dIRUE" + }, + "outputs": [], + "source": [ + "from sklearn.base import BaseEstimator, TransformerMixin" + ] + }, + { + "cell_type": "code", + "execution_count": 41, + "metadata": { + "id": "1DeX_yH_wTIX" + }, + "outputs": [], + "source": [ + "if False: # avoid executing pseudocode\n", + "\n", + " class MultiOutputTransformer(BaseEstimator, TransformerMixin):\n", + " ...\n", + "\n", + "\n", + " class MultiOutputClassifier(KerasClassifier):\n", + "\n", + " @property\n", + " def target_encoder(self):\n", + " return MultiOutputTransformer(...)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "8pwBaT2Qi1U2" + }, + "source": [ + "### get_metadata method" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "N3FYPOwGi7t8" + }, + "source": [ + "SciKeras recognized an optional `get_metadata` on the transformers. `get_metadata` is expected to return a dicionary of with key strings and arbitrary values. SciKeras will set add these items to the wrappers namespace and make them available to your model building function via the `meta` keyword argument:" + ] + }, + { + "cell_type": "code", + "execution_count": 42, + "metadata": { + "id": "Nx2KNaRTi5aY" + }, + "outputs": [], + "source": [ + "if False: # avoid executing pseudocode\n", + "\n", + " class MultiOutputTransformer(BaseEstimator, TransformerMixin):\n", + " def get_metadata(self):\n", + " return {\"my_param_\": \"foobarbaz\"}\n", + "\n", + "\n", + " class MultiOutputClassifier(KerasClassifier):\n", + "\n", + " @property\n", + " def target_encoder(self):\n", + " return MultiOutputTransformer(...)\n", + "\n", + "\n", + " def get_model(meta):\n", + " print(f\"Got: {meta['my_param_']}\")\n", + "\n", + "\n", + " clf = MultiOutputClassifier(model=get_model)\n", + " clf.fit(X, y) # Got: foobarbaz\n", + " print(clf.my_param_) # foobarbaz" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "lNz5uY-v-1TQ" + }, + "source": [ + "## 1. Multiple Outputs" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "W5rrSfES-1TQ" + }, + "source": [ + "Keras makes it striaghtforward to define models with multiple outputs, that is a Model with multiple sets of fully-connected heads at the end of the network. This functionality is only available in the Functional Model and subclassed Model definition modes, and is not available when using Sequential.\n", + "\n", + "In practice, the main thing about Keras models with multiple outputs that you need to know as a SciKeras user is that Keras expects `X` or `y` to be a list of arrays/tensors, with one array/tensor for each input/output.\n", + "\n", + "Note that \"multiple outputs\" in Keras has a slightly different meaning than \"multiple outputs\" in sklearn. Many tasks that would be considered \"multiple output\" tasks in sklearn can be mapped to a single \"output\" in Keras with multiple units. This notebook specifically focuses on the cases that require multiple distinct Keras outputs." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "QTwqF_0UL9qA" + }, + "source": [ + "### 1.1 Define Keras Model" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "4gfMyrIZLjJB" + }, + "source": [ + "Here we define a simple perceptron that has two outputs, corresponding to one binary classification taks and one multiclass classification task. For example, one output might be \"image has car\" (binary) and the other might be \"color of car in image\" (multiclass)." + ] + }, + { + "cell_type": "code", + "execution_count": 43, + "metadata": { + "id": "_JgVNml3yEup" + }, + "outputs": [], + "source": [ + "def get_clf_model(meta, compile_kwargs):\n", + " inp = keras.layers.Input(shape=(meta[\"n_features_in_\"]))\n", + " x1 = keras.layers.Dense(100, activation=\"relu\")(inp)\n", + " out_bin = keras.layers.Dense(1, activation=\"sigmoid\")(x1)\n", + " out_cat = keras.layers.Dense(meta[\"n_classes_\"][1], activation=\"softmax\")(x1)\n", + " model = keras.Model(inputs=inp, outputs=[out_bin, out_cat])\n", + " model.compile(\n", + " loss=[\"binary_crossentropy\", \"sparse_categorical_crossentropy\"],\n", + " optimizer=compile_kwargs[\"optimizer\"]\n", + " )\n", + " return model" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "vWGp30MRk_PN" + }, + "source": [ + "Let's test that this model works with the kind of inputs and outputs we expect." + ] + }, + { + "cell_type": "code", + "execution_count": 44, + "metadata": { + "id": "azbKtjtd-1TR" + }, + "outputs": [], + "source": [ + "X = np.random.random(size=(100, 10))\n", + "y_bin = np.random.randint(0, 2, size=(100,))\n", + "y_cat = np.random.randint(0, 5, size=(100, ))\n", + "y = [y_bin, y_cat]\n", + "\n", + "# build mock meta\n", + "meta = {\n", + " \"n_features_in_\": 10,\n", + " \"n_classes_\": [2, 5] # note that we made this a list, one for each output\n", + "}\n", + "# build mock compile_kwargs\n", + "compile_kwargs = {\"optimizer\": \"sgd\"}\n", + "\n", + "model = get_clf_model(meta=meta, compile_kwargs=compile_kwargs)\n", + "\n", + "model.fit(X, y, verbose=0)\n", + "y_pred = model.predict(X)" + ] + }, + { + "cell_type": "code", + "execution_count": 45, + "metadata": { "colab": { - "name": "DataTransformers.ipynb", - "provenance": [], - "collapsed_sections": [] + "base_uri": "https://localhost:8080/", + "height": 51 + }, + "id": "C4xiVhKjqxzI", + "outputId": "8176d691-bfc7-40dc-bc02-9f92693d69ee" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[[0.4910586 ]\n", + " [0.47602195]]\n" + ] } + ], + "source": [ + "print(y_pred[0][:2, :])" + ] }, - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "id": "z22BE9uhvoxO" - }, - "source": [ - "# Data Transformers" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "hapoJed-voxP" - }, - "source": [ - "Keras support many types of input and output data formats, including:\n", - "\n", - "* Multiple inputs\n", - "* Multiple outputs\n", - "* Higher-dimensional tensors\n", - "\n", - "In this notebook, we explore how to reconcile this functionality with the sklearn ecosystem via SciKeras Data Transformer interface.\n", - "\n", - "\n", - "\n", - "
\n", - "\n", - " Run in Google Colab \n", - "\n", - "View source on GitHub
" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "iT-ibpi7voxQ" - }, - "source": [ - "### Table of contents" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "ekJWKPFMvoxR" - }, - "source": [ - "* [Data transformer interface](#Data-transformer-interface)\n", - " * [get_metadata method](#get_metadata-method)\n", - "* [1. Multiple outputs](#1-1-multiple-outputs)\n", - " * [1.1 Define Keras Model](#1-1-define-keras-model)\n", - " * [1.2 Define output transformer](1-2-define-output-transformer)\n", - " * [1.3 Test classifier](#1-3-test-classifier)\n", - "* [2. Multiple inputs](#2-multiple-inputs)\n", - " * [2.1 Define Keras Model](#2-1-define-keras-model)\n", - " * [2.2 Define input transformer](#2-2-define-input-transformer)\n", - " * [2.3 Test regressor](#2-3-test-regressor)\n", - "* [3. Multidimensional inputs with MNIST dataset](#3-multidimensional-inputs-with-MNIST-dataset)\n", - " * [3.1 Define Keras Model](#3-1-define-keras-model)\n", - " * [3.2 Define transformer](#3-2-define-transformer)\n", - " * [3.3 Test classifier](#3-3-test-classifier)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "6avb3GBQDQyG" - }, - "source": [ - "Install SciKeras" - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "qCcyTjVkvoxR" - }, - "source": [ - "!python -m pip install scikeras" - ], - "execution_count": null, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "EZveNcetDQyL" - }, - "source": [ - "Silence TensorFlow warnings to keep output succint." - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "ekNmO_GPDQyL" - }, - "source": [ - "import warnings\n", - "from tensorflow import get_logger\n", - "get_logger().setLevel('ERROR')\n", - "warnings.filterwarnings(\"ignore\", message=\"Setting the random state for TF\")" - ], - "execution_count": 3, - "outputs": [] - }, - { - "cell_type": "code", - "metadata": { - "id": "Sf4j-x4DvoxV" - }, - "source": [ - "import numpy as np\n", - "from scikeras.wrappers import KerasClassifier, KerasRegressor\n", - "from tensorflow import keras" - ], - "execution_count": 4, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "hCuOBH8AvoxX" - }, - "source": [ - "## Data transformer interface" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "i3fAUKBUvoxY" - }, - "source": [ - "SciKeras enables advanced Keras use cases by providing an interface to convert sklearn compliant data to whatever format your Keras model requires within SciKeras, right before passing said data to the Keras model.\n", - "\n", - "This interface is implemented in the form of two sklearn transformers, one for the features (`X`) and one for the target (`y`). SciKeras loads these transformers via the `target_encoder` and `feature_encoder` methods.\n", - "\n", - "By default, SciKeras implements `target_encoder` for both KerasClassifier and KerasRegressor to facilitate common types of tasks in sklearn. The default implementations are `scikeras.utils.transformers.ClassifierLabelEncoder` and `scikeras.utils.transformers.RegressorTargetEncoder` for KerasClassifier and KerasRegressor respectively. Information on the types of tasks that these default transformers are able to perform can be found in the [SciKeras docs](https://scikeras.readthedocs.io/en/latest/advanced.html#data-transformers).\n", - "\n", - "Below is an outline of the inner workings of the data transfomer interfaces to help understand when they are called:" - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "QM74xeoe-1S-" - }, - "source": [ - "if False: # avoid executing pseudocode\n", - " from scikeras.utils.transformers import (\n", - " ClassifierLabelEncoder,\n", - " RegressorTargetEncoder,\n", - " )\n", - "\n", - "\n", - " class BaseWrapper:\n", - " def fit(self, X, y):\n", - " self.target_encoder_ = self.target_encoder\n", - " self.feature_encoder_ = self.feature_encoder\n", - " y = self.target_encoder_.fit_transform(y)\n", - " X = self.feature_encoder_.fit_transform(X)\n", - " self.model_.fit(X, y)\n", - " return self\n", - " \n", - " def predict(self, X):\n", - " X = self.feature_encoder_.transform(X)\n", - " y_pred = self.model_.predict(X)\n", - " return self.target_encoder_.inverse_transform(y_pred)\n", - "\n", - " class KerasClassifier(BaseWrapper):\n", - "\n", - " @property\n", - " def target_encoder(self):\n", - " return ClassifierLabelEncoder(loss=self.loss)\n", - " \n", - " def predict_proba(self, X):\n", - " X = self.feature_encoder_.transform(X)\n", - " y_pred = self.model_.predict(X)\n", - " return self.target_encoder_.inverse_transform(y_pred, return_proba=True)\n", - "\n", - "\n", - " class KerasRegressor(BaseWrapper):\n", - "\n", - " @property\n", - " def target_encoder(self):\n", - " return RegressorTargetEncoder()" - ], - "execution_count": 39, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "Yg_0PtqhwNSo" - }, - "source": [ - "To substitute your own data transformation routine, you must subclass the wrappers and override one of the encoder defining functions. You will have access to all attributes of the wrappers, and you can pass these to your transformer, like we do above with `loss`." - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "Lb8uZq_dIRUE" - }, - "source": [ - "from sklearn.base import BaseEstimator, TransformerMixin" - ], - "execution_count": 40, - "outputs": [] - }, - { - "cell_type": "code", - "metadata": { - "id": "1DeX_yH_wTIX" - }, - "source": [ - "if False: # avoid executing pseudocode\n", - "\n", - " class MultiOutputTransformer(BaseEstimator, TransformerMixin):\n", - " ...\n", - "\n", - "\n", - " class MultiOutputClassifier(KerasClassifier):\n", - "\n", - " @property\n", - " def target_encoder(self):\n", - " return MultiOutputTransformer(...)" - ], - "execution_count": 41, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "8pwBaT2Qi1U2" - }, - "source": [ - "### get_metadata method" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "N3FYPOwGi7t8" - }, - "source": [ - "SciKeras recognized an optional `get_metadata` on the transformers. `get_metadata` is expected to return a dicionary of with key strings and arbitrary values. SciKeras will set add these items to the wrappers namespace and make them available to your model building function via the `meta` keyword argument:" - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "Nx2KNaRTi5aY" - }, - "source": [ - "if False: # avoid executing pseudocode\n", - "\n", - " class MultiOutputTransformer(BaseEstimator, TransformerMixin):\n", - " def get_metadata(self):\n", - " return {\"my_param_\": \"foobarbaz\"}\n", - "\n", - "\n", - " class MultiOutputClassifier(KerasClassifier):\n", - "\n", - " @property\n", - " def target_encoder(self):\n", - " return MultiOutputTransformer(...)\n", - "\n", - "\n", - " def get_model(meta):\n", - " print(f\"Got: {meta['my_param_']}\")\n", - "\n", - "\n", - " clf = MultiOutputClassifier(model=get_model)\n", - " clf.fit(X, y) # Got: foobarbaz\n", - " print(clf.my_param_) # foobarbaz" - ], - "execution_count": 42, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "lNz5uY-v-1TQ" - }, - "source": [ - "## 1. Multiple Outputs" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "W5rrSfES-1TQ" - }, - "source": [ - "Keras makes it striaghtforward to define models with multiple outputs, that is a Model with multiple sets of fully-connected heads at the end of the network. This functionality is only available in the Functional Model and subclassed Model definition modes, and is not available when using Sequential.\n", - "\n", - "In practice, the main thing about Keras models with multiple outputs that you need to know as a SciKeras user is that Keras expects `X` or `y` to be a list of arrays/tensors, with one array/tensor for each input/output.\n", - "\n", - "Note that \"multiple outputs\" in Keras has a slightly different meaning than \"multiple outputs\" in sklearn. Many tasks that would be considered \"multiple output\" tasks in sklearn can be mapped to a single \"output\" in Keras with multiple units. This notebook specifically focuses on the cases that require multiple distinct Keras outputs." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "QTwqF_0UL9qA" - }, - "source": [ - "### 1.1 Define Keras Model" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "4gfMyrIZLjJB" - }, - "source": [ - "Here we define a simple perceptron that has two outputs, corresponding to one binary classification taks and one multiclass classification task. For example, one output might be \"image has car\" (binary) and the other might be \"color of car in image\" (multiclass)." - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "_JgVNml3yEup" - }, - "source": [ - "def get_clf_model(meta, compile_kwargs):\n", - " inp = keras.layers.Input(shape=(meta[\"n_features_in_\"]))\n", - " x1 = keras.layers.Dense(100, activation=\"relu\")(inp)\n", - " out_bin = keras.layers.Dense(1, activation=\"sigmoid\")(x1)\n", - " out_cat = keras.layers.Dense(meta[\"n_classes_\"][1], activation=\"softmax\")(x1)\n", - " model = keras.Model(inputs=inp, outputs=[out_bin, out_cat])\n", - " model.compile(\n", - " loss=[\"binary_crossentropy\", \"sparse_categorical_crossentropy\"],\n", - " optimizer=compile_kwargs[\"optimizer\"]\n", - " )\n", - " return model" - ], - "execution_count": 43, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "vWGp30MRk_PN" - }, - "source": [ - "Let's test that this model works with the kind of inputs and outputs we expect." - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "azbKtjtd-1TR" - }, - "source": [ - "X = np.random.random(size=(100, 10))\n", - "y_bin = np.random.randint(0, 2, size=(100,))\n", - "y_cat = np.random.randint(0, 5, size=(100, ))\n", - "y = [y_bin, y_cat]\n", - "\n", - "# build mock meta\n", - "meta = {\n", - " \"n_features_in_\": 10,\n", - " \"n_classes_\": [2, 5] # note that we made this a list, one for each output\n", - "}\n", - "# build mock compile_kwargs\n", - "compile_kwargs = {\"optimizer\": \"sgd\"}\n", - "\n", - "model = get_clf_model(meta=meta, compile_kwargs=compile_kwargs)\n", - "\n", - "model.fit(X, y, verbose=0)\n", - "y_pred = model.predict(X)" - ], - "execution_count": 44, - "outputs": [] - }, - { - "cell_type": "code", - "metadata": { - "id": "C4xiVhKjqxzI", - "outputId": "8176d691-bfc7-40dc-bc02-9f92693d69ee", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 51 - } - }, - "source": [ - "print(y_pred[0][:2, :])" - ], - "execution_count": 45, - "outputs": [ - { - "output_type": "stream", - "text": [ - "[[0.4910586 ]\n", - " [0.47602195]]\n" - ], - "name": "stdout" - } - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "GyIq1xXEqyMY", - "outputId": "0cda9bba-4361-46ce-8b00-9669ab5b072e", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 51 - } - }, - "source": [ - "print(y_pred[1][:2, :])" - ], - "execution_count": 46, - "outputs": [ - { - "output_type": "stream", - "text": [ - "[[0.16950993 0.13774167 0.23052557 0.22972858 0.2324943 ]\n", - " [0.1702391 0.13294716 0.20055309 0.25442293 0.24183771]]\n" - ], - "name": "stdout" - } - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "mic4ByPGBwgA" - }, - "source": [ - "As you can see, our `predict` output is also a list of arrays, except it contains probabilities instead of the class predictions.\n", - "\n", - "Our data transormer's job will be to convert from a single numpy array (which is what the sklearn ecosystem works with) to the list of arrays and then back. Additionally, for classifiers, we will want to be able to convert probabilities to class predictions.\n", - "\n", - "We will structure our data on the sklearn side by column-stacking our list\n", - "of arrays. This works well in this case since we have the same number of datapoints in each array." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "2hN9nZiqMNJ9" - }, - "source": [ - "### 1.2 Define output Data Transformer" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "QNBuyxhcMPg8" - }, - "source": [ - "Let's go ahead and protoype this data transformer:" - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "y3E81vxrDGhM" - }, - "source": [ - "from typing import List\n", - "\n", - "from sklearn.base import BaseEstimator, TransformerMixin\n", - "from sklearn.preprocessing import LabelEncoder\n", - "\n", - "\n", - "class MultiOutputTransformer(BaseEstimator, TransformerMixin):\n", - "\n", - " def fit(self, y):\n", - " y_bin, y_cat = y[:, 0], y[:, 1]\n", - " # Create internal encoders to ensure labels are 0, 1, 2...\n", - " self.bin_encoder_ = LabelEncoder()\n", - " self.cat_encoder_ = LabelEncoder()\n", - " # Fit them to the input data\n", - " self.bin_encoder_.fit(y_bin)\n", - " self.cat_encoder_.fit(y_cat)\n", - " # Save the number of classes\n", - " self.n_classes_ = [\n", - " self.bin_encoder_.classes_.size,\n", - " self.cat_encoder_.classes_.size,\n", - " ]\n", - " # Save number of expected outputs in the Keras model\n", - " # SciKeras will automatically use this to do error-checking\n", - " self.n_outputs_expected_ = 2\n", - " return self\n", - "\n", - " def transform(self, y: np.ndarray) -> List[np.ndarray]:\n", - " y_bin, y_cat = y[:, 0], y[:, 1]\n", - " # Apply transformers to input array\n", - " y_bin = self.bin_encoder_.transform(y_bin)\n", - " y_cat = self.cat_encoder_.transform(y_cat)\n", - " # Split the data into a list\n", - " return [y_bin, y_cat]\n", - "\n", - " def inverse_transform(self, y: List[np.ndarray], return_proba: bool = False) -> np.ndarray:\n", - " y_pred_proba = y # rename for clarity, what Keras gives us are probs\n", - " if return_proba:\n", - " return np.column_stack(y_pred_proba, axis=1)\n", - " # Get class predictions from probabilities\n", - " y_pred_bin = (y_pred_proba[0] > 0.5).astype(int).reshape(-1, )\n", - " y_pred_cat = np.argmax(y_pred_proba[1], axis=1)\n", - " # Pass back through LabelEncoder\n", - " y_pred_bin = self.bin_encoder_.inverse_transform(y_pred_bin)\n", - " y_pred_cat = self.cat_encoder_.inverse_transform(y_pred_cat)\n", - " return np.column_stack([y_pred_bin, y_pred_cat])\n", - " \n", - " def get_metadata(self):\n", - " return {\n", - " \"n_classes_\": self.n_classes_,\n", - " \"n_outputs_expected_\": self.n_outputs_expected_,\n", - " }" - ], - "execution_count": 219, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "eKM37jy-HAX9" - }, - "source": [ - "Note that in addition to the usual `transform` and `inverse_transform` methods, we implement the `get_metadata` method to return the `n_classes_` attribute.\n", - "\n", - "Lets test our transformer with the same dataset we previoulsy used to test our model:" - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "yPqhenjpG_qy", - "outputId": "8bbbe68b-7380-431d-bd63-5d23ad7f8b64", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 51 - } - }, - "source": [ - "tf = MultiOutputTransformer()\n", - "\n", - "y_sklearn = np.column_stack(y)\n", - "\n", - "y_keras = tf.fit_transform(y_sklearn)\n", - "print(\"`y`, as will be passed to Keras:\")\n", - "print([y_keras[0][:4], y_keras[1][:4]])" - ], - "execution_count": 220, - "outputs": [ - { - "output_type": "stream", - "text": [ - "`y`, as will be passed to Keras:\n", - "[array([0, 1, 1, 1]), array([2, 4, 2, 0])]\n" - ], - "name": "stdout" - } - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "d_a6JqcDKkTg", - "outputId": "a9d4f3e8-3459-46dc-a8fe-f8ff138389cf", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 119 - } - }, - "source": [ - "y_pred_sklearn = tf.inverse_transform(y_pred)\n", - "print(\"`y_pred`, as will be returned to sklearn:\")\n", - "y_pred_sklearn[:5]" - ], - "execution_count": 221, - "outputs": [ - { - "output_type": "stream", - "text": [ - "`y_pred`, as will be returned to sklearn:\n" - ], - "name": "stdout" - }, - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "array([[0, 3],\n", - " [1, 3],\n", - " [0, 3],\n", - " [0, 3],\n", - " [0, 3]])" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 221 - } - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "9v0RG5ZjPgcy", - "outputId": "7f9a05c9-303b-446e-bca9-42c4f6391e06", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 34 - } - }, - "source": [ - "print(f\"metadata = {tf.get_metadata()}\")" - ], - "execution_count": 222, - "outputs": [ - { - "output_type": "stream", - "text": [ - "metadata = {'n_classes_': [2, 5], 'n_outputs_expected_': 2}\n" - ], - "name": "stdout" - } - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "ScbMdm-9LSx1" - }, - "source": [ - "Since this looks good, we move on to integrating our transformer into our classifier." - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "RKA2U8ANMhn9" - }, - "source": [ - "class MultiOutputClassifier(KerasClassifier):\n", - "\n", - " @property\n", - " def target_encoder(self):\n", - " return MultiOutputTransformer()" - ], - "execution_count": 223, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "1heA-eeTMp3t" - }, - "source": [ - "### 1.3 Test classifier" - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "jXbxL2mnNKkb" - }, - "source": [ - "from sklearn.preprocessing import StandardScaler\n", - "\n", - "# First we build an artifical dataset where the features are highly correlated with the labels\n", - "X = y_sklearn\n", - "X = StandardScaler().fit_transform(X)" - ], - "execution_count": 224, - "outputs": [] - }, - { - "cell_type": "code", - "metadata": { - "id": "MU14rp50Qs-C" - }, - "source": [ - "from sklearn.model_selection import cross_val_score\n", - "from sklearn.metrics import accuracy_score\n", - "\n", - "# We need a custom scorer for this dataset\n", - "# See https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html\n", - "def scorer(estimator, X, y):\n", - " y_pred = estimator.predict(X)\n", - " y_bin, y_cat = y[:, 0], y[:, 1]\n", - " y_pred_bin, y_pred_cat = y_pred[:, 0], y_pred[:, 1]\n", - " return np.mean([accuracy_score(y_bin, y_pred_bin), accuracy_score(y_cat, y_pred_cat)])" - ], - "execution_count": 225, - "outputs": [] - }, - { - "cell_type": "code", - "metadata": { - "id": "G7THvXqaMrSw", - "outputId": "d6ecbd6a-c34e-4296-e3e9-fd8869dc5e30", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 34 - } - }, - "source": [ - "clf = MultiOutputClassifier(model=get_clf_model, verbose=0, random_state=0, epochs=100,)\n", - "\n", - "np.mean(cross_val_score(clf, X, y_sklearn, scoring=scorer))" - ], - "execution_count": 226, - "outputs": [ - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "0.9800000000000001" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 226 - } - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "Pznw-f0v-1TU" - }, - "source": [ - "## 2. Multiple inputs" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "p0uSTuc-voxu" - }, - "source": [ - "The process for multiple inputs is similar, but instead of overriding the transformer in `target_encoder` we override `feature_encoder`." - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "NPOO7tO6-1TV" - }, - "source": [ - "from sklearn.base import BaseEstimator, TransformerMixin\n", - "\n", - "\n", - "class MultiOutputTransformer(BaseEstimator, TransformerMixin):\n", - " ...\n", - "\n", - "\n", - "class MultiOutputClassifier(KerasClassifier):\n", - "\n", - " @property\n", - " def feature_encoder(self):\n", - " return MultiInputTransformer(...)" - ], - "execution_count": 227, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "sp251zciLXAY" - }, - "source": [ - "### 2.1 Define Keras Model" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "1RHjV2Bg902U" - }, - "source": [ - "Let's define a Keras **regression** Model with 2 inputs:" - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "q84Po5aj928v" - }, - "source": [ - "def get_reg_model(compile_kwargs):\n", - "\n", - " inp1 = keras.layers.Input(shape=(1, ))\n", - " inp2 = keras.layers.Input(shape=(1, ))\n", - "\n", - " x1 = keras.layers.Dense(100, activation=\"relu\")(inp1)\n", - " x2 = keras.layers.Dense(50, activation=\"relu\")(inp2)\n", - "\n", - " concat = keras.layers.Concatenate(axis=-1)([x1, x2])\n", - "\n", - " out = keras.layers.Dense(1)(concat)\n", - "\n", - " model = keras.Model(inputs=[inp1, inp2], outputs=out)\n", - " model.compile(loss=\"mse\", optimizer=compile_kwargs[\"optimizer\"])\n", - "\n", - " return model" - ], - "execution_count": 228, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "SXsenn70Alvr" - }, - "source": [ - "And test it with a small mock dataset:" - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "YBAzKT9bAn91" - }, - "source": [ - "X = np.random.random(size=(100, 2))\n", - "y = np.sum(X, axis=1)\n", - "X = np.split(X, 2, axis=1)\n", - "\n", - "# build mock compile_kwargs\n", - "compile_kwargs = {\"optimizer\": \"sgd\"}\n", - "\n", - "model = get_reg_model(compile_kwargs=compile_kwargs)\n", - "\n", - "model.fit(X, y, verbose=0, epochs=100)\n", - "y_pred = model.predict(X).squeeze()" - ], - "execution_count": 230, - "outputs": [] - }, - { - "cell_type": "code", - "metadata": { - "id": "EIKruR9UBzu8", - "outputId": "8c345c10-3a98-4576-b0fc-38c957c1d4ce", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 34 - } - }, - "source": [ - "from sklearn.metrics import r2_score\n", - "\n", - "r2_score(y, y_pred)" - ], - "execution_count": 231, - "outputs": [ - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "0.815902187716304" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 231 - } - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "nLiQPj62CIIO" - }, - "source": [ - "Having verified that our model builds without errors and accepts the inputs types we expect, we move onto integrating a transformer into our SciKeras model." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "6lRbJEbVLqDw" - }, - "source": [ - "### 2.2 Define Data Transformer" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "zPAOsyJ59Ngj" - }, - "source": [ - "Just like for overriding `target_encoder`, we just need to define a sklearn transformer and drop it into our SciKeras wrapper. Since we hardcoded the input\n", - "shapes into our model and do not rely on any transformer-generated metadata, we can simply use `sklearn.preprocessing.FunctionTransformer`:" - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "zbKiSm_B9VKG" - }, - "source": [ - "from sklearn.preprocessing import FunctionTransformer\n", - "\n", - "\n", - "class MultiInputRegressor(KerasRegressor):\n", - "\n", - " @property\n", - " def feature_encoder(self):\n", - " return FunctionTransformer(\n", - " func=lambda X: [X[:, 0], X[:, 1]],\n", - " )" - ], - "execution_count": 13, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "nLvhTvaTCwIZ" - }, - "source": [ - "Note that we did **not** implement `inverse_transform` (that is, we did not pass an `inverse_func` argument to `FunctionTransformer`) because features are never converted back to their original form." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "ngMljBaIL_-3" - }, - "source": [ - "### 2.3 Test regressor" - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "IRyRFjLTEqzE", - "outputId": "7e350740-941a-4571-9c8f-9a5364724b21", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 34 - } - }, - "source": [ - "reg = MultiInputRegressor(model=get_reg_model, epochs=100, verbose=0, random_state=0)\n", - "\n", - "X_sklearn = np.column_stack(X)\n", - "\n", - "np.mean(cross_val_score(reg, X_sklearn, y))" - ], - "execution_count": 233, - "outputs": [ - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "0.9994605932537043" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 233 - } - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "4T753k83IhmW" - }, - "source": [ - "## 3. Multidimensional inputs with MNIST dataset" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "lKNj5QO76UxW" - }, - "source": [ - "In this example, we look at how we can use SciKeras to process the MNIST dataset. The dataset is composed of 60,000 images of digits, each of which is a 2D 28x28 image.\n", - "\n", - "The dataset and Keras Model architecture used come from a [Keras example](https://keras.io/examples/vision/mnist_convnet/). It may be beneficial to understand the Keras model by reviewing that example first." - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "NdDB8dWx8Akc", - "outputId": "c2253bc4-09d4-4c69-80da-a3c10e853782", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 34 - } - }, - "source": [ - "(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()\n", - "x_train.shape" - ], - "execution_count": 22, - "outputs": [ - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "(60000, 28, 28)" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 22 - } - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "aUcI-gys8EvE" - }, - "source": [ - "The outputs (labels) are numbers 0-9:" - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "gqdzgCv18SDw", - "outputId": "19d6a7de-9fea-4345-d332-f25ebe8a212f", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 51 - } - }, - "source": [ - "print(y_train.shape)\n", - "print(np.unique(y_train))" - ], - "execution_count": 23, - "outputs": [ - { - "output_type": "stream", - "text": [ - "(60000,)\n", - "[0 1 2 3 4 5 6 7 8 9]\n" - ], - "name": "stdout" - } - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "DYiQTQH29iMB" - }, - "source": [ - "First, we will \"flatten\" the data into an array of shape `(n_samples, 28*28)` (i.e. a 2D array). This will allow us to use sklearn ecosystem utilities, for example, `sklearn.preprocessing.MinMaxScaler`." - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "3TCV8Uem90rt" - }, - "source": [ - "from sklearn.preprocessing import MinMaxScaler\n", - "\n", - "n_samples_train = x_train.shape[0]\n", - "n_samples_test = x_test.shape[0]\n", - "\n", - "x_train = x_train.reshape((n_samples_train, -1))\n", - "x_test = x_test.reshape((n_samples_test, -1))\n", - "x_train = MinMaxScaler().fit_transform(x_train)\n", - "x_test = MinMaxScaler().fit_transform(x_test)" - ], - "execution_count": 24, - "outputs": [] - }, - { - "cell_type": "code", - "metadata": { - "id": "bvEra5TKA2yi", - "outputId": "32238303-9ca3-406e-95ca-2c59ac8dfc82", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 34 - } - }, - "source": [ - "print(x_train.shape[1:]) # 784 = 28*28" - ], - "execution_count": 25, - "outputs": [ - { - "output_type": "stream", - "text": [ - "(784,)\n" - ], - "name": "stdout" - } - ] - }, - { - "cell_type": "code", - "metadata": { - "id": "f5a02q6tBB3R", - "outputId": "810df38e-a0e6-45bd-ed42-0d7af3a3cbf2", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 34 - } - }, - "source": [ - "print(np.min(x_train), np.max(x_train)) # scaled 0-1" - ], - "execution_count": 26, - "outputs": [ - { - "output_type": "stream", - "text": [ - "0.0 1.0\n" - ], - "name": "stdout" - } - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "aCoUz1qy-rh9" - }, - "source": [ - "Of course, in this case, we could have just as easily used numpy functions to scale our data, but we use `MinMaxScaler` to demonstrate use of the sklearn ecosystem." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "yuR10hymK0dh" - }, - "source": [ - "### 3.1 Define Keras Model" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "rE15zkS4_hGU" - }, - "source": [ - "Next we will define our Keras model (adapted from [keras.io](https://keras.io/examples/vision/mnist_convnet/)):" + { + "cell_type": "code", + "execution_count": 46, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 51 + }, + "id": "GyIq1xXEqyMY", + "outputId": "0cda9bba-4361-46ce-8b00-9669ab5b072e" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[[0.16950993 0.13774167 0.23052557 0.22972858 0.2324943 ]\n", + " [0.1702391 0.13294716 0.20055309 0.25442293 0.24183771]]\n" + ] + } + ], + "source": [ + "print(y_pred[1][:2, :])" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "mic4ByPGBwgA" + }, + "source": [ + "As you can see, our `predict` output is also a list of arrays, except it contains probabilities instead of the class predictions.\n", + "\n", + "Our data transormer's job will be to convert from a single numpy array (which is what the sklearn ecosystem works with) to the list of arrays and then back. Additionally, for classifiers, we will want to be able to convert probabilities to class predictions.\n", + "\n", + "We will structure our data on the sklearn side by column-stacking our list\n", + "of arrays. This works well in this case since we have the same number of datapoints in each array." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "2hN9nZiqMNJ9" + }, + "source": [ + "### 1.2 Define output Data Transformer" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "QNBuyxhcMPg8" + }, + "source": [ + "Let's go ahead and protoype this data transformer:" + ] + }, + { + "cell_type": "code", + "execution_count": 219, + "metadata": { + "id": "y3E81vxrDGhM" + }, + "outputs": [], + "source": [ + "from typing import List\n", + "\n", + "from sklearn.base import BaseEstimator, TransformerMixin\n", + "from sklearn.preprocessing import LabelEncoder\n", + "\n", + "\n", + "class MultiOutputTransformer(BaseEstimator, TransformerMixin):\n", + "\n", + " def fit(self, y):\n", + " y_bin, y_cat = y[:, 0], y[:, 1]\n", + " # Create internal encoders to ensure labels are 0, 1, 2...\n", + " self.bin_encoder_ = LabelEncoder()\n", + " self.cat_encoder_ = LabelEncoder()\n", + " # Fit them to the input data\n", + " self.bin_encoder_.fit(y_bin)\n", + " self.cat_encoder_.fit(y_cat)\n", + " # Save the number of classes\n", + " self.n_classes_ = [\n", + " self.bin_encoder_.classes_.size,\n", + " self.cat_encoder_.classes_.size,\n", + " ]\n", + " # Save number of expected outputs in the Keras model\n", + " # SciKeras will automatically use this to do error-checking\n", + " self.n_outputs_expected_ = 2\n", + " return self\n", + "\n", + " def transform(self, y: np.ndarray) -> List[np.ndarray]:\n", + " y_bin, y_cat = y[:, 0], y[:, 1]\n", + " # Apply transformers to input array\n", + " y_bin = self.bin_encoder_.transform(y_bin)\n", + " y_cat = self.cat_encoder_.transform(y_cat)\n", + " # Split the data into a list\n", + " return [y_bin, y_cat]\n", + "\n", + " def inverse_transform(self, y: List[np.ndarray], return_proba: bool = False) -> np.ndarray:\n", + " y_pred_proba = y # rename for clarity, what Keras gives us are probs\n", + " if return_proba:\n", + " return np.column_stack(y_pred_proba, axis=1)\n", + " # Get class predictions from probabilities\n", + " y_pred_bin = (y_pred_proba[0] > 0.5).astype(int).reshape(-1, )\n", + " y_pred_cat = np.argmax(y_pred_proba[1], axis=1)\n", + " # Pass back through LabelEncoder\n", + " y_pred_bin = self.bin_encoder_.inverse_transform(y_pred_bin)\n", + " y_pred_cat = self.cat_encoder_.inverse_transform(y_pred_cat)\n", + " return np.column_stack([y_pred_bin, y_pred_cat])\n", + " \n", + " def get_metadata(self):\n", + " return {\n", + " \"n_classes_\": self.n_classes_,\n", + " \"n_outputs_expected_\": self.n_outputs_expected_,\n", + " }" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "eKM37jy-HAX9" + }, + "source": [ + "Note that in addition to the usual `transform` and `inverse_transform` methods, we implement the `get_metadata` method to return the `n_classes_` attribute.\n", + "\n", + "Lets test our transformer with the same dataset we previoulsy used to test our model:" + ] + }, + { + "cell_type": "code", + "execution_count": 220, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 51 + }, + "id": "yPqhenjpG_qy", + "outputId": "8bbbe68b-7380-431d-bd63-5d23ad7f8b64" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "`y`, as will be passed to Keras:\n", + "[array([0, 1, 1, 1]), array([2, 4, 2, 0])]\n" + ] + } + ], + "source": [ + "tf = MultiOutputTransformer()\n", + "\n", + "y_sklearn = np.column_stack(y)\n", + "\n", + "y_keras = tf.fit_transform(y_sklearn)\n", + "print(\"`y`, as will be passed to Keras:\")\n", + "print([y_keras[0][:4], y_keras[1][:4]])" + ] + }, + { + "cell_type": "code", + "execution_count": 221, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 119 + }, + "id": "d_a6JqcDKkTg", + "outputId": "a9d4f3e8-3459-46dc-a8fe-f8ff138389cf" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "`y_pred`, as will be returned to sklearn:\n" + ] + }, + { + "data": { + "text/plain": [ + "array([[0, 3],\n", + " [1, 3],\n", + " [0, 3],\n", + " [0, 3],\n", + " [0, 3]])" ] + }, + "execution_count": 221, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "y_pred_sklearn = tf.inverse_transform(y_pred)\n", + "print(\"`y_pred`, as will be returned to sklearn:\")\n", + "y_pred_sklearn[:5]" + ] + }, + { + "cell_type": "code", + "execution_count": 222, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "9v0RG5ZjPgcy", + "outputId": "7f9a05c9-303b-446e-bca9-42c4f6391e06" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "metadata = {'n_classes_': [2, 5], 'n_outputs_expected_': 2}\n" + ] + } + ], + "source": [ + "print(f\"metadata = {tf.get_metadata()}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ScbMdm-9LSx1" + }, + "source": [ + "Since this looks good, we move on to integrating our transformer into our classifier." + ] + }, + { + "cell_type": "code", + "execution_count": 223, + "metadata": { + "id": "RKA2U8ANMhn9" + }, + "outputs": [], + "source": [ + "class MultiOutputClassifier(KerasClassifier):\n", + "\n", + " @property\n", + " def target_encoder(self):\n", + " return MultiOutputTransformer()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "1heA-eeTMp3t" + }, + "source": [ + "### 1.3 Test classifier" + ] + }, + { + "cell_type": "code", + "execution_count": 224, + "metadata": { + "id": "jXbxL2mnNKkb" + }, + "outputs": [], + "source": [ + "from sklearn.preprocessing import StandardScaler\n", + "\n", + "# First we build an artifical dataset where the features are highly correlated with the labels\n", + "X = y_sklearn\n", + "X = StandardScaler().fit_transform(X)" + ] + }, + { + "cell_type": "code", + "execution_count": 225, + "metadata": { + "id": "MU14rp50Qs-C" + }, + "outputs": [], + "source": [ + "from sklearn.model_selection import cross_val_score\n", + "from sklearn.metrics import accuracy_score\n", + "\n", + "# We need a custom scorer for this dataset\n", + "# See https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html\n", + "def scorer(estimator, X, y):\n", + " y_pred = estimator.predict(X)\n", + " y_bin, y_cat = y[:, 0], y[:, 1]\n", + " y_pred_bin, y_pred_cat = y_pred[:, 0], y_pred[:, 1]\n", + " return np.mean([accuracy_score(y_bin, y_pred_bin), accuracy_score(y_cat, y_pred_cat)])" + ] + }, + { + "cell_type": "code", + "execution_count": 226, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 }, + "id": "G7THvXqaMrSw", + "outputId": "d6ecbd6a-c34e-4296-e3e9-fd8869dc5e30" + }, + "outputs": [ { - "cell_type": "code", - "metadata": { - "id": "dBFFXT-__7KU" - }, - "source": [ - "num_classes = 10\n", - "input_shape = (28, 28, 1)\n", - "\n", - "\n", - "def get_model(meta):\n", - " model = keras.Sequential(\n", - " [\n", - " keras.Input(input_shape),\n", - " keras.layers.Conv2D(32, kernel_size=(3, 3), activation=\"relu\"),\n", - " keras.layers.MaxPooling2D(pool_size=(2, 2)),\n", - " keras.layers.Conv2D(64, kernel_size=(3, 3), activation=\"relu\"),\n", - " keras.layers.MaxPooling2D(pool_size=(2, 2)),\n", - " keras.layers.Flatten(),\n", - " keras.layers.Dropout(0.5),\n", - " keras.layers.Dense(num_classes, activation=\"softmax\"),\n", - " ]\n", - " )\n", - " model.compile(\n", - " loss=\"sparse_categorical_crossentropy\", optimizer=\"adam\"\n", - " )\n", - " return model" - ], - "execution_count": 31, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "8aJRGbF1Avpq" - }, - "source": [ - "Now let's define a transformer that we will use to reshape our input from the sklearn shape (`(n_samples, 784)`) to the Keras shape (which we will be `(n_samples, 28, 28, 1)`)." + "data": { + "text/plain": [ + "0.9800000000000001" ] + }, + "execution_count": 226, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "clf = MultiOutputClassifier(model=get_clf_model, verbose=0, random_state=0, epochs=100,)\n", + "\n", + "np.mean(cross_val_score(clf, X, y_sklearn, scoring=scorer))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Pznw-f0v-1TU" + }, + "source": [ + "## 2. Multiple inputs" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "p0uSTuc-voxu" + }, + "source": [ + "The process for multiple inputs is similar, but instead of overriding the transformer in `target_encoder` we override `feature_encoder`." + ] + }, + { + "cell_type": "code", + "execution_count": 227, + "metadata": { + "id": "NPOO7tO6-1TV" + }, + "outputs": [], + "source": [ + "from sklearn.base import BaseEstimator, TransformerMixin\n", + "\n", + "\n", + "class MultiInputTransformer(BaseEstimator, TransformerMixin):\n", + " ...\n", + "\n", + "\n", + "class MultiInputClassifier(KerasClassifier):\n", + "\n", + " @property\n", + " def feature_encoder(self):\n", + " return MultiInputTransformer(...)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "sp251zciLXAY" + }, + "source": [ + "### 2.1 Define Keras Model" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "1RHjV2Bg902U" + }, + "source": [ + "Let's define a Keras **regression** Model with 2 inputs:" + ] + }, + { + "cell_type": "code", + "execution_count": 228, + "metadata": { + "id": "q84Po5aj928v" + }, + "outputs": [], + "source": [ + "def get_reg_model(compile_kwargs):\n", + "\n", + " inp1 = keras.layers.Input(shape=(1, ))\n", + " inp2 = keras.layers.Input(shape=(1, ))\n", + "\n", + " x1 = keras.layers.Dense(100, activation=\"relu\")(inp1)\n", + " x2 = keras.layers.Dense(50, activation=\"relu\")(inp2)\n", + "\n", + " concat = keras.layers.Concatenate(axis=-1)([x1, x2])\n", + "\n", + " out = keras.layers.Dense(1)(concat)\n", + "\n", + " model = keras.Model(inputs=[inp1, inp2], outputs=out)\n", + " model.compile(loss=\"mse\", optimizer=compile_kwargs[\"optimizer\"])\n", + "\n", + " return model" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "SXsenn70Alvr" + }, + "source": [ + "And test it with a small mock dataset:" + ] + }, + { + "cell_type": "code", + "execution_count": 230, + "metadata": { + "id": "YBAzKT9bAn91" + }, + "outputs": [], + "source": [ + "X = np.random.random(size=(100, 2))\n", + "y = np.sum(X, axis=1)\n", + "X = np.split(X, 2, axis=1)\n", + "\n", + "# build mock compile_kwargs\n", + "compile_kwargs = {\"optimizer\": \"sgd\"}\n", + "\n", + "model = get_reg_model(compile_kwargs=compile_kwargs)\n", + "\n", + "model.fit(X, y, verbose=0, epochs=100)\n", + "y_pred = model.predict(X).squeeze()" + ] + }, + { + "cell_type": "code", + "execution_count": 231, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 }, + "id": "EIKruR9UBzu8", + "outputId": "8c345c10-3a98-4576-b0fc-38c957c1d4ce" + }, + "outputs": [ { - "cell_type": "code", - "metadata": { - "id": "CzVwr7glB1tq" - }, - "source": [ - "class MultiDimensionalClassifier(KerasClassifier):\n", - "\n", - " @property\n", - " def feature_encoder(self):\n", - " return FunctionTransformer(\n", - " func=lambda X: X.reshape(X.shape[0], *input_shape),\n", - " )" - ], - "execution_count": 32, - "outputs": [] - }, - { - "cell_type": "code", - "metadata": { - "id": "n1CSzViYDFeK" - }, - "source": [ - "clf = MultiDimensionalClassifier(\n", - " model=get_model,\n", - " epochs=15,\n", - " batch_size=128,\n", - " validation_split=0.1,\n", - " random_state=0,\n", - ")" - ], - "execution_count": 36, - "outputs": [] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "FP_Z5NIuL3bB" - }, - "source": [ - "### 3.2 Test" + "data": { + "text/plain": [ + "0.815902187716304" ] + }, + "execution_count": 231, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "from sklearn.metrics import r2_score\n", + "\n", + "r2_score(y, y_pred)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "nLiQPj62CIIO" + }, + "source": [ + "Having verified that our model builds without errors and accepts the inputs types we expect, we move onto integrating a transformer into our SciKeras model." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "6lRbJEbVLqDw" + }, + "source": [ + "### 2.2 Define Data Transformer" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "zPAOsyJ59Ngj" + }, + "source": [ + "Just like for overriding `target_encoder`, we just need to define a sklearn transformer and drop it into our SciKeras wrapper. Since we hardcoded the input\n", + "shapes into our model and do not rely on any transformer-generated metadata, we can simply use `sklearn.preprocessing.FunctionTransformer`:" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": { + "id": "zbKiSm_B9VKG" + }, + "outputs": [], + "source": [ + "from sklearn.preprocessing import FunctionTransformer\n", + "\n", + "\n", + "class MultiInputRegressor(KerasRegressor):\n", + "\n", + " @property\n", + " def feature_encoder(self):\n", + " return FunctionTransformer(\n", + " func=lambda X: [X[:, 0], X[:, 1]],\n", + " )" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "nLvhTvaTCwIZ" + }, + "source": [ + "Note that we did **not** implement `inverse_transform` (that is, we did not pass an `inverse_func` argument to `FunctionTransformer`) because features are never converted back to their original form." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ngMljBaIL_-3" + }, + "source": [ + "### 2.3 Test regressor" + ] + }, + { + "cell_type": "code", + "execution_count": 233, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 }, + "id": "IRyRFjLTEqzE", + "outputId": "7e350740-941a-4571-9c8f-9a5364724b21" + }, + "outputs": [ { - "cell_type": "markdown", - "metadata": { - "id": "RjmTibFlEr6-" - }, - "source": [ - "Train and score the model (this takes some time)" + "data": { + "text/plain": [ + "0.9994605932537043" ] + }, + "execution_count": 233, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "reg = MultiInputRegressor(model=get_reg_model, epochs=100, verbose=0, random_state=0)\n", + "\n", + "X_sklearn = np.column_stack(X)\n", + "\n", + "np.mean(cross_val_score(reg, X_sklearn, y))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "4T753k83IhmW" + }, + "source": [ + "## 3. Multidimensional inputs with MNIST dataset" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "lKNj5QO76UxW" + }, + "source": [ + "In this example, we look at how we can use SciKeras to process the MNIST dataset. The dataset is composed of 60,000 images of digits, each of which is a 2D 28x28 image.\n", + "\n", + "The dataset and Keras Model architecture used come from a [Keras example](https://keras.io/examples/vision/mnist_convnet/). It may be beneficial to understand the Keras model by reviewing that example first." + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 }, + "id": "NdDB8dWx8Akc", + "outputId": "c2253bc4-09d4-4c69-80da-a3c10e853782" + }, + "outputs": [ { - "cell_type": "code", - "metadata": { - "id": "NaRVWcnZDRlD", - "outputId": "ed105010-c823-438f-a417-6e0fbf7148c4", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 799 - } - }, - "source": [ - "clf.fit(x_train, y_train)" - ], - "execution_count": 37, - "outputs": [ - { - "output_type": "stream", - "text": [ - "Epoch 1/15\n", - "422/422 [==============================] - 39s 94ms/step - loss: 0.3532 - val_loss: 0.0825\n", - "Epoch 2/15\n", - "422/422 [==============================] - 39s 93ms/step - loss: 0.1124 - val_loss: 0.0590\n", - "Epoch 3/15\n", - "422/422 [==============================] - 39s 93ms/step - loss: 0.0870 - val_loss: 0.0483\n", - "Epoch 4/15\n", - "422/422 [==============================] - 39s 93ms/step - loss: 0.0722 - val_loss: 0.0419\n", - "Epoch 5/15\n", - "422/422 [==============================] - 38s 91ms/step - loss: 0.0647 - val_loss: 0.0419\n", - "Epoch 6/15\n", - "422/422 [==============================] - 39s 93ms/step - loss: 0.0583 - val_loss: 0.0361\n", - "Epoch 7/15\n", - "422/422 [==============================] - 39s 93ms/step - loss: 0.0537 - val_loss: 0.0360\n", - "Epoch 8/15\n", - "422/422 [==============================] - 40s 94ms/step - loss: 0.0509 - val_loss: 0.0335\n", - "Epoch 9/15\n", - "422/422 [==============================] - 39s 94ms/step - loss: 0.0470 - val_loss: 0.0321\n", - "Epoch 10/15\n", - "422/422 [==============================] - 39s 93ms/step - loss: 0.0433 - val_loss: 0.0315\n", - "Epoch 11/15\n", - "422/422 [==============================] - 39s 93ms/step - loss: 0.0431 - val_loss: 0.0315\n", - "Epoch 12/15\n", - "422/422 [==============================] - 39s 93ms/step - loss: 0.0410 - val_loss: 0.0297\n", - "Epoch 13/15\n", - "422/422 [==============================] - 40s 95ms/step - loss: 0.0395 - val_loss: 0.0293\n", - "Epoch 14/15\n", - "422/422 [==============================] - 38s 89ms/step - loss: 0.0386 - val_loss: 0.0296\n", - "Epoch 15/15\n", - "422/422 [==============================] - 38s 90ms/step - loss: 0.0362 - val_loss: 0.0284\n" - ], - "name": "stdout" - }, - { - "output_type": "execute_result", - "data": { - "text/plain": [ - "MultiDimensionalClassifier(\n", - "\tmodel=\n", - "\tbuild_fn=None\n", - "\twarm_start=False\n", - "\trandom_state=None\n", - "\toptimizer=rmsprop\n", - "\tloss=None\n", - "\tmetrics=None\n", - "\tbatch_size=128\n", - "\tverbose=1\n", - "\tcallbacks=None\n", - "\tvalidation_split=0.1\n", - "\tshuffle=True\n", - "\trun_eagerly=False\n", - "\tepochs=15\n", - ")" - ] - }, - "metadata": { - "tags": [] - }, - "execution_count": 37 - } + "data": { + "text/plain": [ + "(60000, 28, 28)" ] - }, - { - "cell_type": "code", - "metadata": { - "id": "Mw_nD5FdEzd-", - "outputId": "d23b3c3f-337a-4e27-d7ed-161a7c9a12da", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 51 - } - }, - "source": [ - "score = clf.score(x_test, y_test)\n", - "print(f\"Test score (accuracy): {score:.2f}\")" - ], - "execution_count": 38, - "outputs": [ - { - "output_type": "stream", - "text": [ - "79/79 [==============================] - 2s 26ms/step\n", - "Test score (accuracy): 0.99\n" - ], - "name": "stdout" - } + }, + "execution_count": 22, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()\n", + "x_train.shape" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "aUcI-gys8EvE" + }, + "source": [ + "The outputs (labels) are numbers 0-9:" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 51 + }, + "id": "gqdzgCv18SDw", + "outputId": "19d6a7de-9fea-4345-d332-f25ebe8a212f" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "(60000,)\n", + "[0 1 2 3 4 5 6 7 8 9]\n" + ] + } + ], + "source": [ + "print(y_train.shape)\n", + "print(np.unique(y_train))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "DYiQTQH29iMB" + }, + "source": [ + "First, we will \"flatten\" the data into an array of shape `(n_samples, 28*28)` (i.e. a 2D array). This will allow us to use sklearn ecosystem utilities, for example, `sklearn.preprocessing.MinMaxScaler`." + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": { + "id": "3TCV8Uem90rt" + }, + "outputs": [], + "source": [ + "from sklearn.preprocessing import MinMaxScaler\n", + "\n", + "n_samples_train = x_train.shape[0]\n", + "n_samples_test = x_test.shape[0]\n", + "\n", + "x_train = x_train.reshape((n_samples_train, -1))\n", + "x_test = x_test.reshape((n_samples_test, -1))\n", + "x_train = MinMaxScaler().fit_transform(x_train)\n", + "x_test = MinMaxScaler().fit_transform(x_test)" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "bvEra5TKA2yi", + "outputId": "32238303-9ca3-406e-95ca-2c59ac8dfc82" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "(784,)\n" + ] + } + ], + "source": [ + "print(x_train.shape[1:]) # 784 = 28*28" + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "f5a02q6tBB3R", + "outputId": "810df38e-a0e6-45bd-ed42-0d7af3a3cbf2" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "0.0 1.0\n" + ] + } + ], + "source": [ + "print(np.min(x_train), np.max(x_train)) # scaled 0-1" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "aCoUz1qy-rh9" + }, + "source": [ + "Of course, in this case, we could have just as easily used numpy functions to scale our data, but we use `MinMaxScaler` to demonstrate use of the sklearn ecosystem." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "yuR10hymK0dh" + }, + "source": [ + "### 3.1 Define Keras Model" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "rE15zkS4_hGU" + }, + "source": [ + "Next we will define our Keras model (adapted from [keras.io](https://keras.io/examples/vision/mnist_convnet/)):" + ] + }, + { + "cell_type": "code", + "execution_count": 31, + "metadata": { + "id": "dBFFXT-__7KU" + }, + "outputs": [], + "source": [ + "num_classes = 10\n", + "input_shape = (28, 28, 1)\n", + "\n", + "\n", + "def get_model(meta):\n", + " model = keras.Sequential(\n", + " [\n", + " keras.Input(input_shape),\n", + " keras.layers.Conv2D(32, kernel_size=(3, 3), activation=\"relu\"),\n", + " keras.layers.MaxPooling2D(pool_size=(2, 2)),\n", + " keras.layers.Conv2D(64, kernel_size=(3, 3), activation=\"relu\"),\n", + " keras.layers.MaxPooling2D(pool_size=(2, 2)),\n", + " keras.layers.Flatten(),\n", + " keras.layers.Dropout(0.5),\n", + " keras.layers.Dense(num_classes, activation=\"softmax\"),\n", + " ]\n", + " )\n", + " model.compile(\n", + " loss=\"sparse_categorical_crossentropy\", optimizer=\"adam\"\n", + " )\n", + " return model" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "8aJRGbF1Avpq" + }, + "source": [ + "Now let's define a transformer that we will use to reshape our input from the sklearn shape (`(n_samples, 784)`) to the Keras shape (which we will be `(n_samples, 28, 28, 1)`)." + ] + }, + { + "cell_type": "code", + "execution_count": 32, + "metadata": { + "id": "CzVwr7glB1tq" + }, + "outputs": [], + "source": [ + "class MultiDimensionalClassifier(KerasClassifier):\n", + "\n", + " @property\n", + " def feature_encoder(self):\n", + " return FunctionTransformer(\n", + " func=lambda X: X.reshape(X.shape[0], *input_shape),\n", + " )" + ] + }, + { + "cell_type": "code", + "execution_count": 36, + "metadata": { + "id": "n1CSzViYDFeK" + }, + "outputs": [], + "source": [ + "clf = MultiDimensionalClassifier(\n", + " model=get_model,\n", + " epochs=15,\n", + " batch_size=128,\n", + " validation_split=0.1,\n", + " random_state=0,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "FP_Z5NIuL3bB" + }, + "source": [ + "### 3.2 Test" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "RjmTibFlEr6-" + }, + "source": [ + "Train and score the model (this takes some time)" + ] + }, + { + "cell_type": "code", + "execution_count": 37, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 799 + }, + "id": "NaRVWcnZDRlD", + "outputId": "ed105010-c823-438f-a417-6e0fbf7148c4" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Epoch 1/15\n", + "422/422 [==============================] - 39s 94ms/step - loss: 0.3532 - val_loss: 0.0825\n", + "Epoch 2/15\n", + "422/422 [==============================] - 39s 93ms/step - loss: 0.1124 - val_loss: 0.0590\n", + "Epoch 3/15\n", + "422/422 [==============================] - 39s 93ms/step - loss: 0.0870 - val_loss: 0.0483\n", + "Epoch 4/15\n", + "422/422 [==============================] - 39s 93ms/step - loss: 0.0722 - val_loss: 0.0419\n", + "Epoch 5/15\n", + "422/422 [==============================] - 38s 91ms/step - loss: 0.0647 - val_loss: 0.0419\n", + "Epoch 6/15\n", + "422/422 [==============================] - 39s 93ms/step - loss: 0.0583 - val_loss: 0.0361\n", + "Epoch 7/15\n", + "422/422 [==============================] - 39s 93ms/step - loss: 0.0537 - val_loss: 0.0360\n", + "Epoch 8/15\n", + "422/422 [==============================] - 40s 94ms/step - loss: 0.0509 - val_loss: 0.0335\n", + "Epoch 9/15\n", + "422/422 [==============================] - 39s 94ms/step - loss: 0.0470 - val_loss: 0.0321\n", + "Epoch 10/15\n", + "422/422 [==============================] - 39s 93ms/step - loss: 0.0433 - val_loss: 0.0315\n", + "Epoch 11/15\n", + "422/422 [==============================] - 39s 93ms/step - loss: 0.0431 - val_loss: 0.0315\n", + "Epoch 12/15\n", + "422/422 [==============================] - 39s 93ms/step - loss: 0.0410 - val_loss: 0.0297\n", + "Epoch 13/15\n", + "422/422 [==============================] - 40s 95ms/step - loss: 0.0395 - val_loss: 0.0293\n", + "Epoch 14/15\n", + "422/422 [==============================] - 38s 89ms/step - loss: 0.0386 - val_loss: 0.0296\n", + "Epoch 15/15\n", + "422/422 [==============================] - 38s 90ms/step - loss: 0.0362 - val_loss: 0.0284\n" + ] + }, + { + "data": { + "text/plain": [ + "MultiDimensionalClassifier(\n", + "\tmodel=\n", + "\tbuild_fn=None\n", + "\twarm_start=False\n", + "\trandom_state=None\n", + "\toptimizer=rmsprop\n", + "\tloss=None\n", + "\tmetrics=None\n", + "\tbatch_size=128\n", + "\tverbose=1\n", + "\tcallbacks=None\n", + "\tvalidation_split=0.1\n", + "\tshuffle=True\n", + "\trun_eagerly=False\n", + "\tepochs=15\n", + ")" ] + }, + "execution_count": 37, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" } - ] -} \ No newline at end of file + ], + "source": [ + "clf.fit(x_train, y_train)" + ] + }, + { + "cell_type": "code", + "execution_count": 38, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 51 + }, + "id": "Mw_nD5FdEzd-", + "outputId": "d23b3c3f-337a-4e27-d7ed-161a7c9a12da" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "79/79 [==============================] - 2s 26ms/step\n", + "Test score (accuracy): 0.99\n" + ] + } + ], + "source": [ + "score = clf.score(x_test, y_test)\n", + "print(f\"Test score (accuracy): {score:.2f}\")" + ] + } + ], + "metadata": { + "colab": { + "collapsed_sections": [], + "name": "DataTransformers.ipynb", + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.5" + } + }, + "nbformat": 4, + "nbformat_minor": 1 +} From 3da1bc33768904af846774b95d95626bebf2b2ee Mon Sep 17 00:00:00 2001 From: data hound Date: Tue, 26 Jan 2021 12:15:48 +0530 Subject: [PATCH 2/3] Typo Changes in DataTransformers.md --- docs/source/notebooks/DataTransformers.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/notebooks/DataTransformers.md b/docs/source/notebooks/DataTransformers.md index a8b026f62..8bb008fa7 100644 --- a/docs/source/notebooks/DataTransformers.md +++ b/docs/source/notebooks/DataTransformers.md @@ -354,11 +354,11 @@ if False: from sklearn.base import BaseEstimator, TransformerMixin - class MultiOutputTransformer(BaseEstimator, TransformerMixin): + class MultiInputTransformer(BaseEstimator, TransformerMixin): ... - class MultiOutputClassifier(KerasClassifier): + class MultiInputClassifier(KerasClassifier): @property def feature_encoder(self): From 0026d546b0395d3d80f706e18aa3ba4df3cf5b9e Mon Sep 17 00:00:00 2001 From: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com> Date: Wed, 27 Jan 2021 12:28:48 -0600 Subject: [PATCH 3/3] Revert pre-commit change --- .pre-commit-config.yaml | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index c8552eb7f..e80f5374e 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -1,12 +1,12 @@ -repos: - - repo: https://github.com/psf/black - rev: 19.10b0 - hooks: - - id: black - - - repo: https://github.com/timothycrosley/isort - rev: 5.3.2 - hooks: - - id: isort - additional_dependencies: [toml] - exclude: ^.*/?setup\.py$ +repos: + - repo: https://github.com/psf/black + rev: 19.10b0 + hooks: + - id: black + + - repo: https://github.com/timothycrosley/isort + rev: 5.3.2 + hooks: + - id: isort + additional_dependencies: [toml] + exclude: ^.*/?setup\.py$