AccuracyAssessmentTools.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/boothmanrylan/accuracyAssessmentTools/blob/main/AccuracyAssessmentTools.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9E5EFNO6Dzfx"
      },
      "source": [
        "# Installs and Imports"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ogwaLH3BC4xH",
        "tags": []
      },
      "outputs": [],
      "source": [
        "%cd /content\n",
        "!git clone https://github.com/boothmanrylan/accuracyAssessmentTools.git\n",
        "%cd accuracyAssessmentTools\n",
        "!pip install ."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "from acc_assessment.olofsson import Olofsson\n",
        "from acc_assessment.congalton import Congalton\n",
        "from acc_assessment.stehman import Stehman\n",
        "from acc_assessment.cardille import Cardille"
      ],
      "metadata": {
        "id": "IrLA0qviG2v2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tASR5FX-GITX"
      },
      "source": [
        "# Verify that the code works as expected"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wtKKZboXGHcM",
        "tags": []
      },
      "outputs": [],
      "source": [
        "!pytest"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Cardille Assessment Example\n",
        "\n",
        "The Cardille Assessment is based on ongoing work in the Cardille Computational Landscape Ecology Lab."
      ],
      "metadata": {
        "id": "LFxh2ddz5J9S"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Load data\n",
        "\n",
        "Data should come in two csv files: one containing the map data and one containing the reference data. Each file should have one column containing the point id (to link rows from the map csv file to the reference csv file), and one column for the strata that the point was sampled from, and then one column for each of the possible classes containing the reference/map probability that the point belongs to that class. Column names should match between the two csv files.\n",
        "\n",
        "In addition to the two csv files you also need to supply a dictionary mapping the stratum to their total size."
      ],
      "metadata": {
        "id": "tK5ObHhi5SAO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "map_table = pd.read_csv(\"./tests/map_data_table.csv\")\n",
        "map_table"
      ],
      "metadata": {
        "id": "5wAVv7105Otu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "ref_table = pd.read_csv(\"./tests/ref_data_table.csv\")\n",
        "ref_table"
      ],
      "metadata": {
        "id": "Jf-PDKPB6EII"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "strata_population_dict = {\n",
        "    'a': 5000,\n",
        "    'f': 10000,\n",
        "    'w': 1000,\n",
        "}"
      ],
      "metadata": {
        "id": "_pmh6NXj-odi"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Create the assessment\n",
        "\n",
        "The Cardille accuracy assessment is a class. An explanation of its constructor can be viewed by calling `help` on the `__init__` function."
      ],
      "metadata": {
        "id": "2HCTVamq6L3A"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "help(Cardille.__init__)"
      ],
      "metadata": {
        "id": "9uj8QKkg7RQv"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "assessment = Cardille(\n",
        "    map_data=map_table,\n",
        "    ref_data=ref_table,\n",
        "    strata_col=\"strata\",\n",
        "    id_col=\"id\",\n",
        "    strata_population=strata_population_dict,\n",
        ")"
      ],
      "metadata": {
        "id": "xwb3s4B36Sgh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## View assessment results\n",
        "\n",
        "An overview of the assessment can be seen by printing the assessment object."
      ],
      "metadata": {
        "id": "tjLGKMaL_V6j"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "assessment"
      ],
      "metadata": {
        "id": "zcsQs0K__tzW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Individual accuracies can be accessed by calling the appropriate method.\n",
        "\n",
        "These methods all return a tuple of two floats which are the value and the standard error respectively. If the standard error is not calculable it will be returned as `None`."
      ],
      "metadata": {
        "id": "7VtYfmcUdTv8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "forest_users_accuracy = assessment.users_accuracy(\"Forest\")\n",
        "forest_producers_accuracy = assessment.producers_accuracy(\"Forest\")\n",
        "\n",
        "print(forest_users_accuracy)\n",
        "print(forest_producers_accuracy)"
      ],
      "metadata": {
        "id": "fkP-8QsedRkS"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "To follow the practices outlined in Olofsson 2014 and Stehman 2014, the error matrix is returned as proportion of area by default. You can get an error matrix of point counts by setting `proportions=False`. Note that for the Cardille assessment these counts are scaled by their \"weights\"."
      ],
      "metadata": {
        "id": "Jb9r3wVAd5RY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "assessment.error_matrix()"
      ],
      "metadata": {
        "id": "h5C_tfS0eQ0L"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "assessment.error_matrix(proportions=False)"
      ],
      "metadata": {
        "id": "7T-a3gX8eY6E"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "An overview of all the methods can be seen by calling `help`."
      ],
      "metadata": {
        "id": "0MPt-QAleqRu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "help(assessment)"
      ],
      "metadata": {
        "id": "klkJOHdXem_M"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Stehman Assessment Example\n",
        "\n",
        "The Stehman assessment is based on:\n",
        "Stehman, S.V., 2014. \"Estimating area and map accuracy for stratified\n",
        "random sampling when the strata are different from the map classes\",\n",
        "International Journal of Remote Sensing. Vol. 35 (No. 13).\n",
        "https://doi.org/10.1080/01431161.2014.930207"
      ],
      "metadata": {
        "id": "U6VE_pZeSefM"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Load Data\n",
        "\n",
        "Use the data from Table 2 in Stehamn 2014 as the example assessment. Each row represents one pixel. The table must have at least three columns: one for the stratum each pixel was sampled from, one for the map class of each pixel and one for the reference class of each pixel."
      ],
      "metadata": {
        "id": "iGq4O5gmS1Mp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "stehman_df = pd.read_csv(\"./tests/stehman2014_table2.csv\", skiprows=1)"
      ],
      "metadata": {
        "id": "yKuXwKHLUqM_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "In addition to the table of reference data, the Stehman assessment also needs a dictionary containing the total size of each of the strata. Keys in the dictionary should match the labels used in the strata column of the table."
      ],
      "metadata": {
        "id": "kHG9aNdcV8yH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "stehman_strata_populations= {1: 40000, 2: 30000, 3: 20000, 4: 10000}"
      ],
      "metadata": {
        "id": "dcNdzyzQVEPa"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Create the Assessment"
      ],
      "metadata": {
        "id": "S2iLOTZGS2nw"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "stehman_assessment = Stehman(\n",
        "    data=stehman_df,\n",
        "    strata_col=\"Stratum\",\n",
        "    map_col=\"Map class\",\n",
        "    ref_col=\"Reference class\",\n",
        "    strata_population=stehman_strata_populations\n",
        ")\n",
        "stehman_assessment"
      ],
      "metadata": {
        "id": "1YNJFJhfVqVm"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The Stehman assessment object has the same `user_accuracy`, `producers_accuracy` methods as the Cardille assessment."
      ],
      "metadata": {
        "id": "WJ2JFS6nWPTZ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Olofsson Assessment Example\n",
        "\n",
        "The Olofsson assessment is based on: Olofsson, P., et al., 2014 \"Good practices for estimating area and assessing accuracy of land change\", Remote Sensing of Environment. Vol 148 pp. 42-57 https://doi.org/10.1016/j.rse.2014.02.015"
      ],
      "metadata": {
        "id": "0uS0US9oSle3"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Load Data\n",
        "\n",
        "Use the data from Table 8 in Olofsson et al. 2014 for the example.\n",
        "\n",
        "The Olofsson assessment can either be initialized with an error matrix of pixel counts plus a dictionary of the mapped areas or with a longform table of each pixels map and reference values. Both produce the same results, pick the one that matches the form that your data is in."
      ],
      "metadata": {
        "id": "aWGrQOIXS7z5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "olofsson_mapped_populations = {\n",
        "    \"Deforestation\": 200000,\n",
        "    \"Forest gain\": 150000,\n",
        "    \"Stable forest\": 3200000,\n",
        "    \"Stable non-forest\": 6450000\n",
        "}"
      ],
      "metadata": {
        "id": "BTpb98WEYWUb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# from an error matrix\n",
        "olofsson_data = [\n",
        "    [66, 0, 1, 2],\n",
        "    [0, 55, 0, 1],\n",
        "    [5, 8, 153, 9],\n",
        "    [4, 12, 11, 313],\n",
        "]\n",
        "classes = [\"Deforestation\", \"Forest gain\", \"Stable forest\", \"Stable non-forest\"]\n",
        "olofsson_error_matrix = pd.DataFrame(olofsson_data, index=classes, columns=classes)\n",
        "\n",
        "olofsson_error_matrix"
      ],
      "metadata": {
        "id": "lHsQ-O9sWi8g"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "olofsson_assessment1 = Olofsson(olofsson_error_matrix, olofsson_mapped_populations)\n",
        "olofsson_assessment1"
      ],
      "metadata": {
        "id": "4X412qoeXiM2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# from a lonfrom table\n",
        "# first convert the error matrix to the long form\n",
        "from acc_assessment.utils import _expand_error_matrix\n",
        "olofsson_longform = _expand_error_matrix(olofsson_error_matrix, \"map\", \"ref\")\n",
        "olofsson_longform"
      ],
      "metadata": {
        "id": "H8mLN9hgXoqS"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# to tell it that you are passing a longform table tell it the names of the map\n",
        "# and the reference columns\n",
        "olofsson_assessment2 = Olofsson(\n",
        "    olofsson_longform,\n",
        "    olofsson_mapped_populations,\n",
        "    map_col=\"map\",\n",
        "    ref_col=\"ref\",\n",
        ")\n",
        "olofsson_assessment2"
      ],
      "metadata": {
        "id": "yuZsb5Q0X69P"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Create the Assessment"
      ],
      "metadata": {
        "id": "0h2BhcarTBZ3"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Congalton Assessment Example\n",
        "\n",
        "The Congalton assessment is based on: \"A Review of Assessing the Accuracy of Classifications of Remotely Sensed Data\", Congalton, R. G., 1991. Remote Sensing of Environment, Vol. 37. pp 35-46 https://doi.org/10.1016/0034-4257(91)90048-B"
      ],
      "metadata": {
        "id": "eecFY-NtSpqJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Load Data\n",
        "\n",
        "Use the data from Table 1 in Congalton 1991 to create the example."
      ],
      "metadata": {
        "id": "vVkpSuCgTHng"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "congalton_data = [[65, 4, 22, 24], [6, 81, 5, 8], [0, 11, 85, 19], [4, 7, 3, 90]]\n",
        "congalton_classes = [\"D\", \"C\", \"BA\", \"SB\"]\n",
        "congalton_df = pd.DataFrame(congalton_data, index=congalton_classes, columns=congalton_classes)\n",
        "congalton_table = pd.DataFrame(_expand_error_matrix(congalton_df, \"map\", \"ref\"),)\n",
        "congalton_table"
      ],
      "metadata": {
        "id": "2OkisCJPTOR0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Create the Assessment"
      ],
      "metadata": {
        "id": "HXbIT5MyTJ2b"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "congalton_assessment = Congalton(congalton_table, \"map\", \"ref\")\n",
        "congalton_assessment"
      ],
      "metadata": {
        "id": "q9i1NOjsajEk"
      },
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "colab": {
      "name": "AccuracyAssessmentTools.ipynb",
      "private_outputs": true,
      "provenance": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.16"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}