Understanding_your_Visual_AI_data_with_FiftyOne_.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "In object detection, the quality and characteristics of your training data significantly impact your model's performance.\n",
        "\n",
        "Data curation becomes an important, possibly the most important, part in building a high quality model. Especially when it comes to:\n",
        "\n",
        "#### Improving Data Quality:\n",
        "\n",
        " - **Identifying and Removing Errors:** Data curation helps identify and remove mislabeled images, incorrect bounding boxes, or corrupted data that can mislead the model during training.\n",
        "\n",
        " - **Handling Class Imbalance:** If certain object classes are underrepresented in the dataset, the model might struggle to detect them accurately. Data curation techniques like oversampling or data augmentation can address this imbalance.\n",
        "\n",
        "#### Enhancing Dataset Relevance:\n",
        "\n",
        " - **Focusing on Relevant Data:** Not all data is created equal. Data curation helps select the most relevant subset of data for the specific task, which can improve model efficiency and performance. For example, if you're building a model to detect cars, focusing on images with cars rather than a general dataset with various objects can lead to a more accurate model.\n",
        "\n",
        " - **Filtering Out Noise:** Removing irrelevant or noisy data (e.g., images with poor lighting or occlusions) can help the model focus on the essential features for object detection.\n",
        "\n",
        "#### Optimizing Dataset Size:\n",
        "\n",
        " - **Reducing Training Time:** Large datasets can take a long time to train. Data curation helps create smaller, more efficient datasets that still capture the necessary information for the model to learn effectively. This is the core idea behind the competition you're designing.\n",
        "\n",
        " - **Improving Model Generalizability:** A smaller, well-curated dataset can sometimes lead to better generalizability, forcing the model to learn the essential features rather than overfitting specific examples in a large dataset.\n",
        "\n",
        "#### Data Curation Techniques for Object Detection:\n",
        "\n",
        " - **Data Cleaning:** Correcting labelling errors, removing duplicates, and handling missing values.\n",
        "\n",
        " - **Data Augmentation:** Creating new training examples by applying transformations like flipping, cropping, rotating, or adding noise to existing images.\n",
        "\n",
        " - **Dataset Balancing:** Addressing class imbalance through techniques like oversampling or undersampling.\n",
        "\n",
        " - **Hard Example Mining:** Identifying and focusing on examples the model finds difficult to classify correctly.\n",
        "\n",
        "By applying these techniques, you'll create a cleaner, more relevant, and more efficient dataset, ultimately leading to a better-performing object detection model.\n",
        "\n",
        "This notebook will show you how you can use the open source library, [`fiftyone`](github.com/voxel51/fiftyone), to perform the tasks listed above.\n",
        "\n",
        "Let's get into it!"
      ],
      "metadata": {
        "id": "zcJyWCRzfizN"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5dQeYOdGMDGp"
      },
      "outputs": [],
      "source": [
        "!pip install fiftyone"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Wt58MLl2eKK-"
      },
      "outputs": [],
      "source": [
        "!pip install umap-learn"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Fine-tuning a pretrained model on a new dataset is a common practice in machine learning, especially when dealing with specific domains or limited data availability. Ensuring that the dataset is of high quality is critical for the success of such an endeavor.\n",
        "\n",
        "Here’s a systematic approach to assess and improve the quality of a dataset using `fiftyone` before fine-tuning a model:\n"
      ],
      "metadata": {
        "id": "VT3P8Vsg0zia"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qRN4pLc1NqSJ"
      },
      "outputs": [],
      "source": [
        "import fiftyone as fo\n",
        "import fiftyone.zoo as foz\n",
        "\n",
        "voc_dataset = foz.load_zoo_dataset(\"voc-2012\", split=\"train\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Rgz-zq5TUM7i"
      },
      "outputs": [],
      "source": [
        "voc_dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LV6mzaI9exo3"
      },
      "source": [
        "\n",
        "#**Visual Inspection**\n",
        "\n",
        "Start by visually inspecting the dataset.\n",
        "\n",
        "- Examine the images and their corresponding annotations\n",
        "\n",
        "- Verify the accuracy, consistency, and identify any apparent errors or anomalies\n",
        "\n",
        "- Scrutinize annotations for incorrect labels\n",
        "\n",
        "- Check for missing or inaccurate bounding boxes\n",
        "\n",
        "\n",
        "You can do this via the `fiftyone` app and the Python SDK - both of which allow you to [tag any image issues you encounter during your exploration](https://docs.voxel51.com/user_guide/app.html#tags-and-tagging)!\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DPSq9zLGPJvt"
      },
      "outputs": [],
      "source": [
        "session = fo.launch_app(voc_dataset)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "You can verify the number of images in the training set:"
      ],
      "metadata": {
        "id": "1xDx4p8APLxR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "voc_dataset.count()\n",
        "\n",
        "# note, you can also use len(voc_dataset)"
      ],
      "metadata": {
        "id": "FYwhiOOpPHpk"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "And the number of detections:"
      ],
      "metadata": {
        "id": "h5Ep90YlPQKM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "voc_dataset.aggregate(fo.Count(\"ground_truth.detections\"))"
      ],
      "metadata": {
        "id": "orf3Y1_SOZLV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "You see we have more detections than images, so there will be multiple detections per image. It's a good idea to get some statistics about what is in the dataset."
      ],
      "metadata": {
        "id": "fJvSyO7bPTT9"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2wjLXgH0eqjT"
      },
      "source": [
        "# **Statistical Analysis**\n",
        "\n",
        "When curating a dataset for an object detection task, performing key statistical analyses helps ensure the quality and effectiveness of your data. By understanding and optimizing your dataset, you can significantly improve the performance and robustness of your object detection models.\n",
        "\n",
        "Here are some analyses you can conduct:\n",
        "\n",
        "1. Class Distribution:\n",
        "   - Analyze the distribution of object classes in your dataset\n",
        "   - Identify and address class imbalances to prevent biased models\n",
        "   - Ensure sufficient representation of all classes for accurate detection\n",
        "\n",
        "2. Bounding Box Size Distribution:\n",
        "   - Compute the width and height distributions of bounding boxes\n",
        "   - Identify variations in object sizes and ensure coverage of diverse scales\n",
        "   - Curate a dataset with a representative range of object sizes\n",
        "\n",
        "3. Object Density:\n",
        "   - Calculate the average number of object instances per image\n",
        "   - Ensure a balanced distribution of single-object and multi-object scenes\n",
        "   - Curate a dataset that reflects the expected object density in real-world scenarios\n",
        "\n",
        "4. Image Resolution:\n",
        "   - Analyze the distribution of image resolutions\n",
        "   - Ensure a consistent and appropriate resolution for accurate object detection\n",
        "   - Curate a dataset with images of sufficient detail while considering computational efficiency\n",
        "\n",
        "5. Object Aspect Ratios:\n",
        "   - Compute the aspect ratios (width/height) of bounding boxes\n",
        "   - Identify variations in object shapes and ensure coverage of diverse aspect ratios\n",
        "   - Curate a dataset with a representative range of object shapes\n",
        "\n",
        "6. Object Co-occurrence:\n",
        "   - Analyze the co-occurrence of different object classes within the same image\n",
        "   - Identify potential correlations or dependencies between classes\n",
        "   - Curate a dataset that reflects realistic object co-occurrence patterns\n",
        "\n",
        "By conducting these statistical analyses and curating your dataset accordingly, you can ensure that your object detection models are trained on high-quality, representative data. This data-centric approach leads to improved model performance, increased robustness to real-world variations, and more reliable object detection results.\n",
        "\n",
        "#### You can do this in `fiftyone`!\n",
        "\n",
        "`fiftyone` allows you to analyze your dataset with [pandas style queries](https://docs.voxel51.com/cheat_sheets/pandas_vs_fiftyone.html) and [dataset views](https://docs.voxel51.com/user_guide/using_views.html#using-views) so that you can perform statistical analysis of your dataset.\n",
        "\n",
        "I'll show you how to get started.\n",
        "\n",
        "\n",
        "#### **Distribution of Classes**\n",
        "\n",
        "Analyze the distribution of classes to identify any imbalance. Highly imbalanced datasets can lead to biased models that perform well on frequent classes but poorly on rare classes.\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "voc_dataset.distinct(\"ground_truth.detections.label\")"
      ],
      "metadata": {
        "id": "0jqAnYLq02r3"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "voc_dataset.count_values(\"ground_truth.detections.label\")"
      ],
      "metadata": {
        "id": "ierlnuUHOZUF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### **Annotation Metrics**\n",
        "\n",
        "Calculate metrics such as:\n",
        "\n",
        "• The number of bounding boxes per image\n",
        "\n",
        "• Average area of bounding boxes (as a percentage of total image area)"
      ],
      "metadata": {
        "id": "AVdej43BOH1e"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "FiftyOne provides [`ViewField`](https://docs.voxel51.com/api/fiftyone.core.expressions.html#fiftyone.core.expressions.ViewField) and [`ViewExpression`](https://docs.voxel51.com/api/fiftyone.core.expressions.html#fiftyone.core.expressions.ViewExpression) that give you the power to write custom queries based on information that exists in your dataset.\n"
      ],
      "metadata": {
        "id": "6lMCw0bUUBXI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from fiftyone import ViewField as F"
      ],
      "metadata": {
        "id": "0MQVBjsKPdkL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Get the number of detections per image"
      ],
      "metadata": {
        "id": "-zhPjJXcXfGC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "detection_counter = F(\"ground_truth.detections\").length()\n",
        "num_detections_per_image = voc_dataset.values(detection_counter) #returns a list"
      ],
      "metadata": {
        "id": "hBxVgny-VtFT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "And you can plot them:"
      ],
      "metadata": {
        "id": "Imzgbx-5d1Gj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from fiftyone import ViewField as F\n",
        "from fiftyone.core.plots.views import CategoricalHistogram\n",
        "\n",
        "CategoricalHistogram(\n",
        "    init_view=dataset,\n",
        "    field_or_expr=\"ground_truth\",\n",
        "    expr=F(\"detections\").length(),\n",
        "    title=\"Count of Images by Number of Detections\",\n",
        "    xlabel=\"Number of Detections per image\",\n",
        "    template={\n",
        "        \"layout\": {\n",
        "            \"xaxis\": {\n",
        "                \"range\": [0, 30]  # This sets the x-axis range from 0 to 30\n",
        "            }\n",
        "        }\n",
        "    }\n",
        ")"
      ],
      "metadata": {
        "id": "vI8d4Md2XKR1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "And if you're curious about that samples with more than $n$ number detections, you do that with [filtering and matching](https://docs.voxel51.com/cheat_sheets/filtering_cheat_sheet.html#built-in-filter-and-match-functions)."
      ],
      "metadata": {
        "id": "ZPfjSZNOeXXL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "many_detections_view = voc_dataset.match(detection_counter>20)"
      ],
      "metadata": {
        "id": "-NhDVUitek0d"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "You can programatically get a sense of what's in these images:"
      ],
      "metadata": {
        "id": "d5gxiVLmlU3j"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "many_detections_view.count_values(\"ground_truth.detections.label\")"
      ],
      "metadata": {
        "id": "PcEH6o5hlGp-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "And, of course, you can visually inspect them:"
      ],
      "metadata": {
        "id": "HEZQZyTrlbQd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "fo.launch_app(many_detections_view)"
      ],
      "metadata": {
        "id": "HMwB6sX4eXMo"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Analyze area of bounding boxes per image.\n",
        "\n",
        "First, compute some metadata for each image. Namely, the image height and width.\n"
      ],
      "metadata": {
        "id": "o8dYNNFaRzPz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "voc_dataset.compute_metadata()"
      ],
      "metadata": {
        "id": "PHcoLFxcljqQ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Then, compute the bounding box.\n",
        "\n",
        "Below the absolute and relative bounding box areas are being computed. Absolute bounding box area is the actual pixel dimensions of the bounding box rectangle. Relative bounding box area is area of the bounding box as a percentage of the total image/canvas size.\n",
        "\n",
        "Note: that bounding boxes are in `[top-left-x, top-left-y, width, height]` format.\n"
      ],
      "metadata": {
        "id": "eTgQly6cpzDX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "rel_bbox_area = F(\"bounding_box\")[2] * F(\"bounding_box\")[3]\n",
        "\n",
        "im_width, im_height = F(\"$metadata.width\"), F(\"$metadata.height\")\n",
        "\n",
        "abs_area = rel_bbox_area * im_width * im_height\n",
        "\n",
        "voc_dataset.set_field(\"ground_truth.detections.relative_bbox_area\", rel_bbox_area).save()\n",
        "\n",
        "voc_dataset.set_field(\"ground_truth.detections.absolute_bbox_area\", abs_area).save()"
      ],
      "metadata": {
        "id": "mDhjo2sYljwu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "You can compute the upper and lower [bounds](https://docs.voxel51.com/api/fiftyone.core.aggregations.html#fiftyone.core.aggregations.Bounds) of the bounding box areas as well as the mean for each.\n",
        "\n",
        "Note: these are relative bounding box areas, so they represent the percentage of the total image area.\n",
        "\n",
        "Here's how you can do that:"
      ],
      "metadata": {
        "id": "PeajVGgs3f4m"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "labels = voc_dataset.distinct(\"ground_truth.detections.label\")\n",
        "for label in labels:\n",
        "    view = voc_dataset.filter_labels(\"ground_truth\", F(\"label\") == label)\n",
        "    bounds = view.aggregate(fo.Bounds(\"ground_truth.detections.relative_bbox_area\"))\n",
        "    bounds = (bounds[0]*100, bounds[1]*100)\n",
        "    area = view.mean(\"ground_truth.detections.relative_bbox_area\")*100\n",
        "    print(\"\\033[1m%s:\\033[0m Min: %.2f, Mean: %.2f, Max: %.2f\" % (label, bounds[0], area, bounds[1]))\n"
      ],
      "metadata": {
        "id": "A2fIwQez08p_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Compute overlap over bounding boxes\n",
        "\n",
        "\n",
        "You can use The [`compute_max_ious`](https://docs.voxel51.com/api/fiftyone.utils.iou.html#fiftyone.utils.iou.compute_max_ious) function to compute the maximum Intersection over Union (IoU) between objects in a dataset.\n",
        "\n",
        "IoU is a measure of overlap between two objects, commonly used in object detection tasks.\n"
      ],
      "metadata": {
        "id": "x53oyZuXU_s8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from fiftyone.utils import iou\n",
        "iou.compute_max_ious(voc_dataset, \"ground_truth\")"
      ],
      "metadata": {
        "id": "XAfkG3hy0oot"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "And you can see there now a field in the detections for each object called `max_iou`.\n",
        "\n",
        "The `max_iou` value indicates the highest degree of overlap between an object and any other object in the same sample or frame. A higher `max_iou` value suggests that the object has a significant overlap with at least one other object.\n",
        "\n",
        "You can use this a criterion for filtering or thresholding objects based on their overlap with other objects. For example, you might choose to keep only objects with a `max_iou` value below a certain threshold to remove highly overlapping or duplicate objects.\n",
        "\n",
        "Or, it can be a starting point for further analysis or processing. For instance, you might use the `max_iou` values to identify objects with high overlap and perform additional operations on those objects."
      ],
      "metadata": {
        "id": "mTMzF5mZXxgv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "voc_dataset.first().ground_truth"
      ],
      "metadata": {
        "id": "VoPmbzbwXh5j"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "You can count the number of overlapping objects in the dataset like so. Note that if a detection has no overlap, the `max_iou` value is set to `None` andthe `None` values are ignored in `count`."
      ],
      "metadata": {
        "id": "UiuZGlWEZsrR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "voc_dataset.count(\"ground_truth.detections.max_iou\", safe=True)"
      ],
      "metadata": {
        "id": "jgnLt9N3bchU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "You can get the average intersection over all the detections."
      ],
      "metadata": {
        "id": "jivhX6fEXbfT"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "voc_dataset.mean(\"ground_truth.detections.max_iou\")"
      ],
      "metadata": {
        "id": "BCxj7__jSEbU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "You can do the same for each label as well"
      ],
      "metadata": {
        "id": "UMFSRBizb5iU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "labels = voc_dataset.distinct(\"ground_truth.detections.label\")\n",
        "for label in labels:\n",
        "    view = voc_dataset.filter_labels(\"ground_truth\", F(\"label\") == label)\n",
        "    bounds = view.aggregate(fo.Bounds(\"ground_truth.detections.max_iou\"))\n",
        "    bounds = (bounds[0]*100, bounds[1]*100)\n",
        "    area = view.mean(\"ground_truth.detections.max_iou\")*100\n",
        "    print(\"\\033[1m%s:\\033[0m Min: %.2f, Mean: %.2f, Max: %.2f\" % (label, bounds[0], area, bounds[1]))"
      ],
      "metadata": {
        "id": "c9rpcztdb5md"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gn9XpduXeqs9"
      },
      "source": [
        "# **Quality Checks**\n",
        "\n",
        "### **Duplicate Detection**\n",
        "\n",
        "You can use `fiftyone` to detect and [remove duplicate images](https://docs.voxel51.com/recipes/image_deduplication.html#) or very similar images (which could lead to overfitting).\n",
        "\n",
        "\n",
        "Start by iterating over the samples and compute their file hashes:\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import fiftyone.core.utils as fou\n",
        "\n",
        "for sample in voc_dataset:\n",
        "    sample[\"file_hash\"] = fou.compute_filehash(sample.filepath)\n",
        "    sample.save()"
      ],
      "metadata": {
        "id": "z2HDO3Qaydqy"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The, you can use a simple Python statement to locate the duplicate files in the dataset, i.e., those with the same file hashses:"
      ],
      "metadata": {
        "id": "Amkt4xhKydiP"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from collections import Counter\n",
        "\n",
        "filehash_counts = Counter(sample.file_hash for sample in voc_dataset)\n",
        "dup_filehashes = [k for k, v in filehash_counts.items() if v > 1]\n",
        "\n",
        "print(\"Number of duplicate file hashes: %d\" % len(dup_filehashes))"
      ],
      "metadata": {
        "id": "QSCoEXoLydaG"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Awesome, no duplicates.\n",
        "\n",
        "But if the above indicated that are any duplicates you could visually verify by creating a view that contains only the samples with these duplicate file hashes using the following pattern:\n",
        "\n",
        "```python\n",
        "dup_view = (dataset\n",
        "    # Extract samples with duplicate file hashes\n",
        "    .match(F(\"file_hash\").is_in(dup_filehashes))\n",
        "    # Sort by file hash so duplicates will be adjacent\n",
        "    .sort_by(\"file_hash\")\n",
        ")\n",
        "\n",
        "print(\"Number of images that have a duplicate: %d\" % len(dup_view))\n",
        "print(\"Number of duplicates: %d\" % (len(dup_view) - len(dup_filehashes)))\n",
        "```\n",
        "\n",
        "And then you can inspect this view in the app:\n",
        "\n",
        "```python\n",
        "session.view = dup_view\n",
        "```\n",
        "\n",
        "\n",
        "Alternatively, you can use the [deduplication plugin](https://github.com/jacobmarks/image-deduplication-plugin), which streamlines this workflow!\n"
      ],
      "metadata": {
        "id": "Id-M3w5dOIA-"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **Image quality**\n",
        "\n",
        "Poor image quality can hurt a model's ability to learn meaningful features and generalize well.\n",
        "\n",
        "Some aspects of image quality you want to examine are:\n",
        "\n",
        "- **Image Resolution:** Check for low-resolution images that might lack sufficient detail for the model to learn effectively.\n",
        "\n",
        "- **Image Noise:** Look for images with excessive noise, blur, or artifacts that could negatively impact model performance.\n",
        "\n",
        "- **Lighting Conditions:** Assess the variability of lighting conditions in the images. Extreme lighting variations can pose challenges for the model.\n",
        "\n",
        "`fiftyone` has a robust [plugins ecosystem](https://github.com/voxel51/fiftyone-plugins?tab=readme-ov-file), and you can use the [Image Quality Issues Plugin](https://github.com/jacobmarks/image-quality-issues) to find the following issues:\n",
        "\n",
        "**📏 Aspect ratio**: find images with weird aspect ratios\n",
        "\n",
        "**🌫️ Blurriness**: find blurry images\n",
        "\n",
        "**☀️ Brightness**: find bright and dark images\n",
        "\n",
        "**🌓 Contrast**: find images with high or low contrast\n",
        "\n",
        "**🔀 Entropy**: find images with low entropy\n",
        "\n",
        "**📸 Exposure**: find overexposed and underexposed images\n",
        "\n",
        "**🕯️ Illumination**: find images with uneven illumination\n",
        "\n",
        "**🧂 Noise**: find images with high salt and pepper noise\n",
        "\n",
        "**🌈 Saturation**: find images with low and high saturation\n",
        "\n",
        "Start by **downloading the plug in:**"
      ],
      "metadata": {
        "id": "r3KeEmCeOVoj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!fiftyone plugins download https://github.com/jacobmarks/image-quality-issues/"
      ],
      "metadata": {
        "id": "wAi1khqzmdXj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Note that you can compute image quality issues one at at time, programmatically, like so:"
      ],
      "metadata": {
        "id": "HjkQZp0Hvf7B"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import fiftyone.operators as foo\n",
        "## Access the operator via its URI (plugin name + operator name)\n",
        "compute_brightness = foo.get_operator(\n",
        "    \"@jacobmarks/image_issues/compute_brightness\"\n",
        ")"
      ],
      "metadata": {
        "id": "MkQdndntkp_4"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "compute_brightness(voc_dataset)"
      ],
      "metadata": {
        "id": "eqsSCcdpkqDs"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "And then inspect them in the app.\n",
        "\n",
        "Or, you can hit the backtick (`) key, and pick and choose which one axis of image quality you want to assess. You can do this on the whole image, or patch, level. [Check the documentation for more information](https://github.com/jacobmarks/image-quality-issues)."
      ],
      "metadata": {
        "id": "ZerLqDk5vzrq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "fo.launch_app(voc_dataset)"
      ],
      "metadata": {
        "id": "TLmUJYGCtGtU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Some ideas for overcoming these issues using  Data Augmentation Techniques\n",
        "\n",
        "• Apply random cropping, flipping, and rotations to the images during training.\n",
        "This helps the model learn invariances to these transformations.\n",
        "\n",
        "• Use color jittering to randomly adjust the brightness, contrast, saturation, and hue of the images. This simulates different lighting conditions.\n",
        "\n",
        "• Add random Gaussian noise to the images to improve the model's robustness to noisy input\n",
        "\n",
        "`fiftyone` has an [integration with Albumentations](https://docs.voxel51.com/integrations/albumentations.html) that can help you with this!"
      ],
      "metadata": {
        "id": "3EQp5o_9kqHZ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Using embeddings\n",
        "\n",
        "[Visualizing your dataset in a low-dimensional embedding space](https://docs.voxel51.com/tutorials/image_embeddings.html) can reveal [patterns and clusters](https://docs.voxel51.com/tutorials/clustering.html#Computing-and-Visualizing-Clusters) in your data that can answer important questions about the critical failure modes of your model and how to augment your dataset to address these failures.\n",
        "\n",
        "The [`fiftyone` model zoo](https://docs.voxel51.com/user_guide/model_zoo/models.html) has several embeddings models you can choose from. You can start with the [`clip-vit-base32-torch`](https://docs.voxel51.com/user_guide/model_zoo/models.html#clip-vit-base32-torch) model.\n",
        "\n",
        "\n",
        "For this, you'll need to use the [`compute_visualization`](https://docs.voxel51.com/api/fiftyone.brain.html#fiftyone.brain.compute_visualization) method of the [`FiftyOne Brain`](https://docs.voxel51.com/user_guide/brain.html#brain-embeddings-visualization)"
      ],
      "metadata": {
        "id": "Ts1WNe1vkqOa"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import fiftyone.brain as fob\n",
        "\n",
        "res = fob.compute_visualization(\n",
        "    voc_dataset,\n",
        "    model=\"clip-vit-base32-torch\",\n",
        "    embeddings=\"clip_embeddings\",\n",
        "    method=\"umap\",\n",
        "    brain_key=\"clip_vis\",\n",
        "    batch_size=64,\n",
        "    num_dims=5\n",
        ")\n",
        "\n",
        "voc_dataset.set_values(\"clip_umap\", res.current_points)"
      ],
      "metadata": {
        "id": "ng9RBMvR1qEd"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "And once you have this, you can inspect the embeddings. Check out [the documentation](https://docs.voxel51.com/user_guide/brain.html#brain-embeddings-visualization) for more examples of how to use embeddings to understand your data."
      ],
      "metadata": {
        "id": "pegCAg9R55BS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "session = fo.launch_app(voc_dataset)"
      ],
      "metadata": {
        "id": "CffCDQcD1qJB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### **Outlier Detection**\n",
        "\n",
        "You can use embeddings to look for outliers or anomalous samples in your dataset. Outliers can include images with unusual objects, extreme lighting conditions, or rare poses. Investigate whether these outliers are valid and informative or if they are noisy and potentially harmful to the model's performance.\n",
        "\n",
        "Then, you can decide whether to keep, remove, or separately handle the outliers based on their relevance and impact.\n",
        "\n",
        "You can use the [outlier detection](https://github.com/danielgural/outlier_detection) plugin to do some interesting analysis!\n",
        "\n",
        "\n",
        "Note, the above computed embeddings on the whole image level, but you can also compute embeddings on the patch level. Below is the pattern for that:\n"
      ],
      "metadata": {
        "id": "VLnV5Mzkeqmn"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Tdq9S0gjVjo7"
      },
      "outputs": [],
      "source": [
        "clip_embeddings_model = foz.load_zoo_model('clip-vit-base32-torch')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bK20Xx5vhGCe"
      },
      "outputs": [],
      "source": [
        "gt_patches = voc_dataset.to_patches(\"ground_truth\").clone()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DYzxNsl7UKbc"
      },
      "outputs": [],
      "source": [
        "gt_patches.compute_patch_embeddings(\n",
        "    model=clip_embeddings_model,\n",
        "    patches_field='ground_truth',\n",
        "    embeddings_field = 'patch_embeddings',\n",
        "    batch_size=64,\n",
        "    progress=True\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7d-6lZE-UKU7"
      },
      "outputs": [],
      "source": [
        "import fiftyone.brain as fob\n",
        "\n",
        "fob.compute_visualization(\n",
        "    gt_patches,\n",
        "    embeddings=\"patch_embeddings\",\n",
        "    method=\"umap\",\n",
        "    brain_key=\"umap_clip\",\n",
        "    num_dims=3\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "fo.launch_app(gt_patches)"
      ],
      "metadata": {
        "id": "ekAodi3K9bBh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# The `FiftyOne Brain` is super powerful!\n",
        "\n",
        "Using the Brain, you can also compute:\n",
        "\n",
        "- [Image Uniqueness](https://docs.voxel51.com/user_guide/brain.html#image-uniqueness)\n",
        "\n",
        "- [Label Mistakes](https://docs.voxel51.com/user_guide/brain.html#label-mistakes)\n",
        "\n",
        "Give it try!\n",
        "\n",
        "----\n",
        "\n",
        "Remember, in data-centric AI, the quality and composition of your dataset is the most important thing. By investing time and effort into understanding and optimizing your data, you can unlock the full potential of your object detection models and achieve superior results compared to solely focusing on model architecture and hyperparameters.\n",
        "\n",
        "\n",
        "#### Actionable Steps\n",
        "\n",
        "To operationalize these strategies, consider the following steps:\n",
        "\n",
        "- **Develop a Checklist**: Create a comprehensive checklist based on the points above to ensure thorough review and evaluation of the dataset.\n",
        "\n",
        "- **Automate What You Can**: Develop or use existing tools to automate parts of the quality check, like corruption checks, duplicate detection, and basic annotation verification.\n",
        "\n",
        "- **Document Everything**: Keep detailed records of the quality assessment process, findings, and actions taken. This documentation will be crucial for understanding decisions made during dataset curation and model training.\n",
        "\n",
        "By systematically assessing the data quality through these steps, you can significantly increase the chances of successful model performance when fine-tuning on new datasets."
      ],
      "metadata": {
        "id": "ivN_kmG7kqRk"
      }
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "L4",
      "machine_shape": "hm",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}