YOLO is a powerful tool for object detection and thus, obviously interesting in bio-image analysis. This blog post will show you how to train a YOLO model on an OMERO dataset.
+ +### [Getting started with Miniforge and Python](mara_lampert/getting_started_with_miniforge_and_python/readme)] [Mara Lampert](mara_lampert/readme), July 8th, 2024
This post will help you to get started with Python using Miniforge. More precisely, you will learn how to install Miniforge, how to create and use conda environments and you will get to know some important packages for bio image analysis.
diff --git a/docs/johannes_mueller/Readme.md b/docs/johannes_mueller/Readme.md index 8950e5b..adad6ff 100644 --- a/docs/johannes_mueller/Readme.md +++ b/docs/johannes_mueller/Readme.md @@ -1,4 +1,4 @@ -# Johannes MΓΌller +# Johannes Soltwedel
@@ -17,6 +17,7 @@ educate others to empower them to develop their own workflows and develop new me
* [Advanced GUIs with Qt designer](entry_user_interf2/Readme)
* [Automated package documentation with Sphinx](entry_sphinx/Readme)
* [Getting started with Python and Anaconda](anaconda_getting_started/Readme)
+* [Training a yolo model on an OMERO dataset](ref:yolo_omero)
## Links
* [BiA-PoL group website](https://physics-of-life.tu-dresden.de/en/research/technology-development-groups/bio-image-analysis)
diff --git a/docs/johannes_mueller/yolo_from_omero/detection.yaml b/docs/johannes_mueller/yolo_from_omero/detection.yaml
new file mode 100644
index 0000000..ed61e61
--- /dev/null
+++ b/docs/johannes_mueller/yolo_from_omero/detection.yaml
@@ -0,0 +1,10 @@
+path: E:/BiAPoL/yolo_demo/dataset # dataset root dir
+train: train/images # train images (relative to 'path') 128 images
+val: val/images # val images (relative to 'path') 128 images
+test: test/images # test images (optional)
+
+names:
+ 0: cell
+ 1: compacted
+ 2: spheroid
+ 3: dead
\ No newline at end of file
diff --git a/docs/johannes_mueller/yolo_from_omero/imgs/PR_curve.png b/docs/johannes_mueller/yolo_from_omero/imgs/PR_curve.png
new file mode 100644
index 0000000..09b0873
Binary files /dev/null and b/docs/johannes_mueller/yolo_from_omero/imgs/PR_curve.png differ
diff --git a/docs/johannes_mueller/yolo_from_omero/imgs/annotate.gif b/docs/johannes_mueller/yolo_from_omero/imgs/annotate.gif
new file mode 100644
index 0000000..596c926
Binary files /dev/null and b/docs/johannes_mueller/yolo_from_omero/imgs/annotate.gif differ
diff --git a/docs/johannes_mueller/yolo_from_omero/imgs/confusion_matrix.png b/docs/johannes_mueller/yolo_from_omero/imgs/confusion_matrix.png
new file mode 100644
index 0000000..225cdf6
Binary files /dev/null and b/docs/johannes_mueller/yolo_from_omero/imgs/confusion_matrix.png differ
diff --git a/docs/johannes_mueller/yolo_from_omero/imgs/confusion_matrix_normalized.png b/docs/johannes_mueller/yolo_from_omero/imgs/confusion_matrix_normalized.png
new file mode 100644
index 0000000..62e3db7
Binary files /dev/null and b/docs/johannes_mueller/yolo_from_omero/imgs/confusion_matrix_normalized.png differ
diff --git a/docs/johannes_mueller/yolo_from_omero/imgs/image1.png b/docs/johannes_mueller/yolo_from_omero/imgs/image1.png
new file mode 100644
index 0000000..2ff7b47
Binary files /dev/null and b/docs/johannes_mueller/yolo_from_omero/imgs/image1.png differ
diff --git a/docs/johannes_mueller/yolo_from_omero/imgs/image2.PNG b/docs/johannes_mueller/yolo_from_omero/imgs/image2.PNG
new file mode 100644
index 0000000..1b6d1de
Binary files /dev/null and b/docs/johannes_mueller/yolo_from_omero/imgs/image2.PNG differ
diff --git a/docs/johannes_mueller/yolo_from_omero/imgs/image3.PNG b/docs/johannes_mueller/yolo_from_omero/imgs/image3.PNG
new file mode 100644
index 0000000..36511e1
Binary files /dev/null and b/docs/johannes_mueller/yolo_from_omero/imgs/image3.PNG differ
diff --git a/docs/johannes_mueller/yolo_from_omero/imgs/image4.PNG b/docs/johannes_mueller/yolo_from_omero/imgs/image4.PNG
new file mode 100644
index 0000000..4f15317
Binary files /dev/null and b/docs/johannes_mueller/yolo_from_omero/imgs/image4.PNG differ
diff --git a/docs/johannes_mueller/yolo_from_omero/imgs/image5.png b/docs/johannes_mueller/yolo_from_omero/imgs/image5.png
new file mode 100644
index 0000000..54566e2
Binary files /dev/null and b/docs/johannes_mueller/yolo_from_omero/imgs/image5.png differ
diff --git a/docs/johannes_mueller/yolo_from_omero/imgs/image6.png b/docs/johannes_mueller/yolo_from_omero/imgs/image6.png
new file mode 100644
index 0000000..c028a32
Binary files /dev/null and b/docs/johannes_mueller/yolo_from_omero/imgs/image6.png differ
diff --git a/docs/johannes_mueller/yolo_from_omero/imgs/image7.png b/docs/johannes_mueller/yolo_from_omero/imgs/image7.png
new file mode 100644
index 0000000..096f596
Binary files /dev/null and b/docs/johannes_mueller/yolo_from_omero/imgs/image7.png differ
diff --git a/docs/johannes_mueller/yolo_from_omero/imgs/image8.png b/docs/johannes_mueller/yolo_from_omero/imgs/image8.png
new file mode 100644
index 0000000..7fc3dfd
Binary files /dev/null and b/docs/johannes_mueller/yolo_from_omero/imgs/image8.png differ
diff --git a/docs/johannes_mueller/yolo_from_omero/imgs/labels.jpg b/docs/johannes_mueller/yolo_from_omero/imgs/labels.jpg
new file mode 100644
index 0000000..58841bb
Binary files /dev/null and b/docs/johannes_mueller/yolo_from_omero/imgs/labels.jpg differ
diff --git a/docs/johannes_mueller/yolo_from_omero/imgs/train_batch0.jpg b/docs/johannes_mueller/yolo_from_omero/imgs/train_batch0.jpg
new file mode 100644
index 0000000..2d2eed5
Binary files /dev/null and b/docs/johannes_mueller/yolo_from_omero/imgs/train_batch0.jpg differ
diff --git a/docs/johannes_mueller/yolo_from_omero/imgs/val_batch1_labels.jpg b/docs/johannes_mueller/yolo_from_omero/imgs/val_batch1_labels.jpg
new file mode 100644
index 0000000..d776841
Binary files /dev/null and b/docs/johannes_mueller/yolo_from_omero/imgs/val_batch1_labels.jpg differ
diff --git a/docs/johannes_mueller/yolo_from_omero/imgs/val_batch1_pred.jpg b/docs/johannes_mueller/yolo_from_omero/imgs/val_batch1_pred.jpg
new file mode 100644
index 0000000..101d04b
Binary files /dev/null and b/docs/johannes_mueller/yolo_from_omero/imgs/val_batch1_pred.jpg differ
diff --git a/docs/johannes_mueller/yolo_from_omero/imgs/well_102_t0_x2671.34_y1437.11.png b/docs/johannes_mueller/yolo_from_omero/imgs/well_102_t0_x2671.34_y1437.11.png
new file mode 100644
index 0000000..eea0ff7
Binary files /dev/null and b/docs/johannes_mueller/yolo_from_omero/imgs/well_102_t0_x2671.34_y1437.11.png differ
diff --git a/docs/johannes_mueller/yolo_from_omero/train_yolo.ipynb b/docs/johannes_mueller/yolo_from_omero/train_yolo.ipynb
new file mode 100644
index 0000000..6931878
--- /dev/null
+++ b/docs/johannes_mueller/yolo_from_omero/train_yolo.ipynb
@@ -0,0 +1,741 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "(ref:yolo_omero)=\n",
+ "# Training a (YOLO) object detector using data from OMERO\n",
+ "\n",
+ "In this tutorial, you will learn how to use OMERO and YOLO in conjunction to train an object detector and classifier. But first off, before we start - all data used in this tutorial is [available on Zenodo](https://zenodo.org/records/13329153) under a CC-BY 4.0 license. Now to OMERO and YOLO - what are they and why are they useful?\n",
+ "\n",
+ "## OMERO\n",
+ "\n",
+ "[OMERO](https://www.openmicroscopy.org/omero/) - short for *open microscopy environment remote objects* is a widely used data management platform for image data in the life sciences. It allows you to store, organize, and analyze your images in a secure and scalable way. OMERO is open-source and can be easily extended and integrated with other tools and software.\n",
+ "\n",
+ "## YOLO\n",
+ "\n",
+ "[YOLO](https://docs.ultralytics.com/models/yolov10/) - short for *you only look once* is a class of continuously evolving object detection algorithms. YOLO is known for its speed and accuracy and is widely used in computer vision applications. YOLO is open-source and has been implemented in many programming languages and frameworks. It can be used for a variety of applications such as object detection, object classification, tracking or pose estimation - but I think one of the most bread-and-butter applications is still object detection. Contrary to many common segmentation algorithms in bio-image analysis, it does not seek to classify every pixel, but rather to predict *bounding boxes* around objects of interest.\n",
+ "\n",
+ "This already brings forth one of the key advantages of using YOLO for bio-medical image segmentation, *especially* in instance segmentation problems: Pixel classification, without an additional post-processing step is unable to split pixels into different objects - YOLO does this very natively. A typical result of a YOLO model could look like this:\n",
+ "\n",
+ "![\"YOLO](\"./imgs/image1.png\")
\n",
+ "Images © 2022 Johannes Soltwedel. All rights reserved.
\n", + "\n", + "## Getting started\n", + "\n", + "It all starts with a fresh environment. If you don't have Python installed on your machine installed yet, follow [this](ref:miniforge_python) tutorial by Mara Lampert to get everything up and running. You can omit the last step of the tutorial, as we will be creating a different environment.\n", + "\n", + "To do so, bring up your miniforge prompt and create a new environment:\n", + "\n", + "```bash\n", + "conda create -n omero-yolo python=3.9 ezomero tensorboard scikit-image scikit-learn tqdm ultralytics -c bioconda\n", + "conda activate omero-yolo\n", + "```\n", + "\n", + "```{note}\n", + "Conda or mamba? If you are working with a fresh python installation done according to [the tutorial](ref:miniforge_python), then using conda will be fine. If you are using an older version and were used to mamba, you can use mamba instead of conda. The syntax is the same. You can upgrade your conda installation for conda to be as fast as mamba by following [these instructions](https://www.anaconda.com/blog/a-faster-conda-for-a-growing-community). \n", + "```\n", + "\n", + "Since training this kind of model is using deep learning, we will need to use a CUDA-capable (NVIDIA-manufactured) graphics card. Unfortunately, this is a key requirement for this tutorial. To install the necessary packages (i.e., [pytorch](https://pytorch.org/get-started/locally/)), run the following command:\n", + "\n", + "```bash\n", + "conda install pytorch torchvision pytorch-cuda=12.4 -c pytorch -c nvidia\n", + "```\n", + "\n", + "```{note}\n", + "*Note*: The installation may differ depending on your OS or driver version. To make sure everything runs, update your drivers to the latest version and check out the pytorch-homepage for the [correct installation instructions](https://pytorch.org/get-started/locally/).\n", + "```\n", + "\n", + "## Getting data from omero\n", + "\n", + "Now where does OMERO come into play? Essentially, OMERO can serve as a super-easy tool to do annotations on your images - you can even do so collaboratively. In this example, the data we will work with, looks like this:\n", + "\n", + "\n", + "\n", + "...and this is just a small snippet of the data. The entire dataset comprises more than 1500 images of this kind. What exactly we are looking at is not very important, but the job we'll want to do is essentially this: Detect different objects (cells and spheroids) in each image of a multi-well plate. For reference, these are the objects we are searching for:\n", + "\n", + "\n", + "| Cells | Spheroids |\n", + "|-------|-----------|\n", + "|![\"Cells\"](\"./imgs/image3.PNG\")
| ![\"Spheroids\"](\"./imgs/image4.PNG\")
|\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Annotation\n",
+ "\n",
+ "This brings us to an important step, which one would like to avoid in deep learning if possible, but it's a necessary evil: Annotation. In this case, we will use OMERO to annotate the images. The simplest way to go at this in OMERO, is to just draw bounding boxes and write in the comment what sort of object it is. This is a very simple and straightforward way to do this. Unfortunately, OMERO does not (yet?) support simply tagging ROI (regions of interest) with labels, so we have to do this manually:\n",
+ "\n",
+ "\n",
+ "\n",
+ "Still you can apprecciate that this is **much** faster than doing this in a local image viewer and then saving the annotations in a separate file, let alone annotating single pixels. Still, it's a bit cumbersome so the question of how much annotations are enough annotations naturally arises. Read more on the subject in the [respective section below](#how-much-training-data-is-enough?).\n",
+ "\n",
+ "```{note}\n",
+ "Consistency in the naming here is super-important. Make sure to stick to your conventions (i.e., avoid using \"cell\"/\"cells\"/\"Cell\" in the same dataset).\n",
+ "```\n",
+ "\n",
+ "## Training the model\n",
+ "\n",
+ "Before we can actually proceed to train a model, we need to make the dataset available to us on our local machine. For this, we will use the excellent [ezomero package](https://thejacksonlaboratory.github.io/ezomero/index.html) you already installed. Let's write some code!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 28,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import ezomero\n",
+ "import tqdm\n",
+ "import shutil\n",
+ "import os\n",
+ "from skimage import io\n",
+ "from ultralytics import YOLO\n",
+ "import matplotlib.pyplot as plt\n",
+ "\n",
+ "from sklearn.model_selection import train_test_split"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We check whether pytorch has been installed correctly:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import torch\n",
+ "torch.cuda.is_available()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 54,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "host = # the address of your omero server\n",
+ "user = # your username\n",
+ "secure = True\n",
+ "port = 4064\n",
+ "group = # the group you want to connect to (set to None if you want to connect to the default group)\n",
+ "\n",
+ "conn = ezomero.connect(host=host, user=user, secure=secure, port=port, group=group)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We first need all images and the corresponding annotations in the dataset. This could take a litle as we are sending A LOT of requests to the server. We create ourselves a directory (change the path to your liking) where we save the images and the labels from the OMERO server."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "working_directory = r'E:\\BiAPoL\\yolo_demo\\dataset' # the directory where you want to save the dataset - replace with your own directory\n",
+ "images_dir = os.path.join(working_directory, 'images')\n",
+ "labels_dir = os.path.join(working_directory, 'labels')\n",
+ "os.makedirs(images_dir, exist_ok=True)\n",
+ "os.makedirs(labels_dir, exist_ok=True)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now on to actually downloading - here we have to invest a bit more thought. Why? OMERO and YOLO have different definitions on how a bounding box is defined. The image below illustrates this:\n",
+ "\n",
+ "| | OMERO | YOLO |\n",
+ "|-------|-------|------|\n",
+ "|Anchor|![\"OMERO](\"./imgs/image7.png\")
| ![\"YOLO](\"./imgs/image6.png\")
|\n",
+ "|Units | Pixels | Normalized |\n",
+ "Images © 2022 Johannes Soltwedel. All rights reserved.
\n", + "\n", + "To be compliant with YOLO conventions, we need to convert our annotations into the following format and normalize the positions and sizes of the bounding boxes to the width and height of the image:\n", + "```txt\n", + "class_label1 x_center y_center width height\n", + "class_label2 x_center y_center width height\n", + "...\n", + "```\n", + "\n", + "First, we need to convert the wrtten class labls (e.g., \"cell\" or \"spheroid\") into numerical labels. We can do this by creating a dictionary that maps the class labels to numerical labels:" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [], + "source": [ + "object_classes = {\n", + " 'cell': 0,\n", + " 'compacted': 1,\n", + " 'spheroid': 2,\n", + " 'dead': 3,\n", + "}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Downloading the dataset\n", + "\n", + "Now we iterate over all the images in the dataset, retrieve the corresponding rois a(and the shapes they contain - honestly, I don't fully understand the difference between Shapes and ROIs in OMERO π ). Anyway, we then convert the annotations to YOLO format. This means dividing all coordinates by the width and height of the image as well as adding half the width and half the height to the coordinates of the anchor of the box to move it from the upper-left corner to the center.\n", + "\n", + "```Note\n", + "In the following code, you see image data being imported from `dataset=259` - replace this with the id of the dataset that yu are actually working with.\n", + "```" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "100%|ββββββββββ| 1615/1615 [29:28<00:00, 1.10s/it]\n" + ] + } + ], + "source": [ + "img_ids = ezomero.get_image_ids(conn, dataset=259) # replace 259 with the id of the dataset you want to use\n", + "for img_id in tqdm.tqdm(img_ids):\n", + "\n", + " # get the image and metadata and save the image locally\n", + " metadata, image = ezomero.get_image(conn, image_id=img_id, dim_order='tczyx')\n", + " image_filename = os.path.join(working_directory, 'images', f'{metadata.name}.png')\n", + " labels_filename = os.path.join(working_directory, 'labels', f'{metadata.name}.txt')\n", + " \n", + " image = image.squeeze() # remove singleton dimensions (TCZ)\n", + " io.imsave(image_filename, image.squeeze())\n", + "\n", + " # determine the width and height of the image and get associated rois\n", + " width, height = image.shape[1], image.shape[0]\n", + " roi_ids = ezomero.get_roi_ids(conn, image_id=img_id)\n", + "\n", + " shapes = []\n", + " for roi_id in roi_ids:\n", + " shape_ids = ezomero.get_shape_ids(conn, roi_id=roi_id)\n", + " for shape_id in shape_ids:\n", + " shapes.append(ezomero.get_shape(conn, shape_id=shape_id))\n", + "\n", + " # now to convert the shapes to YOLO format\n", + " with open(labels_filename, 'w') as f:\n", + " for shape in shapes:\n", + " w = shape.width / width\n", + " h = shape.height / height\n", + "\n", + " x = shape.x / width + w / 2\n", + " y = shape.y / height + h / 2\n", + "\n", + " class_id = object_classes[shape.label]\n", + "\n", + " f.write(f'{class_id} {x} {y} {w} {h}\\n')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Before we dive into the training, we have to do one final step. It is common practice in deep learning, to split the data into a training, validation and a testing cohort. This is done to evaluate the model on data it has not seen before. We will do a 70/20/10% split for training, validation and testing, respectively. We first create the folder structure:" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [], + "source": [ + "# Output directories\n", + "output_dirs = {\n", + " 'train': {'images': os.path.join(working_directory, 'train/images'), 'labels': os.path.join(working_directory, 'train/labels')},\n", + " 'val': {'images': os.path.join(working_directory, 'val/images'), 'labels': os.path.join(working_directory, 'val/labels')},\n", + " 'test': {'images': os.path.join(working_directory, 'test/images'), 'labels': os.path.join(working_directory, 'test/labels')}\n", + "}\n", + "\n", + "# Create output directories if they don't exist\n", + "for key in output_dirs:\n", + " os.makedirs(output_dirs[key]['images'], exist_ok=True)\n", + " os.makedirs(output_dirs[key]['labels'], exist_ok=True)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We then sort our images and labels into these folders. For thi we set the random seed to a fixed value (`random_state=42`) for reproducibility:" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [], + "source": [ + "# Get list of all image files\n", + "image_files = [f for f in os.listdir(images_dir) if os.path.isfile(os.path.join(images_dir, f))]\n", + "\n", + "# Split the dataset into train, val, and test\n", + "train_files, test_files = train_test_split(image_files, test_size=0.30, random_state=42)\n", + "val_files, test_files = train_test_split(test_files, test_size=1/3, random_state=42) # 1/3 of 30% => 10%\n", + "\n", + "# Move the files to the respective directories\n", + "for key, files in zip(['train', 'val', 'test'], [train_files, val_files, test_files]):\n", + " for file in files:\n", + " image_file = os.path.join(images_dir, file)\n", + " label_file = os.path.join(labels_dir, file.replace('.png', '.txt'))\n", + "\n", + " shutil.copy(image_file, output_dirs[key]['images'])\n", + " shutil.copy(label_file, output_dirs[key]['labels'])" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Number of samples in training set: 1071\n", + "Number of samples in validation set: 306\n", + "Number of samples in test set: 153\n" + ] + } + ], + "source": [ + "print('Number of samples in training set:', len(train_files))\n", + "print('Number of samples in validation set:', len(val_files))\n", + "print('Number of samples in test set:', len(test_files))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Training\n", + "\n", + "Now for the cool part of all deep-learning frameworks: The *training*. Luckily, YOLO makes this quite easy! (Also see the [documentation](https://docs.ultralytics.com/modes/train/#key-features-of-train-mode) on further hints and settings). Before we can start the training, we need to set a few configuration parameters. YOLO requires us to write this in a separate `yaml` file, which in our case would look something like this. \n", + "\n", + "```{note}\n", + "Make sure to replace `path` in the yaml file with the path to your dataset.\n", + "```\n", + "\n", + "```yaml\n", + "# Train/val/test sets as 1) dir: path/to/imgs, 2) file: path/to/imgs.txt, or 3) list: [path/to/imgs1, path/to/imgs2, ..]\n", + "path: E:/BiAPoL/yolo_demo/dataset # dataset root dir\n", + "train: train/images # train images (relative to 'path') 128 images\n", + "val: val/images # val images (relative to 'path') 128 images\n", + "test: test/images # test images (optional)\n", + "\n", + "names:\n", + " 0: cell\n", + " 1: compacted\n", + " 2: spheroid\n", + " 3: dead" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "For the training, we use a pretrained model. This means that the model has already been trained on a variety of image data (unlike ours, of course), but it is likely already able to distinguish basic shapes." + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": {}, + "outputs": [], + "source": [ + "model = YOLO(\"yolov8n.pt\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We start the training and we add a few parameters to the training run. They are all explained in more detail [here](https://docs.ultralytics.com/modes/train/#train-settings), but here's a quick glance:\n", + "\n", + "- `imgsz`: In order to stack images to batches, yolo needs to reshape them into a common shape, which is defined by this parameter. Since our images are quite small themselves, we can set this to a small value, e.g., 128. The value needs to be a potence of 2.\n", + "- `batch`: Number of images in one batch. The bigger your GPU, the bigger this number can be. For a 8GB GPU, 64 is a good starting point.\n", + "- `epochs`: Number of epochs to train the model. Longer training typically goes hand in hand with better performance, but also with [overfitting](https://en.wikipedia.org/wiki/Overfitting).\n", + "- `flipud`, `fliplr`, `degrees`: These parameters are used to augment the data. [Augmentation](https://en.wikipedia.org/wiki/Data_augmentation) is a technique to artificially increase the size of the dataset by applying transformations (flipping, rotations, etc) to the images. This can help the model to generalize better and prevent overfitting. The values given for these parameters are the probabilities that the transformation is applied to an image.\n", + "- `dropout`: Controls whether certain neurons (part of the neuronal network) are \"[dropped out](https://machinelearningmastery.com/dropout-regularization-deep-learning-models-keras/)\" (muted) during training, which corresponds to the network temporarily forgetting what it has learned. This can help to prevent overfitting as the network learns not to overly rely on certain neurons.\n", + "- `plots`: Plots some metrics.\n", + "- `seed`: Removes the randomness from the training process. This is useful if you want to reproduce the results of a training run.\n", + "\n", + "Anyway - read the docs and play around with the parameters to see what works best for your data.\n", + "\n", + "### Tensorboard\n", + "\n", + "If you don't want to look at endless rows of text in your Jupyter notebook, but rather look at a fancy dashboard, you can use [tensorboard](https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html) to visualize the training process. If you followed the instructions above, you already installed tensorboard. You can start tensorboard by running the following command in your terminal:\n", + "\n", + "```bash\n", + "cd path/to/where/this/notebook/is\n", + "tensorboard --logdir=runs\n", + "```\n", + "\n", + "If you are using [VSCode](https://code.visualstudio.com/), use the built in tensorboard extension by just hitting `Ctrl+Shift+P` and typing `tensorboard` and then selecting the folder `runs`.\n", + "\n", + "**Enough with the talk - Let's start the training!**" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "model.train(data=\"./detection.yaml\", epochs=100, imgsz=128, device=0, batch=64, flipud=0.5, fliplr=0.5,\n", + " dropout=True, optimizer='Adam', seed=42, plots=True)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Lucky for us, YOLO runs a few tests before training starts which we can use to make sure whether we actually got all the classes and the x/y width/heigth conversion stuff right. Starting the training will spawn a folder called `runs` adjacent to this notebook. In this folder, we find an overview of several samples along with our bounding box annotations. \n", + "\n", + "\n", + "\n", + "We are also shown an overview over our input annotations. We essentially see, that we are providing a huge number for the object class `cell`, but hardly any for the other classes in comparison. We also see that all labels are sort of square (top right), as also shown by the 2d histogram on the bottom-right:\n", + "\n", + "![\"labels\"](\"./imgs/labels.jpg\")
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Evaluation\n",
+ "\n",
+ "Once the training is done, we should have a look at some of the metrics yolo generates for us. For instance, we are shown some predictions and the corresponding ground truth labels. This is a good way to see how well the model is doing.\n",
+ "\n",
+ "| Labels | Predictions |\n",
+ "|--------|-------------|\n",
+ "| ![\"labels\"](\"./imgs/val_batch1_labels.jpg\")
| ![\"predictions\"](\"./imgs/val_batch1_pred.jpg\")
|\n",
+ "\n",
+ "At a glance - not half-bad, right? Besides just looking at the predicted bounding boxes we can - and we also should - look at some metrics. What's very important here, is the confusion matrix produced by the training, which essentially tells us how well the model is doing in terms of precision and recall.\n",
+ "\n",
+ "| Counts | Normalized |\n",
+ "|--------|------------|\n",
+ "| ![\"confusion](\"./imgs/confusion_matrix.png\")
| ![\"confusion](\"./imgs/confusion_matrix_normalized.png\")
|\n",
+ "\n",
+ "From this we learn that our model is rather mediocre at detecting the `cell` class: In the normalized confusion matrix, we see that the True positive rate is about 50% for the cell class, whereas the model often (42%) predicts background (no object) where we annotated a cell. Interestingly, the model often predicts cells where the annotator only saw background.\n",
+ "\n",
+ "For the spheroids, we are quite good at finding them, with a true positive rate of 84%. There is one way, of course to improve performance: \n",
+ "\n",
+ "*Annotate more data!*\n",
+ "\n",
+ "But that's a story for another day. For now, we have a working model that can detect cells and spheroids in our images.\n",
+ "\n",
+ "## Applying the model\n",
+ "\n",
+ "Lastly, if you want to apply the model on some new data, this can be done fairly easily with YOLO. In a real applciation, replace `list_of_other_images` with a list of paths to your images. Yolo automatically saves two model checkpoints for us to use: The best model and the last model. Obviously, we'll want to use the best model."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 24,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "model = YOLO(\"./runs/detect/train/weights/best.pt\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "list_of_other_images = [os.path.join(output_dirs['test']['images'], f) for f in test_files]\n",
+ "results = model(list_of_other_images)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "```\n",
+ "Speed: 0.3ms preprocess, 1.4ms inference, 1.3ms postprocess per image at shape (1, 3, 128, 128)\n",
+ "```\n",
+ "\n",
+ "...and that's already it. The last thing we will cover in this tutorial is how to get the predictions and convert them back into something that OMERO could understand. This is a bit more tricky, as we have to convert the normalized bounding boxes back into pixel coordinates. This can be done by multiplying the x/y coordinates with the width and height of the image. \n",
+ "\n",
+ "Let's look at a single example image (e.g., `well_102_t0_x2671.34_y1437.11.png`) and see what the predictions return\n",
+ "\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 69,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "0: 128x128 7 cells, 20.5ms\n",
+ "Speed: 2.0ms preprocess, 20.5ms inference, 6.0ms postprocess per image at shape (1, 3, 128, 128)\n"
+ ]
+ }
+ ],
+ "source": [
+ "test_image = os.path.join(output_dirs['test']['images'], 'well_102_t0_x2671.34_y1437.11.png')\n",
+ "prediction = model([test_image])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "There's a bunch of stuff inside the `prediction` object. What we are looking for is stored in the `boxes` attribute. We can see that yolo offers us the result in a buch of different formats (`boxes.xywh`, `boxes.xywhn`, `boxes.xyxy`, `boxes.xyxyn`). We are interested in the `xywh` format, as this is already given in pixel units. The classes of the respective boxes are stored in the `boxes.cls` attribute. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 65,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "ultralytics.engine.results.Boxes object with attributes:\n",
+ "\n",
+ "cls: tensor([0., 0., 0., 0., 0., 0., 0.], device='cuda:0')\n",
+ "conf: tensor([0.6708, 0.6405, 0.5541, 0.3947, 0.3710, 0.3266, 0.2758], device='cuda:0')\n",
+ "data: tensor([[60.0439, 64.5733, 69.9621, 75.9605, 0.6708, 0.0000],\n",
+ " [83.5776, 55.0671, 95.6779, 67.8791, 0.6405, 0.0000],\n",
+ " [57.3633, 83.5636, 66.9192, 94.2983, 0.5541, 0.0000],\n",
+ " [76.0216, 63.1894, 85.6686, 74.9451, 0.3947, 0.0000],\n",
+ " [76.0707, 53.7196, 86.5719, 65.4781, 0.3710, 0.0000],\n",
+ " [67.7796, 76.9948, 79.4629, 87.6710, 0.3266, 0.0000],\n",
+ " [57.3487, 73.0762, 65.8421, 82.4596, 0.2758, 0.0000]], device='cuda:0')\n",
+ "id: None\n",
+ "is_track: False\n",
+ "orig_shape: (146, 136)\n",
+ "shape: torch.Size([7, 6])\n",
+ "xywh: tensor([[65.0030, 70.2669, 9.9182, 11.3872],\n",
+ " [89.6277, 61.4731, 12.1003, 12.8120],\n",
+ " [62.1412, 88.9310, 9.5560, 10.7346],\n",
+ " [80.8451, 69.0672, 9.6470, 11.7557],\n",
+ " [81.3213, 59.5988, 10.5012, 11.7585],\n",
+ " [73.6213, 82.3329, 11.6833, 10.6762],\n",
+ " [61.5954, 77.7679, 8.4934, 9.3834]], device='cuda:0')\n",
+ "xywhn: tensor([[0.4780, 0.4813, 0.0729, 0.0780],\n",
+ " [0.6590, 0.4210, 0.0890, 0.0878],\n",
+ " [0.4569, 0.6091, 0.0703, 0.0735],\n",
+ " [0.5944, 0.4731, 0.0709, 0.0805],\n",
+ " [0.5980, 0.4082, 0.0772, 0.0805],\n",
+ " [0.5413, 0.5639, 0.0859, 0.0731],\n",
+ " [0.4529, 0.5327, 0.0625, 0.0643]], device='cuda:0')\n",
+ "xyxy: tensor([[60.0439, 64.5733, 69.9621, 75.9605],\n",
+ " [83.5776, 55.0671, 95.6779, 67.8791],\n",
+ " [57.3633, 83.5636, 66.9192, 94.2983],\n",
+ " [76.0216, 63.1894, 85.6686, 74.9451],\n",
+ " [76.0707, 53.7196, 86.5719, 65.4781],\n",
+ " [67.7796, 76.9948, 79.4629, 87.6710],\n",
+ " [57.3487, 73.0762, 65.8421, 82.4596]], device='cuda:0')\n",
+ "xyxyn: tensor([[0.4415, 0.4423, 0.5144, 0.5203],\n",
+ " [0.6145, 0.3772, 0.7035, 0.4649],\n",
+ " [0.4218, 0.5724, 0.4921, 0.6459],\n",
+ " [0.5590, 0.4328, 0.6299, 0.5133],\n",
+ " [0.5593, 0.3679, 0.6366, 0.4485],\n",
+ " [0.4984, 0.5274, 0.5843, 0.6005],\n",
+ " [0.4217, 0.5005, 0.4841, 0.5648]], device='cuda:0')"
+ ]
+ },
+ "execution_count": 65,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "prediction[0].boxes"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Also note, that the values shown for the box anchors say `device='cuda:0'`. This means, that the data is still on the GPU. To use it, we need to detach it from the GPU, move it to the CPU and convert it from a pytorch tensor to a numpy array:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 66,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Object class: 0, Box: [ 65.003 70.267 9.9182 11.387]\n",
+ "Object class: 0, Box: [ 89.628 61.473 12.1 12.812]\n",
+ "Object class: 0, Box: [ 62.141 88.931 9.556 10.735]\n",
+ "Object class: 0, Box: [ 80.845 69.067 9.647 11.756]\n",
+ "Object class: 0, Box: [ 81.321 59.599 10.501 11.759]\n",
+ "Object class: 0, Box: [ 73.621 82.333 11.683 10.676]\n",
+ "Object class: 0, Box: [ 61.595 77.768 8.4934 9.3834]\n"
+ ]
+ }
+ ],
+ "source": [
+ "for obj_class, box in zip(prediction[0].boxes.cls, prediction[0].boxes.xywh):\n",
+ " print(f'Object class: {int(obj_class)}, Box: {box.detach().cpu().numpy()}')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Moving the data back to OMERO\n",
+ "\n",
+ "To come full circle, here's a quick glance on how to move the images along with the predictions back to OMERO. Luckily for us, ezomero makes this relatively straightforward:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 67,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "swapped_object_classes = {value: key for key, value in object_classes.items()}\n",
+ "\n",
+ "rectangles = []\n",
+ "for obj_class, box in zip(prediction[0].boxes.cls, prediction[0].boxes.xywh):\n",
+ " box_numpy = box.detach().cpu().numpy()\n",
+ " x = box_numpy[0] - box_numpy[2] / 2\n",
+ " y = box_numpy[1] - box_numpy[3] / 2\n",
+ " width = box_numpy[2]\n",
+ " heigth = box_numpy[3]\n",
+ " \n",
+ " rectangles.append(ezomero.rois.Rectangle(x, y, width, heigth, label=swapped_object_classes[int(obj_class)]))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Lastly, we could upload this image and the corresponding predictions to a new dataset in OMERO. Note that the image has to be converted in 5d (in our case, we set the format to `tczyx`) to be compatible with OMERO.\n",
+ "\n",
+ "```{note}\n",
+ " The steps up to this point can take a bit of time so you may have to reconnect to the omero server.\n",
+ "```"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "image_to_upload = io.imread(test_image)\n",
+ "image_id = ezomero.post_image(\n",
+ " conn, image=image_to_upload[None, None, None, :], dim_order='tczyx',\n",
+ " dataset_id=260, image_name='example_prediction'\n",
+ " )\n",
+ "\n",
+ "roid_id = ezomero.post_roi(conn, image_id=image_id, shapes=rectangles)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "If we check on our OMERO server we see our predictions:\n",
+ "\n",
+ "![\"OMERO](\"./imgs/image8.png\")
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Final thoughts\n",
+ "\n",
+ "That's it! This is quite a lengthy tutorial, so props if you made it all the way here. \n",
+ "\n",
+ "### Used data\n",
+ "\n",
+ "All the data used in this tutorial is [publically available](https://zenodo.org/records/13329153) at the courtesy of Louis Hewitt ([Grapin-Botton Lab](https://www.mpi-cbg.de/research/researchgroups/currentgroups/anne-grapin-botton/research-focus), MPI-CBG, Dresden, Germany). Without their collaboration, you wouldn't be able to look at this tutorial. So, thank you!\n",
+ "\n",
+ "The bespoke data comes as a packed up OMERO project. You can use it by downloading the stored zip file and using the [omero-cli-transfer](https://github.com/ome/omero-cli-transfer) package to upload it to your OMERO server and then walk yourself through this tutorial from the start. To do so, log into omero from the command line like this:\n",
+ "\n",
+ "```bash\n",
+ "omero login -s