YoloV8 serverless support

CHANGELOG.md

@@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## \[Unreleased]
 ### Added
 - Multi-line text attributes supported (<https://github.com/opencv/cvat/pull/6458>)
+- Added YoloV8 serverless support for semi/automatic annotation (<https://github.com/opencv/cvat/pull/6472>)
 
 ### Changed
 - TDB

README.md

@@ -194,6 +194,7 @@ up to 10x. Here is a list of the algorithms we support, and the platforms they c
 | [Semantic segmentation for ADAS](/serverless/openvino/omz/intel/semantic-segmentation-adas-0001/nuclio) | detector   | OpenVINO   | ✔️  |     |
 | [Text detection v4](/serverless/openvino/omz/intel/text-detection-0004/nuclio)                          | detector   | OpenVINO   | ✔️  |     |
 | [YOLO v5](/serverless/pytorch/ultralytics/yolov5/nuclio)                                                | detector   | PyTorch    | ✔️  |     |
+| [YOLO v8](/serverless/pytorch/ultralytics/yolov8/nuclio)                                                | detector   | PyTorch    | ✔️  | ✔️  |
 | [SiamMask](/serverless/pytorch/foolwood/siammask/nuclio)                                                | tracker    | PyTorch    | ✔️  | ✔️  |
 | [TransT](/serverless/pytorch/dschoerk/transt/nuclio)                                                    | tracker    | PyTorch    | ✔️  | ✔️  |
 | [f-BRS](/serverless/pytorch/saic-vul/fbrs/nuclio)                                                       | interactor | PyTorch    | ✔️  |     |

serverless/pytorch/ultralytics/yolov8/nuclio/function-gpu.yaml

@@ -0,0 +1,131 @@
+metadata:
+  name: pth-ultralytics-yolov8
+  namespace: cvat
+  annotations:
+    name: YOLO v8
+    type: detector
+    framework: pytorch
+    spec: |
+      [
+        { "id": 0, "name": "person" },
+        { "id": 1, "name": "bicycle" },
+        { "id": 2, "name": "car" },
+        { "id": 3, "name": "motorbike" },
+        { "id": 4, "name": "aeroplane" },
+        { "id": 5, "name": "bus" },
+        { "id": 6, "name": "train" },
+        { "id": 7, "name": "truck" },
+        { "id": 8, "name": "boat" },
+        { "id": 9, "name": "traffic light" },
+        { "id": 10, "name": "fire hydrant" },
+        { "id": 11, "name": "stop sign" },
+        { "id": 12, "name": "parking meter" },
+        { "id": 13, "name": "bench" },
+        { "id": 14, "name": "bird" },
+        { "id": 15, "name": "cat" },
+        { "id": 16, "name": "dog" },
+        { "id": 17, "name": "horse" },
+        { "id": 18, "name": "sheep" },
+        { "id": 19, "name": "cow" },
+        { "id": 20, "name": "elephant" },
+        { "id": 21, "name": "bear" },
+        { "id": 22, "name": "zebra" },
+        { "id": 23, "name": "giraffe" },
+        { "id": 24, "name": "backpack" },
+        { "id": 25, "name": "umbrella" },
+        { "id": 26, "name": "handbag" },
+        { "id": 27, "name": "tie" },
+        { "id": 28, "name": "suitcase" },
+        { "id": 29, "name": "frisbee" },
+        { "id": 30, "name": "skis" },
+        { "id": 31, "name": "snowboard" },
+        { "id": 32, "name": "sports ball" },
+        { "id": 33, "name": "kite" },
+        { "id": 34, "name": "baseball bat" },
+        { "id": 35, "name": "baseball glove" },
+        { "id": 36, "name": "skateboard" },
+        { "id": 37, "name": "surfboard" },
+        { "id": 38, "name": "tennis racket" },
+        { "id": 39, "name": "bottle" },
+        { "id": 40, "name": "wine glass" },
+        { "id": 41, "name": "cup" },
+        { "id": 42, "name": "fork" },
+        { "id": 43, "name": "knife" },
+        { "id": 44, "name": "spoon" },
+        { "id": 45, "name": "bowl" },
+        { "id": 46, "name": "banana" },
+        { "id": 47, "name": "apple" },
+        { "id": 48, "name": "sandwich" },
+        { "id": 49, "name": "orange" },
+        { "id": 50, "name": "broccoli" },
+        { "id": 51, "name": "carrot" },
+        { "id": 52, "name": "hot dog" },
+        { "id": 53, "name": "pizza" },
+        { "id": 54, "name": "donut" },
+        { "id": 55, "name": "cake" },
+        { "id": 56, "name": "chair" },
+        { "id": 57, "name": "sofa" },
+        { "id": 58, "name": "pottedplant" },
+        { "id": 59, "name": "bed" },
+        { "id": 60, "name": "diningtable" },
+        { "id": 61, "name": "toilet" },
+        { "id": 62, "name": "tvmonitor" },
+        { "id": 63, "name": "laptop" },
+        { "id": 64, "name": "mouse" },
+        { "id": 65, "name": "remote" },
+        { "id": 66, "name": "keyboard" },
+        { "id": 67, "name": "cell phone" },
+        { "id": 68, "name": "microwave" },
+        { "id": 69, "name": "oven" },
+        { "id": 70, "name": "toaster" },
+        { "id": 71, "name": "sink" },
+        { "id": 72, "name": "refrigerator" },
+        { "id": 73, "name": "book" },
+        { "id": 74, "name": "clock" },
+        { "id": 75, "name": "vase" },
+        { "id": 76, "name": "scissors" },
+        { "id": 77, "name": "teddy bear" },
+        { "id": 78, "name": "hair drier" },
+        { "id": 79, "name": "toothbrush" }
+      ]
+
+spec:
+  description: YOLO v8 via Ultralytics Pytorch
+  runtime: 'python:3.8'
+  handler: main:handler
+  eventTimeout: 30s
+  build:
+    image: cvat.pth.ultralytics.yolov8
+    baseImage: ultralytics/yolov5:latest
+
+    directives:
+      preCopy:
+        - kind: USER
+          value: root
+        - kind: RUN
+          value: apt update && apt install --no-install-recommends -y libglib2.0-0
+        - kind: WORKDIR
+          value: /opt/nuclio
+        - kind: RUN
+          value: pip install supervision
+        - kind: WORKDIR
+          value: /opt/nuclio
+
+  triggers:
+    myHttpTrigger:
+      maxWorkers: 1
+      kind: 'http'
+      workerAvailabilityTimeoutMilliseconds: 10000
+      attributes:
+        maxRequestBodySize: 33554432 # 32MB
+
+  resources:
+    limits:
+      nvidia.com/gpu: 1
+
+  platform:
+    attributes:
+      restartPolicy:
+        name: always
+        maximumRetryCount: 3
+      mountMode: volume

serverless/pytorch/ultralytics/yolov8/nuclio/function.yaml

@@ -0,0 +1,126 @@
+metadata:
+  name: pth-ultralytics-yolov8
+  namespace: cvat
+  annotations:
+    name: YOLO v8
+    type: detector
+    framework: pytorch
+    spec: |
+      [
+        { "id": 0, "name": "person" },
+        { "id": 1, "name": "bicycle" },
+        { "id": 2, "name": "car" },
+        { "id": 3, "name": "motorbike" },
+        { "id": 4, "name": "aeroplane" },
+        { "id": 5, "name": "bus" },
+        { "id": 6, "name": "train" },
+        { "id": 7, "name": "truck" },
+        { "id": 8, "name": "boat" },
+        { "id": 9, "name": "traffic light" },
+        { "id": 10, "name": "fire hydrant" },
+        { "id": 11, "name": "stop sign" },
+        { "id": 12, "name": "parking meter" },
+        { "id": 13, "name": "bench" },
+        { "id": 14, "name": "bird" },
+        { "id": 15, "name": "cat" },
+        { "id": 16, "name": "dog" },
+        { "id": 17, "name": "horse" },
+        { "id": 18, "name": "sheep" },
+        { "id": 19, "name": "cow" },
+        { "id": 20, "name": "elephant" },
+        { "id": 21, "name": "bear" },
+        { "id": 22, "name": "zebra" },
+        { "id": 23, "name": "giraffe" },
+        { "id": 24, "name": "backpack" },
+        { "id": 25, "name": "umbrella" },
+        { "id": 26, "name": "handbag" },
+        { "id": 27, "name": "tie" },
+        { "id": 28, "name": "suitcase" },
+        { "id": 29, "name": "frisbee" },
+        { "id": 30, "name": "skis" },
+        { "id": 31, "name": "snowboard" },
+        { "id": 32, "name": "sports ball" },
+        { "id": 33, "name": "kite" },
+        { "id": 34, "name": "baseball bat" },
+        { "id": 35, "name": "baseball glove" },
+        { "id": 36, "name": "skateboard" },
+        { "id": 37, "name": "surfboard" },
+        { "id": 38, "name": "tennis racket" },
+        { "id": 39, "name": "bottle" },
+        { "id": 40, "name": "wine glass" },
+        { "id": 41, "name": "cup" },
+        { "id": 42, "name": "fork" },
+        { "id": 43, "name": "knife" },
+        { "id": 44, "name": "spoon" },
+        { "id": 45, "name": "bowl" },
+        { "id": 46, "name": "banana" },
+        { "id": 47, "name": "apple" },
+        { "id": 48, "name": "sandwich" },
+        { "id": 49, "name": "orange" },
+        { "id": 50, "name": "broccoli" },
+        { "id": 51, "name": "carrot" },
+        { "id": 52, "name": "hot dog" },
+        { "id": 53, "name": "pizza" },
+        { "id": 54, "name": "donut" },
+        { "id": 55, "name": "cake" },
+        { "id": 56, "name": "chair" },
+        { "id": 57, "name": "sofa" },
+        { "id": 58, "name": "pottedplant" },
+        { "id": 59, "name": "bed" },
+        { "id": 60, "name": "diningtable" },
+        { "id": 61, "name": "toilet" },
+        { "id": 62, "name": "tvmonitor" },
+        { "id": 63, "name": "laptop" },
+        { "id": 64, "name": "mouse" },
+        { "id": 65, "name": "remote" },
+        { "id": 66, "name": "keyboard" },
+        { "id": 67, "name": "cell phone" },
+        { "id": 68, "name": "microwave" },
+        { "id": 69, "name": "oven" },
+        { "id": 70, "name": "toaster" },
+        { "id": 71, "name": "sink" },
+        { "id": 72, "name": "refrigerator" },
+        { "id": 73, "name": "book" },
+        { "id": 74, "name": "clock" },
+        { "id": 75, "name": "vase" },
+        { "id": 76, "name": "scissors" },
+        { "id": 77, "name": "teddy bear" },
+        { "id": 78, "name": "hair drier" },
+        { "id": 79, "name": "toothbrush" }
+      ]
+
+spec:
+  description: YOLO v8 via Ultralytics Pytorch
+  runtime: 'python:3.8'
+  handler: main:handler
+  eventTimeout: 30s
+  build:
+
+    image: cvat.pth.ultralytics.yolov8
+    baseImage: ultralytics/ultralytics:latest-cpu
+
+    directives:
+      preCopy:
+        - kind: USER
+          value: root
+        - kind: RUN
+          value: apt update && apt install --no-install-recommends -y libglib2.0-0
+        - kind: RUN
+          value: pip install supervision
+        - kind: WORKDIR
+          value: /opt/nuclio
+
+  triggers:
+    myHttpTrigger:
+      maxWorkers: 1
+      kind: 'http'
+      workerAvailabilityTimeoutMilliseconds: 10000
+      attributes:
+        maxRequestBodySize: 33554432 # 32MB
+
+  platform:
+    attributes:
+      restartPolicy:
+        name: always
+        maximumRetryCount: 3
+      mountMode: volume

serverless/pytorch/ultralytics/yolov8/nuclio/main.py

@@ -0,0 +1,47 @@
+import json
+import base64
+from PIL import Image
+import io
+from ultralytics import YOLO
+import supervision as sv
+
+def init_context(context):
+    context.logger.info("Init context...  0%")
+    # Change model path for custom model
+    model_path = "yolov8m.pt"
+    model = YOLO(model_path)
+    # Read the DL model
+    context.user_data.model = model
+    context.logger.info("Init context...100%")
+
+def handler(context, event):
+    context.logger.info("Run yolo-v8 model")
+    data = event.body
+    buf = io.BytesIO(base64.b64decode(data["image"]))
+    threshold = float(data.get("threshold", 0.5))
+    context.user_data.model.conf = threshold
+    image = Image.open(buf)
+    yolo_results = context.user_data.model(image, conf=threshold)[0]
+    labels = yolo_results.names
+    detections = sv.Detections.from_yolov8(yolo_results)
+    detections = detections[detections.confidence > threshold]
+    boxes = detections.xyxy
+    conf = detections.confidence
+    class_ids = detections.class_id
+
+    results = []    
+    if boxes.shape[0] > 0:
+        for label, score, box in zip(class_ids, conf, boxes):          
+            xtl = int(box[0])
+            ytl = int(box[1])
+            xbr = int(box[2])
+            ybr = int(box[3])
+
+            results.append({
+                    "confidence": str(score),
+                    "label": labels.get(label, "unknown"),
+                    "points": [xtl, ytl, xbr, ybr],
+                    "type": "rectangle",})
+
+    return context.Response(body=json.dumps(results), headers={},
+                            content_type='application/json', status_code=200)

site/content/en/docs/manual/advanced/ai-tools.md

@@ -204,7 +204,8 @@ see [Automatic annotation](/docs/manual/advanced/automatic-annotation/).
 | Mask RCNN                      | The model generates polygons for each instance of an object in the image. <br><br> For more information, see: <li>[GitHub: Mask RCNN](https://github.com/matterport/Mask_RCNN) <li>[Paper: Mask RCNN](https://arxiv.org/pdf/1703.06870.pdf)                                                                                                                                                                                    |
 | Faster RCNN                    | The model generates bounding boxes for each instance of an object in the image. <br>In this model, RPN and Fast R-CNN are combined into a single network. <br><br> For more information, see: <li>[GitHub: Faster RCNN](https://github.com/ShaoqingRen/faster_rcnn) <li>[Paper: Faster RCNN](https://arxiv.org/pdf/1506.01497.pdf)                                                                                             |
 | YOLO v3                        | YOLO v3 is a family of object detection architectures and models pre-trained on the COCO dataset. <br><br> For more information, see: <li>[GitHub: YOLO v3](https://github.com/ultralytics/yolov3) <li>[Site: YOLO v3](https://docs.ultralytics.com/#yolov3) <li>[Paper: YOLO v3](https://arxiv.org/pdf/1804.02767v1.pdf)                                                                                                      |
-| YOLO v5                        | YOLO v5 is a family of object detection architectures and models based on the Pytorch framework. <br><br> For more information, see: <li>[GitHub: YOLO v5](https://github.com/ultralytics/yolov5) <li>[Site: YOLO v5](https://docs.ultralytics.com/#yolov5)                                                                                                                                                                    |
+| YOLO v5                        | YOLO v5 is a family of object detection architectures and models based on the Pytorch framework. <br><br> For more information, see: <li>[GitHub: YOLO v5](https://github.com/ultralytics/yolov5) <li>[Site: YOLO v5](https://docs.ultralytics.com/#yolov5)               
+| YOLO v8                        | YOLO v8 is a family of object detection architectures and models based on the Pytorch and Ultralytics framework. <br><br> For more information, see: <li>[GitHub: YOLO v8](https://github.com/ultralytics/ultralytics) <li>[Site: YOLO v8](https://docs.ultralytics.com/models/yolov8/)               |
 | Semantic segmentation for ADAS | This is a segmentation network to classify each pixel into 20 classes. <br><br> For more information, see: <li>[Site: ADAS](https://docs.openvino.ai/2019_R1/_semantic_segmentation_adas_0001_description_semantic_segmentation_adas_0001.html)                                                                                                                                                                                |
 | Mask RCNN with Tensorflow      | Mask RCNN version with Tensorflow. The model generates polygons for each instance of an object in the image. <br><br> For more information, see: <li>[GitHub: Mask RCNN](https://github.com/matterport/Mask_RCNN) <li>[Paper: Mask RCNN](https://arxiv.org/pdf/1703.06870.pdf)                                                                                                                                                 |
 | Faster RCNN with Tensorflow    | Faster RCNN version with Tensorflow. The model generates bounding boxes for each instance of an object in the image. <br>In this model, RPN and Fast R-CNN are combined into a single network. <br><br> For more information, see: <li>[Site: Faster RCNN with Tensorflow](https://docs.openvino.ai/2021.4/omz_models_model_faster_rcnn_inception_v2_coco.html) <li>[Paper: Faster RCNN](https://arxiv.org/pdf/1506.01497.pdf) |