Error on training YOLO v8 #16525

lavrentijav · 2024-09-28T18:46:25Z

lavrentijav
Sep 28, 2024

Please help me debag train on CPU

##My code
from ultralytics import YOLO
import cv2
import numpy as np

print("load model")
model = YOLO('yolov8n.yaml')

print("start train")
results = model.train(
data='data.yaml',
epochs=10,
imgsz=480,
name='yolov8n_custom',
device="cpu")
print("the train is finish")

print("start val")
results = model.val()
print("the val is finish")
print("model exported to", model.export())

print()

colors = [
(255, 0, 0), (0, 255, 0), (0, 0, 255), (255, 255, 0), (0, 255, 255),
(255, 0, 255), (192, 192, 192), (128, 128, 128), (128, 0, 0), (128, 128, 0),
(0, 128, 0), (128, 0, 128), (0, 128, 128), (0, 0, 128), (72, 61, 139),
(47, 79, 79), (47, 79, 47), (0, 206, 209), (148, 0, 211), (255, 20, 147)
]

input_video_path = input("enter path to file of video")
capture = cv2.VideoCapture(input_video_path)

fps = int(capture.get(cv2.CAP_PROP_FPS))
width = int(capture.get(cv2.CAP_PROP_FRAME_WIDTH))
height = int(capture.get(cv2.CAP_PROP_FRAME_HEIGHT))

output_video_path = 'detect.mp4'
fourcc = cv2.VideoWriter_fourcc(*'mp4v')
writer = cv2.VideoWriter(output_video_path, fourcc, fps, (width, height))

while True:
ret, frame = capture.read()
if not ret:
break

results = model(frame)[0]

classes_names = results.names
classes = results.boxes.cls.cpu().numpy()
boxes = results.boxes.xyxy.cpu().numpy().astype(np.int32)
for class_id, box, conf in zip(classes, boxes, results.boxes.conf):
    if conf > 0.5:
        class_name = classes_names[int(class_id)]
        color = colors[int(class_id) % len(colors)]
        x1, y1, x2, y2 = box
        cv2.rectangle(frame, (x1, y1), (x2, y2), color, 2)
        cv2.putText(frame, class_name, (x1, y1 - 10), cv2.FONT_HERSHEY_SIMPLEX, 0.5, color, 2)

writer.write(frame)

capture.release()
writer.release()

while True:
command = input("Enter command")
try:
eval(command)
except Exception:
try:
exec(command)
except Exception as error:
print(error)

##My system
Xeon E3-1225 v2
Nvidia GTX 1660 super
Windows 10 22H2
16 GB DDR 3
CUDA: 12.4

##Log
#Start 1
C:\Users\Lavrenitijav\AppData\Local\Microsoft\WindowsApps\python3.12.exe D:\archive_09_03_23\PycharmProjects\YOLO\now_object.py
load model
start train
New https://pypi.org/project/ultralytics/8.2.103 available 😃 Update with 'pip install -U ultralytics'
Ultralytics YOLOv8.2.102 🚀 Python-3.12.6 torch-2.4.1+cu124 CPU (Intel Xeon E3-1225 V2 3.20GHz)
engine\trainer: task=detect, mode=train, model=yolov8n.yaml, data=data.yaml, epochs=10, time=None, patience=100, batch=16, imgsz=480, save=True, save_period=-1, cache=False, device=cpu, workers=8, project=None, name=yolov8n_custom3, exist_ok=False, pretrained=True, optimizer=auto, verbose=True, seed=0, deterministic=True, single_cls=False, rect=False, cos_lr=False, close_mosaic=10, resume=False, amp=True, fraction=1.0, profile=False, freeze=None, multi_scale=False, overlap_mask=True, mask_ratio=4, dropout=0.0, val=True, split=val, save_json=False, save_hybrid=False, conf=None, iou=0.7, max_det=300, half=False, dnn=False, plots=True, source=None, vid_stride=1, stream_buffer=False, visualize=False, augment=False, agnostic_nms=False, classes=None, retina_masks=False, embed=None, show=False, save_frames=False, save_txt=False, save_conf=False, save_crop=False, show_labels=True, show_conf=True, show_boxes=True, line_width=None, format=torchscript, keras=False, optimize=False, int8=False, dynamic=False, simplify=True, opset=None, workspace=4, nms=False, lr0=0.01, lrf=0.01, momentum=0.937, weight_decay=0.0005, warmup_epochs=3.0, warmup_momentum=0.8, warmup_bias_lr=0.1, box=7.5, cls=0.5, dfl=1.5, pose=12.0, kobj=1.0, label_smoothing=0.0, nbs=64, hsv_h=0.015, hsv_s=0.7, hsv_v=0.4, degrees=0.0, translate=0.1, scale=0.5, shear=0.0, perspective=0.0, flipud=0.0, fliplr=0.5, bgr=0.0, mosaic=1.0, mixup=0.0, copy_paste=0.0, auto_augment=randaugment, erasing=0.4, crop_fraction=1.0, cfg=None, tracker=botsort.yaml, save_dir=runs\detect\yolov8n_custom3
Overriding model.yaml nc=80 with nc=4

               from  n    params  module                                       arguments

0 -1 1 1 -1 1 2 -1 1 3 -1 1 4 -1 2 5 -1 1 6 7 8 9 10 -1 1 11 [-1, 6] 1 12 13 -1 1 14 [-1, 4] 1 15 16 17 [-1, 12] 1 18 19 20 [-1, 9] 1 21 22 [15, 18, 21] 1 YOLOv8n summary: 464 ultralytics.nn.modules.conv.Conv [3, 16, 3, 2]
4672 ultralytics.nn.modules.conv.Conv [16, 32, 3, 2]
7360 ultralytics.nn.modules.block.C2f [32, 32, 1, True]
18560 ultralytics.nn.modules.conv.Conv [32, 64, 3, 2]
49664 ultralytics.nn.modules.block.C2f [64, 64, 2, True]
73984 ultralytics.nn.modules.conv.Conv [64, 128, 3, 2]
-1 2 197632 ultralytics.nn.modules.block.C2f [128, 128, 2, True]
-1 1 295424 ultralytics.nn.modules.conv.Conv [128, 256, 3, 2]
-1 1 460288 ultralytics.nn.modules.block.C2f [256, 256, 1, True]
-1 1 164608 ultralytics.nn.modules.block.SPPF [256, 256, 5]
0 torch.nn.modules.upsampling.Upsample [None, 2, 'nearest']
0 ultralytics.nn.modules.conv.Concat [1]
-1 1 148224 ultralytics.nn.modules.block.C2f [384, 128, 1]
0 torch.nn.modules.upsampling.Upsample [None, 2, 'nearest']
0 ultralytics.nn.modules.conv.Concat [1]
-1 1 37248 ultralytics.nn.modules.block.C2f [192, 64, 1]
-1 1 36992 ultralytics.nn.modules.conv.Conv [64, 64, 3, 2]
0 ultralytics.nn.modules.conv.Concat [1]
-1 1 123648 ultralytics.nn.modules.block.C2f [192, 128, 1]
-1 1 147712 ultralytics.nn.modules.conv.Conv [128, 128, 3, 2]
0 ultralytics.nn.modules.conv.Concat [1]
-1 1 493056 ultralytics.nn.modules.block.C2f [384, 256, 1]
752092 ultralytics.nn.modules.head.Detect [4, [64, 128, 256]]
225 layers, 3,011,628 parameters, 3,011,612 gradients, 8.2 GFLOPs

ClearML Task: created new task id=b59a3b9114204a72a98589b1cfc4459d
2024-09-28 21:35:19,210 - clearml.Task - INFO - No repository found, storing script code instead
ClearML results page: https://app.clear.ml/projects/b10f96eb2380448dad26ad1a906b3e0e/experiments/b59a3b9114204a72a98589b1cfc4459d/output/log
ClearML Initialized a new task. If you want to run remotely, please add clearml-init and connect your arguments before initializing YOLO.
TensorBoard: Start with 'tensorboard --logdir runs\detect\yolov8n_custom3', view at http://localhost:6006/
Freezing layer 'model.22.dfl.conv.weight'
train: Scanning D:\archive_09_03_23\PycharmProjects\YOLO\train\labels.cache... 1333 images, 78 backgrounds, 0 corrupt: 100%|██████████| 1333/1333 [00:00<?, ?it/s]
val: Scanning D:\archive_09_03_23\PycharmProjects\YOLO\valid\labels.cache... 693 images, 43 backgrounds, 0 corrupt: 100%|██████████| 693/693 [00:00<?, ?it/s]

Process finished with exit code -1073741819 (0xC0000005)

#Start 2
C:\Users\Lavrenitijav\AppData\Local\Microsoft\WindowsApps\python3.12.exe D:\archive_09_03_23\PycharmProjects\YOLO\now_object.py
load model
start train
New https://pypi.org/project/ultralytics/8.2.103 available 😃 Update with 'pip install -U ultralytics'
Ultralytics YOLOv8.2.102 🚀 Python-3.12.6 torch-2.4.1+cu124 CPU (Intel Xeon E3-1225 V2 3.20GHz)
engine\trainer: task=detect, mode=train, model=yolov8n.yaml, data=data.yaml, epochs=10, time=None, patience=100, batch=16, imgsz=480, save=True, save_period=-1, cache=False, device=cpu, workers=8, project=None, name=yolov8n_custom4, exist_ok=False, pretrained=True, optimizer=auto, verbose=True, seed=0, deterministic=True, single_cls=False, rect=False, cos_lr=False, close_mosaic=10, resume=False, amp=True, fraction=1.0, profile=False, freeze=None, multi_scale=False, overlap_mask=True, mask_ratio=4, dropout=0.0, val=True, split=val, save_json=False, save_hybrid=False, conf=None, iou=0.7, max_det=300, half=False, dnn=False, plots=True, source=None, vid_stride=1, stream_buffer=False, visualize=False, augment=False, agnostic_nms=False, classes=None, retina_masks=False, embed=None, show=False, save_frames=False, save_txt=False, save_conf=False, save_crop=False, show_labels=True, show_conf=True, show_boxes=True, line_width=None, format=torchscript, keras=False, optimize=False, int8=False, dynamic=False, simplify=True, opset=None, workspace=4, nms=False, lr0=0.01, lrf=0.01, momentum=0.937, weight_decay=0.0005, warmup_epochs=3.0, warmup_momentum=0.8, warmup_bias_lr=0.1, box=7.5, cls=0.5, dfl=1.5, pose=12.0, kobj=1.0, label_smoothing=0.0, nbs=64, hsv_h=0.015, hsv_s=0.7, hsv_v=0.4, degrees=0.0, translate=0.1, scale=0.5, shear=0.0, perspective=0.0, flipud=0.0, fliplr=0.5, bgr=0.0, mosaic=1.0, mixup=0.0, copy_paste=0.0, auto_augment=randaugment, erasing=0.4, crop_fraction=1.0, cfg=None, tracker=botsort.yaml, save_dir=runs\detect\yolov8n_custom4
Overriding model.yaml nc=80 with nc=4

               from  n    params  module                                       arguments

0 -1 1 464 ultralytics.nn.modules.conv.Conv [3, 16, 3, 2]
1 -1 1 4672 ultralytics.nn.modules.conv.Conv [16, 32, 3, 2]
2 -1 1 7360 ultralytics.nn.modules.block.C2f [32, 32, 1, True]
3 -1 1 18560 ultralytics.nn.modules.conv.Conv [32, 64, 3, 2]
4 -1 2 49664 ultralytics.nn.modules.block.C2f [64, 64, 2, True]
5 -1 1 73984 ultralytics.nn.modules.conv.Conv [64, 128, 3, 2]
6 -1 2 197632 ultralytics.nn.modules.block.C2f [128, 128, 2, True]
7 -1 1 295424 ultralytics.nn.modules.conv.Conv [128, 256, 3, 2]
8 -1 1 460288 ultralytics.nn.modules.block.C2f [256, 256, 1, True]
9 -1 1 164608 ultralytics.nn.modules.block.SPPF [256, 256, 5]
10 -1 1 0 torch.nn.modules.upsampling.Upsample [None, 2, 'nearest']
11 [-1, 6] 1 0 ultralytics.nn.modules.conv.Concat [1]
12 -1 1 148224 ultralytics.nn.modules.block.C2f [384, 128, 1]
13 -1 1 0 torch.nn.modules.upsampling.Upsample [None, 2, 'nearest']
14 [-1, 4] 1 0 ultralytics.nn.modules.conv.Concat [1]
15 -1 1 37248 ultralytics.nn.modules.block.C2f [192, 64, 1]
16 -1 1 36992 ultralytics.nn.modules.conv.Conv [64, 64, 3, 2]
17 [-1, 12] 1 0 ultralytics.nn.modules.conv.Concat [1]
18 -1 1 123648 ultralytics.nn.modules.block.C2f [192, 128, 1]
19 -1 1 147712 ultralytics.nn.modules.conv.Conv [128, 128, 3, 2]
20 [-1, 9] 1 0 ultralytics.nn.modules.conv.Concat [1]
21 -1 1 493056 ultralytics.nn.modules.block.C2f [384, 256, 1]
22 [15, 18, 21] 1 752092 ultralytics.nn.modules.head.Detect [4, [64, 128, 256]]
YOLOv8n summary: 225 layers, 3,011,628 parameters, 3,011,612 gradients, 8.2 GFLOPs

ClearML Task: created new task id=922263ab550a40ea99557645ab2dfc04
2024-09-28 21:43:33,316 - clearml.Task - INFO - No repository found, storing script code instead
ClearML results page: https://app.clear.ml/projects/b10f96eb2380448dad26ad1a906b3e0e/experiments/922263ab550a40ea99557645ab2dfc04/output/log
ClearML Initialized a new task. If you want to run remotely, please add clearml-init and connect your arguments before initializing YOLO.
TensorBoard: Start with 'tensorboard --logdir runs\detect\yolov8n_custom4', view at http://localhost:6006/
Freezing layer 'model.22.dfl.conv.weight'
train: Scanning D:\archive_09_03_23\PycharmProjects\YOLO\train\labels.cache... 1333 images, 78 backgrounds, 0 corrupt: 100%|██████████| 1333/1333 [00:00<?, ?it/s]
val: Scanning D:\archive_09_03_23\PycharmProjects\YOLO\valid\labels.cache... 693 images, 43 backgrounds, 0 corrupt: 100%|██████████| 693/693 [00:00<?, ?it/s]
Plotting labels to runs\detect\yolov8n_custom4\labels.jpg...
optimizer: 'optimizer=auto' found, ignoring 'lr0=0.01' and 'momentum=0.937' and determining best 'optimizer', 'lr0' and 'momentum' automatically...
optimizer: AdamW(lr=0.00125, momentum=0.9) with parameter groups 57 weight(decay=0.0), 64 weight(decay=0.0005), 63 bias(decay=0.0)
TensorBoard: model graph visualization added ✅
Image sizes 480 train, 480 val
Using 0 dataloader workers
Logging results to runs\detect\yolov8n_custom4
Starting training for 10 epochs...
Closing dataloader mosaic

  Epoch    GPU_mem   box_loss   cls_loss   dfl_loss  Instances       Size
   1/10         0G      4.881      8.367      4.182         33        480:   6%|▌         | 5/84 [00:31<08:20,  6.33s/it]

Process finished with exit code -1073741819 (0xC0000005)

#start 3
C:\Users\Lavrenitijav\AppData\Local\Microsoft\WindowsApps\python3.12.exe D:\archive_09_03_23\PycharmProjects\YOLO\now_object.py
load model
start train
New https://pypi.org/project/ultralytics/8.2.103 available 😃 Update with 'pip install -U ultralytics'
Ultralytics YOLOv8.2.102 🚀 Python-3.12.6 torch-2.4.1+cu124 CPU (Intel Xeon E3-1225 V2 3.20GHz)
engine\trainer: task=detect, mode=train, model=yolov8n.yaml, data=data.yaml, epochs=10, time=None, patience=100, batch=16, imgsz=480, save=True, save_period=-1, cache=False, device=cpu, workers=8, project=None, name=yolov8n_custom5, exist_ok=False, pretrained=True, optimizer=auto, verbose=True, seed=0, deterministic=True, single_cls=False, rect=False, cos_lr=False, close_mosaic=10, resume=False, amp=True, fraction=1.0, profile=False, freeze=None, multi_scale=False, overlap_mask=True, mask_ratio=4, dropout=0.0, val=True, split=val, save_json=False, save_hybrid=False, conf=None, iou=0.7, max_det=300, half=False, dnn=False, plots=True, source=None, vid_stride=1, stream_buffer=False, visualize=False, augment=False, agnostic_nms=False, classes=None, retina_masks=False, embed=None, show=False, save_frames=False, save_txt=False, save_conf=False, save_crop=False, show_labels=True, show_conf=True, show_boxes=True, line_width=None, format=torchscript, keras=False, optimize=False, int8=False, dynamic=False, simplify=True, opset=None, workspace=4, nms=False, lr0=0.01, lrf=0.01, momentum=0.937, weight_decay=0.0005, warmup_epochs=3.0, warmup_momentum=0.8, warmup_bias_lr=0.1, box=7.5, cls=0.5, dfl=1.5, pose=12.0, kobj=1.0, label_smoothing=0.0, nbs=64, hsv_h=0.015, hsv_s=0.7, hsv_v=0.4, degrees=0.0, translate=0.1, scale=0.5, shear=0.0, perspective=0.0, flipud=0.0, fliplr=0.5, bgr=0.0, mosaic=1.0, mixup=0.0, copy_paste=0.0, auto_augment=randaugment, erasing=0.4, crop_fraction=1.0, cfg=None, tracker=botsort.yaml, save_dir=runs\detect\yolov8n_custom5
Overriding model.yaml nc=80 with nc=4

               from  n    params  module                                       arguments

0 -1 1 464 ultralytics.nn.modules.conv.Conv [3, 16, 3, 2]
1 -1 1 4672 ultralytics.nn.modules.conv.Conv [16, 32, 3, 2]
2 -1 1 7360 ultralytics.nn.modules.block.C2f [32, 32, 1, True]
3 -1 1 18560 ultralytics.nn.modules.conv.Conv [32, 64, 3, 2]
4 -1 2 49664 ultralytics.nn.modules.block.C2f [64, 64, 2, True]
5 -1 1 73984 ultralytics.nn.modules.conv.Conv [64, 128, 3, 2]
6 -1 2 197632 ultralytics.nn.modules.block.C2f [128, 128, 2, True]
7 -1 1 295424 ultralytics.nn.modules.conv.Conv [128, 256, 3, 2]
8 -1 1 460288 ultralytics.nn.modules.block.C2f [256, 256, 1, True]
9 -1 1 164608 ultralytics.nn.modules.block.SPPF [256, 256, 5]
10 -1 1 0 torch.nn.modules.upsampling.Upsample [None, 2, 'nearest']
11 [-1, 6] 1 0 ultralytics.nn.modules.conv.Concat [1]
12 -1 1 148224 ultralytics.nn.modules.block.C2f [384, 128, 1]
13 -1 1 0 torch.nn.modules.upsampling.Upsample [None, 2, 'nearest']
14 [-1, 4] 1 0 ultralytics.nn.modules.conv.Concat [1]
15 -1 1 37248 ultralytics.nn.modules.block.C2f [192, 64, 1]
16 -1 1 36992 ultralytics.nn.modules.conv.Conv [64, 64, 3, 2]
17 [-1, 12] 1 0 ultralytics.nn.modules.conv.Concat [1]
18 -1 1 123648 ultralytics.nn.modules.block.C2f [192, 128, 1]
19 -1 1 147712 ultralytics.nn.modules.conv.Conv [128, 128, 3, 2]
20 [-1, 9] 1 0 ultralytics.nn.modules.conv.Concat [1]
21 -1 1 493056 ultralytics.nn.modules.block.C2f [384, 256, 1]
22 [15, 18, 21] 1 752092 ultralytics.nn.modules.head.Detect [4, [64, 128, 256]]
YOLOv8n summary: 225 layers, 3,011,628 parameters, 3,011,612 gradients, 8.2 GFLOPs

ClearML Task: created new task id=25eb8142642b4ef9945869af5eeb44b4
2024-09-28 21:45:09,040 - clearml.Task - INFO - No repository found, storing script code instead
ClearML results page: https://app.clear.ml/projects/b10f96eb2380448dad26ad1a906b3e0e/experiments/25eb8142642b4ef9945869af5eeb44b4/output/log
ClearML Initialized a new task. If you want to run remotely, please add clearml-init and connect your arguments before initializing YOLO.
TensorBoard: Start with 'tensorboard --logdir runs\detect\yolov8n_custom5', view at http://localhost:6006/
Freezing layer 'model.22.dfl.conv.weight'
train: Scanning D:\archive_09_03_23\PycharmProjects\YOLO\train\labels.cache... 1333 images, 78 backgrounds, 0 corrupt: 100%|██████████| 1333/1333 [00:00<?, ?it/s]
val: Scanning D:\archive_09_03_23\PycharmProjects\YOLO\valid\labels.cache... 693 images, 43 backgrounds, 0 corrupt: 100%|██████████| 693/693 [00:00<?, ?it/s]
Plotting labels to runs\detect\yolov8n_custom5\labels.jpg...
optimizer: 'optimizer=auto' found, ignoring 'lr0=0.01' and 'momentum=0.937' and determining best 'optimizer', 'lr0' and 'momentum' automatically...
optimizer: AdamW(lr=0.00125, momentum=0.9) with parameter groups 57 weight(decay=0.0), 64 weight(decay=0.0005), 63 bias(decay=0.0)
TensorBoard: model graph visualization added ✅
Image sizes 480 train, 480 val
Using 0 dataloader workers
Logging results to runs\detect\yolov8n_custom5
Starting training for 10 epochs...
Closing dataloader mosaic

  Epoch    GPU_mem   box_loss   cls_loss   dfl_loss  Instances       Size
   1/10         0G      4.902      8.633      4.178         30        480:   5%|▍         | 4/84 [00:25<08:35,  6.45s/it]

Process finished with exit code -1073741819 (0xC0000005)

UltralyticsAssistant · 2024-09-28T18:46:50Z

UltralyticsAssistant
Sep 28, 2024
Maintainer

👋 Hello @lavrentijav, thank you for bringing this up! 🚀

We recommend checking the Ultralytics Docs for solutions to common issues. If this is a 🐛 Bug Report, please provide a minimum reproducible example so we can assist you better.

Ensure you have the latest version by upgrading your setup:

pip install -U ultralytics

Also, try verifying your setup in a supported environment with all dependencies like CUDA/CUDNN, Python, and PyTorch:

Notebooks with free GPU:
Google Cloud Deep Learning VM. See GCP Quickstart Guide
Amazon Deep Learning AMI. See AWS Quickstart Guide
Docker Image. See Docker Quickstart Guide

For community support, join the Discord 🎧, Discourse, or our Subreddit. An Ultralytics engineer will assist you soon!

0 replies

glenn-jocher · 2024-09-28T23:46:53Z

glenn-jocher
Sep 28, 2024
Maintainer

@lavrentijav it seems you're encountering an error during training. The exit code -1073741819 (0xC0000005) typically indicates an access violation. Ensure your data paths are correct and try reducing the batch size to see if it resolves the issue. If the problem persists, consider updating your Ultralytics package with pip install -U ultralytics.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ultralytics

Error on training YOLO v8 #16525

{{title}}

Replies: 2 comments

{{title}}

{{title}}

Select a reply

Ultralytics

Error on training YOLO v8 #16525

lavrentijav Sep 28, 2024

Replies: 2 comments

UltralyticsAssistant Sep 28, 2024 Maintainer

glenn-jocher Sep 28, 2024 Maintainer

lavrentijav
Sep 28, 2024

UltralyticsAssistant
Sep 28, 2024
Maintainer

glenn-jocher
Sep 28, 2024
Maintainer