Looking for help testing Intel OpenVino support for NCS2

[NateMeyer]

I am working on a build that includes OpenVino support on x86 platforms ( #3768 ). I currently have it working with the iGPU on my i5 1135G7 laptop, more details are in the PR.

I was wondering if there was anyone who had a Neural Compute Stick 2 who wouldn't mind helping me test the VPU device integration?

I have an image pushed to github that should be ready to test. For now, this will only work on an x86 host.

NOW WITH ARM 64/32 BUILDS

docker pull ghcr.io/natemeyer/frigate:0.11.0-openvino-a3afb6f

With this image, you can add the VPU as a detector with the following config

detectors:
  ncs2:
    type: openvino
    device: MYRIAD

Additionally, you will have to add the following configuration to use the included OpenVino model.

model:
  path: /openvino-model/ssdlite_mobilenet_v2.xml
  width: 300
  height: 300
  input_tensor: [B, H, W, C]
  input_pixel_format: "bgr"
  labelmap_path: /openvino-model/coco_91cl_bkgr.txt

WIth this dev image, you need to add the following configuration to docker:

--device-cgroup-rule='c 189:\* rmw' -v /dev/bus/usb:/dev/bus/usb

or in your compose file:

device_cgroup_rules:
  - 'c 189:* rmw'
volumes:
  - /dev/bus/usb:/dev/bus/usb

Thanks!

[NickM-27]

I put this around on a few different existing discussions / issues, hopefully some users will be able to help.

[RLutsch]

happy to help but will have to be arm64 for me though

[RLutsch]

running detectors with ovArm works but the inference is slower than cpu getting 1200

[RLutsch]

lsusb returns

Bus 005 Device 023: ID 03e7:2485 Movidius Ltd. Movidius MyriadX
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00

[NateMeyer]

Ok, looks like you have the right NCS2. It is plugged into a USB2 port? Is that correct? Have you tried removing the devices section of your docker-compose?

I'm not surprised the ovArm detector is slow, I was mostly curious if I had the build correct and everything there to support it. Thanks!

[alucryd]

@RLutsch I second that you should not mount it as a device, rather a volume. Device does not work for some reason, intially the NCS2 is loaded as a high-speed device, the openvino driver actually unloads it and reloads it as a super-speed device to use it, and I guess this doesn't play nice with regular device mounting.

[RLutsch]

I tried all the usb's and all give usb 2.0

I'm using kubernetes and it's mounted as volume so should work, it's the same implimentation as what the coral uses.

I'll have to try compose again....

[MostHated]

Nice, I should be able to help out with this later today as well.

[NateMeyer]

Great! Let me know if you have any questions or problems

[alucryd]

Awesome, will give it a try over the weekend!

[NateMeyer]

Thanks, looks like the files are present in the build artifacts I copied over, just not in the right folder. I'll try getting them to the right place.

[NateMeyer]

@alucryd New image is up.

Since I'm building the openvino python wheel in this image, I'll probably switch to using that directly instead of the one from pypi. Then we should be able to have Arm support as well.

ghcr.io/natemeyer/frigate:0.11.0-openvino-f0d0893

[alucryd]

Apologies for the late reply, been bedridden for a few days. Unfortunately I get exactly the same error as #3797 (reply in thread).

Not sure how I can debug this, won't have time over the weekend but probably next week.

[NateMeyer]

No worries, hope you're feeling better. I should have an image by this weekend that includes the self-built openvino library. I'll take a look at what else could be causing that error.

Aside, building this library for arm takes forever on my laptop, but I finally got it to complete without error yesterday. Now we'll see if it works 😆 Might need to find a better way to manage this build.

[NateMeyer]

@alucryd Do you have the Myriad udev rules installed on your host? Have you successfully run other models on the NCS2 stick before?

[titilambert]

Hello, I just tested with ghcr.io/natemeyer/frigate:0.11.0-openvino-f0d0893 with core I7 3770S

And I add got:

E: [ncAPI] [    189878] [frigate.detecto] ncDeviceOpen:1013     Failed to find booted device after boot
Process detector:ncs2:
Traceback (most recent call last):
  File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/frigate/frigate/object_detection.py", line 121, in run_detector
    object_detector = LocalObjectDetector(
  File "/opt/frigate/frigate/object_detection.py", line 66, in __init__
    self.detect_api = OvDetector(
  File "/opt/frigate/frigate/detectors/openvino.py", line 16, in __init__
    self.interpreter = self.ov_core.compile_model(
  File "/usr/local/lib/python3.9/dist-packages/openvino/runtime/ie_api.py", line 266, in compile_model
    super().compile_model(model, device_name, {} if config is None else config)
RuntimeError: Failed to allocate graph: MYRIAD device is not opened.
[2022-09-24 12:49:51] frigate.watchdog               INFO    : Detection appears to have stopped. Exiting frigate...

[alucryd]

@NateMeyer Thanks for the new image, still getting the same issue though. I do have the udev rules installed on the host, and the device is detected by the query_device sample program.

I'm a complete rookie when it comes to neural stuff and I can't find an easy way to test the device, tutorials I find all seem outdated, the first that pops up is ncappzoo, the documentation mentions a make run command, except run isn't even a valid target in the Makefile... This neural stuff looks like it's being made hard on purpose :/

I'll install the newest OpenVINO release, see if I can go further at least on the host side.

[NateMeyer]

I found this note regarding the issues with using the NCS2 inside Docker:
Use a Docker Image for Intel Movidius Neural Compute Stick and Intel Neural Compute Stick 2

Looks like I may have to rebuild libusb without udev support to make it work? The other workaround (not preferred, but faster to test) is to launch the docker image with --privileged -v /dev:/dev --network=host (gross)

[NateMeyer]

Another version of that page:
Running the Image on Intel® Neural Compute Stick 2

This page provides a sample command to test running the MYRIAD device in a prebuilt docker image. Could try this to see if it is working?

docker run -itu root:root --rm --device-cgroup-rule='c 189:\* rmw' -v /dev/bus/usb:/dev/bus/usb <image_name> \
/bin/bash -c "omz_downloader --name googlenet-v1 --precisions FP16 && omz_converter --name googlenet-v1 --precision FP16 && curl -O https://storage.openvinotoolkit.org/data/test_data/images/car_1.bmp && python3 samples/python/hello_classification/hello_classification.py public/googlenet-v1/FP16/googlenet-v1.xml car_1.bmp MYRIAD"

Also:

Only one NCS2 device connected to the host can be used when running inference in a container

[NateMeyer]

@alucryd, @titilambert, I updated the image once more, testing out building libusb without udev per the Intel documentation above. I also noted down what I expect you need to add to your docker-compose.yml file to pass through the device? See the updated instructions at the top of this thread and let me know if that makes sense. Thanks!

[alucryd]

@NateMeyer Thanks ! Still no luck though, I also tried intel's docker image and it doesn't even work either... So that's not on you.

I opened a bug report upstream, hopefully someone can help: openvinotoolkit/openvino#13288

[indieOR]

I was able to run this image with my Neural Compute Stick 2, and Frigate is reporting an inference speed of 10.

One word of caution if you are running the container inside of a VM. The Neural Compute Stick 2 does some odd things when in a VM. When the driver instantiates the device the device ID seems to change in /dev ont he host, making the mapping into the VM no longer relevant and the device disappears. This could explain some issues.

I have the container running directly on bare metal with the NCS2 and it's working without a hitch. I actually gave up Frigate earlier this year due to Coral unavailability, but have had this NCS2 sitting in my closet for almost 2 years. Glad to finally put it to use and be using Frigate again.

[NickM-27]

I fixed the formatting for you 👍

[NickM-27]

also 10 is the default value of inference speed, otherwise it is a calculated rolling average so odds are it may have just been showing the default before.

[NateMeyer]

@indieaz Is the NCS2 plugged into a USB 2 or 3 port? How many cameras are you running detection on? Curious as a reference for the latency number.

[NickM-27]

Yeah, now that we have multiple detectors will probably want to add inference numbers for each one to help users decide as multiple options may exist.

[indieOR]

@indieaz Is the NCS2 plugged into a USB 2 or 3 port? How many cameras are you running detection on? Curious as a reference for the latency number.

@NateMeyer I am running against 7 cameras right now.

NCS2 is plugged into a USB 3 port, and lsusb reports USB 3.2 link rate.

Bus 002 Device 003: ID 03e7:f63b Intel Myriad VPU [Movidius Neural Compute Stick                                                                                                                                                             ]
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               3.20

[n0rthking]

I don't have NCS2, but I have got it working on my server with Intel J4105 using the GPU device as detector. Interface speed is now 25 ms which is a nice improvement from the default CPU detector, which had about 150 ms interface speed. Additionally the CPU usage is now very low (about 15% in idle and 30% during detection, much better than full load with the default CPU detector).

[alucryd]

@NateMeyer I finally got it working! I tried with privileged, and with the cgroup rule, both worked, so cgroup rule it is.

I believe the issue all this time is that I was mounting /dev/bus/usb as a device instead of a volume. I've always passed devices that way so didn't think twice about it, but it seems that this doesn't play well with the NCS2.

Getting an inference speed of about 62, which is in line with @indieaz's results.

[NateMeyer]

@alucryd, for reference how many cameras were you running detection on?

[alucryd]

I only have 2 cameras.

[Jachimo]

Just wondering, is there an 0.11.1 version of the OpenVINO Docker image? Or should I still use 0.11.0? I have an NCS2 kicking around that I'd be happy to plug in and be able to use.

[NateMeyer]

Hey @Jachimo, this has now been merged into the dev branch and is available in the 0.12 Beta. Take a look at the release notes and documentation in #5156

[clamel77]

Great job, thank you very much. It was exactly what I was looking for.. I installed it on raspberry pi3 with Intel ncs2 usb stick.. :)

[amzaldua]

Can someone explain to me how to do this with an Intel HDU like I'm a baby?
My hardware is a NUC with a Celeron N5105 (10th generation)

[NickM-27]

it is all covered in the docs https://deploy-preview-4055--frigate-docs.netlify.app/configuration/detectors#openvino-detector

all you need to do is add that part to your config (assuming your GPU is visible in the container, if you are using hwaccel then it is already)

[amzaldua]

Thanks NickM-27.
It's just what I needed. I reduced my inference speed from 130-100ms to 32ms with this. Awesome.

[FrigateEnthusiast]

Is that 32ms with the Compute Stick? I had one sitting a drawer but thought but thought it would not compete with the Coral since it was USB2. Both are supposed to be 4 trillion ops once the bits arrive.

[amzaldua]

In my case, this value is only for the CPU.

[alucryd]

Has anyone used the stick on an aarch64 SBC recently ? I can't get it to work on a Pi 4 or Vim 3, both running a fully up to date Debian Bookworm. It's working fine on an Intel server with all parameters strictly identical.