[BUG] Recording neural network output throws KeyError: '2_out;0_preview'

[takgoya]

Following your documentation for Recording (https://docs-old.luxonis.com/projects/sdk/en/latest/quickstart/#recording), I have tried to record both the color camera and the neural network outputs.

Code is as simple as in your example:

from depthai_sdk import OakCamera
from depthai_sdk.record import RecordType

with OakCamera() as oak:
    color = oak.create_camera('color', resolution='1080p')
    yolo = oak.create_nn('yolov6n_coco_640x640', input=color)

    oak.record([color, yolo], path='./records', record_type=RecordType.VIDEO)
    oak.start(blocking=True)

It throws the following error:
File: "..\site-packages\depthai_sdk\recorders\video_recorder.py", line 98, in write
self._writers[name].write(frame)
KeyError: '2_out;0_preview'

It happens with any neural network, I have tried with my own customized model, and it gives the same error.

[brmarkus]

Which camera, which OAK device are you using?
Does the camera with its sensors behave normal and works as expected, for consistency check?

Which version of depthai do you use?

Which version of Python do you use?

What does your environment look like (like which OS), are you running in a virtual-environment?

[takgoya]

Camera: OAK-D W PoE OV9782

Conda environment in Windows, with:

• Python version: 3.10.14
• depthai==2.28.0.0
• depthai-sdk==1.15.0

Without recording camera works properly, running the neural network with no problem.
Everything works fine.

[brmarkus]

Without the second component recording is working:

from depthai_sdk import OakCamera, RecordType

with OakCamera() as oak:
    color = oak.create_camera('color', resolution='1080p')
    yolo = oak.create_nn('yolov6n_coco_640x640', input=color)

    #oak.record([color.out.encoded, yolo], path='./records', record_type=RecordType.VIDEO)
    oak.record([color], path='./records', record_type=RecordType.VIDEO)
    oak.start(blocking=True)

Sorry, haven't used the SDK since quite some time now... can't remember the exact API...

[takgoya]

Yep, the color channel has no problem with the recording.

The problems appears when the channel to be recorded is the one of the neural network.

[brmarkus]

The xout out name of the YOLO object is "NN results".
If I set it manually, it still didn't get it, and still throws KeyError: 'NN results'.
When I, however, hard-code "take the first key in the dictionary and use it to access the dict" (or second, when adding color and yolo), then it's working:

In file "depthai_sdk\recorders\video_recorder.py":

def write(self, name: str, frame: Union[np.ndarray, dai.ImgFrame]):
    first_key = next(iter(self._writers))
    print( "First key chosen: <" + first_key + ">" )
    self._writers[first_key].write(frame)

Not sure there is a "hidden" (invisble) "extra-character" added to the xout-name of the NNComponent.

[brmarkus]

There is a loop in "depthai_sdk\record.py" to iterate over the components.
Adding print()s and I can see it is iterating over both:

Write a frame for <CAM_A_bitstream>
Write a frame for <NN Results>

And the write() method then receives the correct names (at least what can be seen in the console), when printing the keys of the dictionary::

Writer-Dict:{'CAM_A_bitstream': <depthai_sdk.recorders.video_writers.av_writer.AvWriter object at 0x000001D27202ED50>, 'NN results': <depthai_sdk.recorders.video_writers.av_writer.AvWriter object at 0x000001D223E85B50>}
Writer-Dict-key:dict_keys(['CAM_A_bitstream', 'NN results'])

Without hard-coding the dict-key-instance, it throws KeyError: 'NN results', but the key "NN results" gets printed.

=> is there a hidden, invisible character/byte/token added?

[brmarkus]

NN Results versus NN results!

Looks like it's hard-coded somewhere when creating the VideoWriter (and name gets replaced at runtime after calling start(), if not set manually)...

Finally, I got it working with this:

from depthai_sdk import OakCamera, RecordType

with OakCamera() as oak:
    color = oak.create_camera('color', resolution='1080P', fps=20, encode='H264')
    yolo = oak.create_nn('yolov6n_coco_640x640', input=color)
    yolo.out.encoded.set_name( "NN results" )

    # Synchronize & save all (encoded) streams
    oak.record([color.out.encoded, yolo.out.encoded], './records', RecordType.VIDEO)
    # Show color stream and camera&inference-results
    oak.visualize([color, yolo], scale=2/3, fps=True)

    oak.start(blocking=True)

=> two video files created (first the elementary streams, then muxed into two MP4 container video files):

• CAM_A_bitstream.mp4
• NN results.mp4

Something for the Luxonis-team to have a closer look into.

[brmarkus]

(I used MS-Win11, comand-prompt (not Powershell), created a virt-env, using Python 3.12.4)
(installed DepthAI with py -m pip install depthai-sdk)

Looks like the (av-)VideoWriter instance gets created under the initial xout-out name NN results.
But at runtime - if no name is set manually before calling oak.start() - the dynamic output 2_out;0_preview (in this pipeline example) is used, which isn't present in the dictionary of all initially created video-writer instances.

[takgoya]

Cool! Many thanks.
I will try it.

[takgoya]

I have tried it and for me it works setting the encode to H265. With H264 it throws an error when closing the video.

What I have seen in the recorded neural network video is that it does not contain any detection, although there are some. It looks like the raw video.

I have also seen that there is some parameter that can be passed to Visualize to set a record_path, but it is not further used.

It seems that this recording feature is not developed.

[brmarkus]

How have you initially installed DepthAI?

From another issue like "#1171" it looks like we are using the wrong DepthAI...

I was following "https://docs-old.luxonis.com/projects/sdk/en/latest/quickstart/#installation" with py -m pip install depthai-sdk.
But looks like we need to go with DepthAI-V3...

@themarpe @Erol444 can you confirm we used the wrong DepthAI??

[Erol444]

Hi @takgoya ,
Video recorder can't record NN outputs, only raw sensor streams (camera sensors, imu..). For now, I'd suggest using callback for yolo output, and add cv2.videoWriter inside it.

oak.visualize(yolo, callback=cb_function)

@brmarkus , could you please contact me at [email protected]?

[takgoya]

Thanks for the replies.

Yes, I have implemented a callback to call cv2.Videowriter in order to record.