[BUG] SDK - Replay occasionally takes depth recording for color camera

[RiccardoGrieco]

Describe the bug

When replaying recorded data, occasionally 6_depth is used for color camera.

Minimal Reproducible Example

Recording script:

from depthai_sdk import OakCamera
from depthai import Device, ColorCameraProperties, MonoCameraProperties
fps = 30.0
record_dir="/tmp/oak_recording/24-04-30-15_08_17/1-1844301011A3860E00/1-1844301011A3860E00"
with OakCamera() as oak:
        # create streams
        color = oak.camera('color', resolution=ColorCameraProperties.SensorResolution.THE_1080_P, fps=fps, encode=None)
        stereo = oak.stereo(resolution=MonoCameraProperties.SensorResolution.THE_720_P, fps=fps, encode=None)
        stereo.config_stereo(align=color)

        # configure recording
        oak.record([color, stereo, stereo.left, stereo.right], path=record_dir)

        # start recording. Use CTRL+C to stop.
        oak.start(blocking=True)

Replay script:

from depthai_sdk import OakCamera
import depthai as dai

with OakCamera(replay='/tmp/oak_recording/24-04-30-15_08_17/1-1844301011A3860E00/1-1844301011A3860E00') as oak:
    oak.replay.set_loop(True)  # <--- Enable looping of the video, so it will never end

    color = oak.create_camera('color')
    
    oak.visualize([color], fps=True)
    oak.start(blocking=True)

Expected behavior

CAM_A_video is used for color camera.

Screenshots

Here a screenshot of the replay script and the visualization:

Additional context

Library versions:

depthai                   2.25.0.0
depthai-pipeline-graph    0.0.5
depthai-sdk               1.13.1

I dig a bit into the issue and looks like the problem lies in this piece of code, where 6_depth is selected as stream name:

depthai/depthai_sdk/src/depthai_sdk/components/camera_component.py

Lines 78 to 86 in f4a0d3d

	if self.is_replay():
	stream_name = None
	for name, stream in self._replay.streams.items():
	if stream.get_socket() == self._socket:
	stream_name = name
	break
	if stream_name is None:
	raise Exception(f"{source} stream was not found in specified depthai-recording!")
	self._source = stream_name

Reason behind this is that when streams are initialized, 6_depth is assigned a None socket:

depthai/depthai_sdk/src/depthai_sdk/readers/videocap_reader.py

Lines 39 to 67 in f4a0d3d

	for fileName in os.listdir(str(path)):
	f_name, ext = os.path.splitext(fileName)
	if ext not in _videoExt:
	continue
	# Check if name of the file starts with left.. right.., or CameraBoardSocket
	if f_name.startswith('CAM_'):
	# Remove everything after CAM_x
	f_name = f_name[:5]
	socket = None
	try:
	socket = parse_camera_socket(f_name)
	except ValueError:
	# Invalid file name
	pass
	# TODO: avoid changing stream names, just use socket
	# stream = str(socket)
	# if socket == dai.CameraBoardSocket.CAM_A:
	# stream = 'color'
	# elif socket == dai.CameraBoardSocket.CAM_B:
	# stream = 'left'
	# elif socket == dai.CameraBoardSocket.CAM_C:
	# stream = 'right'
	self.videos[f_name.lower()] = {
	'reader': cv2.VideoCapture(str(path / fileName)),
	'socket': socket
	}

And if socket is None in ReplayStream, method get_socket returns by default CAM_A socket:

depthai/depthai_sdk/src/depthai_sdk/replay.py

Lines 50 to 59 in f4a0d3d

	def get_socket(self) -> dai.CameraBoardSocket:
	if self.camera_socket is not None:
	return self.camera_socket
	if 'left' in self.stream_name.lower():
	return dai.CameraBoardSocket.LEFT
	elif 'right' in self.stream_name.lower():
	return dai.CameraBoardSocket.RIGHT
	else:
	return dai.CameraBoardSocket.CAM_A
	# raise Exception("Please specify replay stream CameraBoardSocket via replay.specify_socket()")

⚠️
NOTE that this issue is strictly dependent on the order of streams returned by self._replay.streams.items(). I tried with two different PCs and one has the correct behaviour, because CAM_A_video is read first.

[MartinMotycka]

@themarpe Check this, please.

[themarpe]

@Erol444 do you intend we check this behavior out?

[Erol444]

Hi @RiccardoGrieco ,
As we're working on depthai v3 (which will supersede depthai-sdk), we, unfortunately, won't be focusing on SDK anymore.
Kind regards, Erik