"Frame Didn't Arrive Within 5000" issue with D455 in Python 3.6 built with Pip

[Chris45215]

---

Required Info
Camera Model	D455
Firmware Version	05.12.14.50
Operating System & Version	{Win 10
Platform	PC
SDK Version	2.47.0.3313
Language	Python
Computer model	MSI GS 63

Issue Description

This issue is similar to the several others that report the same problem (such as #8691), with the exception that we are using the SDK in Python 3.6 (with Visual Studio as the development environment) 64-bit, with the latest version of pyrealsense2 installed via the command "pip install pyrealsense2". This matches the suggestion for issue #8691, except we are using a slightly older version of Python. Also, the computer makes the "device disconnected" sound whenever it hits the error, as though the sensor has been disconnected. This symptom has also been reported as issue #5181, #2168, #6628, and #927. The particular camera the issue happens with was delivered today, and we used the cable that was included with the sensor.

One of our computers, the MSI GS 63, gets the "Frame Didn't Arrive Within 5000" issue when running the Quick Start code at https://pypi.org/project/pyrealsense2/. But, please note that the error message doesn't appear after 5 seconds as we would expect - it appears after around 15 or 20 seconds. We tried to ensure that there are no resource conflicts for the sensor by adding this hard reset command before that test code:
ctx = rs.context()
devices = ctx.query_devices()
for dev in devices:
dev.hardware_reset()

This allowed the sensor to run and output for a few frames (but definitely not the 100 specified in the code), then it stopped and reported the same "Frame Didn't Arrive Within 5000" error.

This problem does not occur when using a same-model sensor on an MSI GS 65 - that computer runs the code exactly as expected. However, we were able to reproduce the symptoms by using the camera as the video source in a Zoom chat and then running the code, as that creates a resource conflict - however this produced the error report within the expected 5 seconds, rather than the much longer wait on the MSI GS 63 which has the problem.

The apparently-effective advice we've seen in other threads is "try a different computer" - but that's not a viable solution for us. We are not in a position to arbitrarily buy new computers for our employees, especially when we have no way to know if the new computers will be compatible either. And this would be especially problematic when installing systems for customer applications, in which we can't have mysterious compatibility issues. Does anyone know the specific incompatibility? If the issue exists with one RealSense model, will it exist with all of them? Will the issue exist for every request for frames from the sensor, or just some requests?

[MartyG-RealSense]

Hi @Chris45215 Thanks very much for the very detailed information about your problem. It sounds as though the test code is fine if it can run without problems on your MSI GS 65. If the newly-arrived D455 camera also works without problems on the MSI GS 65, this suggests that the camera hardware and the currently installed version of its firmware driver are fine too.

Regarding your Zoom video test: it would be expected behaviour for a particular stream type to become unavailable if another application was already 'claiming' ownership of a particular stream from a particular camera and therefore blocking access to it until the program using it closes that stream and 'releases' it for another application to use.

This can be demonstrated by launching a RealSense example program such as rs-capture and then launching the RealSense Viewer and trying to enable the stream that the other application is using - the result is a No Frames Received message being displayed in the Viewer stream panel of the blocked stream. The same blocking effect is observed if enabling a stream in the Viewer first and then launching an example program - the example is the one whose stream access is blocked. The rules for accessing a stream are discussed in the SDK's Multi-Streaming Model documentation in the link below.

https://github.com/IntelRealSense/librealsense/blob/master/doc/rs400_support.md#multi-streaming-model

It may be helpful to test the GS 63 computer with a simple Python wrapper example from the SDK called python-tutorial-1-depth.py that is specifically configured to use depth only at a specific resolution and FPS speed, rather than letting the default stream configuration of the particular camera model be applied automatically. The Quick Start program is likely using the D455's default stream configuration because the program does not use a custom cfg or config stream definition.

Since the python-tutorial-1-depth.py example program does not access the color stream due to the specific depth-only configuration, if the program consistently works on the GS 63 then this would lend weight to the possibility of the problem being a color-stream related issue.

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/python-tutorial-1-depth.py

[Chris45215]

Thanks @MartyG-RealSense, and I do have an update on the issue.
We were able to get some metadata from a single frame when we added code call .hardreset() to the beginning of the program; it was an attempt to ensure nothing else was monopolizing the sensor. But it hit the same problem after returning the first two lines of printout for the sample program.

Then, one day later - after the computer was ignored overnight - it seems to works fine. We will test it more today to ensure the issue is resolved, and that it doesn't re-appear when pulling the actual frames. We also have another sensor on order for another office, and can see what happens when we plug that in.

I have suspicion that there's some USB issue or driver issue which Microsoft detects and queues a small update for, but doesn't show the update within Windows Update or anywhere else, and it just silently installs that update when the computer is idle. I suspect this because the same thing happened with the Kinect For XBox One when attached to the PC (so technically that product name was "Kinect For Windows v2"), back in 2018. That Kinect issue was resolved by running Windows update twice (and ensuring the last time showed no updates), rebooting, and returning a few days later. That's the best guess I have.

I'm going to leave this issue open in case we encounter the problem when pulling image frames, or if it happens with the next setup. Even if it does resolve after a day, that's far from an ideal solution, as we don't want to promise deliveries of our system (which includes the sensor) to our customers when we don't know the source of the problem and we can't be certain it will work.

[MartyG-RealSense]

Thanks again @Chris45215 for the highly detailed feedback. It's good to hear that you have had apparent positive progress. It is fine to leave this case open for a further time period whilst you perform additional tests. Good luck!

[MartyG-RealSense]

Hi @Chris45215 Do you have an update that you can provide about this case, please? Thanks!

[Chris45215]

@MartyG-RealSense we do have a bit of an update - we made a version of our program (which uses PCL) that runs on Linux, and started getting the exact same error again. And unfortunately, we can't expect a magic overnight update to fix this one like it does with Windows.
So I still don't know exactly what is causing it, but the issue is definitely there. If it can't be resolved or avoided, it's a critical issue that will prevent us from using these sensors in customer applications. If it's a 'simple' USB port problem, the SDK or driver should throw a clear explanation saying what the problem is and how to fix it.

[MartyG-RealSense]

An instruction that can be useful when the frame queue gets jammed up and is not receiving any new frames is Keep(). This stores frames in memory. Because it progressively consumes a computer / computing device's memory capacity as it stores the frames, it is usually useful for short-duration recordings such as 30 seconds.

If you run a Keep() test script and do not experience your usual time-out problems during the test recording period, this may provide a clue about the problem that your Python application is having.

#6146

[MartyG-RealSense]

Hi @Chris45215 Do you require further assistance with this case, please? Thanks!

[Chris45215]

Our latest update is that we took the RealSense aspects out of our project, because the addition of the pyrealsense2 library created some conflict with the pclpy library (and specifically the Laspy library that it requires) that we are using, and that prevented pclpy from working at all. We use sensors from multiple sources, so we can't depend exclusively on the RealSense, or its own flavor of the PCL components.
Unfortunately, the projects will not run even after pyrealsense2 is removed from the environment.

So, whether that latest proposed solution works or not is still unknown. We need to resolve the PCLpy/Laspy issues first. The specific error that we get is on the first line of the program, "import pclpy", and the error is "type object 'DimensionInfo' has no attribute 'from_dtype'". That error refers to a class within the Laspy library. I have a write-up of that bug in the PCLpy project at davidcaron/pclpy#92

[MartyG-RealSense]

I couldn't find any examples of pyrealsense2 and pclpy / Laspy being used together in a project, which may be a reflection of the difficulties that you have had with combining them together. pclpy is apparently a binding for the PCL library though, and the RealSense SDK already has its own PCL wrapper (see the link below). Would it therefore be practical to access PCL through that instead of pclpy?

https://github.com/IntelRealSense/librealsense/tree/master/wrappers/pcl

The link below describes attempts by a RealSense user to access PCL through the RealSense SDK.

#2483

If you do need to use pclpy, the link below provides a script that a RealSense user wrote to save a point cloud as a pcd file through pclpy (I note that they apparently do not import pyrealsense2 in their script):

#4210 (comment)

[Chris45215]

We need as many features from PCL as possible, like voxelgrids and octrees, which fortunately PCLpy has but it seems that the Realsense PCL wrapper doesn't include all of those - maybe it does, but I couldn't find documentation or examples of it. Regardless, we are using several types of depth cameras and so we can't be tied completely to a Realsense-focused implementation.

We can use C++ rather than Python, but installing the C++ version of PCL for Windows is.... lets just say that there is more documentation saying that it's a nightmare than there is saying how to actually do it successfully.

[MartyG-RealSense]

I reviewed this case completely from the beginning again in search of further insights. As I have already linked to the very few references available about using a RealSense camera with pclpy, I was not able to make much progress though.

The last idea that I have is that if pclpy's installation is dependent on Conda then it might make a difference if the pyrealsense2 wrapper was built from source code with Conda instead of a pip installation. The link below contains guides for installing the Python wrapper on Windows with Conda that RealSense users have created.

#1657 (comment)

https://rahulvishwakarma.wordpress.com/2019/08/17/realsense-435i-depth-rgb-multi-camera-setup-and-opencv-python-wrapper-intel-realsense-sdk-2-0-compiled-from-source-on-win10/

[MartyG-RealSense]

Hi @Chris45215 Do you require further assistance with this case, please? Thanks!

[Chris45215]

Fortunately, I can give good news in the update. We copied the Python environment from one of the computers that was working properly, and it runs without error on the other computers.

I've not been able to find a difference between any of the packages or versions in the environments, but I presume some differences must exist.
We may be able to post a small example of the RealSense working with PCLpy as a public repository at some point, using some of the features that aren't in examples anywhere else.

So, we can probably close this case. The original frame issue resolved itself after a few days; and the Laspy/PyRealsense issue is resolved as long as we have our backup of the working environment. Thanks for your help throughout it; these Heisenbugs are the hardest to handle.

[MartyG-RealSense]

You're very welcome - it's excellent to hear that your situation worked out for you. Thanks for the update :)