With multiple cameras, frames drop sporadically and inconsistently

[banuprathap]

---

Required Info
Camera Model	D435
Firmware Version	05.12.09.00
Operating System & Version	Ubuntu 14.04
Kernel Version (Linux Only)	4.4
Platform	Intel Com Xpress
SDK Version	2.40.0
Language	C++
Segment	Robot

Issue Description

We have three D435 cameras connected to a USB3 root hub on the com express. We power these USB ports externally with dedicated 1A per port on our mother board.

We stream only depth frames from these cameras at 1280x720 @15fps. After extensive testing, we discovered that the sensors stop streaming after a long while. We haven't spent anytime in understanding the duration but this can be sometime between several minutes to a day or two. We were temporarily able to workaround this by resetting power (simulate unplug and replug) via the external power supply to these USB ports. Are there any other reports on this issue? We are interested in finding a long term solution for this.

A showstopper for us is the frame rate drop. We notice that sporadically the frame rate drops to as low as 6FPS, as estimated by our logic in the frame callback handler (adapted from the comment here). This adversely affects our robot's ODOA.

I see a similar issue on the firmware release document (DSO-10591) marked as No Fix. Can you share the details on why this issue happens and why realsense team deems it as No Fix so we can make some informed tradeoff analysis? Also can you suggest methods or steps to mitigate this issue?

Thanks.

[MartyG-RealSense]

Hi @banuprathap Do all of the cameras stop streaming simultaneously?

An event that may fit these circumstances is if the camera connection become mis-identified as USB 2.0 instead of USB 3 for some reason. 1280x720 at 15 FPS is unsupported when the camera is in USB 2.0 mode, with the highest FPS available at that resolution being 6 FPS.

So if the camera was reset after the problem occurs and is still mis-identified as being on a USB 2.0 connection after the reset then if 1280x720 is still being used then 6 FPS may become the frame rate that is applied (the only FPS available for a D435 at 1280x720 on a USB 2 connection).

Are you streaming in the RealSense Viewer tool or using your own script, please?

[banuprathap]

Hi @MartyG-RealSense. Yes, all three cameras are stopping at the same time. We use the C++ API with a registered callback handler. See a sample snippet below. Please note that the reset happens after the stream stops and the sensors connect back as USB3.2 devices almost everytime.

std::vector<rs2::stream_profile> stream_profiles = sensor.get_stream_profiles();
for(auto&& sp : stream_profiles)
{
      if(matching profile)
      {
            sensor.open(sp);
            sensor.start(bind(&RsSensorServer::frameCallback, this, _1));
      }
}

I tried with the realsense viewer and the viewer fps always lingers around ~6FPS. I understand (from other threads) that this could be attributed to the UI render overhead.

Do you have any info on the frame drops and the seemingly related DSO-10591 item in the release notes?

[MartyG-RealSense]

I do not have information about aspects of firmware drivers that are documented in the firmware release notes. I do apologize.

Do you have auto-exposure enabled or are you using a manual exposure value, please? In either mode (automatic or manual exposure), there are circumstances in which FPS may fall, but there are ways to enforce a constant FPS rate. These are described in the link below.

#1957 (comment)

The link mentions how, when auto-exposure is enabled, an option called Auto-Exposure Priority can be disabled to enforce a constant FPS. You can test this in the RealSense Viewer by disabling Auto-Exposure Priority under the RGB section of the Viewer's options side-panel, in the 'Controls' category. Left-click on the box beside the option to turn it from blue color (On) to black (Off).

[banuprathap]

Hi @MartyG-RealSense, I forgot to mention that we stream only the depth data. No IR or RGB streams are enabled.

I disabled auto exposure and I still observe the same issue of frame drops.

Apologies for being direct but could you put me in touch with someone who is familiar about DSO-10591 ?

[MartyG-RealSense]

The contents of internal processes such as DSOs are not able to be discussed, unfortunately.

Auto-Exposure Priority is the option that should be disabled. This is separate from the Auto-Exposure option, which should be left enabled when Auto-Exposure Priority is disabled. It may not make a difference if you are only streaming depth alone though.

Re-reading this discussion, I note your mention in the opening comment about your 1 Amp per port power allocations on the powered hub. I recommend budgeting for each camera on the hub to consume 2 Watts. If your ports are powered by your motherboard and that power in turn comes from the device's power supply unit then the stability of power supply to the USB ports may be less than if the cameras were attached to a mains electricity powered external USB 3 hub.

[MartyG-RealSense]

I will post here the additional information that you provided in another case so that the discussion can be kept within the same case.

---

We started noticing that the sensor would sometime not connect back and fails with Resource temporarily unavailable exception.

Here is a snippet of how we initialize the device.

rs2::context m_ctx;
m_ctx.set_devices_changed_callback(bind(&RsSensorServer::changeDeviceCallback, this, _1));

void RsSensorServer::changeDeviceCallback(rs2::event_information& info)
{
    if(info.was_removed(m_device))
    {
        m_device = rs2::device();
    }
    if(!m_device)
    {
        rs2::device_list new_devices = info.get_new_devices();
        if (new_devices.size() > 0)
        {
            initDeviceHandle(new_devices);
            // if we have matching device start streaming
            if(m_device) startDevice();
        }
    }
}

void RsSensorServer::initDeviceHandle(rs2::device_list list)
{
    if(!m_device)
    {
        // process only if we don't already have initialized       
        if(list.size() == 0)
        {
           LOG() << "No Realsense devices were found!";
        }
        else
        {
            // iterate over all available devices
            for(size_t i = 0; i < list.size(); i++)
            {
                rs2::device dev;
                try
                {
                    dev = list[i];
                }
                catch(const std::exception& e)
                {
                    LOG() << "Device " << i+1 << "/" << list.size() 
                                        << " failed with exception: " << e.what();
                    continue;
                }
                <do more stuff here> 
             }
    }
}

[banuprathap]

Hi @MartyG-RealSense, I wanted to clarify on #9428. It is not clear to me why it is marked duplicate. To me #9385 is about the frame rate drops and eventual stream stalls where as #9428 is about occasional device initialization failures. In other words, if we reinitialize the cameras even without the stream stalling, we notice the init failure with resource unavailable exception.

Autoexposure priority had no effect on FPS either. Since our board supplies 1A per port at 5V, our budget is 5W (I'm not an electrical engineer).

Do you have any suggestions on either issues?

I'd be happy to run more tests or generate debug logs if that'd be helpful.

[MartyG-RealSense]

Aside from the script and an added couple of lines, the text of #9428 was almost the same as this case. If you make some edits to #9428 to make it clear that it is a separate question then I am happy to re-open it and treat it separately from this case.

I see that you are using Ubuntu 14.04 in the information that you kindly provided at the top of this case. Given that the Debian packages list support for Ubuntu 16/18/20, did you build librealsense from source code instead (a method that supports Ubuntu 14.04) and use the instruction for 14.04 (below)?

[banuprathap]

I clarified the description for #9428.

Correct. I followed instructions to patch Kernel and install dependencies. I downloaded the source code, compiled and installed it with steps below:

Note: I edit the unix_config.make as a workaround for #8031.

cd librealsense2 && sed '/-mssse3/d' -i CMake/unix_config.cmake
mkdir build
cd build && \
  cmake .. \
  -DCMAKE_C_COMPILER=gcc-6 \
  -DCMAKE_CXX_COMPILER=g++-6 \
  -DCMAKE_BUILD_TYPE=Release \
  -DBUILD_EXAMPLES=false \
  -DBUILD_GRAPHICAL_EXAMPLES=false \
  -DBUILD_GLSL_EXTENSIONS=false \
  -DBUILD_PYTHON_BINDINGS=true \
  -DPYTHON_EXECUTABLE=/usr/bin/python \
  -DBUILD_EASYLOGGINGPP=false \
  -DBUILD_WITH_TM2=false \
  -DIMPORT_DEPTH_CAM_FW=false \
  -DFORCE_RSUSB_BACKEND=false
make
make install

[MartyG-RealSense]

I have re-opened the closed case and taken it on. Thanks very much for the edit.

I looked at the DSO-10591 reference that you made at the start of this case in the firmware specification document. The firmware notes are a public document and so that information source at least can be quoted. 'No fix' means that there are no plans to create a fix for that particular issue.

The SDK already introduced improvements to multicam in SDK version 2.35.2, though these primarily address multicam issues related to rs2::pipeline. Some multicam problems may instead be caused by factors external to the SDK like a specific model of USB hub, and the SDK cannot address such problems that are outside of its control.

[MartyG-RealSense]

Hi @banuprathap Do you require further assistance with this case, please? Thanks!

[banuprathap]

Hi, yes. We are still facing the frame drop issue and it is not clear how we would deploy a multicam application with reliable FPS streaming. We are well below the recommended bandwidth per document here.
https://dev.intelrealsense.com/docs/multiple-depth-cameras-configuration#section-a-bandwidth

Any inputs or recommendations from the realsense team on how to improve?

[MartyG-RealSense]

Given that all three cameras experience the frame drop simultaneously, have you tried a different hub to test whether the problem is with the hardware of the currently used USB3 root hub?

A model that Intel tested successfully in their 400 Series multiple camera white-paper document was Amazon's AmazonBasics powered USB 3 hub. I have one of these myself based on that recommendation and it has been problem-free.

https://dev.intelrealsense.com/docs/multiple-depth-cameras-configuration#section-2-multi-camera-considerations

[banuprathap]

We are connecting the three cameras directly to the PC (root hub). Our processor has 4x USB3.0 ports exposed (see).

Let me check if we can order one and give it a shot. Just so I understand, the AmazonBasics USB hub should still be connected to one of these 4 ports, correct?

Also, the document I linked above states In general connecting cables straight to the PC gives better results when pushing the bandwidth limits.

So it should only be worse with the AmazonBasics hub, correct?

[MartyG-RealSense]

The AmazonBasics hub has a connector for a mains power supply 'brick' and a USB port for connecting a USB lead between the hub and a port on the computer (this port is for the hub-to-computer cable and not for devices). The hub has USB-A (full size connector) and USB-C (micro-connector) versions available. I have the version that plugs into a USB-C micro port on my computer. So make sure to order the USB-A version if you do not have a USB-C micro port on your computer.

So yes, the hub-to-computer lead should connect to one of the four USB 3.0 ports on your computer.

Connecting cameras directly to the USB port of a computer instead of via a hub has the advantage that each individual port on the computer typically has a dedicated USB controller, whereas a hub may have USB controllers that handle a couple of ports instead of just one. Using a mains-powered hub may outweigh the possibility of a modest drop in performance though in regards to the stability of power delivery to the USB port on a powered hub.

[banuprathap]

Thanks.

To summarise, we are agreeing the current configuration of connecting 3 sensors directly to PC has the optimal performance. We have verified that each port can receive a maximum of 1Amps. We stream these cameras at 1280x720 resolution and 15FPS but still see low (~6FPS) frame rate sporadically.

It would be of great help if you could provide pointers on where to look next.

[MartyG-RealSense]

Are you able to confirm that the FPS is actually dropping to 6 FPS? As mentioned in the #7488 (comment) link that you quoted at the beginning of this case, values produced by the frame-drop calculator do not necessarily mean that the FPS is dropping.

[banuprathap]

Apologies for inaccurate terminology.

We notice that the frames are getting dropped and our application receives ~6 frames per second (6 invocations of frame callbacks every second).

[MartyG-RealSense]

In my mind I cannot eliminate the possibility that some of the instability may be due to using Ubuntu 14.04 and kernel 4.4, which is extremely old in terms of modern librealsense SDK / firmware setups.

Is the use of 14.04 compelled by some other non-RealSense aspect of your project that needs 14.04 to maintain compatibility with it, please?

[banuprathap]

Thanks for the suggestion about kernel version. We are unfortunately tied to Ubuntu 14.04 with 4.4 kernel due to other dependencies of the project.

I ran a quick test capturing the different frame metadata including timestamps and notice that the frame ids are "skipping". Does this mean the frames are lost in transit (i.e USB) or am I misinterpreting it?

FRAME_ID	SENSOR_TIMESTAMP	FRAME_TIMESTAMP	BACKEND_TIMESTAMP	TIME_OF_ARRIVAL		FRAME_TIMESTAMP - SENSOR_TIMESTAMP
0	2185072929	2185079714	1627652355121	1627652355134		6785
1	2185139593	2185146378	1627652355187	1627652355219		6785
2	2185206257	2185213042	1627652355253	1627652355270		6785
3	2185272921	2185279706	1627652355319	1627652355472		6785
4	2185334051	2185346369	1627652355386	1627652355715		12318
5	2185400715	2185413033	1627652355453	1627652355910		12318
6	2185467379	2185479697	1627652355520	1627652356162		12318
9	2185672557	2185679688	1627652355720	1627652356501		7131
12	2185872723	2185879679	1627652355920	1627652356705		6956
16	2186139378	2186146334	1627652356187	1627652356824		6956
21	2186472697	2186479653	1627652356520	1627652356984		6956
24	2186672688	2186679644	1627652356720	1627652357203		6956

When the frames are dropped, usbtop reports a drop in the device bandwidth. All three USB devices in table below are D435.

Bus ID 8 (USB bus number 8)	To device	From device
Device ID 11 :	53.18 kb/s	20776.28 kb/s
Device ID 12 :	53.17 kb/s	20746.79 kb/s
Device ID 13 :	66.16 kb/s	29354.61 kb/s

[MartyG-RealSense]

You do indeed seem to be dropping frames. You may be able to reduce frame drops if you try increasing the frame queue capacity from its default of '1'. This introduces latency , which reduces the likelihood of frame drops occurring.

https://github.com/IntelRealSense/librealsense/wiki/Frame-Buffering-Management-in-RealSense-SDK-2.0#latency-vs-performance

[MartyG-RealSense]

Hi @banuprathap Do you require further assistance with this case, please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.