Multi-camera issue: After connecting the sixth camera, its motion frame doesn't arrive #5828
Comments
|
Hello, I have the same problem. I am running Ubuntu 18.04.4 LTS - 5.3.0-28-generic. The installation of realsense 2.32.1 via the Debian package worked without error messages. Also the kernel patch: Whether I use the sensors inside ROS or just the realsense-viewer. Starting with the 4th sensor I get the error message: I have exchanged the cables, as well as the ports, the error always comes with the 4th connected sensor regardless of the configuration. I also have the problem that if I connected the sensors via a Thunderbolt 3 hub like before, there are no more frames when the sensor is turned off and on again. Operation via the hub was no problem before. Despite the error message starting with the 4th sensor, I get data in realsense-viewer as long as I do not run the sensors over the hub. What does not work is to work with the sensors in ROS. The nodes that subscribe to the corresponding channels of the sensors don't get any data anymore. This was running without problems before and no changes were made at this point by me. In about one of 30 attempts my ROS configuration works and I can work with the sensor data as usual. Why it sometimes works and sometimes not is not reproducible for me. If it does not work I either get the following error message from t265:
Or I get an error message that the camera_manager has died. Unfortunately the log file cannot be found. I have also noticed that the physical port on the T265 is e.g. With the t265 I also noticed that the serial number has changed since realsense 2.32.1. Four zeros were added at the front. I hope you can help me, I'm a bit confused why my system doesn't work as usual anymore. |
|
Hello @schmiran , Today we rebuilt the library from development branch (Latest commit d3980b2). We also apply the patch manually. However the motion frame is still not arrived from the sixth cameras |
|
Well... I found out the same error |
|
Hey @wangroger0801 , so installing the development branch didn't help after all? I'll try it this afternoon and let you know. Regarding your error message with the frames:
I get the same error message when I run sensors across the Hub. Even on the first sensor. If I connect them directly to the system, I don't get that error message. |
|
For the problem with the wrong physical ID I found this IntelRealSense/realsense-ros#1051 (comment) and will test it later. |
|
Hey @wangroger0801 , I have now also tested the developer branch, without changes. The other hint about the physical ID couldn't help me either. Do you have new information? Or maybe @RealSenseCustomerSupport has an idea what causes the error messages? |
|
Hello @schmiran , Thanks for your reply. We did a new test. It fixed the motion frame not arrived problem. But more errors pop out. We still can't using 6 cameras. I am sure it not hub related it works fine on Windows. We added Although we did enable all the frames in realsense viewer, our multi-cam program failed to start. It reported @RealSenseCustomerSupport any idea how we can fix this issue? |
|
With the new Realsense version it works again. I still have the problem, that if I switch the sensor off and on again, no more frames are coming but this can be solved with an I hope the new version has helped you too. It's a pity that there was no support for this topic. Greetings |
|
We're facing the same issue. When we connect the 2nd camera, we get "Cannot set power state" error.
|
|
I can't seem to find documentation regarding low power mode 672b33b In our 2nd camera, we need to run only RGB stream. Can we run it in low power mode to get around the cannot set power state issue? |
|
Interestingly when I run the RGB only streaming camera first and then the first camera which does RGB + Depth streaming, both the cameras run fine. Seems like when i run the RGB+Depth camera first, it takes up more power than it needs and then does not leave any for the RGB only streaming Realsense camera. Need some way to limit the resources consumed by each Realsense camera. |
void rs_uvc_device::set_power_state(power_state state)
{
_action_dispatcher.invoke_and_wait([&, this](dispatcher::cancellable_timer c)
{
if(state != _power_state)
{
switch(state)
{
case D0:
_messenger = _usb_device->open(_info.mi);
if (_messenger)
{
listen_to_interrupts();
_power_state = D0;
}
break;
case D3:
if(_messenger)
{
close_uvc_device();
_messenger.reset();
}
_power_state = D3;
break;
}
}
}, [this, state](){ return state == _power_state; });
if(state != _power_state)
throw std::runtime_error("failed to set power state");
}The 2nd camera (RGB only) is unable to open the USB device for some reason when the 1st camera (RGB + Depth) is fired up first. From the above function, seems like D0 is the power state. |
|
@MartyG-RealSense any thoughts on this one? |
|
@tispratik This link may give you a useful lead: |
|
Thanks Marty. I had seen that, but does not seem to apply to our case. Our case is 1 camera streaming depth+RGB connected to USB 3.1 port and another camera streaming only RGB stream connected to a USB 3.0 4Amps hub which is in turn connected to another USB 3.0 port on the same motherboard. We have the exact same issue on 3 sets of our machines, so it is a reproducible problem. Interesting to note is that when we run the RGB only camera first, then the RGB+depth camera, both play along nicely on power. But if we run the RGB+depth camera first, then the RGB only camera gives the error of "cannot set power state". |
|
@tispratik It sounds as though one camera is connected to the hub and one is connected directly to the computing device. If that is the case, connecting both to the hub should allow both cameras to be handled by the hub's USB controllers. |
|
Yes, we tried connecting both cameras to the hub as well as both cameras directly to the motherboard. We see the same issue. |
|
Update: Even though we could run the two cameras by starting the RGB camera first and then the RGB+depth one, the RGB camera dies after some time. Following is what i see in dmesg. |
|
A little research on the error points me to https://www.spinics.net/lists/linux-usb/msg102772.html The usb_status usb_messenger_libusb::reset_endpoint(const rs_usb_endpoint& endpoint, uint32_t timeout_ms)
{
int ep = endpoint->get_address();
auto sts = libusb_clear_halt(_handle->get(), ep);
if(sts < 0)
{
std::string strerr = strerror(errno);
LOG_WARNING("reset_endpoint returned error, index: " << ep << ", error: " << strerr << ", number: " << int(errno));
return libusb_status_to_rs(sts);
}
return RS2_USB_STATUS_SUCCESS;
} |
|
@tispratik These 'lowering kernel' messages are a Linux kernel issue rather than a Librealsense error. A cause of the messages you experienced in the post above can apparently be high CPU load. If you are having kernel and patching problems then maybe you could consider the backend SDK installation method that does not rely on kernel versions and patching. You will need an internet connection though. https://support.intelrealsense.com/hc/en-us/community/posts/360037657234/comments/360009799533 |
|
Thanks Marty. Do you mean to say that we should set the flag
|
|
Yes, set -DFORCE_RSUSB_BACKEND=true |
|
@MartyG-RealSense Thanks for your support. I also tired |
|
My understanding is that a mains-powered USB 3 hub can supply about 12 W of power, and each individual RealSense 400 Series camera requires 2 W. So I wonder if your sixth camera has insufficient power available. If each hub has 12 W available, using more than one powered hub may help, putting some of the cameras on the second hub. In the USB specification, up to five hubs can be connected together (a setup known as '5 deep'). |
|
We have a PCI-E USB adapter which should give sufficient power. And It doesn't explain the reason why it works on windows with the exact configuration. |
|
A PCI-E USB adapter will still be drawing the power for the port from the computer's power supply unit in the same way that the built-in USB ports do (i.e they are 'passive' ports on the adapter and vulnerable to instabilities like the built-in ports). Are you using Windows and Ubuntu in dual-boot on the same computer, please? |
|
It is using PCI-E x4 which should give 25W power to the USB ports. |
|
The amount of wattage is important but not the only factor. I have seen cases involving 650 W power supply units on enterprise servers where camera disconnections could still occur. The stability of the power supplied to the port matters as much as the amount of wattage supplied. Mains-powered external USB hubs give greater power stability than the internal power supply units of computers. At this point, I will refer the case internally to a support team manager to see if they can provide advice. I apologise for the delay in the meantime. |
|
I could reliably reproduce the problem with the below script. You will need two cameras for running the script. import pyrealsense2 as rs
import numpy as np
import cv2
devices = rs.context().query_devices()
pipeline_1 = rs.pipeline()
pipeline_2 = rs.pipeline()
config_1 = rs.config()
config_2 = rs.config()
config_1.enable_device(devices[0].get_info(rs.camera_info.serial_number))
config_2.enable_device(devices[1].get_info(rs.camera_info.serial_number))
config_1.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 60)
config_1.enable_stream(rs.stream.color, 640, 480, rs.format.rgb8, 60)
config_2.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 60)
config_2.enable_stream(rs.stream.color, 640, 480, rs.format.rgb8, 60)
pipeline_1.start(config_1)
pipeline_2.start(config_2)
while True:
frames_1 = pipeline_1.wait_for_frames()
frames_2 = pipeline_2.wait_for_frames()
color_frame_1 = frames_1.get_color_frame()
color_frame_2 = frames_2.get_color_frame()
color_image_1 = np.asanyarray(color_frame_1.get_data())
color_image_2 = np.asanyarray(color_frame_2.get_data())
cv2.namedWindow('CAM 1', cv2.WINDOW_AUTOSIZE)
cv2.imshow('CAM 1', color_image_1)
cv2.namedWindow('CAM 2', cv2.WINDOW_AUTOSIZE)
cv2.imshow('CAM 2', color_image_2)
cv2.waitKey(1)
if cv2.waitKey(1) & 0xFF == ord('q'):
breakThe above script as is may or may not work for you. What is very whacky about the whole issue is that if you swap the order of starting the pipelines 1 and 2......it works!! The order in which we start the two pipelines matters for some unknown reason which the Realsense team needs to dig into. Try: pipeline_1.start(config_1)
pipeline_2.start(config_2)vs: Swap pipelines start: pipeline_2.start(config_2)
pipeline_1.start(config_1) |
|
The compilation error is related to #6573 and will be addressed in PR #6581. |
|
Any update on this issue? The compilation issue is separate. We see that this hasn't been fixed yet. |
|
@tispratik If you are getting that compilation error when trying to build SDK 2.35.2 from source with BUILD_WITH_CUDA=true, 2.35.2 has a bug that causes build to fail when CUDA is true in the CMake build statement. A fix has been developed for the next version of the SDK. The bug can also be corrected with a method in the link below: |
|
We are not getting the compilation failure after the fix we applied based on an earlier comment in this ticket. This issue #5828 (comment) still persists. |
|
The multi-camera improvements were designed to address multicam problems related to rs::pipeline. It was acknowledged though that it may not correct every multicam problem, especially if the problem was related to the model of USB hub being used. |
|
Marty. I am frustrated that we are dilly dallying the issue. There is no connection to USB Hub. It was a test we did to see if it helps to add a hub. The hub does not help as well. It is such an easily reproducible issue. We have bought 100 intel cameras so far during our R&D phase, now we are in production, but still facing the issue. We have been forced to use a regular RGB camera as our second camera to pair up with the 1st Realsense camera. Why can't we have a 2 Realsense camera setup going?? |
|
An approach that other Python users have used when configuring cameras 1 and 2 is to structure the lines like this: config_1.enable_device('013102060174') In comparison, you use this line: config_1.enable_device(devices[0].get_info(rs.camera_info.serial_number)) So if you test your script using the exact serial numbers of the two cameras that you are using and there are no problems, this would suggest that the problem may be with how your line is retrieving the serial number. A single RealSense camera does not need a serial number. A serial number is only needed if there is more than one camera, to differentiate between the devices. If it cannot recognise more than one RealSense camera serial number, it suggests that it is not successfully checking the serial numbers and so behaving in its default mode (using only one camera in the absence of a serial number). |
|
The list of devices is coming from Realsense API itself. Not sure what wrong we can do retrieving the serial numbers from the |
|
We got the list of serial numbers using the below command: |
|
The pyrealsense2 documentation for the format of enable_device instructions is here: |
|
RuntimeError: failed to set power state is happening for me on OSX with 2 x 415 cameras on FW 05.12.06.00, from latest code today (2.36.0). I modified the device manager to use serial number strings as suggested above but it didn't help. I'm trying to run box_dimensioner_multicam_demo.py before starting a multi camera project. Code runs with either camera connected but not both. |
|
@keithahern Does use of the serial numbers make a difference if you unplug the two cameras and then re-insert them in a particular order, as suggested earlier in this discussion: |
|
@MartyG-RealSense no difference. Tried with the both the api serial id code and the hardcoded id code. Both plugged into 2019 MacBook Pro. |
|
@keithahern Are you able to test the cameras with another multicam program such as rs-multicam on your Mac, or stream two cameras at the same time in RealSense Viewer, to confirm whether your computer can stream two cameras simultaneously? |
|
Yes it works fine on rs-multicam and realsense viewer with 2 cameras.
… On 16 Jul 2020, at 13:08, MartyG-RealSense ***@***.***> wrote:
@keithahern Are you able to test the cameras with another multicam program such as rs-multicam on your Mac, or stream two cameras at the same time in RealSense Viewer, to confirm whether your computer can stream two cameras simultaneously?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
@keithahern My research indicates that when failed to set power state occurs with multiple cameras on Python, it tends to be related to the device manager script in box_dimensioner_multicam or self-created projects that use an adaptation of that device manager script. I tracked down an alternative Python multicam script that you could test to see if it can be used as a basis for your project. |
|
thanks Marty, The code you reference actually has the same problem: |
|
ok, on a hunch I changed the cables I am using, which are these high quality cables, recommended by RealSense. I am using the 5m cables. Now the interesting thing is that running 2 x 415s with these cables in the RealSense Viewer showing both depth and rgb at max res is fine. In fact I have run 4 cameras/cables without problems. So here are my results: It's weird that opencv_viewer_example.py is now working with my long cables. Perhaps they weren't inserted properly (but RealSense viewer worked on the 1st run...). Either way the example code in the distribution is not working. Perhaps this helps. |
|
@keithahern Great news that you achieved positive results from your cable change! The high-quality USB cable supplier now officially recommended by the official data sheet document for the 400 Series cameras is Newnex. I have seen past reports from RealSense users that NTC Distributing cables have worked for them too though.
|
|
I've been following this thread for over a month because I have been having the same issue. The solve was changing the cables? I'm already using the cables that came with my cameras (d435 and d435i). I purchased a powered USB hub in case my nuc wasn't providing enough power for both. I am still having this issue. |
|
Today, after not running from my code for a week I get the power error with the supplied cables. It worked last week! I am wondering if it’s to do with the USB A -> USB C adapters I use to connect it to my MacBook. The stiff bundled cables and the adaptors are under a bit of strain - however the librealsense python multi cam demo now works, which didn’t last week!
There is something very strange going on.
… On 29 Jul 2020, at 19:58, alexisburns91 ***@***.***> wrote:
I've been following this thread for over a month because I have been having the same issue. The solve was changing the cables? I'm already using the cables that came with my cameras (d435 and d435i). I purchased a powered USB hub in case my nuc wasn't providing enough power for both. I am still having this issue.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
Hardware: i7-9800X, Qudro RTX4000 , Nvidia driver 435.21
Issue Description
Hello,
We are going to use six or more cameras in our system. On windows10, all demos from sdk are fine without problem.
On Ubuntu, different errors start appearing after plug in more cameras.
The main issue which stop us developing is we cam not use the motion frame from the sixth camera. (In demo rs-multicam, it only show 5 cameras when 6 are plugged in).
Could your team check how we could fix this issue? Thanks!
Monitoring the frame data from the Realsense Viewer.
Testing procedures:
First CAM
Second CAM
Third CAM
4th CAM
5th CAM
6th CAM
After enabling the Motion Frame from the 6th CAM
All the frame reading disappeared on the viewer, shows:
Error from realsense viewer log
Samples from dmesg errors:
After searching on the closed issues here, it seems to be a kernel patch problem. But we did install librealsense-dkms package.
verified by:
modinfo uvcvideo | grep "version:"We also tried install patch manually under Ubuntu (the official guide) and Arch Linux, under kernel 5.0 and 5.1. following the guide from closed issue #4271 , but we have the same problem.
It seems not caused by a bandwidth or cable problem. 4 in 6 cameras are connected to dedicated USB3.1 ports and everything works fine on windows
USB connection verified by
lsusb --treeThe text was updated successfully, but these errors were encountered: