Multiple D435i cameras are synchronized in depth, can the master enable imu without damaging the synchronization function #12072
Comments
|
Hi @chenxiaocongAI The IMU cannot be hardware-synced with a master trigger. However, each individual camera's IMU data packet is timestamped using the depth sensor hardware clock on that particular camera to allow temporal synchronization between gyro, accel and depth frames. |
If it's just a single camera, I think your statement is fine, but what I need to do now is to enable the imu function when synchronizing the depth of 5 cameras. After successfully synchronizing the depth of 5 cameras, I enabled the Imu function on the slave camera, but this will affect the synchronization of the depth of 5 cameras. Have you tested the depth synchronization of 5 cameras and enabled the imu function of any camera at the same time? If tested, can I show you the demo |
|
Depth should be the only stream that is being hardware-synced to the master. However, a RealSense user at IntelRealSense/realsense-ros#2648 found that the infrared stream was also apparently being affected. So there is a precedent for a stream type that is not directly being synced to a master being affected by the sync process. RealSense users usually do not use IMU in multiple camera hardware sync setups, so I do not recall a previous case where another user has experienced the same issue as you and I do not know of an official Intel test that has done so. I do not have the equipment at my location to replicate your test myself. |
There is a problem that I have never understood, since IMU also uses the timestamp of the depth sensor. Why am I unable to synchronize depth sensors with other cameras when I turn on IMU on one camera |
|
The hardware sync system was designed for syncing depth between multiple cameras and has never supported use of IMU streams during sync, unfortunately. Hardware sync is not always necessary for a project though and satisfactory results may stll be able to be achieved with a multiple set of unsynced cameras. |
|
Hi @chenxiaocongAI Do you require further assistance with this case, please? Thanks! |
|
Which camera has depth and color sensors on the same motherboard
发自我的iPhone
…------------------ Original ------------------
From: MartyG-RealSense ***@***.***>
Date: Sun,Aug 13,2023 3:02 PM
To: IntelRealSense/librealsense ***@***.***>
Cc: chenxiaocongAI ***@***.***>, Mention ***@***.***>
Subject: Re: [IntelRealSense/librealsense] Multiple D435i cameras aresynchronized in depth, can the master enable imu without damaging thesynchronization function (Issue #12072)
Hi @chenxiaocongAI Do you require further assistance with this case, please? Thanks!
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
The D415 and D455 camera models have RGB and depth sensors on the same board. The images below show the circuit boards of D415 and D455 with the RGB sensor integrated on them. D415
D455
|
|
I successfully synchronized depth using 5 cameras (including d435i and d455), but when I turned on the color sensor on the master camera, I found that the arrival timestamp of the master camera sometimes differs by about 30ms from the depth data received by my program. May I ask if anyone has encountered a problem like mine and has addressed it
发自我的iPhone
…------------------ Original ------------------
From: MartyG-RealSense ***@***.***>
Date: Sun,Aug 13,2023 10:01 PM
To: IntelRealSense/librealsense ***@***.***>
Cc: chenxiaocongAI ***@***.***>, Mention ***@***.***>
Subject: Re: [IntelRealSense/librealsense] Multiple D435i cameras aresynchronized in depth, can the master enable imu without damaging thesynchronization function (Issue #12072)
The D415 and D455 camera models have RGB and depth sensors on the same board. The images below show the circuit boards of D415 and D455 with the RGB sensor integrated on them.
D415
D455
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
When using a single camera and turning on the depth and color sensors, I found that the arrival time of the depth sensor sometimes lags behind the PC time when I received the callback by 30ms. When I receive data on the PC, I record the current time and then obtain the time of the depth sensor. When I compare the two, I sometimes find that it lags by 30ms. Why is this happening@MartyG-RealSense |
|
When both depth and RGB are enabled, there will be a temporal offset of one frame, as advised by a RealSense team member at #1548 (comment) If the wait_for_frames() instruction is used in a script then the RealSense SDK should automatically try to find the best timestamp match between depth and RGB. The SDK should also attempt to automatically sync the two streams when they have the same FPS. A way to help match the FPS is to disable an RGB option called Auto-Exposure Priority. If Auto-Exposure is enabled and Auto-Exposure Priority is disabled then the SDK will attempt to enforce a constant FPS rate for both streams. |
|
@MartyG-RealSense I'm not asking about the time difference between the depth sensor and the color sensor in the same frame. What I want to ask is, when a PC receives depth sensor data, it records the current PC time and subtracts the arrival time of the depth sensor from the PC time, resulting in a delay of over 30ms. |
|
Timestamps are a complex subject, with different types of timestamp generating their time at different points (such as at the beginning of data transmission to USB, or at the beginning of exposure). Some are generated by the camera hardware's firmware driver whilst others are derived from the computer's system clock. At #2188 (comment) a RealSense team member explains the differences in operation between timestamp types such as frame_timestamp and time_of_arrival. I feel that if you want the depth data arrival time to match up with the system clock of the computer then retrieving the time_of_arrival timestamp is going to work best for you so that it can take account of the time required for a frame to be captured, processed and passed to the user interface. Bear in mind that if a hardware sync 'master and slave' multicam configuration is being used then the slave cameras will try to replicate the timestamp timing of the master camera. Also, it is expected that over time the slave cameras' timestamps will begin to deviate from the master camera's over a long period of tens of minutes. When this occurs, it shows that hardware sync is actually working correctly. This principle is explained in the section of the hardware sync white-paper document linked to below, under the heading Now to the somewhat counter intuitive aspect of time stamps. https://dev.intelrealsense.com/docs/multiple-depth-cameras-configuration#3-multi-camera-programming |
|
#8419 Has this feature been implemented? Can this function be used on multiple types of devices |
|
Is there an example of multiple camera soft synchronization
发自我的iPhone
…------------------ Original ------------------
From: MartyG-RealSense ***@***.***>
Date: Tue,Aug 15,2023 3:17 PM
To: IntelRealSense/librealsense ***@***.***>
Cc: chenxiaocongAI ***@***.***>, Mention ***@***.***>
Subject: Re: [IntelRealSense/librealsense] Multiple D435i cameras aresynchronized in depth, can the master enable imu without damaging thesynchronization function (Issue #12072)
Timestamps are a complex subject, with different types of timestamp generating their time at different points (such as at the beginning of data transmission to USB, or at the beginning of exposure). Some are generated by the camera hardware's firmware driver whilst others are derived from the computer's system clock.
At #2188 (comment) a RealSense team member explains the differences in operation between timestamp types such as frame_timestamp and time_of_arrival.
I feel that if you want the depth data arrival time to match up with the system clock of the computer then retrieving the time_of_arrival timestamp is going to work best for you so that it can take account of the time required for a frame to be accessible by the user after it has been captured, processed and passed to the user interface.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
This may be more complex than what you had in mind, but the CONIX Center at Carnegie Mellon created a system for networking up to 20 RealSense cameras simultaneously and combining their individual outputs into a single one. https://github.com/conix-center/pointcloud_stitching Intel's hardware sync white-paper guide also provides advice about software sync. It is also possible to align frames in software using either their time-stamp or frame-number. Both approaches are valid but you do need to be careful during the initial step of subtracting the offsets that the frames were not read across a frame boundary – for example one camera could have a frame belonging to a previous frame time. The benefit of using the frame counters is that they will not walk off over time More information can be found at #2148 |
|
. D435 allows Depth to Color sync, and Depth to another Depth sync, but not Color to another Color, so at least one (or both) of the color sensors will not be synced. |
|
I would like to ask if the external HW sync, implicitly offers internal RGB-D sync (which is still in Intel's known bug lists) (on D435i or D455 devices). |
|
Color to color sync has never been supported by hardware sync. It may be possible to instead sync two streams from different cameras using the time_of_arrival timestamp though, as described at #2186 (comment) Depth + RGB sync has never been supported for D435i. Intel explored implementing RGB sync for D455 but decided not to proceed. In theory, D415 supports depth + RGB hardware sync in Inter Cam Sync Mode 3 ('Full Slave') but in practice RGB sync never worked well and is no longer supported by Intel (though mode 3 is still selectable). |
|
Why cannot depth and color sensors be synchronized in time and not placed on official documents. As an ordinary person, buying an RGBD camera assumes that the depth and color sensors are synchronized by default |
|
On most RealSense camera models except D415 and D455, the depth and RGB sensors are not mounted on the same circuit board inside the camera and the RGB sensor is attached separately to the circuit board via a cable. This means that sync between depth and color has to be performed by software mechanisms in the RealSense SDK. |
|
But you don't have a method to synchronize depth sensors and color sensors on D455 either! |
|
Hardware sync requires support in the firmware driver. Support of RGB sync for D455 in the firmware was considered but implementation was not proceeded with. |
|
Do you have a white paper for multi camera soft synchronization that you can check |
|
No unfortunately, the only reference to multicam software sync is the one that I linked to earlier. |
|
How to ensure that the order of their arrival is consistent when using multi camera synchronization. For example, when synchronizing for the first time, if the data from five cameras arrives in the order of 1, 2, 3, 4, and 5, will all synchronized data arrive in this order. The slave receives a trigger signal and my program receives data. The time difference between the slave's data arrival and the master's data arrival on the PC is 12ms. Can you reduce this time. |
|
It is possible that the color stream could start okay (with similar timings between depth and color) and update less and less frequently until it stops updating. Which method did you use to install the librealsense SDK, please? If it was built from source code with CMake with the build flag -DFORCE_RSUSB_BACKEND=TRUE and you are using multiple D435i cameras then the SDK will not perform as well as it would if the SDK was installed from packages or built with RSUSB = false. This is because an RSUSB = true build of the SDK is suited for use with a single camera rather than multiple cameras. |
|
@MartyG-RealSense I installed SDK using apt-install. Have there been any previous issues with this issue? If so, how can we solve it |
|
There was a Python case at #5417 where a RealSense user was using depth and RGB in a callback but said that the callback only contained data from one of the two streams. The user found that in their particular case they needed to "call as_frameset() on the frame the callback was processing and then iterate over the frames". |
|
@MartyG-RealSense Are the officials aware of this bug and have they addressed it in the new version |
|
It may not be a bug but an issue in the script. You could test the SDK's official rs-callback C++ program which streams depth and color to see whether you experience any problems with color frame updating in that script too. https://github.com/IntelRealSense/librealsense/tree/master/examples/callback |
|
@MartyG-RealSense |
|
My callback programming knowledge is admittedly limited so I do not have an answer for that question, unfortunately. I do apologize. |
|
Why is this problem not a bug? It has already affected the normal operation of my program, and the official has not provided a solution to the problem. What should you do if the customer encounters this problem? Can relevant developers be asked to answer this question head-on?
发自我的iPhone
…------------------ Original ------------------
From: MartyG-RealSense ***@***.***>
Date: Tue,Nov 21,2023 10:00 PM
To: IntelRealSense/librealsense ***@***.***>
Cc: chenxiaocongAI ***@***.***>, Mention ***@***.***>
Subject: Re: [IntelRealSense/librealsense] Multiple D435i cameras aresynchronized in depth, can the master enable imu without damaging thesynchronization function (Issue #12072)
My callback programming knowledge is admittedly limited so I do not have an answer for that question, unfortunately. I do apologize.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
I tested the rs-callback example program and both depth and color functioned correctly, showing the same frame count without any slowdown of the update. This suggests that there is not a bug affecting the color frame, at least not in this particular callback program. I do not exclude the possibility that problems might occur if a depth-color callback was implemented in a different way to how rs-callback works. My recommendation would be to not use callback in a program unless there is a vital reason for doing so, such as the IMU needing to be streamed in addition to depth + color (since depth-color-IMU does not work well without callback). I will highlight your case to my Intel RealSense colleagues to seek their development advice. Thanks very much for your patience. |
|
@MartyG-RealSense @martinus If you recommend using your callback, it can solve depth and color data problems. But if I need to use imu, depth, and color in the future, I will still come back and solve this problem. So the problem needs to be fundamentally solved. Instead of going around and using another method to solve it |
|
My Intel RealSense colleagues have reviewed this case, including the image that you provided earlier.
They confirmed that the screenshot appears to show that the color stream stops arriving but the depth stream continues. They agreed with my suggestion to test the rs-callback C++ example program to see how the depth and color frames behave. Since rs-callback has been confirmed to be functioning correctly, if the color frame is not being delivered when tested on your own computer then it could indicate an issue with your computer's configuration or your SDK installation. They also suggested testing depth and color streaming in the RealSense Viewer to see whether color stream slowdown occurs in your Viewer. Lastly, they suggested disabling Auto-Exposure Priority - as mentioned earlier in this discussion at #12072 (comment) - to see whether it forces the color stream to keep up with the depth stream. |
|
@MartyG-RealSense |
|
In order to open an investigation into a potential bug, it is important to first establish evidence that a problem may be a bug and not an issue related to one particular customer's computer or scripting that is not occurring for other users. The requests to perform testing of rs-callback and the RealSense Viewer on your computer are important to this evidence process, because if the color stream was also lagging / non-updating in these programs then it could indicate a problem with color streaming in general on your particular system rather than an SDK callback bug. My colleagues provided further advice after reviewing your case. They noted your mentions in October of using genlock hardware sync with multiple cameras. If the camera is still on a genlock sync connection when testing your depth+color callback program then they suggest testing with just one camera or without hardware sync enabled. |
|
I did do this experiment based on hardware synchronization. I tried using only one camera and turning on external hardware synchronization, but the problem still exists. |
|
If you test with hardware sync off and you still have the color problem then I can certainly follow up again with my colleagues in the coming week to continue to explore your issue. They provided some further advice. If the RGB stream is stopped or drops a lot of frames, the SDK keeps using the last available frame. One way to narrow down the problem would be to start simple by using only one camera without hardware sync and see if the problem persists. Then gradually move to multi camera and hardware sync mode. |
|
External hardware signal generators need to have a frequency close to 30Hz and a high level lasting for 100us. How long does it take for the rise delay to reach the high level? What else should be noted? If the external frequency is 30Hz and the depth output is 30Hz, then the depth frequency should be set to 60Hz. If you want the color output to be 30Hz, should its frequency be set to 60 or 30？ @MartyG-RealSense |
|
My undertanding is that when a waveform is square, the time for the trigger pulse to reach the 'high' state is instantaneous, and the height of the trigger is determined by the voltage (1.8 V in this case), with the base of the trigger representing 0 V and the peak of the trigger representing 1.8 V. The width of the trigger (the length of time that it remains at the peak of 1.8 V before it falls back down to 0) is determined by the us value. When using an external trigger, the trigger width is recommended to be at least 100 us. When a master camera is generating the trigger pulse instead of an external device, the trigger width is just 50 us. Below is an image from the genlock external sync paper.
In the genlock hardware sync system, the FPS that the camera is set to should be 2x the trigger frequency. So if the trigger is 30 Hz, set the camera FPS to 60. |
|
I am in external mode and have five cameras (2 D455 and 3 D435i) participating in hardware synchronization. They all enable depth, while D455 also enables color sensors. I found that in external mode, one of the cameras lost one frame every five frames in depth, but the color was not lost. Has anyone else encountered this situation? But when this camera does not participate in hardware synchronization, its depth and color are normal, and there is no frame loss in depth. But only the depth of the D455 camera has intermittent frame loss. @MartyG-RealSense |
|
Whilst you can use different models of RealSense camera in a hardware sync setup, ideally they should all be the same model otherwise differences in the hardware design of the different models could cause issues with achieving sync. Inter Cam Sync Modes usually do not have color sync, only depth. To sync both RGB and color, only mode 3 (Full Slave) can be used. Slave modes 2 and 3 operate under different rules to mode 4+. The camera does not wait indefinitely for a trigger pulse, but instead only waits for a certain period of time on each frame when listening for a trigger pulse before giving up listening on that frame and triggering capture independently (unsyched). Also, mode 3 was intended for use only with the D415 camera model and even then it was unreliable. Intel eventually dropped support for genlock sync (though it remains accessible in the SDK), and modes 1 and 2 (Master and Slave) became the only officially supported Inter Cam Sync Modes. |
|
@MartyG-RealSense below is my log.Finally, the color is not updated, and the depth is constantly updated. sm = 252122300018 ;depth_device_time = 1318333894 ;color_device_time = 1318080334 ;depth_backend_time = 1700909059.980000color_current_time = 1700909059.727526 |
|
CallbackFinished,Depth,#40,@1700912620955.000732, callback duration: 2.043701 ms |
|
Genlock hardware sync mode does not have technical support provided by Intel for it any more, and so if someone chooses to use genlock then the Intel RealSense support team unfortunately cannot advise about new problems that arise from it that are not already referenced in previous cases. If color sync is not important to you then technical support can only be provided for an external sync problem if all of the cameras use mode 2 (Slave), the camera FPS is set to 30 and the trigger frequency is matched as closely as possible to the actual FPS of the camera (for example, 30.02 Hz) using an external generator device such as an oscilloscope. The white-paper guide for use of mode 2 is at the link below. https://dev.intelrealsense.com/docs/multiple-depth-cameras-configuration |
|
@MartyG-RealSense |
|
Yes, our team certainly can provide advice if it also occurs in Mode 2. |
|
Although my problem appears in the external model, it is very similar to the master slave mode, except that external does not generate data flow without triggering signals. Are you not helping me solve this problem just because I am in the external model mode? You don't even know the cause of the problem, and just because I am an external model, you don't help me solve it? |
|
External trigger sync is still supported but only when using Inter Cam Sync mode 1 and 2. Support for genlock sync was withdrawn by removing its white-paper document from the directory page of white papers so that it was no longer accessible. Mode 3 and above continues to be supported in the SDK for those who need it though. https://dev.intelrealsense.com/docs/whitepapers A PDF copy of the withdrawn paper is provided by me for those who need to use genlock because mode 1 and 2 cannot meet their project requirements, but on the understanding that genlock sync is no longer officially supported. If you can test in mode 2 yourself and confirm that color is still not updating correctly on that particular D455 then I will be pleased to seek further advice from the RealSense support team about your problem. |
|
Can you help me ask the realsensen team this question first? If they don't support me, I can continue to verify whether this problem exists in Model 2 mode |
|
I have contacted them with the details of your described problem. That you have a five camera setup of mixed D435i and D455 and are performing synced depth from all cameras, and unsynced color from the two D455. And with one of the two D455, the color stream stops updating but works normally when not part of the hardware sync setup. I will update you with their response as soon as I receive it, which should be during the coming week. Thank you for your patience. |
|
May I ask why the official does not support external mode? Is there any defect in this method with the camera? We want to understand the issue with using external on our end
发自我的iPhone
…------------------ Original ------------------
From: MartyG-RealSense ***@***.***>
Date: Sat,Nov 25,2023 8:42 PM
To: IntelRealSense/librealsense ***@***.***>
Cc: chenxiaocongAI ***@***.***>, Mention ***@***.***>
Subject: Re: [IntelRealSense/librealsense] Multiple D435i cameras aresynchronized in depth, can the master enable imu without damaging thesynchronization function (Issue #12072)
I have contacted them with the details of your described problem. That you have a five camera setup of mixed D435i and D455 and are performing synced depth from all cameras, and unsynced color from the two D455. And with one of the two D455, the color stream stops updating but works normally when not part of the hardware sync setup.
I will update you with their response as soon as I receive it, which should be during the coming week. Thank you for your patience.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
Hi @chenxiaocongAI My Intel RealSense colleagues have followed up on your case. They checked hardware sync with a scope and did .not find any probems with the hardware. They then found that the issue seems to be related to the RealSense Viewer software. According to their experiment, if you enable the RGB in a second Viewer then D455 and D435 can be synched correctly in the Viewer. So they ask that you do not enable RGB in the same Viewer as depth but instead try opening an additional Viewer for RGB streaming only. |
Before opening a new issue, we wanted to provide you with some useful suggestions (Click "Preview" above for a better view):
All users are welcomed to report bugs, ask questions, suggest or request enhancements and generally feel free to open new issue, even if they haven't followed any of the suggestions above :)
Issue Description
<Describe your issue / question / feature request / etc..>
I have five D435I cameras and have achieved depth synchronization between them. The implementation mainly involves using one as the master and the other as the slave. But when I enable the IMU function on the master, the depth synchronization of the five cameras mentioned above cannot be synchronized, and the master's frametime will deviate from the other four cameras. My question is: Can you enable the IMU function of the master while synchronizing five depth cameras
The text was updated successfully, but these errors were encountered: