High variance in observed FPS in multi-camera setup #11321
Comments
|
Hi @michaelcukier As the number of RealSense cameras that are simultaneously active on the same computer is increased, the processing burden on the computer of handling those multiple cameras will increase, necessitating a higher-specification processor. For example, the Up Squared single-board computer has a 1.8 Ghz processor and can handle two RealSense 400 Series cameras simultaneously. The AGX Orin has a 2.2 Ghz processor. For four simultaneously active cameras on the same computer, a processor equivalent to an Intel Core i7 or better is recommendable. It is possible that whilst your board can handle two cameras simultaneously at almost 30 FPS, three cameras is a processing tipping-point where performance begins to degrade. You could test whether performance improves if the FPS for all cameras is reduced from 30 to 15. |
|
Thanks but I am monitoring CPU usage and when running three cameras at the highest resolution and 30FPS, it never goes above 40%.
On 13 Jan 2023, at 15:19, MartyG-RealSense ***@***.***> wrote:
Hi @michaelcukier<https://github.com/michaelcukier> As the number of RealSense cameras that are simultaneously active on the same computer is increased, the processing burden on the computer of handling those multiple cameras will increase. For example, the Up Squared single-board computer has a 1.8 Ghz processors and can handle two RealSense 400 Series cameras simultaneously. The AGX Orin has a 2.2 Ghz processor.
It is possible that whilst your board can handle two cameras simultaneously at almost 30 FPS, three cameras is a processor load tipping-point where the performance of the cameras begins to degrade. You could test whether performance improves if the FPS for all cameras is reduced from 30 to 15.
—
Reply to this email directly, view it on GitHub<#11321 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AH6VJ7ID7PUMUSHMLYELZTLWSFW6FANCNFSM6AAAAAAT2PODFA>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
Are you capturing only depth frames, or both depth and color frames please? |
|
Both depth and color.
On 13 Jan 2023, at 17:25, MartyG-RealSense ***@***.***> wrote:
Are you capturing only depth frames, or both depth and color frames please?
—
Reply to this email directly, view it on GitHub<#11321 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AH6VJ7MHVWIOVXQKCQ43IFLWSGF2DANCNFSM6AAAAAAT2PODFA>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
If both depth and color are enabled then there may sometimes be a drop in FPS. A fix for this can be to have auto-exposure enabled and have an RGB option called Auto-Exposure Priority disabled. This causes the RealSense SDK to try to enforce a constant FPS for both depth and color streams. A simple Python code snippet for disabling Auto-Exposure Priority can be found at #5885 (comment) Edit: The D405 model does not have the Auto Exposure Priority setting, so that excludes that particular solution in this case. If you are using manual exposure values then there can be a drop in FPS if exposure goes into a certain range, as described at #1957 (comment) |
|
I'm not using manual exposure values. I still tried adding the code snippet you sent and I'm getting "RuntimeError: object doesn't support option #30".
On 13 Jan 2023, at 17:36, MartyG-RealSense ***@***.***> wrote:
If both depth and color are enabled then there may sometimes be a drop in FPS. A fix for this can be to have auto-exposure enabled and have an RGB option called Auto-Exposure Priority disabled. This causes the RealSense SDK to try to enforce a constant FPS for both depth and color streams. A simple Python code snippet for disabling Auto-Exposure Priority can be found at #5885 (comment)<#5885 (comment)>
If you are using manual exposure values then there can be a drop in FPS if exposure goes into a certain range, as described at #1957 (comment)<#1957 (comment)>
—
Reply to this email directly, view it on GitHub<#11321 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AH6VJ7P6IFI6KPVFVIBOQ33WSGHDFANCNFSM6AAAAAAT2PODFA>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
I added an update that the D405 model does not have the Auto Exposure Priority setting, so that excludes that particular solution in this case. As message edits do not go to your email, I will add additional thoughts to a new comment. :) |
|
A way to confirm whether the FPS is slowing due to both depth and color being enabled would be to test with only depth frames and see if the third camera still has the FPS drop. |
|
This is correct. I have just tried now running all 3 cameras with only depth enabled, and I'm getting exactly 30FPS for all three cameras, so no drop in FPS at all.
On 13 Jan 2023, at 17:45, MartyG-RealSense ***@***.***> wrote:
A way to confirm whether the FPS is slowing due to both depth and color being enabled would be to test with only depth frames and see if the third camera still has the FPS drop.
—
Reply to this email directly, view it on GitHub<#11321 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AH6VJ7IR6VHZCJO6F3YTYSLWSGIDRANCNFSM6AAAAAAT2PODFA>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
Thanks very much for the confirmation. If you need to have both depth and color enabled, please next test whether performance of the third camera improves if the FPS for all cameras is reduced from 30 to 15. |
|
At 15 FPS performance seems better. I ran 3 tests:
1st test: 15.0/15.0/15.0
2nd test: 13.5/13.9/15.0
3rd test: 15.0/15.0/15.0
However we do need 30 FPS for this application.
On 13 Jan 2023, at 18:08, MartyG-RealSense ***@***.***> wrote:
Thanks very much for the confirmation. If you need to have both depth and color enabled, please next test whether performance of the third camera improves if the FPS for all cameras is reduced from 30 to 15.
—
Reply to this email directly, view it on GitHub<#11321 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AH6VJ7NLGIPIM2HGYCB3M4LWSGKZPANCNFSM6AAAAAAT2PODFA>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
Are you able to reduce the resolution instead of the FPS in order to reduce the volume of data that needs to be processed? |
|
Reducing the resolution also improves things a lot, and we I get 30 FPS in my tests. But we need the highest resolution possible, 1280*720.
On 13 Jan 2023, at 18:43, MartyG-RealSense ***@***.***> wrote:
Are you able to reduce the resolution instead of the FPS in order to reduce the volume of data that needs to be processed?
—
Reply to this email directly, view it on GitHub<#11321 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AH6VJ7M7STWJWVBUEFCT3KDWSGO4RANCNFSM6AAAAAAT2PODFA>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
As you are using the color stream, potentially you could get some extra performance by building the librealsense SDK on your Jetson with CUDA support enabled if you have not done so already, as the SDK's CUDA support automatically offloads the processing of YUY to RGB color conversion from the Jetson's CPU to its Nvidia graphics GPU so the CPU does not have to do that color processing. |
michaelcukier
changed the title
|
Hi @michaelcukier Do you require further assistance with this case, please? Thanks! |
|
Hi @MartyG-RealSense Thank you. I compiled using CUDA support and the problem still persist. GPU usage however seems rather low (around 30%) when running three cameras. I just tried buying a PCIe USB 3.2 gen2 expansion card for my Jetson Orin and I connected a single D405 to it. It behaves strangely: the camera captures a couple frames and then freezes completely. Note that if I connect a USB webcam to the PCI it works fine. Any idea? |
|
Which Ubuntu version and JetPack version are you using with Orin, please? D405 has been able to be successfully used by RealSense users with Jetson Orin, Ubuntu 20.02 and JetPack 5.0.2, as described at #10416 |
|
I have the ones you have just mentioned. |
|
Thank you. Have you been able to test D405 successfully with other RealSense programs such as realsense-viewer? Also, official D405 support was introduced in librealsense version 2.51.1, so 2.51.1 or newer should be used with D405. |
|
Yes, with the realsense-viewer it seems to work fine. The librealsense I'm using is the latest one |
|
Does the camera still freeze if used with the basic Python multiple camera script at #1735 (comment) |
|
The camera freezes when used in single-camera and/or multi-camera setup. Again this is only when connected through a PCIe USB expansion card. |
|
It is difficult to advise about PCIe USB expansion cards as I have not used one for years, and it was on a desktop PC rather than a Jetson. On the USB expansion cards at that time though, they had an optional connector for linking the expansion board to the computer's power supply so that it could handle devices with high power draw instead of just low-draw devices such as keyboard and mouse. It would be worth checking whether your own card has some form of external power connector in case your expansion card is providing insufficient power to the RealSense camera for it to be able to operate. |
|
Ive just check and it seems the max supply for my PCIe card on the USB-A port is 5W. Is this enough for a single d405 ? |
|
You should budget 2W for each RealSense camera. So if the card supplies 5W shared between all devices attached to it then it should be sufficient for a single D405 so long as there are not additional devices attached to the card that are drawing power. |
|
Ok then we discard this problem, as in my tests the only device connected to the PCIe is a single D405. |
|
What happens if you use realsense-viewer with the camera attached to the PCIe card? |
|
It shows me the first frame and then freezes. I get a "no frame received" in red error after a bit. |
|
In a Jetson discussion about PCIe USB expansion cards at the link below, one commenter suggests adding pcie_aspm=off to the kernel command line in case the card has a problem with low power states. https://forums.developer.nvidia.com/t/third-party-pci-e-usb-3-0-expansion-card-for-xavier-board/ |
|
Hi @michaelcukier Did the advice in the above comment make a difference, please? Thanks! |
|
Hi @michaelcukier Do you require further assistance with this case, please? Thanks! |
|
Case closed due to no further comments received. |
Issue Description
I'm running three D405 using
pyrealsense2. I'm running a simple python code that turns on the three cameras and count how many frames were received, and I sometimes get 4 FPS on average for all three cameras, and some other times 29 FPS, even though the setup is exactly the same. I don't understand why.I tried running my code for a very long time (1 hour) twice, compared the results, and still got very different observed FPS. I am using the exact same three USB cables. I've also noticed that, very often, one camera has a significantly superior FPS compared to the two others.
What could be the reason(s) for this massive variance in FPS between each tests? Why is the observed FPS not stable at all? I could understand a difference of 1 or 2 FPS during each test, but a difference of 25 is really strange.
Note: when trying two or one D405, this problem seems to go away and the FPS is always at nearly 30FPS.
The text was updated successfully, but these errors were encountered: