Consistent out of memory errors when connecting multiple NT clients

[LightspeedLazer]

(Copied from this CD post)
Our team is currently working on building a robot for an offseason competition and we’ve run into several out of memory errors when testing code on the robot. I believe I’ve narrowed down the issue to the number of NT clients connected because of some tests that I’ve conducted. The issues started when we added a Limelight 2 (v2023.6.0) for Apriltag vision and connected it to the RoboRIO through a network switch. When we turned on the robot it would work fine until the point where I connected either AdvantageScope (v3.0.0-beta-5) or Shuffleboard (v2024.1.1-beta-3). The robot would continue running for about 2-4 seconds until it threw a memory error. It would then restart the robot code and then throw another memory error after ~2 seconds of running. I tried disconnecting the Limelight (and removing references to it in the code) and was able to connect two instances of both AS and SB to the robot without issue for several minutes. With AS connected I tried reconnecting the Limelight and once again it would error after a couple seconds. I retried this test with SB connected instead of AS and it had the same result. We had a Limelight 1 available so I tried connecting that alongside the Limelight 2 without connecting anything else to NT and yet again it threw a memory error.

I don’t really know how to solve this issue besides not connecting AS or SB to the robot but that’s a huge detriment to the software team and the drivers. I know that other teams are able to run a lot more cameras than we are and still connect AS and SB to their robot and be perfectly fine, so I’m very curious as to what is the actual issue with our robot. One of our programmers also brought up the fact that the Limelight 2 connects with NT3 instead of NT4 which might be an issue if NT for some reason doesn’t like using both NT3 and NT4 at the same time. I’ll have more info on the memory errors later today when I recreate the issue and record what gets printed to the console.

Here’s our robot code repo if somehow we’re doing something very wrong in our code that’s causing the memory errors. We’re using WPILib v2024.1.1-beta-3.

[PeterJohnson]

Are you rebooting the Rio between the test runs?

[LightspeedLazer]

My apologies for not responding until now, we just had the aforementioned competition and I just got back
No the rio was not rebooted between test runs but I can rerun the tests again rebooting each time
I can almost guarantee that it will have the same result but if you think it might do something different then I can rerun the tests

[LightspeedLazer]

I decided to run some tests without the Limelight but with some OrangePis running PhotonVision (which is an NT4 client as opposed to the Limelights which are NT3). Unfortunately it still has memory errors, but they're a lot more inconsistent than the Limelight errors. Sometimes it would error within seconds of booting, and other times it wouldn't error for several minutes. Another inconsistency is that some of the errors are std::bad_alloc errors instead of os::commit_memory errors. Shuffleboard would also constantly disconnect and reconnect because of an incomplete fragment. Unfortunately the logs don't really reveal anything besides the constant NT disconnects and reconnects, but if you want them I have them downloaded.

[PeterJohnson]

Several people have reported the incomplete fragment issue but I’ve not been able to isolate the cause yet. I’m planning on adding more logging to help isolate the problem, but in the meantime getting a wireshark capture of the network traffic from the desktop to the Rio would help.

Regarding the out of memory errors, one change we made was to change the default garbage collector. It might be worth trying to change it back for testing purposes; instructions for modifying build.gradle to do this are here: https://github.com/wpilibsuite/GradleRIO#using-alternate-garbage-collector