Strange state: can't restart robot code and riolog pane in VSCode not working

[chauser]

I'm going to start this as a discussion since it isn't reproducible, but it may be helpful if someone else sees something similar occur.

I unplugged a camera (Microsoft HD-3000) from a RoboRio running the NI 2024 Beta 2 image while I had robot code running and streaming the camera, and then plugged the camera back in. The Restart Robot Code button on the DS (Also NI 2024 Beta 2) would then do nothing and the RioLog pane in VS Code appeared frozen. Usually, pushing the Restart Robot Code produces a message in the RioLog about running a kill script. The camera stream continued working after plugging in the camera. At some point the Disconnect button in the RioLog turned red.

I was able to deploy (from VS Code) and the usual startup messages appeared in the DS output view. The RioLog pane remained frozen. Closing the RioLog pane and deploying again did not result in any change. Closing and reopening VS Code and deploying again did not produce any change.

My notes don't say anything about having rebooted the RoboRio from the DS but I did reboot it with the Reset button on the RoboRio. That cleared the issue.

I have a vague recollection of having encountered something similar before but have no notes about it. If you see something similar please comment with as many details as you can. Similar failure of "Restart Robot Code" was mentioned in #39 without much detail in the context of memory (actually process) leaks arising from deployment.

[TylerSeiford]

I've run into similar intermittent issues on the beta 1 image with the riolog freezing up and unable to restart robot code. I've only spent ~20 minutes on the beta 2 image but didn't see this issue come up yet, but it sounds like it's still around.

Are there any useful debug steps we can do when this happens to gather more info?

[chauser]

After more time to think about this, one important thing to check is if the enable/disable buttons on the DS work when in this state. Being unable to disable a running robot would be a really bad thing.
Other things I plan to do if I see it again: try to ssh into the rio and see if there are any indications of out-of-memory (if so you probably won't be able to start an ssh session but the result of trying would useful). If an ssh session is already open it may be possible to gather more information. The state of the NetComm process on the rio is probably very relevant but I don't know how to capture it.

[chauser]

It happened again -- and it was again when I unplugged a USB camera and plugged another one in. I confirmed that the DS robot enable/disable buttons still worked. And that the button to reboot the Rio worked. Before rebooting I looked at the memory and cpu use using top and observed nothing terribly unusual. The cpu utilization of the (java) robot process was maybe a bit higher than usual (28% vs what I recall as a more normal 14% according to top) but nothing stood out. Given that the other DS buttons still worked I am less suspicious of a major problem with the NetComm Daemon.