"Start training" stops early, renderer disconnects

[chigozienri]

Describe the bug
"Start Training" cell runs for only a very short period of time, during which time the viewer shows only a small subset of the training images. After the "Start Training" cell has has finished running, the viewer shows "Renderer disconnected". This may be related to #2144 and #1832, although unlike the behaviour reported in those issues, there is some time where the viewer shows as connected.

To Reproduce
Environment: Google Colab free tier, t4 GPU, Python 3.10.10
Steps to reproduce the behavior:

1. Run cells up to "Downloading Data"
2. Upload custom images (in this case I had 39 frames)
3. Run cells up and including to "Start training"
4. Click "refresh page" on viewer, and see images start to appear
5. 15 seconds later, script exits, "renderer exits" on viewer
6. No errors appear, but no way to interact with images, no checkpoints generated

Expected behavior
Viewer to connect in a persistent way, nerfacto to start training and generating checkpoints

Screenshots

[tancik]

It sounds like you are being kicked off of the port. Is it possible there is a firewall that blocks large traffic spikes?

[sudoku-lord]

@tancik This looks like the same issue I'm getting on Google Colab (I'm assuming the poster is also using the Colab notebook). Have there been any updates on this bug? Thank you!

[chigozienri]

I am not seeing this issue any more, and I am not sure why. I will keep the issue open for now and update if it reoccurs.

I was indeed using the Colab notebook.

[tancik]

Might be related to this
#2328 (comment)

[aliabusaleh]

Is there any update on how to fix that?