Dirty editors cannot be saved after application recovers from "offline" mode

[kenneth-marut-work]

Bug Description:

If a backend process hogs resources and blocks frontend<->backend communication the application will go "offline" and eventually recover. During this time, the frontend can be slow to recognize that connection is interrupted as per #12194 and a user may continue to make edits to a file, putting it into a dirty state. When the application finally recovers from "offline" mode, it is impossible to save any new changes to the existing editor. The editor must be closed and reopened.

Steps to Reproduce:

1. Build this branch which adds a command to hang the backend for 3 minutes.
2. Start Theia for browser and open an editor (FileA.txt)
3. Run the command "hang backend"
4. Start making edits to FileA.txt and try to save them (they will fail because the backend is blocked, notice also that there is no indication that there is a failure)
5. Eventually (about 1 min) the status bar will turn yellow (ConnectionStatusService is slow to recognize "offline" when backend processes are consuming resources #12194), after 3 minutes the "hang backend" command will complete and the status bar will return to blue. The application should be interactive again
6. notice that you still cannot save your edits you made to FileA.txt

Additional Information

• Operating System:
• Linux
• Theia Version:
  Master

[colin-grant-work]

@tortmayr, this and #12194 seems in part to be a weakness in the RPC system. It seems that if the backend is 'busy' long enough, the websocket connections providing communication between front and backends will be closed, but when that happens, pending requests won't be rejected. In the case of editors open against files, when they try to save, a request is made to update the file resource, but eventually the channels that would inform the resource that its request had succeded are disposed of entirely, leaving the associated Promise hanging indefinitely. Although it's easy to trigger in the editor system, and it appears that the ConnectionStatusService is also subject to the same issue, it's likely that those aren't the only place RPC calls can hang even though we know that they can no longer succeed.

[tortmayr]

@colin-grant-work Good catch, you are right pending requests of RPC calls are currently not properly rejected when underlying channels is closed/disposed. I'm going to have a look at it

[tortmayr]

@colin-grant-work #12581 improves the RPC system. With this change pending requests are now properly rejected if the underlying channel is closed (i.e. the connection to the backend is lost)

[msujew]

@tortmayr The issue seems to persist (at least in 1.47), see #13702.