Replay of buffered messages never happen

[kevin-bates]

There are currently two issues with respect to the replay of buffered messages - one on the server-side, the other client-side. This issue corresponds to the server-side issue.

When the WebSocket connection is closed while a kernel is still active (e.g., closing the active browser tab), the kernel manager rightly starts buffering messages. Upon detecting a new WebSocket connection, the handler checks if there are any buffered messages so that it can replay them. However, it never discovers the buffered messages because it enforces an additional check that the buffered messages found must correspond to the same 'session_key'. Since the session key is really a function of the Session class instance (the session key is composed of the kernel_id and the Session class instance's session variable - which is, by default, a randomly generated id), the correspondence between the current "session" and the buffered "session" never materializes. As a result, all of the buffered messages are discarded and never replayed.

I'm not sure why that particular aspect of the session was used as part of the session key, but using only kernel_id seems sufficient. Is there some collaboration aspect to this, although I'm not sure why we wouldn't want to replay messages for the kernel relative to kernel_id (one replay is better than zero).

As a result, I'd like to propose we drop the use of the session_key when handling buffered messages - otherwise no replays occur. If we wanted to use a broader-scoped 'key' (that spans WS connections), we need to remember that Kernel/Enterprise Gateway only performs kernel-management and their clients aren't required to manage sessions.

The client-side issue is that once replays are fixed (I did this temporarily), replayed messages are sent to the client, but never displayed. I believe this is because the java-script code somewhere is only displaying messages with parent-header:session matches and since that aspect of the replayed messages is what is different (btw, the header:session values are consistent across the WS connections), then those messages are ignored.

I'm hoping someone will read this and tell me that I'm misunderstanding things, but this is what I see happening right now.

[kevin-bates]

After debugging this more closely. The session key used for buffering is a function of the Session class instance associated with the handler ZMQChannelsHandler. There's also a session.session member hanging off the kernel manager instance. This string is persistent across connections which is what we want. I'll look into this as an approach to resolve this.

[tswsl1989]

I believe this is the issue I'm seeing as well, although admittedly this is through jupyterlab
Specifically, I close a tab containing the lab instance the notebook continues to run in the background (correctly), but when I open another tab to that instance, I never see the updated messages.

This also appears to be the case if the network connection drops - once the connection is restored, the output from the currently running cell does not get updated.

[kevin-bates]

When the connection is closed due to closing the browser tab, the buffering logic is triggered and buffered messages are discarded when the same running notebook is opened in a subsequent tab.

However, my experiences with the network connection drops is that the buffering logic is NOT triggered AND any messages not initially displayed as the result of the network drop DO get displayed upon the network's enablement. During this entire period, however, none of the buffered message logic is performed. This leads me to believe the 'buffering' is occurring at the ZMQ level.

Both of these scenarios are clear when debug logging is enabled. Please try your scenarios with --debug or --log-level=DEBUG enabled on the command line.

I've added an example of the buffered message output to #4110 .

[kevin-bates]

Please see the discussion in #4110. I am unable to trigger the replay, but I'm probably not doing the right thing to emulate network disruption. At any rate, Min was able to show the log statements - which resolved my angst 😃 on this issue.

Learning the intended behavior relative to connection sessions was a huge help, albeit not necessarily what most would expect, but understandable nonetheless.