SQL activity pages (sessions, transactions and statements) are empty during an active workload

[kevin-v-ngo]

CC dedicated clusters show no activity in the Statements, Sessions, or Transactions pages within the CC console. There’s only a red info icon with no text. This was during an active workload (TPCC).

Note these pages were working after initially creating the cluster. Screenshot:

[Azhng]

Spent some times investigation this. This is the result of backend not properly surfacing the error message to frontend.

Specifically in this case, the intrusion was using an expired kubenet cookie, which resulted in a error in the backend DB Console proxy. (This can be reproduced using a hacked version of intrusion). As the result, it simply returns a HTTP response with 502 error code without any error message. This causes the frontend to surface an empty error string.

However, our error handling is severely flawed here due to the API calls within componentDidUpdate() method. This triggers an infinite loop that essentially causes the entire webpage to crash.

PR here addressed that issue.

[kevin-v-ngo]

Thanks for the update Archer! Should there even be an error message surfaced? What's the expected behavior?

I would expect Statements/Transactions to still be populated in this view.

Also did you forget to link the PR?

[Azhng]

Should there even be an error message surfaced?

This should definitely be an error. Though I am expecting the error should be coming from Console API backend.

What's the expected behavior?

Well, ideally the expected behavior is there's never going to be an error :P . But in this case, it's hard to say. If either the Console -> Intrusion connection or Intrusion -> Cluster breaks, this would means something really wrong is happening in the cloud. Last few times this happened we both declared incidents.

I would expect Statements/Transactions to still be populated in this view.

If the connection borked, we won't be able to get data from the backend. I suppose we can still load the page with our cached stale data until the cache TTL expires. But once the cache TTL expires there's not much we can do. Also we won't be able to change selected time ranges or go into statement detail page, since they would need additional API call.

[Azhng]

Chatted with @maryliag offline. To summarize:

1. In normal case, we would like to poll the backend API at a fixed interval (5 minutes).
2. If we encounter an error, we would stop polling and display the error.

• when we display the error, we also want to display a "Retry" button along with the error message. One user click on the Retry button, we will retry the calling the backend API:
  • if the retry API call succeeds, we would resume the polling
  • else, we do nothing

@Annebirzin , do you mind create a design for the error box with a Retry button?

[kevin-v-ngo]

Thanks Archer. I was more wondering why we're consistently getting this (unexpected) error related cookie expiration and how we can make this more resilient.

So it seems like we'll be polling at a fixed interval which should reduce the likelihood of running into this error (expired cookie) but for edge cases, we need an error message and retry button. Correct?

[Azhng]

why we're consistently getting this (unexpected) error related cookie expiration

This was due to erroneous logic within the Intrusion, which was recently patched.

So it seems like we'll be polling at a fixed interval which should reduce the likelihood of running into this error (expired cookie) but for edge cases, we need an error message and retry button. Correct?

Hmm running at a fixed interval is not going to reduce the likelihood of running into this error, since it's something completely out of our control.

This is more of a technical issue of how we are managing out API calls. Currently, we would attempt to issue an API call as soon as anything on the page has changed (e.g. filter, search bar, sorting order). This attempt is aborted if we have already requested for data from the API within last cache TTL (5 minutes).

This means we would constantly attempt to issue API calls, keep getting aborted in a normal case. But in unhappy case, er would keep attempt to issue API calls, keep getting errors and retry them. This is extremely wasteful.

The new approach would be instead of running this loop of:

• attempt to call API
• gets aborted

We simply poll at a fixed interval, and pause the polling if an error is encountered.

[kevin-v-ngo]

Got it! Thanks for providing the context Archer.

[Annebirzin]

Here are error states for the statements and transactions pages, including a link to reload the page: https://www.figma.com/file/MoSPFkEyJfGipCmQvvLg1a/21.2_SQL-obsrv_query-performance?node-id=3524%3A74970

Not sure if we want to be more specific around the messaging cc @ianjevans

Also, does the same thing happen on the transaction page?

[Azhng]

Yes, same thing would also happen on the Transaction Page

[ianjevans]

I have a slight preference for "error" over "issue" on wording, and we can be more clear that the error isn't in the cluster but the console.
"This page had an unexpected error while loading transactions/statements. Reload this page."

[Annebirzin]

Sounds good, updated the designs to reflect those changes