Avoid sudden dataset reordering in dashboard

[philippotto]

This PR fixes #4461 in a rudimentary way. I thought quite a while about this and I think this solution is fine. In the most common use case, the user will open the dashboard and it will simply work (be fast and won't show any sudden changes). If the user wants to open a new dataset (which was just added), they will likely click the "refresh" button, anyway, and then the new dataset will appear. There is the chance that a "soft reload" (instead of having the back-end rescan all directories) would also do the trick, but integrating this into the UI is likely quite confusing. I briefly thought about updating the dataset list with the updated cache content when the mouse is at a good position (e.g., when it hovers over the fresh button), but in the end this is probably too weird.

URL of deployed dev instance (used for testing):

• https://___.webknossos.xyz

Steps to test:

• it's a bit hard to test which is why I monkey-patched the getDatasets api to drop (or not drop) some elements between page refreshs
• during that, I could verify that a change in the dataset list "arrived" on the second page refresh (since a page load will always show the cache contents which get updated when loading the dataset)

Issues:

• contributes to Dataset list reorders when refresh request completes, leads to misclicks #4461

---

• Updated (unreleased) changelog
• Updated (unreleased) migration guide if applicable
• Updated documentation if applicable
• Adapted wk-connect if datastore API changes
• Needs datastore update after deployment
• Ready for review

[daniel-wer]

Great, this works well! I was able to test it fairly easily, as opening another than the first dataset moves it to the very top of the dataset list (due to the LRU sorting), which now only happens after a reload as intended. :)

I still like the possible solution from the issue

Show an explicit update indicator: We could tell the user that an async update is in progress and once it is finished, the indicator could turn into a button saying "Show 5 new datasets available". Similar to this picture: https://i.stack.imgur.com/cWkTc.png

(The image is a nice find btw, alram 😁)

So maybe we can keep a variation of this issue open as a possible future improvement :)

[philippotto]

I still like the possible solution from the issue

Show an explicit update indicator: We could tell the user that an async update is in progress and once it is finished, the indicator could turn into a button saying "Show 5 new datasets available". Similar to this picture: https://i.stack.imgur.com/cWkTc.png

(The image is a nice find btw, alram )

So maybe we can keep a variation of this issue open as a possible future improvement :)

Yeah, I agree! The problem which I have with this, though, is that it's difficult to communicate to the user that there's still a chance to do a "hard" refresh. So, if the indicator says "Show 2 new datasets" and then these datasets don't contain the expected dataset, the user might think that there is no use in clicking "refresh datasets" (because he just got an update one second ago). Instead, they might try to debug the directory on disk or something like that.
This problem is likely solvable, but I couldn't come up with an elegant solution, yet. Maybe a "Try harder" button for finding new datasets 😆 But honestly, we probably have to make the different "refresh" strategies obvious in the UI.

I'll cross-post this in the original issue leave that open, too.