Create delta callbacks, control execution order of added, updated, pruned #749

kinow · 2021-09-02T06:05:03Z

These changes close #747

Used the workflow provided by @hjoliver to reproduce and fix the issue. See screenshot of the UI after fixing the issue, no browser console, no alerts 🙆‍♂️

Requirements check-list

I have read CONTRIBUTING.md and added my name as a Code Contributor.
Contains logically grouped changes (else tidy your branch by rebase).
Does not contain off-topic changes (use other PRs for other changes).
Appropriate tests are included (unit and/or functional).
Appropriate change log entry included.
No documentation update required.

kinow · 2021-09-03T03:03:56Z

I finished the core changes, pending to fix the tests. But before that I will think about it over the weekend, and see if there are other ways to solve the #747 bug.

We have 2 callbacks for the Tree view. The first callback populates a workflow lookup, used by Tree view (and Graph view, and Table view, in the future). The second callback creates the workflow tree structure, where the nodes of the tree contain pointers to the workflow lookup created by the first callback.

The #747 bug happens when we have processed the first callback, applying some pruned deltas that removed nodes from the global lookup. And then the second callback has the same nodes in the updated delta, but now it fails to locate the nodes that have been just removed.

This pull request splits the execution of deltas, creating a common interface for each delta callback, where we have, in order:

before()
onAdded()
onUpdated()
onPruned()
after()
commit()
tearDown()

For each delta, then, we have in pseudo-code:

for (callback of callbacks) {
  callback.before(deltasData)
  callback.onAdded(deltasData)
  callback.onUpdated(deltasData)
  callback.commit()
}
for (callback of [...callbacks].reverse()) {
  callback.onPruned()
  callback.commit()
}

And when the component is destroyed, tearDown is called to clear the Vuex store.

The second for is fixing this bug, by applying pruned deltas in reverse order… we add & update everything, everywhere. Then we can safely remove the pruned data. This way we always apply every delta, with no risk of trying to access data that was already removed.

codecov-commenter · 2021-09-03T03:04:36Z

Codecov Report

Merging #749 (317c280) into master (a555e92) will increase coverage by 0.03%.
The diff coverage is 95.68%.

❗ Current head 317c280 differs from pull request most recent head c71ed5d. Consider uploading reports for the commit c71ed5d to get more accurate results

@@            Coverage Diff             @@
##           master     #749      +/-   ##
==========================================
+ Coverage   90.98%   91.01%   +0.03%     
==========================================
  Files          84       87       +3     
  Lines        1675     1703      +28     
  Branches      105      105              
==========================================
+ Hits         1524     1550      +26     
- Misses        121      123       +2     
  Partials       30       30

Flag	Coverage Δ
e2e	`79.15% <81.29%> (+1.30%)`	⬆️
unittests	`78.07% <55.39%> (-3.59%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/components/cylc/gscan/GScan.vue	`90.62% <ø> (ø)`
src/views/Dashboard.vue	`100.00% <ø> (ø)`
src/views/Tree.vue	`100.00% <ø> (ø)`
src/components/cylc/gscan/deltas.js	`83.33% <82.35%> (-16.67%)`	⬇️
src/services/workflow.service.js	`93.13% <93.10%> (-0.99%)`	⬇️
src/components/cylc/common/deltas.js	`96.42% <96.42%> (ø)`
src/components/cylc/common/callbacks.js	`100.00% <100.00%> (ø)`
src/components/cylc/gscan/callbacks.js	`100.00% <100.00%> (ø)`
src/components/cylc/tree/callbacks.js	`100.00% <100.00%> (ø)`
src/components/cylc/tree/deltas.js	`100.00% <100.00%> (ø)`
... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 27b8c33...c71ed5d. Read the comment docs.

kinow · 2021-09-05T22:46:58Z

Compilation errors fixed, now just need to fix these 32 unit tests that are broken... errr.. ☕ 😵

hjoliver · 2021-09-05T23:05:07Z

Only 32! try messing with the core of cylc-flow ... 😭

kinow · 2021-09-06T01:05:41Z

src/services/workflow.service.js

+      }
+      this.mergeQueries(baseSubscriber.query.query, subscriber.query.query)
+      subscription.callbacks = union(subscription.callbacks, subscriber.query.callbacks)
+    }


for-loops are easier to debug 😬

kinow · 2021-09-06T01:06:49Z

src/views/Dashboard.vue

-        []
+        [
+          new GScanCallback()
+        ]


I think this is clearer. Before it was using the same Vuex action that is used by GScan. But now it's even clearer that for now the Dashboard and WorkflowsTable are re-using the query & callbacks of GScan.

kinow · 2021-09-06T01:07:44Z

src/services/workflow.service.js

+        throw new Error('Error recomputing subscription: Query variables do not match.')
+      }
+      this.mergeQueries(baseSubscriber.query.query, subscriber.query.query)
+      subscription.callbacks = union(subscription.callbacks, subscriber.query.callbacks)


Note to self: manual test, and confirm this is working OK.

Ha! I knew it. Quickly wrote a unit test to verify it.

Let me fix it, and make the test permanent.

kinow · 2021-09-06T01:17:06Z

Just pending some manual testing, review if I didn't miss any parts of the code to be tested, and then it should be ready for review 😪 but will probably leave the manual test for tomorrow morning.

kinow · 2021-09-06T23:14:27Z

Tested locally with some unit tests, found one bug, fixed. Tested locally again with cylc hub and 4 workflows, no issues found 🎉 Had 4 workflows running, with and w/o hierarchy, and with and w/o families. I believe I found a bug, but in cylc-flow (will confirm & report later).

Ready for review! 🎉

kinow · 2021-09-10T06:15:34Z

Rebased fixing conflicts.

hjoliver · 2021-09-20T08:28:57Z

(Testing this now, next major review priority)

hjoliver · 2021-09-20T08:42:40Z

Hmm, opening a second tree view on this branch causes the tree to freeze 🤔

hjoliver · 2021-09-20T08:44:01Z

(although that doesn't seem to be the case in your screen recording above!)

kinow · 2021-09-20T09:20:58Z

I've been keeping this PR up to date with master, maybe something changed and caused the freeze. Will try to reproduce in a bit (or in the morning, on a Windows computer trying to get a nvidia computer board to boot 🤓 )

kinow · 2021-09-20T09:23:34Z

But if you are investigating it, what I would look at are

anything in the browser console?
does it matter if you try on a different browser? (I have ubuntu-lts firefox I use for everyday tasks & a chromium browser that I use only for cylc ui dev)
or with a different workflow?
any chance there's cache?
nothing in the UIS or flow logs?

Then I'd start looking at vue dev utils to see what's the state of the Tree view (literally, look for the Tree component/view, which must contain a TreeItem which is the top of the tree). And also look at the Network tab to confirm the subscription is going OK. Then from there it'd be the IDE to see what could be wrong with the code.

kinow · 2021-09-20T21:15:10Z

Working OK for me @hjoliver

checked out branch (double-checked it was up to date)
yarn install
yarn run build:watch
Started UIS & five

(the quick blank state is due to the subscription being reloaded; I'll check why that's happening)

hjoliver · 2021-09-20T21:50:30Z

Looks good. Does it carry on to the next cycle points though, with multiple tabs open?

kinow · 2021-09-20T21:57:48Z

Looks good. Does it carry on to the next cycle points though, with multiple tabs open?

I think so. Two trees, CP 180.

Some seconds later, CP 182.

I found why it's restarting the subscription, but that's a bug in master too. Will create an issue for that one and start preparing a fix and an e2e test to prevent that regression 👍

kinow · 2021-09-20T21:58:29Z

If you would like I can try with the same workflow you are using and see if it works?

hjoliver · 2021-09-20T21:59:24Z

Huh, strange (but good). I've tried with several workflows. Let me just nuke everything and start from scratch again...

kinow · 2021-09-20T22:05:01Z

In case it helps, heres five

[scheduling]
  cycling mode = integer
  initial cycle point = 1
  [[queues]]
     [[[default]]]
       limit = 1
  [[graph]]
    R1 = "prep => foo"
    P1 = "foo[-P1] => foo => bar"

[runtime]
  [[root]]
      script="sleep 5"
  [[prep]]
  [[foo]]
  [[bar]]

kinow · 2021-09-20T22:41:43Z

@hjoliver shared his screen showing what this PR looks like.

yarn run build
start UIS + five
add a second Tree View

Now the tree views are both frozen, not being updated. The network tab shows 2 GraphQL subscriptions, 1 being GScan as always, and 3 the tree views. Normally the ID for the tree view would be 2, so I guess the subscription was reloaded but then the UI got confused somehow.

We tested on Chrome and Firefox on Linux.

hjoliver · 2021-09-21T02:30:14Z

(Result: the bug was only triggered in production builds, something to do with lodash's unionBy())

hjoliver · 2021-09-21T02:37:08Z

I can confirm the latest commit fixes it 👍

kinow · 2021-09-21T02:53:27Z

@hjoliver sent me a workflow that shows errors due to missing parents. It happens when a task cannot find the parent family proxy, and also when a job cannot find the parent task.

I've created a gist with the deltas collected, the diff used to fire the debugger and to log more info, and also to remove GScan to have one less subscription to filter: https://gist.github.com/kinow/817205bbbef7684722fce5cce1939764

One parent task only appears as updated, skipping the added status. But I'm still investigating it that's the cause of the errors with that workflow.

added and updated deltas for each callback, commit. Then the pruned in reverse order, commit.

kinow · 2021-10-03T23:32:40Z

(rebased)

oliver-sanders

I'm still investigating

Not sure what the status of this is.

The code looks good, I ran some local tests and everything worked fine.

kinow · 2021-10-05T08:18:52Z

Not sure what the status of this is.

Just pending review.

The code looks good, I ran some local tests and everything worked fine.

Thanks! @hjoliver or @dwsutherland can sign it off then (I think they were the first ones to report the issue)

hjoliver

👍

kinow added this to the 0.6.0 milestone Sep 2, 2021

kinow self-assigned this Sep 2, 2021

kinow force-pushed the vuex-updates-order branch from 960d8cf to 70296b4 Compare September 3, 2021 02:54

kinow changed the title ~~WIP executing mutations in different order~~ Create delta callbacks, control execution order of added, updated, pruned Sep 3, 2021

kinow commented Sep 6, 2021

View reviewed changes

kinow force-pushed the vuex-updates-order branch from 8788df1 to c1022fa Compare September 6, 2021 21:14

kinow marked this pull request as ready for review September 6, 2021 23:14

kinow requested review from oliver-sanders and hjoliver September 6, 2021 23:16

kinow force-pushed the vuex-updates-order branch from 72d38a1 to b76ac13 Compare September 10, 2021 06:14

kinow force-pushed the vuex-updates-order branch 2 times, most recently from 98460b6 to e5c3f95 Compare September 19, 2021 22:39

kinow mentioned this pull request Sep 20, 2021

Subscription being reloaded unnecessarily #778

Closed

kinow added 4 commits October 4, 2021 12:32

Executing callbacks in different order. First execute all the

73d9e21

added and updated deltas for each callback, commit. Then the pruned in reverse order, commit.

Fix compilation problems and unit tests

0c731cc

Prevent adding duplicate callbacks

5330dcd

Replace unionBy by for-loop

c71ed5d

kinow force-pushed the vuex-updates-order branch from 317c280 to c71ed5d Compare October 3, 2021 23:32

oliver-sanders approved these changes Oct 5, 2021

View reviewed changes

hjoliver approved these changes Oct 13, 2021

View reviewed changes

hjoliver merged commit a3005b0 into cylc:master Oct 13, 2021

kinow deleted the vuex-updates-order branch October 13, 2021 01:10

hjoliver mentioned this pull request Dec 7, 2021

2021 Cylc Meetings cylc/cylc-admin#139

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create delta callbacks, control execution order of added, updated, pruned #749

Create delta callbacks, control execution order of added, updated, pruned #749

kinow commented Sep 2, 2021 •

edited

Loading

kinow commented Sep 3, 2021

codecov-commenter commented Sep 3, 2021 •

edited

Loading

kinow commented Sep 5, 2021

hjoliver commented Sep 5, 2021

kinow Sep 6, 2021

kinow Sep 6, 2021

kinow Sep 6, 2021

kinow Sep 6, 2021

kinow commented Sep 6, 2021

kinow commented Sep 6, 2021

kinow commented Sep 10, 2021 •

edited

Loading

hjoliver commented Sep 20, 2021

hjoliver commented Sep 20, 2021

hjoliver commented Sep 20, 2021

kinow commented Sep 20, 2021

kinow commented Sep 20, 2021 •

edited

Loading

kinow commented Sep 20, 2021 •

edited

Loading

hjoliver commented Sep 20, 2021

kinow commented Sep 20, 2021

kinow commented Sep 20, 2021

hjoliver commented Sep 20, 2021

kinow commented Sep 20, 2021

kinow commented Sep 20, 2021

hjoliver commented Sep 21, 2021 •

edited

Loading

hjoliver commented Sep 21, 2021

kinow commented Sep 21, 2021

kinow commented Oct 3, 2021

oliver-sanders left a comment

kinow commented Oct 5, 2021

hjoliver left a comment

Create delta callbacks, control execution order of added, updated, pruned #749

Create delta callbacks, control execution order of added, updated, pruned #749

Conversation

kinow commented Sep 2, 2021 • edited Loading

kinow commented Sep 3, 2021

codecov-commenter commented Sep 3, 2021 • edited Loading

Codecov Report

kinow commented Sep 5, 2021

hjoliver commented Sep 5, 2021

kinow Sep 6, 2021

Choose a reason for hiding this comment

kinow Sep 6, 2021

Choose a reason for hiding this comment

kinow Sep 6, 2021

Choose a reason for hiding this comment

kinow Sep 6, 2021

Choose a reason for hiding this comment

kinow commented Sep 6, 2021

kinow commented Sep 6, 2021

kinow commented Sep 10, 2021 • edited Loading

hjoliver commented Sep 20, 2021

hjoliver commented Sep 20, 2021

hjoliver commented Sep 20, 2021

kinow commented Sep 20, 2021

kinow commented Sep 20, 2021 • edited Loading

kinow commented Sep 20, 2021 • edited Loading

hjoliver commented Sep 20, 2021

kinow commented Sep 20, 2021

kinow commented Sep 20, 2021

hjoliver commented Sep 20, 2021

kinow commented Sep 20, 2021

kinow commented Sep 20, 2021

hjoliver commented Sep 21, 2021 • edited Loading

hjoliver commented Sep 21, 2021

kinow commented Sep 21, 2021

kinow commented Oct 3, 2021

oliver-sanders left a comment

Choose a reason for hiding this comment

kinow commented Oct 5, 2021

hjoliver left a comment

Choose a reason for hiding this comment

kinow commented Sep 2, 2021 •

edited

Loading

codecov-commenter commented Sep 3, 2021 •

edited

Loading

kinow commented Sep 10, 2021 •

edited

Loading

kinow commented Sep 20, 2021 •

edited

Loading

kinow commented Sep 20, 2021 •

edited

Loading

hjoliver commented Sep 21, 2021 •

edited

Loading