Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KM upgrade E2E test #2920

Merged
merged 1 commit into from
May 21, 2020
Merged

KM upgrade E2E test #2920

merged 1 commit into from
May 21, 2020

Conversation

ptrus
Copy link
Member

@ptrus ptrus commented May 18, 2020

Closes: #2517

TODO:

@ptrus ptrus changed the title w1p ptrus/feature/e2e-km-upgrade May 18, 2020
@ptrus ptrus changed the title ptrus/feature/e2e-km-upgrade KM upgrade E2E test May 18, 2020
@ptrus ptrus force-pushed the ptrus/feature/e2e-km-upgrade branch 11 times, most recently from 2f826af to e72e3fb Compare May 19, 2020 14:55
@codecov
Copy link

codecov bot commented May 19, 2020

Codecov Report

Merging #2920 into master will increase coverage by 0.35%.
The diff coverage is 78.57%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2920      +/-   ##
==========================================
+ Coverage   68.13%   68.48%   +0.35%     
==========================================
  Files         360      360              
  Lines       34934    34947      +13     
==========================================
+ Hits        23802    23935     +133     
+ Misses       8041     7933     -108     
+ Partials     3091     3079      -12     
Impacted Files Coverage Δ
go/worker/keymanager/worker.go 65.95% <76.92%> (+0.65%) ⬆️
go/worker/keymanager/init.go 70.21% <100.00%> (+0.64%) ⬆️
go/common/cbor/codec.go 77.14% <0.00%> (-5.72%) ⬇️
go/runtime/tagindexer/tagindexer.go 68.47% <0.00%> (-4.35%) ⬇️
go/worker/common/committee/group.go 79.27% <0.00%> (-1.32%) ⬇️
go/storage/mkvs/iterator.go 84.37% <0.00%> (-1.25%) ⬇️
go/runtime/transaction/transaction.go 75.75% <0.00%> (-1.22%) ⬇️
go/worker/compute/merge/committee/node.go 75.87% <0.00%> (-0.75%) ⬇️
go/runtime/client/client.go 72.99% <0.00%> (ø)
...o/consensus/tendermint/apps/staking/state/state.go 57.87% <0.00%> (+0.40%) ⬆️
... and 29 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6ccba9c...92b62f2. Read the comment docs.

@ptrus ptrus marked this pull request as ready for review May 19, 2020 16:23
// TODO: with the change that keymanager initially registers without
// status, before it is initialized, we should wait here that keymanager
// is initialized. Add a method to wait for KM initialization?
time.Sleep(10 * time.Second)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the "register as a keymanager on the first status update" fix is ok (see changelog fragment), i'll fix this properly by waiting for KM initialization, but didn't want to implement that before the fix is acked, as it might not be needed if we decide for a different fix (e.g. maybe just retrying the initialization)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't something be needed in either case with the "key manager always registers"? E.g., even if you retry, the retry could happen later than the key manager actually registering with correct ExtraInfo? So maybe we should do #2130 and then you could call WaitReady on the key manager node(s) to check if they are really ready?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah would be, but if i revert the "km always registers" and only replace it it with retrying the initialization no changes are needed here - so i think i'll do just that.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once we have #2130 we should change the code here to wait for ready, and we can also bring back the "km always registers" fix, since i think is generally a bit nicer than the current "hack" to only register after initial failure :-)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe mention this in #2130 (and add a comment in the code) so that we don't forget about it?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done, i'll mention on the issue once this is merged so i can permalink to code

go/oasis-test-runner/oasis/fixture.go Outdated Show resolved Hide resolved
go/oasis-test-runner/oasis/fixture.go Outdated Show resolved Hide resolved
go/worker/keymanager/worker.go Outdated Show resolved Hide resolved
// TODO: with the change that keymanager initially registers without
// status, before it is initialized, we should wait here that keymanager
// is initialized. Add a method to wait for KM initialization?
time.Sleep(10 * time.Second)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't something be needed in either case with the "key manager always registers"? E.g., even if you retry, the retry could happen later than the key manager actually registering with correct ExtraInfo? So maybe we should do #2130 and then you could call WaitReady on the key manager node(s) to check if they are really ready?

@ptrus ptrus force-pushed the ptrus/feature/e2e-km-upgrade branch 3 times, most recently from 98ecaeb to b8d456f Compare May 20, 2020 10:51

set
},
threshold: 9002,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At some point, we probably should get rid of the hack that sets the threshold to 2 when using test keys. Also either leave a comment here, or pick a different way to force the MRENCLAVE to change.

@ptrus ptrus force-pushed the ptrus/feature/e2e-km-upgrade branch from b8d456f to 40ca1f3 Compare May 20, 2020 11:44
go/worker/keymanager/worker.go Outdated Show resolved Hide resolved
go/worker/keymanager/worker.go Show resolved Hide resolved
@ptrus ptrus force-pushed the ptrus/feature/e2e-km-upgrade branch from 40ca1f3 to 1165a40 Compare May 20, 2020 15:14
.changelog/2517.bugfix.md Outdated Show resolved Hide resolved
go/oasis-test-runner/oasis/oasis.go Outdated Show resolved Hide resolved
go/oasis-test-runner/scenario/e2e/keymanager_upgrade.go Outdated Show resolved Hide resolved
go/oasis-test-runner/scenario/e2e/keymanager_upgrade.go Outdated Show resolved Hide resolved
go/oasis-test-runner/scenario/e2e/runtime.go Outdated Show resolved Hide resolved
go/worker/keymanager/worker.go Outdated Show resolved Hide resolved
@ptrus ptrus force-pushed the ptrus/feature/e2e-km-upgrade branch from 1165a40 to de2f992 Compare May 20, 2020 16:08
go/oasis-test-runner/oasis/oasis.go Outdated Show resolved Hide resolved
go/oasis-test-runner/oasis/oasis.go Outdated Show resolved Hide resolved
go/oasis-test-runner/oasis/oasis.go Outdated Show resolved Hide resolved
@ptrus ptrus force-pushed the ptrus/feature/e2e-km-upgrade branch 2 times, most recently from 4adae78 to c822ade Compare May 21, 2020 08:55
@ptrus ptrus force-pushed the ptrus/feature/e2e-km-upgrade branch from c822ade to 92b62f2 Compare May 21, 2020 08:58
@ptrus
Copy link
Member Author

ptrus commented May 21, 2020

thx for the reviews

@ptrus ptrus merged commit 3d5e7dc into master May 21, 2020
@ptrus ptrus deleted the ptrus/feature/e2e-km-upgrade branch May 21, 2020 09:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Tests for key manager upgrades
3 participants