Support KVS no merge flag #982
Conversation
Yes, please. |
|
I got lost in #813, so sorry if this should be obvious, but is the |
|
This is cool! I haven't had a chance to review in detail but my first thought is to go with integer flags in both API and wire protocol, and to avoid optional JSON fields (personal pref, but it tends to make encode/decode slightly more complicated when they are optional) I second @lipari's comment - what we have done in the past is to submit a sched PR with an "apply after flux-core pr number whatever is merged" note. |
Sorry, #813 was kind of all over the place and I got lost at one point too. I think the answer is no. This seems to me to be a useful way to allow the commit merging optimization to remain for the common case, but provide a way to disable it on a commit basis for special cases, e.g. the "state machine tracking" case that came up before. |
Sorry, another dumb question: Just to make sure I understand, will this work for multiple handles doing kvs commits, e.g. if I want to disable commit merging within my handle (for a specific commit), but in the meantime some other actor within an instance calls I'd feel better if there was a test case for that if possible |
The flag should protect your commit from being merged with any other commit in the master (regardless of where the other commits originate). |
|
Great. Thanks for the clarification. |
|
@garlick Ahh, I now see that you used an integer "flags" in the wire protocol for kvs get. I'll follow the pattern used there. |
Add a flags argument to kvs_commit, kvs_commit_begin, kvs_fence, and kvs_fence_begin functions. Fix ABI across all of remaining flux-core.
Add commit/fence flags internally to wire protocol and internal fence data structures.
Update tests for flags arguments in kvs wire protocol.
Support kvs flag KVS_NO_MERGE in commit and fence functions to inform KVS to not merge commits with other ones when commits are finalized. Fixes flux-framework#813
Add additional KVS_NO_MERGE test in kvs/commitmerge. If KVS commit-merge=1 module option is set, and all kvs_commit() calls set the KVS_NO_MERGE flag, the behavior should be similar to when commit-merge=0.
Add additional KVS_NO_MERGE test in kvs/commit tests.
Add additional KVS_NO_MERGE tests in kvs/commit test by mixing both mergeable and non-mergeable commits.
chu11
force-pushed
the
issue813-kvsnomergeflag
branch
from
fa3d15d to
83706c0
Compare
Codecov Report
@@ Coverage Diff @@
## master #982 +/- ##
==========================================
+ Coverage 75.92% 75.93% +<.01%
==========================================
Files 152 152
Lines 25936 25946 +10
==========================================
+ Hits 19692 19702 +10
Misses 6244 6244
Continue to review full report at Codecov.
|
|
Round 2 of this PR. Didn't take as long to cleanup as I anticipated. Basically it's what I did before, but cleaned up w/ appropriate tinier commits. Instead of using a boolean in the wire protocol to indicate "no merge", I put in an int flag. The int flag got carried into appropriate data structures. Didn't support "flags" in lua. Noticed flags weren't supported in I'm not a fan of the flag name "KVS_NO_MERGE", but couldn't really think of anything better. Any better ideas? I didn't add the ability to "no merge" w/ the |
|
This looks great @chu11! Nice work. I don't really have a better suggestion on the flag name. Also, I think it's probably fine to leave the flag off the lua bindings for now. We still have work to do on the KVS API and @grondo has been holding off until that settles down to move the lua bindings forward. I gather you covered the test @grondo suggested with your "mixed" mode test above? |
|
@garlick I didn't add that specific test, as I felt it was more or less handled by the last test case I added in: In this one I add a special "modulo" option to the |
|
@chu11: that test seems great to me! Also, sorry about the state of the lua binding! |
|
Thanks Al! Merging. |
Here's an initial try at issue #813. Note that I duplicated a bunch of functions by appending
_flagsto the name (e.g.kvs_commit_flags()). I didn't want to break ABI for the time being. Will convert appropriately later on. In hindsight, I think I should have two commits, one introducing a "flags", then another introducing the flagKVS_NO_MERGE. Will redo that when I cleanup.I elected to the make the "mergeable" boolean in each fence wire packet optional. I can sort of go either way on this. It could be made a mandatory boolean in each packet?
Probably need some additional tests too, minimally to test flags on the fence functions.
Wasn't sure how to go about the ABI breakage. Obviously I can fix everything in flux-core, but should I just send PR to flux-sched as well at the appropriate time?