Add v0 of versioning to AAE hashtree locks, upgrades, and object hashing #1473

bsparrow435 · 2016-09-02T13:22:33Z

This PR adds version 0 of AAE hashing to riak_kv_entropy_manager, riak_kv_index_hashtree, riak_object, and the riak_kv_vnode. Specifically, we've added logic which requires getting a hashtree lock to provide an appropriate version and then our hashtree version determines what version of riak_object:hash will be used for the incoming data.

Upgrading is completed automatically for the user. The upgrade uses a combination of riak_core_capability with a new riak_kv variable object_hash_version and logic within the riak_kv_entropy manager to intelligently upgrade hashtrees once all conditions have been met. THe upgrade flow is as follows:

User upgrade all nodes to 2.2.
Each node locally starts checking to begin upgrade in riak_kv_entropy_manager within the management tick.
IIF all partitions are built and have exchanged with all sibling partitions, we'll mark the riak_kv_entropy_manager state with a new pending_version.(need to exchange all siblings to catch any pre 2.1 data potentially suffering from kv679).
Each poke on a riak_kv_index_hashtree check the pending version of the manager and if it's set, calls the vnode to upgrade it's local tree.
Vnode destroys and makes a new hashtree with the upgrade atom included in the start opts.
Once all upgraded hashtrees have registered with entropy_manager, flip our entropy_manager version to 0 to indicate this node has completed the upgrade for all local nodes.

Open areas for discussion:

How best to handle default case of riak_object:hash without provided version(yz hashing). Current logic flips the hash when the entropy_manager notes all local trees have upgraded.
hashtree:destroy only removes data within the partition dir(i.e. ./data/antie_entropy/PARTID/*). The partition ID's still need to be cleaned up out of the anti-entropy directory.

…atch pre 2.1 vclocks which require a rewrite to be safe Added riak_kv boolean environment variable `hash_only_vclock` which enables/disabled the vclock hashing logic. Additionally, when enabled, we will no longer do the expensive compare/logging on read-repair to catch old pre2.1 vclocks who require a rewrite. Additional logic must be added to `yz_kv.erl` so the hashes in the entropy_data element of the Solr documents match what is stored in KV AAE. Upgrading the hash logic requires clearing all YZ documents and re-indexing everything. This is because we store the obj hash in the Solr document. This will hopefully change in the future to allow for easier AAE changes.

Add a static version “v0” to the data_root of the riak_kv AAE directory. This allows us to isolate old/newer trees as well as support downgrades which will not allow old and new trees to exchange either locally or across MDC.

New error `bad_version` added to the return from `riak_kv_index_hashtree:do_get_lock` if the requested version and the local version of the hash tree do not match. Requires changes in yokozuna and riak_repl to fully function.

Change boolean configuration variable to integer version starting with 0. Change all specs from atom() to non_neg_integer() for version number. Add extra logging for invalid non-integer hash version configurations.

Was not correctly handling case where Result list was empty of valid object responses or only a single valid response.

Add version 0 of object hashing(config `object_hash_version`) which hashes only the vclock to represent a version of an object. Also add versioning to index_hashtree get_lock API which now takes a Version parameter. The absence of a version is assumed to be `undefined` and will only be allowed to get a lock if the hash on disk is in legacy format. Upgraded hash trees are located within the anti_entropy data_dir under the directory v0. Currently the only supported version is 0 but the code allows for support of increasing integer versions. Add verbose riak_object:equals check to get_core for all read operations who do not trigger read repair. This logic is to detect potentially pre-2.1 data which may be impacted by old vector clock bugs and should be rewritten before upgrading the hash version for safety.

WIP commit in case of computer failure or something like that.

…l recovering some history

Also, add some testing logs and debug functions.

nickelization · 2016-09-02T19:52:56Z

src/riak_kv_entropy_manager.erl

@@ -116,16 +124,27 @@ release_lock(Pid) ->
 %%      will try to acquire a concurrency lock. If successsful, the request is
 %%      then forwarded to the relevant index_hashtree to acquire a tree lock.
 %%      If both locks are acquired, the pid of the remote index_hashtree is
-%%      returned.
+%%      returned. This function assumes an undefined version of the hashtree


Instead of stating what the function does, I feel like this comment would be more helpful if it explained why it does it. Something like "this is left over for compatibility in mixed clusters with pre-2.2 nodes, from before we added versioning to the hashtree."

Will change.

nickelization · 2016-09-02T20:53:28Z

src/riak_kv_get_core.erl

+            case [RObj || {_Idx, {ok, RObj}} <- Results] of
+                [] ->
+                    ok;
+                [_|[]] ->


Very small nitpick, but the pattern I normally see to match a singleton list is just [_]. I'm sure this works just as well, but feels a little bit unidiomatic.

Additionally, cleaned up the anode code which killed the hash tree on upgrade. Removed one binary layer from the riak_object:hash and added the build locks(different from concurrency lock(doh)) to the upgrade procedure so we won’t kill trees and then have them sit empty until we can get a build lock.

nickelization · 2016-09-12T15:48:01Z

@bsparrow435 So it occurred to me this morning that we should probably have an option to disable automatically upgrading to the new tree format. It doesn't necessarily have to be documented or supported, but I think it would be good to provide an escape hatch. If nothing else, it may make testing much easier if we want to, say, write a riak_test module that starts with 2.2 and then performs a downgrade (e.g. something like verify_riak_object_reformat).

nickelization · 2016-09-12T21:17:37Z

+1 0e6bb3e

Add v0 of versioning to AAE hashtree locks, upgrades, and object hashing Reviewed-by: nickelization

nickelization · 2016-09-12T21:43:40Z

@borshop merge

Brian Sparrow added 19 commits June 14, 2016 03:06

Fix two unbounded variables

7a02691

Add version to index hash tree data root

3f94aef

Add a static version “v0” to the data_root of the riak_kv AAE directory. This allows us to isolate old/newer trees as well as support downgrades which will not allow old and new trees to exchange either locally or across MDC.

Allow for version-less lock for subsystems who don't care about version

2f627fb

Fix spec

bf7fa07

Fix specs and export arity

b69cbd8

Address review comments

14bd8e5

Change boolean configuration variable to integer version starting with 0. Change all specs from atom() to non_neg_integer() for version number. Add extra logging for invalid non-integer hash version configurations.

Handle cases where KV is unavailable during an exchange

542aebb

Was not correctly handling case where Result list was empty of valid object responses or only a single valid response.

Clarity that the first element of the result is a partition Idx

010a9ff

WIP commit

a3c393b

WIP commit in case of computer failure or something like that.

WIP push of many commits rebased. This broke some things and I'm stil…

c8b708f

…l recovering some history

Finish grabbing omitted final commits from previous rebase

98e05af

Fix compilation errors

cbeb66e

Fix pattern match, compiler issues

c77b12f

Also, add some testing logs and debug functions.

Fix errors and remove some log spam by checking too frequently

94c17f5

Add some debug logging

e13b5a4

Fix replies for vnode:command handling hash tree upgrade

48f5e54

This was referenced Sep 2, 2016

WIP - Use riak_object:hash instead of duplicating the logic basho/yokozuna#655

Merged

WIP - consistent object hashing using vclocks basho/riak_repl#745

Merged

nickelization reviewed Sep 2, 2016
View reviewed changes

Brian Sparrow added 4 commits September 8, 2016 14:28

Change some logging to debug to reduce log spam during upgrade

1848c00

Address Dialyzer Errors

79e910b

Defined orddict as a type for dialyzer

5166583

Fix dialyzer errors from riak_repl

0e6bb3e

borshop added a commit that referenced this pull request Sep 12, 2016

Merge pull request #1473 from basho/feature-bjs-aae-hashing-v0

222447c

Add v0 of versioning to AAE hashtree locks, upgrades, and object hashing Reviewed-by: nickelization

borshop merged commit 0e6bb3e into develop-2.2 Sep 12, 2016

martinsumner mentioned this pull request Aug 9, 2017

On a fresh Riak 2.2 cluster - AAE won't work for most of the first day #1656

Open

martincox deleted the feature-bjs-aae-hashing-v0 branch June 14, 2019 09:16

martinsumner mentioned this pull request Feb 17, 2020

Infinite waiting handoff basho/riak#1006

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add v0 of versioning to AAE hashtree locks, upgrades, and object hashing #1473

Add v0 of versioning to AAE hashtree locks, upgrades, and object hashing #1473

bsparrow435 commented Sep 2, 2016

nickelization Sep 2, 2016

bsparrow435 Sep 2, 2016

nickelization Sep 2, 2016

nickelization commented Sep 12, 2016

nickelization commented Sep 12, 2016

nickelization commented Sep 12, 2016

Add v0 of versioning to AAE hashtree locks, upgrades, and object hashing #1473

Add v0 of versioning to AAE hashtree locks, upgrades, and object hashing #1473

Conversation

bsparrow435 commented Sep 2, 2016

nickelization Sep 2, 2016

Choose a reason for hiding this comment

bsparrow435 Sep 2, 2016

Choose a reason for hiding this comment

nickelization Sep 2, 2016

Choose a reason for hiding this comment

nickelization commented Sep 12, 2016

nickelization commented Sep 12, 2016

nickelization commented Sep 12, 2016