Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add v0 of versioning to AAE hashtree locks, upgrades, and object hashing #1473

Merged
merged 24 commits into from
Sep 12, 2016

Commits on Jun 14, 2016

  1. Added logic to hash only vclock for object hashes. Added logging to c…

    …atch pre 2.1 vclocks which require a rewrite to be safe
    
    Added riak_kv boolean environment variable `hash_only_vclock` which
    enables/disabled the vclock hashing logic. Additionally, when enabled,
    we will no longer do the expensive compare/logging on read-repair to
    catch old pre2.1 vclocks who require a rewrite.
    
    Additional logic must be added to `yz_kv.erl` so the hashes in the
    entropy_data element of the Solr documents match what is stored in KV
    AAE. Upgrading the hash logic requires clearing all YZ documents and
    re-indexing everything. This is because we store the obj hash in the
    Solr document. This will hopefully change in the future to allow for
    easier AAE changes.
    Brian Sparrow committed Jun 14, 2016
    Configuration menu
    Copy the full SHA
    51af50c View commit details
    Browse the repository at this point in the history
  2. Fix two unbounded variables

    Brian Sparrow committed Jun 14, 2016
    Configuration menu
    Copy the full SHA
    7a02691 View commit details
    Browse the repository at this point in the history

Commits on Jun 16, 2016

  1. Add version to index hash tree data root

    Add a static version “v0” to the data_root of the riak_kv AAE
    directory. This allows us to isolate old/newer trees as well as support
    downgrades which will not allow old and new trees to exchange either
    locally or across MDC.
    Brian Sparrow committed Jun 16, 2016
    Configuration menu
    Copy the full SHA
    3f94aef View commit details
    Browse the repository at this point in the history

Commits on Jun 22, 2016

  1. Add version to index_hashtree state, path, and get_lock interface

    New error `bad_version` added to the return from
    `riak_kv_index_hashtree:do_get_lock` if the requested version and the
    local version of the hash tree do not match. Requires changes in
    yokozuna and riak_repl to fully function.
    Brian Sparrow committed Jun 22, 2016
    Configuration menu
    Copy the full SHA
    52a0ad0 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    2f627fb View commit details
    Browse the repository at this point in the history
  3. Fix spec

    Brian Sparrow committed Jun 22, 2016
    Configuration menu
    Copy the full SHA
    bf7fa07 View commit details
    Browse the repository at this point in the history
  4. Fix specs and export arity

    Brian Sparrow committed Jun 22, 2016
    Configuration menu
    Copy the full SHA
    b69cbd8 View commit details
    Browse the repository at this point in the history

Commits on Jul 20, 2016

  1. Address review comments

    Change boolean configuration variable to integer version starting with
    0.
    
    Change all specs from atom() to non_neg_integer() for version number.
    
    Add extra logging for invalid non-integer hash version configurations.
    Brian Sparrow committed Jul 20, 2016
    Configuration menu
    Copy the full SHA
    14bd8e5 View commit details
    Browse the repository at this point in the history
  2. Handle cases where KV is unavailable during an exchange

    Was not correctly handling case where Result list was empty of valid
    object responses or only a single valid response.
    Brian Sparrow committed Jul 20, 2016
    Configuration menu
    Copy the full SHA
    542aebb View commit details
    Browse the repository at this point in the history

Commits on Aug 15, 2016

  1. Configuration menu
    Copy the full SHA
    010a9ff View commit details
    Browse the repository at this point in the history
  2. Forward merge aae hashing change v0

    Add version 0 of object hashing(config `object_hash_version`) which
    hashes only the vclock to represent a version of an object. Also add
    versioning to index_hashtree get_lock API which now takes a Version
    parameter. The absence of a version is assumed to be `undefined` and
    will only be allowed to get a lock if the hash on disk is in legacy
    format.
    
    Upgraded hash trees are located within the anti_entropy data_dir under
    the directory v0. Currently the only supported version is 0 but the
    code allows for support of increasing integer versions.
    
    Add verbose riak_object:equals check to get_core for all read
    operations who do not trigger read repair. This logic is to detect
    potentially pre-2.1 data which may be impacted by old vector clock bugs
    and should be rewritten before upgrading the hash version for safety.
    Brian Sparrow committed Aug 15, 2016
    Configuration menu
    Copy the full SHA
    910adb1 View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2016

  1. WIP commit

    WIP commit in case of computer failure or something like that.
    Brian Sparrow committed Aug 22, 2016
    Configuration menu
    Copy the full SHA
    a3c393b View commit details
    Browse the repository at this point in the history

Commits on Aug 31, 2016

  1. WIP push of many commits rebased. This broke some things and I'm stil…

    …l recovering some history
    Brian Sparrow committed Aug 31, 2016
    Configuration menu
    Copy the full SHA
    c8b708f View commit details
    Browse the repository at this point in the history

Commits on Sep 1, 2016

  1. Configuration menu
    Copy the full SHA
    98e05af View commit details
    Browse the repository at this point in the history
  2. Fix compilation errors

    Brian Sparrow committed Sep 1, 2016
    Configuration menu
    Copy the full SHA
    cbeb66e View commit details
    Browse the repository at this point in the history
  3. Fix pattern match, compiler issues

    Also, add some testing logs and debug functions.
    Brian Sparrow committed Sep 1, 2016
    Configuration menu
    Copy the full SHA
    c77b12f View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    94c17f5 View commit details
    Browse the repository at this point in the history
  5. Add some debug logging

    Brian Sparrow committed Sep 1, 2016
    Configuration menu
    Copy the full SHA
    e13b5a4 View commit details
    Browse the repository at this point in the history

Commits on Sep 2, 2016

  1. Fix replies for vnode:command handling hash tree upgrade

    Brian Sparrow committed Sep 2, 2016
    Configuration menu
    Copy the full SHA
    48f5e54 View commit details
    Browse the repository at this point in the history

Commits on Sep 8, 2016

  1. Address review comments

    Additionally, cleaned up the anode code which killed the hash tree on
    upgrade. Removed one binary layer from the riak_object:hash and added
    the build locks(different from concurrency lock(doh)) to the upgrade
    procedure so we won’t kill trees and then have them sit empty until we
    can get a build lock.
    Brian Sparrow committed Sep 8, 2016
    Configuration menu
    Copy the full SHA
    586ea62 View commit details
    Browse the repository at this point in the history

Commits on Sep 9, 2016

  1. Configuration menu
    Copy the full SHA
    1848c00 View commit details
    Browse the repository at this point in the history
  2. Address Dialyzer Errors

    Brian Sparrow committed Sep 9, 2016
    Configuration menu
    Copy the full SHA
    79e910b View commit details
    Browse the repository at this point in the history
  3. Defined orddict as a type for dialyzer

    Brian Sparrow committed Sep 9, 2016
    Configuration menu
    Copy the full SHA
    5166583 View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2016

  1. 5 Configuration menu
    Copy the full SHA
    0e6bb3e View commit details
    Browse the repository at this point in the history