Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tolerate b1 shard index inconsistencies during TSM conversion #5647

Closed
wants to merge 2 commits into from
Closed

Tolerate b1 shard index inconsistencies during TSM conversion #5647

wants to merge 2 commits into from

Commits on Feb 12, 2016

  1. Tolerate shard meta data corruption 

    A case (influxdata#5606) was found where a lot of data unexpectedly disappeared from a database
following a TSM conversion.

The proximate cause was an inconsistency between the root Bolt DB bucket list
and the meta data in the "series" bucket of the same shard. There were apparently valid
series in Bolt DB buckets that were no longer referenced by the meta data
in the "series" bucket - so-called orphaned series; since the conversion
process only iterated across the series found in the meta data, the conversion process
caused the orphaned series to be removed from the converted shards. This resulted in the
unexpected removal of data from the TSM shards that had previously been accessible
(despite the meta data inconsistency) in the b1 shards.

The root cause of the meta data inconsistency in the case above is not understood but, in the
case above, removal of the orphaned series wasn't the appropriate resolution of the
inconsistency.

This change detects occurrences of meta data inconsistency in shards during conversion
and, by default, will cause the conversion process to fail (--repair-index=fail) if any such
inconsistency is found.

The user can force the conversion to occur by choosing to resolve the inconsistency
either by assuming the meta data is incorrect and attempting to convert the orphaned
series (--repair-index=repair) or by assuming the meta data is correct and ignoring the
orphaned series (--repair-index=ignore).

Currently detection and recovery of orphaned series is only supported for b1 shards;
bz1 shards are not currently supported.

Signed-off-by: Jon Seymour <[email protected]>
    @jonseymour
    jonseymour committed Feb 12, 2016
    Configuration menu
    Copy the full SHA
    090dbe1 View commit details
    Browse the repository at this point in the history

  2. If Open failed, then r.tx may not be initialized. 

    Signed-off-by: Jon Seymour <[email protected]>
    @jonseymour
    jonseymour committed Feb 12, 2016
    Configuration menu
    Copy the full SHA
    21c4b93 View commit details
    Browse the repository at this point in the history