[0.10.0] TSM conversion reproducibly drops data silently #5606
Comments
|
Can you post the output of the migration tool? Also, how are you determining points are dropped? Some data such as fields with NaN or Inf will be dropped because they are not supported field values. These values were able to be inserted in past versions of some endpoints inadvertently. |
|
The output of the TSM conversion can be found here - https://gist.github.com/jonseymour/1dd3f092eea45c1ab3b5 The technique for detecting missing data was initially a case of using grafana to review a date of significance to us (13 Jan 2016). All the timeseries data for that date was missing in the migrated database, but was present in the restored database. For the purposes of investigation, I reduced the query to this: select count(current) from "foo.bar.state" where time >= '2016-01-13' and time < '2016-01-20' group by time(1d) I should get numbers of around ~800k for each day, but for the missing days was instead getting 0 for 5 of the 7 days. (I got the expected values of ~800k for 2 of the 7 days in the time period above). |
|
I should clarify: all the data for the periods with missing data was missing from one measurement. There were other measurements that had data for that date. |
|
Can you run |
|
show shards output -> https://gist.github.com/jonseymour/ca81c516bd192a0a8119 An example of a measurement point: time,channel,controlled,current,event,observedMax,offTicks,onTicks,peak,period,schema,thingType,uncontrolled,user 2016-01-13T00:00:02.471130001Z,"demand",0,0,"state",2368.2,52507,,0,4,"http://foobar.com/protocol/demand","aircon",0,"8e47ec63-9784-46ae-8042-224166f5d26b" The tags for this measurement are: The other columns in the sample are fields. (This example was extracted from a 0.10.0 system running against a restored b1 database) |
|
For reference, here is a counting query for a different measurement run against both the restored database and the converted database: (good - restored database) (bad - converted database) So, in some cases there are no data points in the periods of interest, and in other cases there about 1/35th of the expected number. |
|
It looks like that data should be in shard 14, 15, 16. You should have some directories under you data dir like |
|
Here is the file listing, I'll collect the dump now. |
|
@jwilder - could you e-mail [email protected] and I'll reply with a dropbox link to the dumps for 14, 15 and 16 since I am not comfortable sharing the contents of the dumps in a public forum. |
|
Sure. Thanks. |
|
@jwilder It looks like the issue with the dropped data is that the shard series index has lost track of series that are stored in the shard, so these aren't exported. This doesn't affect access by the b1 engine itself because the series are accessed directly by bucket name without reference to whether the series is listed in the series index. I am going to have a crack at greedily trying to resurrect orphaned series by using a heuristic to discover them, irrespective of the state of the series index in the shard meta data. |
|
Ah... so the |
|
Yeah, that's what I think is happening. |
|
So, what I plan to do is this. If I find series-like buckets (e.g. names containing ,) in the shard that aren't referenced by the series index then I will abort, unless --repair-series-index is specified in which case I will use an augmented series index that includes the series I find from the bucket list. How does that sound, @jwilder? |
|
Sounds reasonable. |
|
Ok, I have a candidate fix here - https://github.com/jonseymour/influxdb/tree/repair-b1-shard which I have used to successfully recover the data I expected to recover from shard 16. I'll run the modified code against the full database before I issue a pull request. FWIW: the claimed space reduction for this shard dropped from 99% to 81%. |
|
I have a hypothesis about how the corruption might have been introduced, although not enough evidence to confirm it at this point. If a write for a set of points that introduces new series to a shard fails then the in-memory copy of an index will record that the series has been created even though the change to on disk version of the index will be rolled back. Subsequent writes into the series will succeed, but the on disk index will be inconsistent with the disk and this condition will never be fixed unless the server is restarted while the shard is still hot - in other words, a failed write might cause the in memory meta data index to become dirty with respect to the disk. If someone else confirms that this could be a problem, it might be worth raising a separate issue to document the possibility of meta data corruption during a failed write. |
A case (influxdata#5606) was found where a lot of data unexpectedly disappeared from a database following a TSM conversion. The proximate cause was an inconsistency between the root Bolt DB bucket list and the meta data in the "series" bucket of the same shard. There were apparently valid series in Bolt DB buckets that were no longer referenced by the meta data in the "series" bucket - so-called orphaned series; since the conversion process only iterated across the series found in the meta data, the conversion process caused the orphaned series to be removed from the converted shards. This resulted in the unexpected removal of data from the TSM shards that had previously been accessible (despite the meta data inconsistency) in the b1 shards. The root cause of the meta data inconsistency in the case above is not understood but, in the case above, removal of the orphaned series wasn't the appropriate resolution of the inconsistency. This change detects occurrences of meta data inconsistency in shards during conversion and, by default, will cause the conversion process to fail (--repair-index=fail) if any such inconsistency is found. The user can force the conversion to occur by choosing to resolve the inconsistency either by assuming the meta data is incorrect and attempting to convert the orphaned series (--repair-index=repair) or by assuming the meta data is correct and ignoring the orphaned series (--repair-index=ignore). Currently detection and recovery of orphaned series is only supported for b1 shards; bz1 shards are not currently supported. Signed-off-by: Jon Seymour <[email protected]>
|
This is somewhat circumstantial, but I plotted the number of original points, the number of points in the broken conversion and the delta against the beginning of each shard's 7 day capture period and also the version numbers that were installed during that period: The elimination of broken shards for weeks in which 0.9.6.1 was partially installed would tend to suggest that whatever issue was causing the shard index to become corrupted was fixed between version 0.9.2 and 0.9.6.1 |
|
Yeah, I reckon the corruption was caused by the problem fixed with 3348dab which arrived in the tree after v0.9.2 |
A case (#5606) was found where a lot of data unexpectedly disappeared from a database following a TSM conversion. The proximate cause was an inconsistency between the root Bolt DB bucket list and the meta data in the "series" bucket of the same shard. There were apparently valid series in Bolt DB buckets that were no longer referenced by the meta data in the "series" bucket - so-called orphaned series; since the conversion process only iterated across the series found in the meta data, the conversion process caused the orphaned series to be removed from the converted shards. This resulted in the unexpected removal of data from the TSM shards that had previously been accessible (despite the meta data inconsistency) in the b1 shards. The root cause of the meta data inconsistency in the case above is not understood but, in the case above, removal of the orphaned series wasn't the appropriate resolution of the inconsistency. This change detects occurrences of meta data inconsistency in shards during conversion and, by default, will cause the conversion process to fail (--repair-index=fail) if any such inconsistency is found. The user can force the conversion to occur by choosing to resolve the inconsistency either by assuming the meta data is incorrect and attempting to convert the orphaned series (--repair-index=repair) or by assuming the meta data is correct and ignoring the orphaned series (--repair-index=ignore). Currently detection and recovery of orphaned series is only supported for b1 shards; bz1 shards are not currently supported. Signed-off-by: Jon Seymour <[email protected]>
|
@jonseymour I added a fix for both I did some local testing, but would you be able to try the updated fix on your shards that were having some of the series ignored? |
A case (influxdata#5606) was found where a lot of data unexpectedly disappeared from a database following a TSM conversion. The proximate cause was an inconsistency between the root Bolt DB bucket list and the meta data in the "series" bucket of the same shard. There were apparently valid series in Bolt DB buckets that were no longer referenced by the meta data in the "series" bucket - so-called orphaned series; since the conversion process only iterated across the series found in the meta data, the conversion process caused the orphaned series to be removed from the converted shards. This resulted in the unexpected removal of data from the TSM shards that had previously been accessible (despite the meta data inconsistency) in the b1 shards. The root cause of the meta data inconsistency in the case above was a failure, in versions prior to v0.9.3 (actually 3348dab) to update the "series" bucket with series that had been created in previous shards during the life of the same influxd process instance. This fix is required to avoid data loss during TSM conversions for shards that were created with versions of influx that did not include 3348dab (e.g. prior to v0.9.3). Analysis-by: Jon Seymour <[email protected]>
A case (influxdata#5606) was found where a lot of data unexpectedly disappeared from a database following a TSM conversion. The proximate cause was an inconsistency between the root Bolt DB bucket list and the meta data in the "series" bucket of the same shard. There were apparently valid series in Bolt DB buckets that were no longer referenced by the meta data in the "series" bucket - so-called orphaned series; since the conversion process only iterated across the series found in the meta data, the conversion process caused the orphaned series to be removed from the converted shards. This resulted in the unexpected removal of data from the TSM shards that had previously been accessible (despite the meta data inconsistency) in the b1 shards. The root cause of the meta data inconsistency in the case above was a failure, in versions prior to v0.9.3 (actually 3348dab) to update the "series" bucket with series that had been created in previous shards during the life of the same influxd process instance. This fix is required to avoid data loss during TSM conversions for shards that were created with versions of influx that did not include 3348dab (e.g. prior to v0.9.3). Analysis-by: Jon Seymour <[email protected]>
|
Fixed by #5675. Thanks @jwilder, @joelegasse! |
A case (#5606) was found where a lot of data unexpectedly disappeared from a database following a TSM conversion. The proximate cause was an inconsistency between the root Bolt DB bucket list and the meta data in the "series" bucket of the same shard. There were apparently valid series in Bolt DB buckets that were no longer referenced by the meta data in the "series" bucket - so-called orphaned series; since the conversion process only iterated across the series found in the meta data, the conversion process caused the orphaned series to be removed from the converted shards. This resulted in the unexpected removal of data from the TSM shards that had previously been accessible (despite the meta data inconsistency) in the b1 shards. The root cause of the meta data inconsistency in the case above was a failure, in versions prior to v0.9.3 (actually 3348dab) to update the "series" bucket with series that had been created in previous shards during the life of the same influxd process instance. This fix is required to avoid data loss during TSM conversions for shards that were created with versions of influx that did not include 3348dab (e.g. prior to v0.9.3). Analysis-by: Jon Seymour <[email protected]>
G'day I recently upgraded my influx instance from 0.9.6.1 to 0.10.0
To perform the upgrade I did the following:
The following day I ran did some sanity checks against the migrated data and discovered that some data had gone missing.
To verify that the data was, indeed, present prior to the upgrade, I restored the original b1 databases to another influx instance (using a filesystem restore), restarted a copy of 0.9.6.1 against the restored copy of the database and re-ran the queries to confirm that the data I was expecting to be there was in fact there.
I then re-ran the migration process on the restored copy of the database, double checking that the migration ran without reporting errors. I then restarted a copy of 0.10.0 against the migrated database and observed that the same sequences of data that went missing from the first migration was also missing in the repeated migration.
In other words, I have now run this migration twice from the same starting data and in both cases, data went missing. The data was present in the source prior to migration, as verified by the steps I took to do this.
I reviewed the output of influx_tsm and there was nothing in the output that suggested that data would be lost.
Let me know if you want me to enable additional diagnostics on a further attempt to restore the database. For the moment, I am planning to revert our production system to the b1 engine because I can't afford to lose access to the data that the migration process appears to silently drop.
jon.
The text was updated successfully, but these errors were encountered: