Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem: versiondb write wrong data #1683

Closed
yihuang opened this issue Nov 11, 2024 · 0 comments · Fixed by #1685
Closed

Problem: versiondb write wrong data #1683

yihuang opened this issue Nov 11, 2024 · 0 comments · Fixed by #1685
Assignees

Comments

@yihuang
Copy link
Collaborator

yihuang commented Nov 11, 2024

suspected to be related to a rocksdb version upgrade happens in the past, which breaks user-defined timestamp feature. will develop a fix tool, should not happen again in the future.

Description

versiondb using rocksdb's user-defined timestamp feature, which assign a timestamp with a key, the timestamp is encoded with little endianness and append to the key as postfix, and the UDT feature will handle the timestamp specially, like a tuple (key, timestamp).
But for the wrong blocks, the data are apparently written without the UDT special handling, and read out as (key | timestamp, 0).

Fix

We adopt a non-intrusive strategy to fix:

  • add cronosd changeset fixdata <dir> command to re-insert the wrong data.
  • we change the versiondb to ignore timestamp 0 data.

In this way, we don't automatically trim the timestamp 0 data, so if the same issue happen again in the future, we can still apply the same command to recover it.

@yihuang yihuang self-assigned this Nov 11, 2024
yihuang added a commit to yihuang/cronos that referenced this issue Nov 11, 2024
Closes: crypto-org-chain#1683

Solution:
- add fix command to fix corrupted data in versiondb
yihuang added a commit to yihuang/cronos that referenced this issue Nov 11, 2024
Closes: crypto-org-chain#1683

Solution:
- add fix command to fix corrupted data in versiondb
yihuang added a commit to yihuang/cronos that referenced this issue Nov 11, 2024
…pto-org-chain#1685)

Closes: crypto-org-chain#1683

Solution:
- add fix command to fix corrupted data in versiondb

rename

support SkipVersionZero

support SkipVersionZero

cleanup
yihuang added a commit to yihuang/cronos that referenced this issue Nov 12, 2024
…pto-org-chain#1685)

Closes: crypto-org-chain#1683

Solution:
- add fix command to fix corrupted data in versiondb

rename

support SkipVersionZero

support SkipVersionZero

cleanup

cleanup

cleanup

fix test

cleanup

destroy

log store name

fix data manually

cli

Update versiondb/client/fixdata.go

Signed-off-by: yihuang <[email protected]>

Update versiondb/client/fixdata.go

Signed-off-by: yihuang <[email protected]>

rnemae

Update CHANGELOG.md

Signed-off-by: yihuang <[email protected]>

fix test

don't return nil as empty slice

add dryrun mode

separete read from iteration

add stores flag

debug

add timestamp api
yihuang added a commit that referenced this issue Nov 12, 2024
…) (#1686)

* Problem: nixpkgs-fmt is deprecated (backport: #1677)

Solution:
- switch to nixfmt-rfc-style

* Problem: no command to fix corrupted data in versiondb (backport: #1685)

Closes: #1683

Solution:
- add fix command to fix corrupted data in versiondb

rename

support SkipVersionZero

support SkipVersionZero

cleanup

cleanup

cleanup

fix test

cleanup

destroy

log store name

fix data manually

cli

Update versiondb/client/fixdata.go

Signed-off-by: yihuang <[email protected]>

Update versiondb/client/fixdata.go

Signed-off-by: yihuang <[email protected]>

rnemae

Update CHANGELOG.md

Signed-off-by: yihuang <[email protected]>

fix test

don't return nil as empty slice

add dryrun mode

separete read from iteration

add stores flag

debug

add timestamp api

* update rocksdb

* changelog

* fix flag name conflict

* validate timestamp

* Update CHANGELOG.md

Signed-off-by: yihuang <[email protected]>

* skip non-zero version

* flush after fix

* only flush if not dry-run

---------

Signed-off-by: yihuang <[email protected]>
github-merge-queue bot pushed a commit that referenced this issue Nov 12, 2024
* Problem: no command to fix corrupted data in versiondb

Closes: #1683

Solution:
- add fix command to fix corrupted data in versiondb

* rename

* support SkipVersionZero

* support SkipVersionZero

* cleanup

* cleanup

* cleanup

* destroy

* fix data manually

* Update CHANGELOG.md

Signed-off-by: yihuang <[email protected]>

* Update versiondb/tsrocksdb/store.go

Co-authored-by: mmsqe <[email protected]>
Signed-off-by: yihuang <[email protected]>

* cli

* cleanup

* Update versiondb/client/fixdata.go

Signed-off-by: yihuang <[email protected]>

* rename

* fix test

* check nil

* don't return nil as empty slice

* add dryrun mode

* cleanup

* separete read from iteration

* store name flag

* validate timestamp

* skip non-zero version

* update gomod2nix

* fix build

* flush after fix

* fix

* revert

---------

Signed-off-by: yihuang <[email protected]>
Co-authored-by: mmsqe <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant