Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

go/badger: enable truncate to recover from corrupted value log file #2732

Merged
merged 1 commit into from
Feb 27, 2020

Conversation

ptrus
Copy link
Member

@ptrus ptrus commented Feb 27, 2020

Encountered corrupted badger value log file when running txsource tests with node restarts. Apparently badger is not at all resilient to crashes unless the truncate option is set on (which could result in data loss, but i think this should be safe as this means the crash happened while it was being written to the value log, so not yet committed).

{"caller":"node.go:741","err":"tendermint: failed to create node: tendermint/db/badger: failed to open database: During db.vlog.open: Value log truncate required to run DB. This might result in data loss","level":"error","module":"oasis-node","msg":"failed to start tendermint service","ts":"2020-02-27T09:42:18.599026044Z"}

This was also default behavior before an configuration option was added: dgraph-io/badger#452 dgraph-io/badger#740

@ptrus ptrus force-pushed the ptrus/fix/badger-enable-truncation branch from 9defea7 to 3a296de Compare February 27, 2020 11:13
@codecov
Copy link

codecov bot commented Feb 27, 2020

Codecov Report

Merging #2732 into master will increase coverage by 0.18%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff            @@
##           master   #2732      +/-   ##
=========================================
+ Coverage   62.72%   62.9%   +0.18%     
=========================================
  Files         381     381              
  Lines       35753   35757       +4     
=========================================
+ Hits        22426   22494      +68     
+ Misses      10512   10434      -78     
- Partials     2815    2829      +14
Impacted Files Coverage Δ
go/storage/mkvs/urkel/db/badger/badger.go 69.3% <100%> (+0.06%) ⬆️
go/runtime/localstorage/localstorage.go 56.14% <100%> (+0.78%) ⬆️
go/common/persistent/persistent.go 84.31% <100%> (+0.31%) ⬆️
go/runtime/history/db.go 82.64% <100%> (+0.14%) ⬆️
go/consensus/tendermint/db/badger/badger.go 68.21% <100%> (+0.12%) ⬆️
go/worker/storage/service_external.go 47.31% <0%> (-6.46%) ⬇️
go/runtime/committee/nodes.go 80.64% <0%> (-2.16%) ⬇️
go/worker/compute/txnscheduler/committee/node.go 62.34% <0%> (-1.86%) ⬇️
go/worker/registration/worker.go 66.34% <0%> (-0.49%) ⬇️
go/oasis-node/cmd/root.go 86.48% <0%> (-0.36%) ⬇️
... and 20 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b731c5a...5a388df. Read the comment docs.

@ptrus ptrus force-pushed the ptrus/fix/badger-enable-truncation branch 2 times, most recently from 87e7d44 to 88c90e3 Compare February 27, 2020 11:32
@ptrus ptrus force-pushed the ptrus/fix/badger-enable-truncation branch from 88c90e3 to e58392b Compare February 27, 2020 11:56
@ptrus ptrus force-pushed the ptrus/fix/badger-enable-truncation branch from e58392b to 5a388df Compare February 27, 2020 12:54
@ptrus ptrus self-assigned this Feb 27, 2020
@ptrus ptrus merged commit 074978b into master Feb 27, 2020
@ptrus ptrus deleted the ptrus/fix/badger-enable-truncation branch February 27, 2020 13:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants