-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix windows dataloss issue #1134
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed 2 of 2 files at r1.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @ashish-goswami)
Windows doesn't allow memory mapping a file to a size greater than the file's actual size. To circumvent this, we increase the file size by truncating it. https://github.com/dgraph-io/badger/blob/f5b63211d7f3e2f5f8b698893313b2a54e4df7de/y/mmap_windows.go#L41-L48 When badger would re-open, we try to replay this "truncated" file. Since this truncated file consists of all zeros, the replay would return the last offset as `zero` and then we would truncate the original file to size `zero`. The replay function would return `zero` as the last valid offset which was wrong. The last valid offset is start offset plus the forward movements of the file offset. So instead of https://github.com/dgraph-io/badger/blob/f5b63211d7f3e2f5f8b698893313b2a54e4df7de/value.go#L433 ```go var validEndOff uint32 // notice we're starting from zero, not the start point. ``` we should be doing ```go var validEndOff uint32 = offset ``` Fixes - #1126 (cherry picked from commit 969a8e8)
I see that this issue has been resolved, but currently I am encountering the same phenomenon on Windows 10 Golang 1.21.0. When the program interrupts abnormally and fails to close properly, the vlog size becomes 1kb and prompts "Value log truncate required to run DB. This might result in data loss" @jarifibrahim |
Windows doesn't allow memory mapping a file to a size greater than the file's actual size. To circumvent this, we increase the file size by truncating it.
badger/y/mmap_windows.go
Lines 41 to 48 in f5b6321
When badger would re-open, we try to replay this "truncated" file. Since this truncated file consists of all zeros, the replay would return the last offset as
zero
and then we would truncate the original file to sizezero
.The replay function would return
zero
as the last valid offset which was wrong. The last valid offset is start offset plus the forward movements of the file offset.So instead of
badger/value.go
Line 433 in f5b6321
we should be doing
Fixes - #1126
This change is