Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Partially fix data corruption due to canceling an asynchronous operation #228

Closed
ikopylov opened this issue Jan 3, 2023 · 0 comments · Fixed by #250
Closed

Partially fix data corruption due to canceling an asynchronous operation #228

ikopylov opened this issue Jan 3, 2023 · 0 comments · Fixed by #250
Assignees
Labels
bug Something isn't working enhancement New feature or request

Comments

@ikopylov
Copy link
Member

ikopylov commented Jan 3, 2023

Unfortunately, async operation cancellation can corrupt data on disk and break the consistency of data in memory. We can't fix this completely, because cancellation can happen at any point where .await is used. But we can improve the situation and prevent data corruption on disk:

  1. We track offset within file in 2 places: in Blob (https://github.com/qoollo/pearl/blob/master/src/blob/core.rs#L30) and in File (https://github.com/qoollo/pearl/blob/master/src/blob/file.rs#L14). They are easily can diverge. We can choose not to track offset on Blob level and only track it on File level
  2. Update File offset at the place where cancellation is not possible. Currently it updates here: https://github.com/qoollo/pearl/blob/master/src/blob/file.rs#L62 and right on the next line we use .await. We have to move the offset update right into place before the synchronous write happens: https://github.com/qoollo/pearl/blob/master/src/blob/file.rs#L96
  3. We need to set an offset in the record header (https://github.com/qoollo/pearl/blob/master/src/blob/core.rs#L274), but with new approach it becomes harder. We need to transfer a special structure directly to the place where the data is written to the file, which allows to patch the data with the offset value. Such struct will be added in this PR: 188 serialize outside lock #220. Also we should return the offset value to the caller, so it can also change value within header struct
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants