Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include memtable in the snapshot for update #6045

Closed
breezewish opened this issue Sep 27, 2022 · 0 comments · Fixed by #6146
Closed

Include memtable in the snapshot for update #6045

breezewish opened this issue Sep 27, 2022 · 0 comments · Fixed by #6146
Labels
type/enhancement The issue or PR belongs to an enhancement.

Comments

@breezewish
Copy link
Member

breezewish commented Sep 27, 2022

Enhancement

Currently, when snapshot is generated for update, memtable is not included. This means, we need to ensure data is flushed in many scenarios. This characteristic has caused several issues previously:

The reason that memtable is not included for update is to ensure that no data will be written to the shared memtable, because there may be appendable column files. As an improvement, we could mark column files to be not appendable, when generating a snapshot for update. In this way, we could also ensure that new writes will not change the result of the snapshot.

After this improvement takes effect, may be we could revert PRs like #4959 to reduce unnecessary flushes. Leaving unflushed data for long time is still a problem, because it means data will be lost after a power lost (when persisted index has been pushed forward). Instead of trying to flush the data in more cases, we could also use some background threads to better discover them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant