You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, when snapshot is generated for update, memtable is not included. This means, we need to ensure data is flushed in many scenarios. This characteristic has caused several issues previously:
The reason that memtable is not included for update is to ensure that no data will be written to the shared memtable, because there may be appendable column files. As an improvement, we could mark column files to be not appendable, when generating a snapshot for update. In this way, we could also ensure that new writes will not change the result of the snapshot.
After this improvement takes effect, may be we could revert PRs like #4959 to reduce unnecessary flushes. Leaving unflushed data for long time is still a problem, because it means data will be lost after a power lost (when persisted index has been pushed forward). Instead of trying to flush the data in more cases, we could also use some background threads to better discover them.
The text was updated successfully, but these errors were encountered:
Enhancement
Currently, when snapshot is generated for update, memtable is not included. This means, we need to ensure data is flushed in many scenarios. This characteristic has caused several issues previously:
The reason that memtable is not included for update is to ensure that no data will be written to the shared memtable, because there may be appendable column files. As an improvement, we could mark column files to be not appendable, when generating a snapshot for update. In this way, we could also ensure that new writes will not change the result of the snapshot.
After this improvement takes effect, may be we could revert PRs like #4959 to reduce unnecessary flushes. Leaving unflushed data for long time is still a problem, because it means data will be lost after a power lost (when persisted index has been pushed forward). Instead of trying to flush the data in more cases, we could also use some background threads to better discover them.
The text was updated successfully, but these errors were encountered: