You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, we would use the following steps to write kv data within a transaction.
Main thread:
Write a real key/value pair into the storage.
Write a commit mark to indicate that the pair is committed.
KvGarbageCollector will periodically scan the storage. If it finds a key/value pair that without a corresponding commit mark for it, KvGarbageCollector will review it as data that is uncommitted, then KvGarbageCollector will clean the data.
As the time of the writing process in RocksDB is relatively fast, we can make KvGarbageCollector only remove uncommitted data that was written into storage 10 minutes ago, then things will be solved.
The text was updated successfully, but these errors were encountered:
What would you like to be improved?
Currently, we would use the following steps to write kv data within a transaction.
Main thread:
KvGarbageCollector
will periodically scan the storage. If it finds a key/value pair that without a corresponding commit mark for it, KvGarbageCollector will review it as data that is uncommitted, thenKvGarbageCollector
will clean the data.There is a chance that the "KvGarbageCollector" thread operates between steps 1 and 2. Fore more, please see the code. https://github.com/datastrato/gravitino/blob/99f2ede127c6977c0cf7c2a137166a8ca780c369/core/src/main/java/com/datastrato/gravitino/storage/kv/KvGarbageCollector.java#L91-L117
How should we improve?
As the time of the writing process in RocksDB is relatively fast, we can make
KvGarbageCollector
only remove uncommitted data that was written into storage 10 minutes ago, then things will be solved.The text was updated successfully, but these errors were encountered: