Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Improvement] Possible concurrency issues for KvGrabageCollector #2888

Closed
yuqi1129 opened this issue Apr 11, 2024 · 0 comments · Fixed by #2890
Closed

[Improvement] Possible concurrency issues for KvGrabageCollector #2888

yuqi1129 opened this issue Apr 11, 2024 · 0 comments · Fixed by #2890
Assignees
Labels
improvement Improvements on everything

Comments

@yuqi1129
Copy link
Contributor

yuqi1129 commented Apr 11, 2024

What would you like to be improved?

Currently, we would use the following steps to write kv data within a transaction.

Main thread:

  1. Write a real key/value pair into the storage.
  2. Write a commit mark to indicate that the pair is committed.

KvGarbageCollector will periodically scan the storage. If it finds a key/value pair that without a corresponding commit mark for it, KvGarbageCollector will review it as data that is uncommitted, then KvGarbageCollector will clean the data.

There is a chance that the "KvGarbageCollector" thread operates between steps 1 and 2. Fore more, please see the code. https://github.com/datastrato/gravitino/blob/99f2ede127c6977c0cf7c2a137166a8ca780c369/core/src/main/java/com/datastrato/gravitino/storage/kv/KvGarbageCollector.java#L91-L117

How should we improve?

As the time of the writing process in RocksDB is relatively fast, we can make KvGarbageCollector only remove uncommitted data that was written into storage 10 minutes ago, then things will be solved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
improvement Improvements on everything
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants