Skip to content
This repository has been archived by the owner on Sep 17, 2024. It is now read-only.

Chunked storage type #68

Open
kirillt opened this issue Jul 4, 2023 · 0 comments
Open

Chunked storage type #68

kirillt opened this issue Jul 4, 2023 · 0 comments

Comments

@kirillt
Copy link
Member

kirillt commented Jul 4, 2023

We have two storage implementations:

  • File-based: all key-value pairs are stored in a single file.
  • Folder-based: each key gets its individual file in a designated folder.

File-based storages tend to fail when dealing with map sizes nearing 10,000 entries, often resulting in slow, sometimes even flawed writing. On the other hand, folder-based storages must be inefficient when handling smaller values.

Chunked storage blends both methods: it utilizes a folder containing files, with each file holding multiple entries. We need a strategy to identify the chunks requiring updates after a storage model modification. The use of Merkle trees might be necessary for efficient synchronization of external updates.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

1 participant