Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[POC] Add a GrowingByteBuffer for VFS implementations #13

Closed
wants to merge 2 commits into from

Conversation

rkistner
Copy link

sqlite3.dart has two main VFS implementations apart from the OPFS-based ones:

  1. InMemoryFileSystem
  2. IndexedDbFileSystem

The IndexedDbFileSystem loads the files into memory, and is then just an InMemoryFileSystem + background write support.

The InMemoryFileSystem keeps a Uint8Array buffer per file. When writing to the file, it grows the buffer by creating a new Uint8Array and copying the data over. This gets really slow when doing that one page at a time with a 10MB+ database size.

This adds a new GrowingByteBuffer implementation, that grows the size by a factor of 2 each time. The significantly reduces the overhead of resizing the buffer, at the cost of using more memory.

In my tests of syncing a 30MB / 50k rows database, this reduced the sync time from 120s to 10s with IndexedDbFileSystem, and 5.5s with InMemoryFileSystem. This is compared to around 5s with a :memory: database (not sure if there is any significant difference between that and InMemoryFileSystem after this change).

While it worked in my tests, this still has to be cleaned up and tested properly, before making an upstream PR.

@rkistner
Copy link
Author

rkistner commented Oct 21, 2024

Refactored and merged upstream in simolus3#258; will likely be released in v2.4.7.

@rkistner rkistner closed this Oct 21, 2024
@rkistner rkistner deleted the bytebuffer branch October 21, 2024 10:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant