[POC] Add a GrowingByteBuffer for VFS implementations #13
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
sqlite3.dart has two main VFS implementations apart from the OPFS-based ones:
The IndexedDbFileSystem loads the files into memory, and is then just an InMemoryFileSystem + background write support.
The InMemoryFileSystem keeps a Uint8Array buffer per file. When writing to the file, it grows the buffer by creating a new Uint8Array and copying the data over. This gets really slow when doing that one page at a time with a 10MB+ database size.
This adds a new
GrowingByteBuffer
implementation, that grows the size by a factor of 2 each time. The significantly reduces the overhead of resizing the buffer, at the cost of using more memory.In my tests of syncing a 30MB / 50k rows database, this reduced the sync time from 120s to 10s with IndexedDbFileSystem, and 5.5s with InMemoryFileSystem. This is compared to around 5s with a :memory: database (not sure if there is any significant difference between that and InMemoryFileSystem after this change).
While it worked in my tests, this still has to be cleaned up and tested properly, before making an upstream PR.