[Miniflare 3] Implement simulators as Durable Objects #656
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
mrbbot
force-pushed
the
bcoll/tre-magic-proxy
branch
2 times, most recently
from
681ff91
to
da9e434
Compare
mrbbot
force-pushed
the
bcoll/tre-durable-object-simulators
branch
from
120a0d0
to
6cf2b79
Compare
mrbbot
commented
Aug 14, 2023
mrbbot
added
the
multiple reviews suggested
|
oh my god yes please, this will make our installs so much faster and reliable |
mrbbot
force-pushed
the
bcoll/tre-magic-proxy
branch
2 times, most recently
from
6498024
to
cabbecb
Compare
mrbbot
force-pushed
the
bcoll/tre-magic-proxy
branch
2 times, most recently
from
152e2de
to
62c842e
Compare
mrbbot
force-pushed
the
bcoll/tre-durable-object-simulators
branch
6 times, most recently
from
6eb5436
to
9bd3a27
Compare
mrbbot
added a commit
that referenced
this pull request
Aug 21, 2023
This is a temporary fix to allow Workers Sites tests cherrypicked from #656 to pass. Listing keys in Workers Sites `__STATIC_CONTENT` namespace was previously broken, due to a type error in the inline namespace worker script. This change also means we can avoid the `Function#toString()`ing to "bundle" code in inline worker scripts.
mrbbot
added a commit
that referenced
this pull request
Aug 21, 2023
This is a temporary fix to allow Workers Sites tests cherrypicked from #656 to pass. Listing keys in Workers Sites `__STATIC_CONTENT` namespace was previously broken, due to a type error in the inline namespace worker script. This change also means we can avoid the `Function#toString()`ing to "bundle" code in inline worker scripts.
mrbbot
added a commit
that referenced
this pull request
Aug 21, 2023
This is a temporary fix to allow Workers Sites tests cherrypicked from #656 to pass. Listing keys in Workers Sites `__STATIC_CONTENT` namespace was previously broken, due to a type error in the inline namespace worker script. This change also means we can avoid the `Function#toString()`ing to "bundle" code in inline worker scripts.
penalosa
reviewed
Aug 21, 2023
This replaces the loopback server's support for logging requests with a more generic endpoint, that supports logging any arbitrary message. This will be needed by Durable Object simulators that want to write to the console, without `workerd`'s verbose prefixing. In particular, Queues logs dispatches, and Cache logs first usage when the `cacheWarnUsage` option is enabled.
Previously, we were only testing the `KVGateway`. In preparation for porting the KV implementation to a worker, rewrite the tests to test the full KV flow, including the router.
...and move dependencies only used in Workers to `devDependencies`. They're bundled in with the Worker code, so don't need to be installed again. Importantly, this change removes `better-sqlite3`, meaning Miniflare no longer has any native dependencies other than `workerd`. Fixes #599 Fixes cloudflare/workers-sdk#3423 Fixes cloudflare/workers-sdk#3449 Fixes cloudflare/workers-sdk#3531 Fixes cloudflare/workers-sdk#3534 Fixes cloudflare/workers-sdk#3708 Fixes cloudflare/workers-sdk#3746 Closes cloudflare/cloudflare-docs#10241
When combined with running tests in serial, this should ensure we only have at most two TCP connections open to `workerd` at a time.
mrbbot
force-pushed
the
bcoll/tre-durable-object-simulators
branch
from
ca5ba92
to
4b4dc30
Compare
mrbbot
added a commit
that referenced
this pull request
Sep 6, 2023
With #656, the Queues dispatcher is now implemented as part of the broker Durable Object. We no longer send message batches directly from Node.js, so can remove queue handling from the entry worker. Note the magic proxy enqueues messages through queue producer bindings like regular workers, so never used this endpoint directly.
mrbbot
added a commit
that referenced
this pull request
Sep 11, 2023
With #656, the Queues dispatcher is now implemented as part of the broker Durable Object. We no longer send message batches directly from Node.js, so can remove queue handling from the entry worker. Note the magic proxy enqueues messages through queue producer bindings like regular workers, so never used this endpoint directly.
2 tasks
This was referenced Sep 25, 2023
Closed
mrbbot
added a commit
to cloudflare/workers-sdk
that referenced
this pull request
Nov 17, 2023
cloudflare/miniflare#656 introduced a bug in the SQL statement used for getting the old blob ID of entries to delete when overriding keys. This meant if a key was overridden, the old blob pointed to by that key was not deleted. This lead to an accumulation of garbage when `kvPersist` and `cachePersist` were enabled.
6 tasks
mrbbot
added a commit
to cloudflare/workers-sdk
that referenced
this pull request
Dec 4, 2023
cloudflare/miniflare#656 introduced a bug in the SQL statement used for getting the old blob ID of entries to delete when overriding keys. This meant if a key was overridden, the old blob pointed to by that key was not deleted. This lead to an accumulation of garbage when `kvPersist` and `cachePersist` were enabled.
mrbbot
added a commit
to cloudflare/workers-sdk
that referenced
this pull request
Dec 4, 2023
cloudflare/miniflare#656 introduced a bug in the SQL statement used for getting the old blob ID of entries to delete when overriding keys. This meant if a key was overridden, the old blob pointed to by that key was not deleted. This lead to an accumulation of garbage when `kvPersist` and `cachePersist` were enabled.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hey! 👋 This PR ports all of Miniflare's simulators from Node.js to Durable Objects running in
workerd. There should be practically no1 user-facing feature changes: this is purely an internal refactoring. This refactor brings a few advantages:Removing native dependencies: Miniflare previously depended on
better-sqlite3for an SQLite implementation.workerdalready ships with SQLite, andbetter-sqlite3requires a native add-on. If prebuilt binaries were not found, this was compiled from source. On my 2021 M1 Pro MacBook, this takes just under 30s. Becauseminiflareis a hard-dependency ofwrangler, users installingwranglerwould often hit this, slowing down their installs.better-sqlite3is a great library, but if we're already shipping SQLite withworkerd, ideally we shouldn't need to ship it again.better-sqlite3also ships with a fair few transitive dependencies. By removing our dependency on it, we're able to cut Miniflare's total dependency count in half:(generated by https://npm.anvaka.com/#/view/2d/miniflare)
Running
workerdstandalone: we're now able to run Miniflare's simulators inworkerdalone, without Node.js involvement. This meansworkerdusers can benefit from Miniflare's simulators to run Workers written using Cloudflare's services. In a future PR, we'll add some samplecapnpconfigs for doing this, allowing you to do things likeusing Miniflare = import "./node_modules/miniflare/miniflare.capnp";to add simulators to yourworkerdconfig. These configs could be used in conjunction withworkerd compileto build single-file-binaries for running Cloudflare Workers too.Opening the door for production code sharing: lots of Cloudflare's Developer Platform is built on Cloudflare. By porting Miniflare's simulators to Workers, we're making it easier for us to use parts of the actual production implementations of KV, R2, Queues, and D1 in the future.
This PR is best reviewed commit-by-commit, I've added comments for particular areas of concern.
Footnotes
Technically Queues is now supported on Node versions < 18, and the D1 database API has been updated to match the new, no-longer "experimental" backend ↩