-
-
Notifications
You must be signed in to change notification settings - Fork 125
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chore(worker-api/impl): pass stream to parent #324
Conversation
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
This pull request is automatically built and testable in CodeSandbox. To see build info of the built libraries, click here or the icon next to each commit SHA. Latest deployment of this branch, based on commit 204a34e:
|
Upstream issue: DefinitelyTyped/DefinitelyTyped#67980 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good so far!
Can we do the same thing for main -> worker too?
Co-authored-by: Daishi Kato <[email protected]>
…aku into chore/pass-stream-to-parent
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi, this looks great. I found a way to make it much simpler.
import type { Worker as WorkerOrig } from 'node:worker_threads'; | ||
import type { | ||
TransferListItem, | ||
Worker as WorkerOrig, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
btw, I recently use WorkerType
for the naming convention instead of WorkerOrig
. What do you think?
const err = | ||
mesg.err instanceof Error ? mesg.err : new Error(String(mesg.err)); | ||
controller.error(err); | ||
mesg.stream.pipeTo( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think this is necessary.
So, we don't need mesg.type=pipe at all, and attach the stream
for mesg.type=render.
Co-authored-by: Daishi Kato <[email protected]>
const controller = controllerMap.get(mesg.id)!; | ||
controller.close(); | ||
} else if (mesg.type === 'err') { | ||
|
||
const controller = controllerMap.get(mesg.id)!; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we need controllerMap
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, if my previous comment was unclear.
const handleRender = async (mesg: MessageReq & { type: 'render' }) => { | ||
const handleRender = async ( | ||
mesg: MessageReq & { type: 'render' }, | ||
stream: ReadableStream, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was thinking we can simply use rr.stream
. Any issue with it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you mean just use what's coming from the message?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, can we do it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll try my best!
}, | ||
}); | ||
|
||
mesg.stream?.pipeThrough(bridge); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This bridge is almost no-op, no?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you clarify what a no-op is?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I mean it doesn't do anything. "no operation".
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
isn't it converting array buffer also to Uint8Array?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As we no longer use post message for stream, we shouldn't need any conversion.
Nice, is there any resource that shows why postMessage needs the
conversion! Just curious.
And sure, I'll handle this.
…On Mon, 1 Jan 2024, 10:37 Daishi Kato, ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In packages/waku/src/lib/handlers/dev-worker-impl.ts
<#324 (comment)>:
> - const err =
- mesg.err instanceof Error ? mesg.err : new Error(String(mesg.err));
- controller.error(err);
+ const bridge = new TransformStream({
+ transform(chunk, controller) {
+ if (chunk instanceof Uint8Array) {
+ controller.enqueue(chunk);
+ } else if (chunk instanceof ArrayBuffer) {
+ controller.enqueue(new Uint8Array(chunk, 0, chunk.byteLength));
+ } else {
+ controller.error(new Error('Unexepected buffer type'));
+ }
+ },
+ });
+
+ mesg.stream?.pipeThrough(bridge);
As we no longer use post message for stream, we shouldn't need any
conversion.
—
Reply to this email directly, view it on GitHub
<#324 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AJBMICG3DZJEOXQSHXHRNBLYMJOBTAVCNFSM6AAAAABBEWC7IOVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTOOJZG4ZTMNBVGY>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
I don't know if it's related with conversion. Never mind. |
@@ -50,36 +44,21 @@ const handleRender = async (mesg: MessageReq & { type: 'render' }) => { | |||
searchParams: new URLSearchParams(rr.searchParamsString), | |||
method: rr.method, | |||
context: rr.context, | |||
body: rr.stream, | |||
body: rr.stream!, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How can we assure that it's not undefined?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To be honest, I thought about this, and I could not find a solution, it required so many types gymnastics which I then could not get it right. Let me know if you have an idea.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, yeah, it's optional. 204a34e should do for now. We can probably accept undefined
so that we don't need such syntax.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks great! Thanks for your work.
Resolves #308
We implemented our messaging method, but streams are already transferrable between the worker and the main thread. The only issue is that the types are not supporting them although the implementation does support them, that's why I used
as unknown as TransferListItem