-
Notifications
You must be signed in to change notification settings - Fork 29.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
for await & Readable #29428
Comments
I think this is probably correct (although probably surprising) since readable is supposed to be greedy. In order to get the behaviour you want you need to set for await (const d of Readable.from(generate(), { highWaterMark: 1 })) {
console.log(d)
} |
@mcollina: Thoughts? In the given example I believe the behaviour is correct since the generator is However, using a sync generator has the same behavior and I'm not sure that is correct? |
We might need to think about it a bit more. Look at the following: async function* generate() {
yield 1
yield 2
throw new Error('Boum')
}
;(async () => {
try {
for await (const d of generate()) {
console.log(d)
}
} catch (e) {
console.log(e)
}
})() The result is:
I think this might be something we can fix. Specifically, the following is making the iterator exit eagerly: node/lib/internal/streams/async_iterator.js Lines 61 to 86 in 63b056d
I think we might want to put the promise rejection in the node/lib/internal/streams/async_iterator.js Line 155 in 63b056d
@benjamingr what do you think? |
On second thought, I might be wrong, and this might be due for something else, or something that we cannot really fix. Streams are greedy into emitting Lines 1223 to 1236 in 63b056d
destroy eagerly and this is what triggers it.
As a short term measure, we should definitely document it. |
That streams do watermarking and that they are greedy :] To avoid the stream behaviour one can simple not convert their AsyncIterator to a stream. Now I do believe we can "fix" this (on the async iterator side) by only emitting the error and destroying after we are done emitting data on the iterator but honestly I am not sure that would be better from a stream consumer PoV. In fact it would likely be worse. So +1 on a docs change, -0.5 on changing the |
I think I’d be in favour of that, fwiw. |
That was my first intuition too but I am having a hard time coming up with a real use-case in which this (buffering) behaviour is better. Namely, in all cases I considered I would rather recover from the error sooner rather than later. The cases I came up with were:
Edit: the theme of these is having to reconnect to establish synchronisation with the producer later on anyway. On the other hand if I look at async iterators without streams, such use cases are easy to come up with and are abundant in reactive systems (like the whole server flow being accepting requests from clients in a for...await ) The biggest issue I see here is that there is missing data that we are dropping that the user might be interested in. I am also not sure how else we can expose it. Basically the constraints are:
|
Yeah, that’s what I’m thinking too. I’d also consider it more expected for (async) iterators to throw the exception only after providing data that was produced before the error, because in the typical (async) generator function approach, that’s when that exception would be have been created. |
Note that we can consider this to be a problem for our |
Note that running this other snippet produces a different output function makeStreamV2() {
return new Readable({
objectMode: true,
read() {
if (this.reading) {
return
}
this.reading = true
this.push(1)
this.push(2)
this.emit('error', 'boum')
}
})
}
;(async () => {
try {
for await (const d of makeStreamV2()) {
console.log(d)
}
} catch (e) {
console.log(e)
}
})()
|
@mcollina did we resolve this or is this something someone needs to further look into? |
This needs some decision to be made, and possibly some docs to be added, or the behavior changed. |
Currently from will eagerly buffer up items which means that errors are also eagerly encountered and items which are buffer when an error occurs will be discarded, which is inconsistent with how generators work. Fixes: nodejs#29428
Currently from will eagerly buffer up items which means that errors are also eagerly encountered and items which are buffer when an error occurs will be discarded, which is inconsistent with how generators work. Fixes: #29428 PR-URL: #33201 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]>
Currently from will eagerly buffer up items which means that errors are also eagerly encountered and items which are buffer when an error occurs will be discarded, which is inconsistent with how generators work. Fixes: #29428 PR-URL: #33201 Reviewed-By: Matteo Collina <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]>
Hi, I am trying to consume a readable with a
for await
loop.When I create the simple stream below, the console output the error straight away instead of logging the data events first. Is it the expected behavior
Version: v12.9.1
Platform: Darwin Kernel Version 18.7.0
the output is
instead of the expected
The text was updated successfully, but these errors were encountered: