stream: ensure pipeline always destroys streams

[ronag]

There was an edge case where an incorrect assumption was made
in regards to whether eos/finished means that the stream is
actually destroyed or not.

Checklist

• make -j4 test (UNIX), or vcbuild test (Windows) passes
• tests and/or benchmarks are included
• documentation is changed or added
• commit message follows commit guidelines

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/29328/

[ronag]

This is blocking backport of #31223 to node V13 which does not have autoDestroy enabled by default, thus some tests fail.

[mcollina]

lgtm

[mcollina]

If this is a backport it should target the relative branch (v13-staging in this case). Moreover I would expect to see a rather different change compared to what is on master.

[ronag]

If this is a backport it should target the relative branch (v13-staging in this case). Moreover I would expect to see a rather different change compared to what is on master.

This is not a backport. It fixes a problem on master for streams that do not autoDestroy. However, it should be backported as well and is required for backporting #31223 to v13. Since in master we changed default autoDestroy to true which is not (and will not be) included in v13.

[mcollina]

lgtm

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/29334/

[ronag]

Landed in b2be348

[arcanis]

@ronag It seems this diff has introduced a major regression in Got:
sindresorhus/got#1107 (comment)

Note that I don't know if it's Got that relied on a buggy behavior or if it's an actual regression, but I figured I should ref the issues together 🙂

[targos]

Should this land on v12.x-staging? If so, please open a backport PR.

[ronag]

Should not land on 12.x. Has indirect consequences that can cause breakage.

[mafintosh]

Does this work with things like a TCP stream where destroy actually destroys the underlying fd was finish still can mean its waiting for FIN ack?

[ronag]

Does this work with things like a TCP stream where destroy actually destroys the underlying fd was finish still can mean its waiting for FIN ack?

Sorry, I didn't get that?

[mafintosh]

simple example pipeline(socket, socket). This calls destroy automatically after eos now yes? Ie basically forces autoDestroy if I’m understanding that correctly.

For streams that have postfinish state this can be problematic. For example TCP (this may have changed since I looked at it last), the stream emits finish when all pending data has been sent but end() has a side effect. Its sends a FIN packet also. Now you auto destroy it meaning FIN wont get resend if that packet is lost.

My utp-native (tcp like stream over udp) module will def break because of this due to o/

[ronag]

For streams that have postfinish state this can be problematic. For example TCP (this may have changed since I looked at it last), the stream emits finish when all pending data has been sent but end() has a side effect. Its sends a FIN packet also. Now you auto destroy it meaning FIN wont get resend if that packet is lost.

I'm not familiar with this so I can't really answer. Maybe @addaleax or @jasnell?

If more work needs to be done, e.g. send a FIN packet, I think that should be handled by _destroy alternatively 'finish' should be delayed until the FIN stuff is complete (I think this might be what currently happens?).

Though, in v14 pipeline won't call destroy as it assumes that "standard" streams (such as net.Socket since #31806) will handle it themselves (#32158).

[mafintosh]

follow up: wont this break all duplex streams?

net.createServer(function (socket) {
  pipeline(socket, a)
  pipeline(b, socket)
})

Wont this destroy the socket when either of those pipelines are done?

[mafintosh]

I can see tcp uses autoDestroy so it that part is prob fine, re my first q

[ronag]

Wont this destroy the socket when either of those pipelines are done?

There has been follow up PRs that ensure that if:

• The first stream is writable, it won't be destroyed
• The last stream is readable, it won't be destroyed

The current implementation is a little bit messy, but if you look at this PR which cleans it up a bit it might be clear.

[ronag]

@mafintosh Anything still left unanswered?

[mafintosh]

@ronag i think the other pr solves my above issue. Didn’t realise this was older.

[mafintosh]

Have you tested this with http servers? Calling destroy on a req destroys the full socket I think

[ronag]

Have you tested this with http servers? Calling destroy on a req destroys the full socket I think

Yes, I noticed

[mafintosh]

Heads up, this breaks tar-stream also due to the same reasons as HTTP. Just verified on latest 13.

[mafintosh]

Test case

echo hello > hello.txt
echo world > world.txt
tar c hello.txt world.txt > test.tar

const tar = require('tar-stream')
const fs = require('fs')
const path = require('path')
const pipeline = require('stream').pipeline

fs.createReadStream('test.tar')
  .pipe(tar.extract())
  .on('entry', function (header, stream, done) {
    console.log(header.name) // in 13 this will only unpack one file due to
                                                 // pipeline destroying the entire stream
    pipeline(stream, fs.createWriteStream(path.join('/tmp', header.name)), done)
  })

[mafintosh]

Let me put that into a proper issue

[mafintosh]

Thanks for the navigation help @ronag :)