-
Notifications
You must be signed in to change notification settings - Fork 7.3k
child_process: stdout data is lost if not read quick enough #6595
Comments
I'm not sure whether this is a valid issue, other committers should weigh in. My observations:
/cc @isaacs @tjfontaine |
Thanks for the response.
The problem is there's no way to know when the 'close' event is going to come. The only way to prevent reading data after the 'close' event is to always read all the available data in the same tick that you receive the 'readable' event. And if a 'readable' event always forces you to read straight away, doesn't that make it effectively the same as the 'data' event? Both events result in an immediate read (but 'data' saves you having to write the line to do it). This makes it hard to process the data asynchronously. Currently I'm having to push the data into an array when it's available, then pop the data off again when my async function is ready for it. But if the process produces output quicker than the async func consumes it then my array/buffer grows indefinitely. Also, for comparison, net sockets do allow you to read the data after the 'close' event, and file streams only emit 'end' after all data has been read() |
I also hit this bug, and additionally noticed that you can miss the data entirely if you remove the var childProcess = require('child_process');
var inspect = require('util').inspect;
var child = childProcess.spawn('echo', ['1234'], {stdio: ['ignore', 'pipe',2]});
child.stdout.setEncoding('ascii');
child.on('exit', function(code) {
console.log("CHILD EXIT", code);
});
var reader = function() {
console.log(" (data available)");
var chunk;
while (null !== (chunk = child.stdout.read())) {
console.log('CHUNK: ' + inspect(chunk));
}
// pretend we're not ready to read data for a short time,
// even though this is in the same tick:
child.stdout.removeListener('readable', reader);
child.stdout.on('readable', reader);
};
child.stdout.on('readable', reader); Output:
So the stream emits 3 "readable" events, but none of them actually have any data available. So it seems that any non-flowing use of child streams cannot be trusted to not drop data. Also note that the (node v0.10.26) |
I've noticed similar behavior when using unbuffered output on a child process (specifically "python -u") and the 'data' event. When I do this, my node process reads a portion of the data, but drops the last several writes from the child process (which we do control, and we are even flushing stdout after each write just to make sure). Specifically, our child process outputs yaml documents over a period of time (status updates, essentially), and when the process exits not all of the yaml has been received by my 'data' listener prior to the 'close' event. As mentioned above, no data exists in child.stdout for reading, so it appears that the data is permanently missing. |
@trevnorris ... any thoughts on this one? |
I think there was a libuv patch that went in recently that helped with this. |
Unread data that is still in the child_process stdout stream when the process exits seems to be deleted. So if you attempt to read the data after the process has exited then the read() function returns null rather than the expected data.
For example, the following code always gets null from proc.stdout.read(), even though the child process is actually outputting data.
The problem here is this means you must read the data as quickly as it's produced, so there's no opportunity to put back pressure on the stream.
version: 0.10.22-linux-x64
The text was updated successfully, but these errors were encountered: