Tracking issue: stdio problems #6980

Fishrock123 · 2016-05-25T21:24:24Z

This is a meta-issue for tracking multiple related but distinct stderr/stdout issues.

Below are checklists indicating the progress of these issues & pull requests.
A checkmark indicates it is resolved (or merged).

This issue encompasses the following issues:

Chunked stdout/stderr drops writes if terminated early. Chunked stdout/stderr drops writes if terminated early. #784
Interleaved stdout/stderr on terminals (TTYs)
- Part of: stdio buffered writes (chunked) issues & process.exit() truncation stdio buffered writes (chunked) issues & process.exit() truncation #6456
Missing or truncated error message Missing or truncated error message #6867

And the following pull requests:

process: flush stdout/stderr upon process.exit() process: flush stdout/stderr upon process.exit() #6773
tty: set the handle to blocking mode tty: set the handle to blocking mode #6816
tty: use blocking mode on OS X tty: use blocking mode on OS X #6895
unix: add uv_flush_sync unix: add uv_flush_sync libuv/libuv#876
unix,stream: fix getting the correct fd for a handle unix,stream: fix getting the correct fd for a handle #6838
- Should be backported to v4.x (and maybe v5.x)

The current item in focus is:
n/a

The TTY (Terminal) issues have been resolved. Pipes still remain a problem.

This pr was a result of the discussion in #6968 to better track all the efforts going on here.

[EDIT by @Trott: Did some edits around the 6773 PR checklist item, but undid them]

The text was updated successfully, but these errors were encountered:

Fishrock123 · 2016-05-25T21:28:56Z

Current plan is to make a release candidate (today/tomorrow) as discussed in #6934 with the OS X patch from #6895 to fix the largest issue (OS X stdio interleaving to TTYs) asap.

kzc · 2016-05-25T21:36:33Z

You can add this one to the list:

#6867 Missing or truncated error message

It's a variation of the process.exit() stdio truncating problem but with uncaught exceptions.

Edit: reproduced on node 5.x on Linux. Same issue likely exists on node 4.x.

kzc · 2016-05-25T22:33:56Z

This related merged PR for process.stdout._handle.fd should also be mentioned here for completeness:

unix,stream: fix getting the correct fd for a handle #6838

It ought to be back ported to node 5.x and 4.x.

Fishrock123 · 2016-05-25T23:23:06Z

No guarantees on v5.x since it's now in critical-maintinence-only mode, but we'll try to get in v4.x soon.

Fishrock123 · 2016-05-26T23:41:29Z

RC for #6895 building at https://ci-release.nodejs.org/job/iojs+release/921/ (Updated: new ci (2))

Fishrock123 · 2016-05-27T02:38:36Z

I have an RC up at https://nodejs.org/download/rc/v6.2.1-rc.1/ that should fix the OS X problems.
Please test.

cc-ing the people from other threads: @isaacs, @nodejs/npm, @Qix-, @eljefedelrodeodeljefe, @bcoe, @mikemaccana, @mgol

Qix- · 2016-05-27T03:25:08Z

@Fishrock123

Qix- is missing the Overall/Read permission

Fishrock123 · 2016-05-27T21:16:01Z

@Qix- Are you getting that attempting to access the directory, or the files themselves?

(or is node making that error?)

Qix- · 2016-05-27T22:54:07Z

@Fishrock123 Works now, not sure what changed. :) I'll give it a try.

OS X has a tiny 1kb hard-coded buffer size for stdout / stderr to TTYs (terminals). Output larger than that causes chunking, which ends up having some (very small but existent) delay past the first chunk. That causes two problems: 1. When output is written to stdout and stderr at similar times, the two can become mixed together (interleaved). This is especially problematic when using control characters, such as \r. With interleaving, chunked output will often have lines or characters erased unintentionally, or in the wrong spots, leading to broken output. CLI apps often extensively use such characters for things such as progress bars. 2. Output can be lost if the process is exited before chunked writes are finished flushing. This usually happens in applications that use `process.exit()`, which isn't infrequent. See nodejs#6980 for more info. This became an issue as result of the Libuv 1.9.0 upgrade. A fix to an unrelated issue broke a hack previously required for the OS X implementation. This resulted in an unexpected behavior change in node. The 1.9.0 upgrade was done in c3cec1e, which was included in v6.0.0. Full details of the Libuv issue that induced this are at nodejs#6456 (comment) Refs: nodejs#1771 Refs: nodejs#6456 Refs: nodejs#6773 Refs: nodejs#6816 PR-URL: nodejs#6895 Reviewed-By: Rod Vagg <[email protected]> Reviewed-By: Anna Henningsen <[email protected]>

Many thanks to thefourtheye and addaleax who helped make the python bits of this possible. See nodejs#6980 for more info regarding the related TTY issues. Refs: nodejs#6456 Refs: nodejs#6773 Refs: nodejs#6816 PR-URL: nodejs#6895 Reviewed-By: Rod Vagg <[email protected]> Reviewed-By: Anna Henningsen <[email protected]>

OS X has a tiny 1kb hard-coded buffer size for stdout / stderr to TTYs (terminals). Output larger than that causes chunking, which ends up having some (very small but existent) delay past the first chunk. That causes two problems: 1. When output is written to stdout and stderr at similar times, the two can become mixed together (interleaved). This is especially problematic when using control characters, such as \r. With interleaving, chunked output will often have lines or characters erased unintentionally, or in the wrong spots, leading to broken output. CLI apps often extensively use such characters for things such as progress bars. 2. Output can be lost if the process is exited before chunked writes are finished flushing. This usually happens in applications that use `process.exit()`, which isn't infrequent. See #6980 for more info. This became an issue as result of the Libuv 1.9.0 upgrade. A fix to an unrelated issue broke a hack previously required for the OS X implementation. This resulted in an unexpected behavior change in node. The 1.9.0 upgrade was done in c3cec1e, which was included in v6.0.0. Full details of the Libuv issue that induced this are at #6456 (comment) Refs: #1771 Refs: #6456 Refs: #6773 Refs: #6816 PR-URL: #6895 Reviewed-By: Rod Vagg <[email protected]> Reviewed-By: Anna Henningsen <[email protected]>

Many thanks to thefourtheye and addaleax who helped make the python bits of this possible. See #6980 for more info regarding the related TTY issues. Refs: #6456 Refs: #6773 Refs: #6816 PR-URL: #6895 Reviewed-By: Rod Vagg <[email protected]> Reviewed-By: Anna Henningsen <[email protected]>