AIX: Intermittent failure on pseudo-tty/test-tty-wrap #9728

mhdawson · 2016-11-21T22:54:09Z

Version: master and 4.x
Platform: AIX
Subsystem: pseudo-tty

Seen a few intermittent failures in the pseudo-tty/test-tty-wrap test today

https://ci.nodejs.org/job/node-test-commit-aix/1998/nodes=aix61-ppc64/
https://ci.nodejs.org/job/node-test-commit-aix/1988/nodes=aix61-ppc64/

length differs.
expect=2
actual=0
patterns:
pattern = ^hello\ world\ 1$
pattern = ^hello\ world\ 2$
outlines:
not ok 1242 pseudo-tty/test-tty-wrap
  ---
  duration_ms: 0.150
  severity: fail
  stack: |-
  ...

The text was updated successfully, but these errors were encountered:

mhdawson · 2016-11-21T22:56:23Z

stress run - https://ci.nodejs.org/job/node-stress-single-test/nodes=aix61-ppc64/1067/

Trott · 2016-11-21T23:07:59Z

/cc @Fishrock123

Fishrock123 · 2016-11-21T23:21:28Z

I am very confused. What? How?

Fishrock123 · 2016-11-21T23:22:33Z

Oh I see, no output is produced? Oh dear.

I guess you've triggered the exit bug with load alone?? ¯\_(ツ)_/¯

gibfahn · 2016-11-22T11:00:12Z

cc/ @nodejs/platform-aix @Akpotohwo

mhdawson · 2016-11-22T13:47:02Z

Passed 200 runs in stress build.

Akpotohwo · 2016-11-25T15:14:04Z

Failed 24 times in 3100 on one of our aix machines... It's the same failure as shown earlier.

mhdawson · 2016-11-25T15:50:39Z

Good to hear you can recreate.

mhdawson · 2016-12-08T14:09:30Z

Failed again in the build today. https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix61-ppc64/2433/

@Akpotohwo any update on the investigation ?

mhdawson · 2016-12-15T17:17:19Z

Failed again https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix61-ppc64/2566/console

Trott · 2016-12-23T21:40:43Z

Again today:

https://ci.nodejs.org/job/node-test-commit-aix/2736/nodes=aix61-ppc64/console:

length differs.
expect=2
actual=0
patterns:
pattern = ^hello\ world\ 1$
pattern = ^hello\ world\ 2$
outlines:
not ok 1288 pseudo-tty/test-tty-wrap
  ---
  duration_ms: 0.158
  severity: fail
  stack: |-

Trott · 2016-12-23T21:41:49Z

I guess you've triggered the exit bug with load alone?? ¯_(ツ)_/¯

@Fishrock123 (or anyone else that knows): Can you elaborate a bit on "the exit bug"? Is there a link or something?

Trott · 2016-12-23T21:45:18Z

Should this be marked as flaky on AIX similar to no_dropped_stdio and no_interleaved_stdio? Is it likely failing for the same reasons?

mhdawson · 2017-01-03T21:55:19Z

@Trott Given that we have seen relatively regularly I'll plan to ahead and mark it as flaky.

mhdawson · 2017-01-04T19:04:21Z

@gibfahn I'll submit a PR to excluded, can you follow up on where our investigation stands on this one.

We have had nodejs#9728 open for a while but the frequency of the failures seems to be such that we should mark it as flaky while we continue to investigate.

mhdawson · 2017-01-04T20:41:34Z

PR to exclude: https://ci.nodejs.org/job/node-test-pull-request/5702/

gibfahn · 2017-01-04T20:55:10Z

PR: #10618

We have had #9728 open for a while but the frequency of the failures seems to be such that we should mark it as flaky while we continue to investigate. PR-URL: #10618 Reviewed-by: Colin Ihrig <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Sakthipriyan Vairamani <[email protected]> Reviewed-By: James M Snell <[email protected]>

gibfahn · 2017-01-16T14:13:46Z

@Akpotohwo if you're looking at pseudo-tty test failures on AIX it might be worth looking at #7973 as well.

We have had nodejs#9728 open for a while but the frequency of the failures seems to be such that we should mark it as flaky while we continue to investigate. PR-URL: nodejs#10618 Reviewed-by: Colin Ihrig <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Sakthipriyan Vairamani <[email protected]> Reviewed-By: James M Snell <[email protected]>

gireeshpunathil · 2017-02-16T11:21:31Z

Investigated further, and got these inferences:

There are 2 problems with the pseudo-tty tests:
i) child (node) data getting truncated at parent (python),
ii) hang in parent.

This subject test (tty-wrap) mostly undergoes (i) and occasionally, (ii).

Root cause for (i):

Passing behavior:

2687288:        16056741: kopen("/dev/pts/4", O_RDWR)           = 14
...
2687288:        16056741: kwrite(14, " h e l l o   w o r l d  ".., 14) = 14
4850562: 10945517: kread(3, " h e l l o   w o r l d  ".., 9999) = 15
2687288:        16056741: kwrite(14, " h e l l o   w o r l d  ".., 14) = 14
4850562: 10945517: kread(3, " h e l l o   w o r l d  ".., 9999) = 15

Failing behavior:

852692: kopen("/dev/pts/4", O_RDWR)                     = 14
...
852692: kwrite(14, " h e l l o   w o r l d  ".., 14)    = 14
852692: kwrite(14, " h e l l o   w o r l d  ".., 14)    = 14
852692: _exit(0)
4325988: 13435549: kread(3, "\0\0\0\0\0\0\0\0\0\0\0\0".., 9999) Err#5  EIO

By the time the child is completely terminated, having its side of the fd closed. Subsequent read by the parent results in EIO, equivalent to a broken pipe.

Hang root cause is not identified, but recreated in a small python + C test case:

bash-4.3$ cat parent.py
#!/usr/bin/env python
import errno
import os
import pty
from subprocess import Popen, STDOUT

master_fd, slave_fd = pty.openpty()
proc = Popen(['./a.out'],stdout=slave_fd, close_fds=True)
os.close(slave_fd)
data = os.read(master_fd, 512)
print('got ' + repr(data))

bash-4.3$ cat child.cc
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>

int main()
{
        printf("hello world\n");
        exit(0);
}
bash-4.3$

This issue is raised with python community: python 29545 , but no one seem to have picked it up yet.

For (i): I see an issue with the test case - it assumes that the parent can read from the fd irrespective of the child's life span and exit status, however it works always in Linux, magically. AIX uses different process scheduling policies that Linux, and gets affected by this. We have seen many issues in other tests, failure reason a function of the order of execution of child vs. parent soon after the fork.

A 100 ms delay after the child write and before exit consistently resolves the issue. This test and no_dropped_stdio.js - 7973 are affected by this.

@mhdawson @Fishrock123 @gibfahn - please let me know what you think, if you agree I will come up with a PR to this effect.

gibfahn · 2017-02-16T13:49:38Z

@gireeshpunathil if a 100ms delay fixes this consistently on AIX then that seems reasonable to me.

mhdawson · 2017-02-16T20:47:02Z

Is there a way to have the parent signal the child when it has completed its write so that we can avoid a timeout ? For example if the parent could write 'done' to the childs stdin when it had read. Delays are always problematic so want to be sure we don't have any other way to achieve the same result.

gireeshpunathil · 2017-02-17T08:30:19Z

@mhdawson - while it is technically possible for the parent to acknowledge the data and make it a mandate to receive the ack for child to exit, the python parent (testcfg.py) is the main drver also for the rest of the tests in test/pseudo-tty, furthermore it delegates the child spawning work to tools/test.py which is much more generic (drives the whole test). So I guess changing the parent logic to ack. can have side effects elsewhere.
/cc @Fishrock123

gireeshpunathil · 2017-03-02T11:00:36Z

So the approach of synchronizing between parent child has issues:

The python driver's child spawing routine (RunProcess()) is not designed to write back to child.
On the other hand, none of the children (node test cases) are designed to live indefinitely or listen for events / data from parent.
If we re-design this to communicate between them, the tests will hang. Here is why:

The only way parent finishes reading data from the child is when it encounters EIO (when child exits) and break-exit the tight read loop. (or timeout expiry)
There is no way for the parent to otherwise understand the child has finished writing. This is because each child has different data and difference sequence of writing the data.
If we insert a process.stdin.on('data') event on child in expecation of an acknowledgement from parent, this is never going to happen - the parent continues to attempt reading until it times out - as the read error will never occur (child is alive)

We can send a 'special data' indicating the write-finish from the child to circumvent this, but given that this will warrant a change in all the children (node test cases) and the issue occurs only in AIX, I would think we address the race condition in the failing tests alone, by adding a reasonable delay before exit. [ we have done this in many other test cases where AIX-(other os) disparity was visible w.r.t process scheduling.

possibilities:

Modify the python parent to read a 'special data' to terminate the read loop. Then acknowledge the data reception back to the child. Modify 'every' child programs to add the 'special data' at the end of the test, and then setup a read loop to receive the ACK, before exiting.
Modify affected test cases to provide additional delay, supplying necessary leeway for the parent to read the data before exiting.

I am fine in either way, suggestions are much appreciated.
/cc @mhdawson @Fishrock123 @gibfahn

gibfahn · 2017-03-02T11:23:33Z

I'm still +1 on adding a delay, it seems like a vastly simpler option.

We have had #9728 open for a while but the frequency of the failures seems to be such that we should mark it as flaky while we continue to investigate. PR-URL: #10618 Reviewed-by: Colin Ihrig <[email protected]> Reviewed-By: Rich Trott <[email protected]> Reviewed-By: Sakthipriyan Vairamani <[email protected]> Reviewed-By: James M Snell <[email protected]>

gibfahn · 2017-03-30T09:31:17Z

@gireeshpunathil so to clarify, would adding a delay fix the issue for pseudo-tty/test-tty-wrap, or do we need to get the underlying Python bug fixed first?

gireeshpunathil · 2017-03-30T09:49:22Z

@gibfahn - as per clarified here - this issue is root caused by the python bug in question, and cannot be circumvented through time delay. The python parent gets trapped in the system on an fd close call, without recovery

Trott · 2017-07-31T04:34:42Z

This hasn't recurred in some time. Feel free to re-open (or leave a comment requesting that it be re-opened) if you disagree, but I'm inclined to close at this time. I'm just tidying up and not acting on a super-strong opinion or anything like that.

sam-github · 2019-06-07T19:51:47Z

This is still referenced in

node/test/pseudo-tty/pseudo-tty.status

Line 4 in 6447be6

# being investigated under https://github.com/nodejs/node/issues/9728

If we close it, shouldn't we also remove the reference to it? Or perhaps change the ref to nodejs/build#1820?

Or perhaps, I misunderstand the meaning of test-tty-wrap : FAIL, PASS? That seems to imply the test is considered sucessful if it passes or if it fails, which seems to cover all the options, and not be so useful.

Trott · 2019-06-08T02:17:35Z

@sam-github Yes, you are correct: That entry should be removed. #28129

The test is believed to no longer be unreliable on AIX. Remove the flaky designation from the appropriate status file. Closes: nodejs#9728 PR-URL: nodejs#28129 Fixes: nodejs#9728 Reviewed-By: Gireesh Punathil <[email protected]> Reviewed-By: Richard Lau <[email protected]> Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]> Reviewed-By: Trivikram Kamat <[email protected]> Reviewed-By: James M Snell <[email protected]>

The test is believed to no longer be unreliable on AIX. Remove the flaky designation from the appropriate status file. Closes: #9728 PR-URL: #28129 Fixes: #9728 Reviewed-By: Gireesh Punathil <[email protected]> Reviewed-By: Richard Lau <[email protected]> Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]> Reviewed-By: Trivikram Kamat <[email protected]> Reviewed-By: James M Snell <[email protected]>

mhdawson changed the title ~~AIX: Intermittent failure on~~ AIX: Intermittent failure on pseudo-tty/test-tty-wrap Nov 21, 2016

mscdex added the aix Issues and PRs related to the AIX platform. label Nov 21, 2016

Trott added the tty Issues and PRs related to the tty subsystem. label Nov 21, 2016

mhdawson added a commit to mhdawson/io.js that referenced this issue Jan 4, 2017

test: mark test-tty-wrap as flaky for AIX

d2cbcf2

We have had nodejs#9728 open for a while but the frequency of the failures seems to be such that we should mark it as flaky while we continue to investigate.

mhdawson mentioned this issue Jan 4, 2017

test: mark test-tty-wrap as flaky for AIX #10618

Closed

2 tasks

gibfahn mentioned this issue Feb 24, 2017

AIX CI failure: pseudo-tty/test-stderr-stdout-handle-sigwinch #11541

Closed

gireeshpunathil mentioned this issue Mar 6, 2017

test: delay child exit in AIX for pseudo-tty tests #11715

Closed

4 tasks

Trott closed this as completed Jul 31, 2017

richardlau mentioned this issue Jun 4, 2019

Intermittent tty timeouts on AIX nodejs/build#1820

Closed

sam-github reopened this Jun 7, 2019

sam-github added the flaky-test Issues and PRs related to the tests with unstable failures on the CI. label Jun 7, 2019

Trott mentioned this issue Jun 8, 2019

test: remove test-tty-wrap from status file #28129

Merged

2 tasks

Trott closed this as completed in #28129 Jun 10, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AIX: Intermittent failure on pseudo-tty/test-tty-wrap #9728

AIX: Intermittent failure on pseudo-tty/test-tty-wrap #9728

mhdawson commented Nov 21, 2016

mhdawson commented Nov 21, 2016 •

edited

Loading

Trott commented Nov 21, 2016

Fishrock123 commented Nov 21, 2016

Fishrock123 commented Nov 21, 2016 •

edited

Loading

gibfahn commented Nov 22, 2016

mhdawson commented Nov 22, 2016

Akpotohwo commented Nov 25, 2016

mhdawson commented Nov 25, 2016

mhdawson commented Dec 8, 2016

mhdawson commented Dec 15, 2016

Trott commented Dec 23, 2016 •

edited

Loading

Trott commented Dec 23, 2016

Trott commented Dec 23, 2016

mhdawson commented Jan 3, 2017

mhdawson commented Jan 4, 2017

mhdawson commented Jan 4, 2017

gibfahn commented Jan 4, 2017

gibfahn commented Jan 16, 2017

gireeshpunathil commented Feb 16, 2017 •

edited by gibfahn

Loading

gibfahn commented Feb 16, 2017

mhdawson commented Feb 16, 2017

gireeshpunathil commented Feb 17, 2017

gireeshpunathil commented Mar 2, 2017

gibfahn commented Mar 2, 2017

gibfahn commented Mar 30, 2017

gireeshpunathil commented Mar 30, 2017

Trott commented Jul 31, 2017

sam-github commented Jun 7, 2019

Trott commented Jun 8, 2019

AIX: Intermittent failure on pseudo-tty/test-tty-wrap #9728

AIX: Intermittent failure on pseudo-tty/test-tty-wrap #9728

Comments

mhdawson commented Nov 21, 2016

mhdawson commented Nov 21, 2016 • edited Loading

Trott commented Nov 21, 2016

Fishrock123 commented Nov 21, 2016

Fishrock123 commented Nov 21, 2016 • edited Loading

gibfahn commented Nov 22, 2016

mhdawson commented Nov 22, 2016

Akpotohwo commented Nov 25, 2016

mhdawson commented Nov 25, 2016

mhdawson commented Dec 8, 2016

mhdawson commented Dec 15, 2016

Trott commented Dec 23, 2016 • edited Loading

Trott commented Dec 23, 2016

Trott commented Dec 23, 2016

mhdawson commented Jan 3, 2017

mhdawson commented Jan 4, 2017

mhdawson commented Jan 4, 2017

gibfahn commented Jan 4, 2017

gibfahn commented Jan 16, 2017

gireeshpunathil commented Feb 16, 2017 • edited by gibfahn Loading

gibfahn commented Feb 16, 2017

mhdawson commented Feb 16, 2017

gireeshpunathil commented Feb 17, 2017

gireeshpunathil commented Mar 2, 2017

gibfahn commented Mar 2, 2017

gibfahn commented Mar 30, 2017

gireeshpunathil commented Mar 30, 2017

Trott commented Jul 31, 2017

sam-github commented Jun 7, 2019

Trott commented Jun 8, 2019

mhdawson commented Nov 21, 2016 •

edited

Loading

Fishrock123 commented Nov 21, 2016 •

edited

Loading

Trott commented Dec 23, 2016 •

edited

Loading

gireeshpunathil commented Feb 16, 2017 •

edited by gibfahn

Loading