investigate flaky test-domain-error-types

[Trott]

• Test: test-domain-error-types
• Platform: ubuntu1804_sharedlibs_debug
• Console Output:

not ok 655 parallel/test-domain-error-types
  ---
  duration_ms: 0.812
  severity: crashed
  exitcode: -11
  stack: |-

• Build Links: https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_debug_x64/26102/consoleText

[Trott]

This one has been happening pretty relentlessly. It feels superfluous to add more examples, but here's one anyway.

https://ci.nodejs.org/job/node-test-commit-linux-containered/26170/nodes=ubuntu1804_sharedlibs_debug_x64/consoleText

not ok 735 parallel/test-domain-error-types
  ---
  duration_ms: 0.913
  severity: crashed
  exitcode: -11
  stack: |-
  ...

[Trott]

(Seems to only happen in debug builds but happens there a lot.)

[Trott]

The first time this appears to happen in v8 canary build 3 days ago: https://ci.nodejs.org/job/node-test-commit-node-v8/765/

That's a sub-task of https://ci.nodejs.org/job/node-update-v8-canary/1714/. Weirdly, the main task is green even though the subtask is red.

@nodejs/v8

[Trott]

The first time this shows up in an actual pull request CI seems to be https://ci.nodejs.org/job/node-test-commit-linux-containered/26102/ which is a CI run for #37991 which is surely unrelated. So...something in V8 that slipped in at some point?

[Trott]

(Or maybe some other change messed with garbage collection or something in a way that surfaces a bug that was masked previously?)

[Trott]

From #37970 (comment):

---

I'm fairly certain this is causing #38063 (frequent test-domain-error-types failures on debug build in CI).

Here are 8 clean debug CI runs on e0eb515 the commit immediately preceding this one in master.

• https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_debug_x64/26311/
• https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_debug_x64/26310/
• https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_debug_x64/26309/
• https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_debug_x64/26308/
• https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_debug_x64/26307/
• https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_debug_x64/26306/
• https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_debug_x64/26305/
• https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_debug_x64/26304/

But here it is failing on f0bf373, the commit in this PR:

• https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_debug_x64/26244/

I'm going to run it a bunch more times to confirm that wasn't a fluke.

Conversation to be continued in #38063, although I'll circle back here if I find out I'm wrong and this PR isn't the cause of that issue. I'm pretty sure it is, though.

---

[Trott]

Here are 8 more CI runs to verify that f0bf373 is indeed the change where this issue starts showing up:

• https://ci.nodejs.org/job/node-test-commit-linux-containered/26312/nodes=ubuntu1804_sharedlibs_debug_x64/
• https://ci.nodejs.org/job/node-test-commit-linux-containered/26313/nodes=ubuntu1804_sharedlibs_debug_x64/
• https://ci.nodejs.org/job/node-test-commit-linux-containered/26314/nodes=ubuntu1804_sharedlibs_debug_x64/
• https://ci.nodejs.org/job/node-test-commit-linux-containered/26315/nodes=ubuntu1804_sharedlibs_debug_x64/
• https://ci.nodejs.org/job/node-test-commit-linux-containered/26316/nodes=ubuntu1804_sharedlibs_debug_x64/
• https://ci.nodejs.org/job/node-test-commit-linux-containered/26317/nodes=ubuntu1804_sharedlibs_debug_x64/
• https://ci.nodejs.org/job/node-test-commit-linux-containered/26318/nodes=ubuntu1804_sharedlibs_debug_x64/
• https://ci.nodejs.org/job/node-test-commit-linux-containered/26319/nodes=ubuntu1804_sharedlibs_debug_x64/

[Trott]

Now they're all green. Starting to seem like I was wasting my time bisecting because the issue was a CI issue and not an issue with the test?

[Trott]

Optimistically closing.

[Trott]

Argh, just did a resume build on node-daily-master and it showed up again.

[Trott]

Crashed again today. https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_debug_x64/27078/testReport/(root)/test/parallel_test_domain_error_types/

[mhdawson]

Failed again today: https://ci.nodejs.org/job/node-test-commit-linux-containered/27102/

[joyeecheung]

This now has been consistently failing https://ci.nodejs.org/job/node-test-commit-arm-debug/ (now on the equinix-ubuntu1804_sharedlibs_container-arm64 containers)