fix: worker being killed after being spawned and other worker bugs #13107

phawxby · 2022-08-08T22:19:33Z

This adds a test case for the issue described above and can be demonstrated with.

cd e2e/worker-restarting 
node ../../packages/jest-cli/bin/jest.js

phawxby · 2022-08-09T12:50:12Z

@backmask @SimenB I've found the problem and now I've seen it 🤦

I'm writing some more/better tests to try and catch this in future and should have the PR ready to go in an hour or so

…ng on

phawxby · 2022-08-10T06:16:17Z

PR is mostly done and I solved a few other bugs that I found while I was at it. I'm nos just trying to resolve the Windows weirdness that doesn't occur on my Windows machine.

e2e/worker-restarting/package.json

e2e/__tests__/worker-restarting.test.ts

…g noise

phawxby · 2022-08-11T07:05:24Z

FINALLY!!!

SimenB

yay!

packages/jest-worker/src/types.ts

SimenB · 2022-08-11T08:39:48Z

packages/jest-worker/src/workers/ChildProcessWorker.ts

+    const silent = this._options.silent ?? true;
+
+    if (!silent) {
+      console.warn('Unable to detect out of memory event if silent === false');


we should only log this if _childIdleMemoryUsageLimit is passed I think? Then we can probably also make it an error instead of warning

same comment applies to thread worker - shouldn't we only monitor for OOM if the option is passed, otherwise keep old behavior?

we should only log this if _childIdleMemoryUsageLimit is passed I think? Then we can probably also make it an error instead of warning

Yes to the first bit, i'm inclined to say no to the second. It can be useful for debugging to see the child output when a memory limit is in place, for example to see the exact thrown error message.

same comment applies to thread worker - shouldn't we only monitor for OOM if the option is passed, otherwise keep old behavior?

Actually, ignore my previous comment....

Detecting an out of memory event and the worker crashing is independent of monitoring of the idle memory usage and rebooting as necessary. The _childIdleMemoryUsageLimit applies to the latter case, the former is building and improving on #13054

Do I need better docs to explain that?

I've added some comments to explain

SimenB · 2022-08-11T08:40:42Z

packages/jest-worker/src/workers/ChildProcessWorker.ts

-    const handler = (chunk: any) => {
-      if (this._state !== WorkerStates.OUT_OF_MEMORY) {
-        let str: string | undefined = undefined;
+    this._detectOutOfMemoryCrash();


here as well, only if _childIdleMemoryUsageLimit is passed?

SimenB · 2022-08-11T08:42:37Z

packages/jest-worker/src/workers/WorkerAbstract.ts

+
+export default abstract class WorkerAbstract
+  extends EventEmitter
+  implements Pick<WorkerInterface, 'waitForWorkerReady' | 'state'>


Create a new interface within types.ts instead of Picking here?

It's an extremely limited use case just to prevent the class throwing typing errors, it didn't really seem necessary to create a new class just for that as it would have no utility elsewhere. Someone could create a different worker type in future and create a different abstract with totally different base props.

packages/jest-worker/src/workers/WorkerAbstract.ts

packages/jest-worker/src/workers/__tests__/WorkerEdgeCases.test.js

phawxby · 2022-08-11T10:02:00Z

These test failures don't appear to be related to my changes, are they known to be flaky?

SimenB · 2022-08-11T12:01:57Z

packages/jest-worker/src/workers/WorkerAbstract.ts

@@ -44,12 +48,14 @@ export default abstract class WorkerAbstract

    if (typeof options.on === 'object') {
      for (const [event, handlers] of Object.entries(options.on)) {
-        if (Array.isArray(handlers)) {
+        // Can't do Array.isArray on a ReadonlyArray<T>.
+        // https://github.com/microsoft/TypeScript/issues/17002


SimenB · 2022-08-11T12:02:55Z

These test failures don't appear to be related to my changes, are they known to be flaky?

Yeah, at times 😞 I restarted the failing builds now

SimenB

looks great!

packages/jest-worker/src/workers/__tests__/WorkerEdgeCases.test.js

phawxby · 2022-08-11T12:15:36Z

Hopefully this is the PR done

SimenB · 2022-08-11T13:41:59Z

https://github.com/facebook/jest/releases/tag/v29.0.0-alpha.5

phawxby · 2022-08-11T14:04:25Z

@SimenB we're getting closer, tests are running locally in our main project on the alpha release and heap usage is remaining stable. What has me confused is I'm getting a tonne of snapshot failures, not sure if there's beena. chance to how snapshots are encoded between 27 - 29 which is causing it. Investigating now

 PASS   unit  src/common/helpers/context/Site/mapSiteUrlContextData.test.ts (8.602 s, 240 MB heap size)
 PASS   unit  src/common/helpers/support/getFiltersForContent/__tests__/index.test.ts (5.026 s, 228 MB heap size)
 PASS   unit  src/common/helpers/context/Site/SiteContext.test.ts (8.986 s, 253 MB heap size)
 PASS   unit  src/common/helpers/sku/__tests__/getSkuPricing.test.ts (8.824 s, 240 MB heap size)
 PASS   unit  src/common/helpers/tokens/lib/__tests__/traverseTokens.test.ts (7.575 s, 202 MB heap size)
 PASS   unit  src/common/helpers/tokens/xrx/category/__tests__/index.test.ts (7.508 s, 206 MB heap size)
 PASS   unit  src/common/helpers/contentful/__tests__/getContentfulCascade.test.ts (7.459 s, 211 MB heap size)
 PASS   unit  src/common/helpers/support/__tests__/buildSupportDownloadUrl.test.ts (7.689 s, 208 MB heap size)
 PASS   unit  src/common/helpers/sitemap/generators/__tests__/SupportDndProductSitemapGenerator.test.ts (7.701 s, 213 MB heap size)
 PASS   unit  src/common/helpers/product/findRelatedProducts/index.test.ts (16.946 s, 238 MB heap size)
 PASS   unit  src/common/helpers/locale/localeMatch/__tests__/beMatch.test.ts (98 MB heap size)
 PASS   unit  src/common/helpers/locale/localeMatch/__tests__/xeroxMatch.test.ts (109 MB heap size)
 PASS   unit  src/common/helpers/product/__tests__/performProductFooterNavCascade.test.ts (8.692 s, 230 MB heap size)
 PASS   unit  src/common/helpers/sitemap/generators/__tests__/MarketingProductSkuSitemapGenerator.test.ts (233 MB heap size)
 PASS   unit  src/common/helpers/contentful/__tests__/getMasterGlossary.test.ts (8.866 s, 241 MB heap size)
 PASS   unit  src/common/helpers/context/Site/mapSiteHeaderFooterContextData/__tests__/mapSecondaryLink.test.ts (8.78 s, 239 MB heap size)
 PASS   unit  src/common/helpers/url/__tests__/chaseUrl.test.ts (8.204 s, 207 MB heap size)
 PASS   unit  src/common/helpers/support/__tests__/getPagesForProduct.test.ts (8.222 s, 205 MB heap size)
 PASS   unit  src/common/helpers/akamai/__tests__/invalidate.test.ts (105 MB heap size)
 PASS   unit  src/common/helpers/context/Locale/mapEntryToLocaleContext.test.ts (144 MB heap size)
...

Look at that heap usage :D

phawxby · 2022-08-11T14:14:36Z

Ah, it appears to be this change.
#13036

This is looking good but I'd prefer more external validation

SimenB · 2022-08-11T14:42:28Z

Ah, it appears to be this change. #13036

Yep.

https://jestjs.io/docs/next/upgrading-to-jest29#snapshot-format

phawxby · 2022-08-15T13:51:03Z

@SimenB I know you were hoping for a stable release last week had I not broken jest, I assume that means a stable release this week instead?

SimenB · 2022-08-15T14:05:22Z

Yep, that's the plan! Mostly waiting for a resolution on #12856

github-actions · 2022-09-15T00:09:56Z

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.

test: ad integration test case for worker not executing test

71d0080

facebook-github-bot added the cla signed label Aug 8, 2022

phawxby mentioned this pull request Aug 8, 2022

fix: pipe workerIdleMemoryLimit to globalConfig #13106

Merged

phawxby changed the title ~~test: ad integration test case for worker not executing test~~ test: fix worker being killed after being spawned Aug 9, 2022

phawxby added 19 commits August 9, 2022 14:02

test: add test case for worker being killed just once

8c6ab73

fix: worker being killed after spawning

86acdf1

Merge branch 'master' into worker-restart-bug

718425b

fix: missing license

de0c0e8

fix: flaky tests

d83aa84

chore: add some diagnostic logging

9435d0b

chore: break apart tests to see where the problem is occuring

29e1193

fix: handling of work ready status

c23f581

chore: add fallback timeout

e8a8058

refactor: add abstract class with common functions and vars

4e0b399

fix: restarting on init

790f772

refactor: event emitting is a much cleaner way of tracking what's goi…

b4151ea

…ng on

chore: better handle shutdown events

2753069

fix: only set out of memory when necessary

617e102

chore: windows debugging

28230da

chore: try and debug single test

60b5b1b

chore: remove debugging

a9a3779

chore: remove logging

cdf9627

chore: simplify tests and move more to abstract worker

7a79c92

SimenB reviewed Aug 10, 2022

View reviewed changes

e2e/worker-restarting/package.json Show resolved Hide resolved

phawxby added 2 commits August 10, 2022 07:57

chore: only run flaky test

11fd6f8

chore: enable tests again

55f4097

SimenB reviewed Aug 10, 2022

View reviewed changes

e2e/__tests__/worker-restarting.test.ts Outdated Show resolved Hide resolved

fix: missing snapshot

723951b

phawxby added 4 commits August 10, 2022 22:29

chore: use up memory quicker and more logging

8e4a31e

chore: try removing force exit to see if that helps with the confusin…

b974eea

…g noise

chore: different strat

1e1023d

chore: remove debugging

ab2931d

phawxby marked this pull request as ready for review August 11, 2022 07:05

phawxby requested a review from SimenB August 11, 2022 07:05

chore: revert to silent worker

c610bdb

phawxby changed the title ~~test: fix worker being killed after being spawned~~ fix: worker being killed after being spawned and other worker bugs Aug 11, 2022

Update CHANGELOG.md

ee20013

SimenB reviewed Aug 11, 2022

View reviewed changes

chore: pr feedback

bd786b1

phawxby requested a review from SimenB August 11, 2022 11:01

SimenB mentioned this pull request Aug 11, 2022

[Feature]: Add a setting to limit worker memory utilization #12893

Closed

SimenB reviewed Aug 11, 2022

View reviewed changes

SimenB approved these changes Aug 11, 2022

View reviewed changes

packages/jest-worker/src/workers/__tests__/WorkerEdgeCases.test.js Outdated Show resolved Hide resolved

packages/jest-worker/src/workers/__tests__/WorkerEdgeCases.test.js Outdated Show resolved Hide resolved

chore: pr changes

613fcf2

phawxby requested a review from SimenB August 11, 2022 12:15

SimenB merged commit d6ad15b into jestjs:main Aug 11, 2022

github-actions bot locked as resolved and limited conversation to collaborators Sep 15, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: worker being killed after being spawned and other worker bugs #13107

fix: worker being killed after being spawned and other worker bugs #13107

phawxby commented Aug 8, 2022 •

edited

Loading

phawxby commented Aug 9, 2022

phawxby commented Aug 10, 2022

phawxby commented Aug 11, 2022

SimenB left a comment

SimenB Aug 11, 2022

SimenB Aug 11, 2022

phawxby Aug 11, 2022

phawxby Aug 11, 2022 •

edited

Loading

phawxby Aug 11, 2022

SimenB Aug 11, 2022

SimenB Aug 11, 2022

phawxby Aug 11, 2022

phawxby commented Aug 11, 2022

SimenB Aug 11, 2022

SimenB commented Aug 11, 2022

SimenB left a comment

phawxby commented Aug 11, 2022

SimenB commented Aug 11, 2022

phawxby commented Aug 11, 2022 •

edited

Loading

phawxby commented Aug 11, 2022

SimenB commented Aug 11, 2022 •

edited

Loading

phawxby commented Aug 15, 2022

SimenB commented Aug 15, 2022

github-actions bot commented Sep 15, 2022

fix: worker being killed after being spawned and other worker bugs #13107

fix: worker being killed after being spawned and other worker bugs #13107

Conversation

phawxby commented Aug 8, 2022 • edited Loading

phawxby commented Aug 9, 2022

phawxby commented Aug 10, 2022

phawxby commented Aug 11, 2022

SimenB left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

phawxby Aug 11, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

phawxby commented Aug 11, 2022

Choose a reason for hiding this comment

SimenB commented Aug 11, 2022

SimenB left a comment

Choose a reason for hiding this comment

phawxby commented Aug 11, 2022

SimenB commented Aug 11, 2022

phawxby commented Aug 11, 2022 • edited Loading

phawxby commented Aug 11, 2022

SimenB commented Aug 11, 2022 • edited Loading

phawxby commented Aug 15, 2022

SimenB commented Aug 15, 2022

github-actions bot commented Sep 15, 2022

phawxby commented Aug 8, 2022 •

edited

Loading

phawxby Aug 11, 2022 •

edited

Loading

phawxby commented Aug 11, 2022 •

edited

Loading

SimenB commented Aug 11, 2022 •

edited

Loading