jest-circus runs children in shuffled order #12922

jhwang98 · 2022-06-08T00:06:58Z

Summary

Resolves #4386

Test plan

This PR is a work in progress based on #4386

facebook-github-bot · 2022-06-08T00:07:02Z

Hi @jhwang98!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at [email protected]. Thanks!

facebook-github-bot · 2022-06-08T12:58:30Z

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

SimenB

super exciting stuff!

packages/jest-config/src/normalize.ts

packages/jest-core/src/cli/index.ts

packages/jest-core/src/runJest.ts

packages/jest-core/src/watch.ts

packages/jest-util/src/shuffleArray.ts

Co-authored-by: Simen Bekkhus <[email protected]>

…ests

Smrtnyk · 2022-09-15T11:41:25Z

Hi, any update on future of this, will this PR get adopted in the future?
We have thousands of tests at the company where I work and some of them are order dependent as it seems.
To not go to every test suite and try to find them, this feature would help us spot such tests and fix them.

Joshua-Hwang · 2022-09-26T12:02:46Z

Hi @SimenB is there anything else that I need to change before we can get this merged?

Happy to improve this PR 👍

SimenB

code is shaping up well!

Main thing missing is tests and docs 🙂 Tests for the shffleArray function would be great, plus of course that seed works and works consistently. Probably integration test?

Also, can you update the changelog?

mrazauskas

Thanks for working on this.

If that is only jest-circus option, it would be good to mentioned that in documentation. For example, like here: https://github.com/facebook/jest/blob/e57496455437a1d3d0d9cafc4cbc6a8eb90bcfa7/docs/GlobalAPI.md?plain=1#L752-L756

docs/CLI.md

mrazauskas · 2022-09-30T04:11:29Z

docs/CLI.md

@@ -28,6 +28,18 @@ Run tests related to changed files based on hg/git (uncommitted files):
 jest -o
 ```

+Run tests in a file in a random order (this will print the seed used for random number generation):


I get the idea, but what if this would be moved to --randomize? If a user is interested to read about --randomize, current text does not provide usage details. It is not really clear where to look for it and the details here are important.

I've copied what I've written here to be part of the CLI description of the --randomize option.
I've done the same with --seed as well.

Was this what you were looking for?

Thanks. I was trying to say that this usage information could be more useful, if placed next to --randomize description. That’s there I would expect to find it, but this is just an opinion (;

ahh gotcha. Yeah that makes sense.
I initially put a description near the beginning because this might be a popular features. On further thinking it's probably not that popular.

Joshua-Hwang · 2022-11-20T20:51:37Z

Hey @SimenB were there any additional changes you want made to this PR?
I noticed you still have changes requested but I think I've adjusted to all the existing comments.

dubzzz · 2023-01-12T23:35:38Z

@jhwang98 I just released pure-rand v6, no change expected on your code but probably worth to update it before merging. It brings a slight performance benefit on uniform distributions.

jhwang98 · 2023-02-13T04:18:30Z

This is the second time the tests have been failing 😕.

I'm not sure what the problem could be here.

jayeclark · 2023-02-13T13:09:24Z

Hey @jhwang98 , the tests are failing because of a conflict between the lock file (yarn.lock) and your package.json. I took a quick glance & it looks like when you merged the recent changes from main, those included a yarn.lock file that overwrote your previous one. As a result, when the CI/CD install script runs in the GitHub actions build environment, it sees that your package.json is instructing it to install pure-rand, but finds that that package doesn’t exist in the corresponding yarn.lock file, and throws an error as a result.

Running yarn install locally with the most recent version of your branch should alter the lock file locally to include pure-rand. If you then commit & push that updated lock file I believe the broken CI should resolve.

Joshua-Hwang · 2023-02-17T04:25:40Z

The check that failed was for installing Node.

Don't think it was related to my change.

SimenB · 2023-02-23T07:52:39Z

I recently was sent this and it made me think of this PR 😅 let's get it in!

SimenB

just a few nits 🙂

packages/jest-config/src/normalize.ts

SimenB · 2023-02-23T07:57:12Z

packages/jest-circus/src/shuffleArray.ts

+  const upperBoundSeedValue = 2 ** 31;
+  if (seed < -upperBoundSeedValue || seed > upperBoundSeedValue - 1) {
+    throw new Error(
+      `seed value must be between \`-0x80000000\` and \`0x7fffffff\` inclusive instead it is ${seed}`,
+    );
+  }


do we need this check? isn't seed from config already guaranteed to be within these bounds?

I guess so.

I was thinking if the shuffleArray ever gets used elsewhere. If not then I guess we're fine.

it's not exported outside of jest circus, so we should be fine

packages/jest-circus/src/shuffleArray.ts

packages/jest-core/src/cli/index.ts

packages/jest-circus/src/shuffleArray.ts

packages/jest-circus/src/run.ts

Co-authored-by: Simen Bekkhus <[email protected]>

packages/jest-circus/src/run.ts

github-actions · 2023-03-26T00:08:48Z

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.

jest-circus runs children in shuffled order

e43ca99

added shuffle and pseudorandom number generator

bbb1cf3

facebook-github-bot added the cla signed label Jun 8, 2022

Joshua-Hwang added 4 commits August 25, 2022 22:42

merged master

4c1d07c

added cli argv

99ac4cd

removed unnecessary dependencies

882f920

add console logs about the seed

4df23a9

SimenB reviewed Sep 3, 2022

View reviewed changes

Joshua-Hwang and others added 8 commits September 3, 2022 23:44

More readable Math.pow()

dc6e64f

Co-authored-by: Simen Bekkhus <[email protected]>

Used recommended logging approach

3d09cfb

Co-authored-by: Simen Bekkhus <[email protected]>

Used recommended logging approach

f8860e2

Co-authored-by: Simen Bekkhus <[email protected]>

added prando rng

0bec19f

bug where rng was recreated every run

3a1d712

swapped order of logs

95590dc

added typing for the rng

14349bd

Merge branch 'main' of github.com:facebook/jest into feat-randomise-t…

549988d

…ests

SimenB requested changes Sep 28, 2022

View reviewed changes

jhwang98 and others added 8 commits September 29, 2022 11:05

linting issues fixed

4ae8ff2

unit tests are green

967b32e

used snapshots instead

561146c

added e2e tests

2f95b65

changed valid seeds

50bada0

added changelog message

b78fdc2

improved cli help and added documentation

3f33508

Merge branch 'main' into feat-randomise-tests

d84caa1

mrazauskas reviewed Sep 30, 2022

View reviewed changes

jhwang98 added 2 commits January 13, 2023 10:58

merged main

4705449

bumped pure-rand package

af913e2

jhwang98 requested a review from SimenB January 23, 2023 06:14

jhwang98 added 2 commits January 26, 2023 00:52

Merge branch 'main' into feat-randomise-tests

a6b4d06

Merge branch 'main' into feat-randomise-tests

e4f0b24

jhwang98 and others added 3 commits February 14, 2023 12:55

yarn.lock updated

c4baaaa

fixed copyright headers

87547f3

Merge branch 'main' into feat-randomise-tests

8bfa4c8

Merge branch 'main' into feat-randomise-tests

0202230

SimenB approved these changes Feb 23, 2023

View reviewed changes

jhwang98 and others added 6 commits February 23, 2023 20:23

Update packages/jest-config/src/normalize.ts

3a0a4f1

Co-authored-by: Simen Bekkhus <[email protected]>

Update packages/jest-core/src/cli/index.ts

43d6f8b

Co-authored-by: Simen Bekkhus <[email protected]>

Update packages/jest-circus/src/shuffleArray.ts

7bd3fe1

Co-authored-by: Simen Bekkhus <[email protected]>

remove bounds check as it's being done at the normalize level

cff3995

changed error message slightly

7c8bfc1

swapped order for generating rng

c3c6516

SimenB reviewed Feb 23, 2023

View reviewed changes

packages/jest-circus/src/run.ts Outdated Show resolved Hide resolved

Update packages/jest-circus/src/run.ts

d57bf1c

SimenB approved these changes Feb 23, 2023

View reviewed changes

packages/jest-circus/src/run.ts Outdated Show resolved Hide resolved

Update packages/jest-circus/src/run.ts

1137ce1

SimenB merged commit cbe0ac1 into jestjs:main Feb 23, 2023

github-actions bot locked as resolved and limited conversation to collaborators Mar 26, 2023

Joshua-Hwang deleted the feat-randomise-tests branch September 15, 2023 16:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

jest-circus runs children in shuffled order #12922

jest-circus runs children in shuffled order #12922

jhwang98 commented Jun 8, 2022 •

edited by SimenB

Loading

facebook-github-bot commented Jun 8, 2022

facebook-github-bot commented Jun 8, 2022

SimenB left a comment

Smrtnyk commented Sep 15, 2022

Joshua-Hwang commented Sep 26, 2022

SimenB left a comment

mrazauskas left a comment

mrazauskas Sep 30, 2022

jhwang98 Sep 30, 2022

mrazauskas Sep 30, 2022

jhwang98 Sep 30, 2022

Joshua-Hwang commented Nov 20, 2022

dubzzz commented Jan 12, 2023

jhwang98 commented Feb 13, 2023

jayeclark commented Feb 13, 2023

Joshua-Hwang commented Feb 17, 2023

SimenB commented Feb 23, 2023 •

edited

Loading

SimenB left a comment

SimenB Feb 23, 2023

jhwang98 Feb 23, 2023

SimenB Feb 23, 2023

github-actions bot commented Mar 26, 2023

jest-circus runs children in shuffled order #12922

jest-circus runs children in shuffled order #12922

Conversation

jhwang98 commented Jun 8, 2022 • edited by SimenB Loading

Summary

Test plan

facebook-github-bot commented Jun 8, 2022

Action Required

Process

facebook-github-bot commented Jun 8, 2022

SimenB left a comment

Choose a reason for hiding this comment

Smrtnyk commented Sep 15, 2022

Joshua-Hwang commented Sep 26, 2022

SimenB left a comment

Choose a reason for hiding this comment

mrazauskas left a comment

Choose a reason for hiding this comment

mrazauskas Sep 30, 2022

Choose a reason for hiding this comment

jhwang98 Sep 30, 2022

Choose a reason for hiding this comment

mrazauskas Sep 30, 2022

Choose a reason for hiding this comment

jhwang98 Sep 30, 2022

Choose a reason for hiding this comment

Joshua-Hwang commented Nov 20, 2022

dubzzz commented Jan 12, 2023

jhwang98 commented Feb 13, 2023

jayeclark commented Feb 13, 2023

Joshua-Hwang commented Feb 17, 2023

SimenB commented Feb 23, 2023 • edited Loading

SimenB left a comment

Choose a reason for hiding this comment

SimenB Feb 23, 2023

Choose a reason for hiding this comment

jhwang98 Feb 23, 2023

Choose a reason for hiding this comment

SimenB Feb 23, 2023

Choose a reason for hiding this comment

github-actions bot commented Mar 26, 2023

jhwang98 commented Jun 8, 2022 •

edited by SimenB

Loading

SimenB commented Feb 23, 2023 •

edited

Loading