Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CLI --seed without value and files argument is counterintuitive #1691

Closed
Krinkle opened this issue May 1, 2022 · 1 comment · Fixed by #1800
Closed

CLI --seed without value and files argument is counterintuitive #1691

Krinkle opened this issue May 1, 2022 · 1 comment · Fixed by #1800
Assignees
Labels
Component: CLI Type: Enhancement New idea or feature request.
Milestone

Comments

@Krinkle
Copy link
Member

Krinkle commented May 1, 2022

Usage: qunit [options] [files]

[…]

Options:
  --seed [value]         specify a seed to re-order your tests;
                         if specified without a value, a seed will be generated
  […]

When using qunit --seed it will default to running test/**.js and will print the randomly selected seed for future re-use. This works as expected:

$ qunit --seed
Running tests with seed: 3vjdr2nhlga
TAP version 13
ok 1 Single > has a test
1..1
# pass 1
# skip 0
# todo 0
# fail 0

The problem is, when using a files argument, the command becomes ambiguous and it naturally becomes the value for the seed instead:

$ qunit --seed test/single.js
Running tests with seed: test/single.js
[…]

This can be worked around currently by using -- to mark the end of the options and the start of the arguments:

$ qunit --seed -- test/single.js 
Running tests with seed: 3vjdr2nhlga
TAP version 13
ok 1 Single > has a test
1..1
# pass 1
# skip 0
# todo 0
# fail 0
@Krinkle Krinkle added Type: Enhancement New idea or feature request. Component: CLI labels May 1, 2022
@Krinkle
Copy link
Member Author

Krinkle commented Sep 8, 2024

Approaches to consider:

  1. Introduce --newseed CLI option (along with ?newseed parameter, and QUnit.config.newseed = true). This will instruct QUnit to generate a new seed, and assign it to QUnit.config.seed. This can be backported to QUnit 2.x to ease the transition and allow for forward-compatible adoption.
  2. Change --seed CLI option to require a value, thus qunit --seed and qunit --seed -- test/ would fail due to a missing option value. This could be perceived as a breaking change, however, while --seed with value is likely used in automation, without a seed seems exclusively used by hand. Given how the current behaviour is (to my knowledge) exclusively perceived as a bug, I'm willing to change this in a patch release, as long as it can be done without introducing a new option. The question is how...
  3. Change --seed CLI option to treat a reserved value like new as instruction to produce a new random seed. This means --seed=new will no longer be treated as actual seeds but instead as instruction to generate a new seed. I think this will be more intuitive than introducing a separate option like --newseed. If we use true as the reserved value, like --seed=true, then this would also align the CLI with our other configuration interfaces (preconfig, config, URL param) where QUnit.config.seed = true already carries this meaning.

After this, we'll want to improve the HTML Reporter and TAP Reporter, to present these to users to help reproduce a specific failure.

@Krinkle Krinkle added this to the 3.0 release milestone Sep 8, 2024
Krinkle added a commit to Krinkle/qunit that referenced this issue Sep 16, 2024
@Krinkle Krinkle self-assigned this Sep 16, 2024
Krinkle added a commit to Krinkle/qunit that referenced this issue Sep 16, 2024
Krinkle added a commit to Krinkle/qunit that referenced this issue Sep 16, 2024
Krinkle added a commit that referenced this issue Dec 6, 2024
* Fix `--seed <file>` arg confusion. This is now correctly interpreted
  as a file.

* Fix presence of "Running tests with seed:" message on CLI when preconfig
  qunit_config_seed is set via environment variable. Previously the
  seed was applied, but not printed because the printing logic only
  checked the CLI options, not QUnit.config.

* Add `--seed=true` to generate a new seed.
  This is backported from QUnit 3.0.0-alpha, and aligns the CLI with
  what was already supported via Preconfig, QUnit.config, and URL
  params.

Cherry-picked from 390a5b7 (3.0.0-dev).
> Ref #1691.
@Krinkle Krinkle modified the milestones: 3.0 release, 2.x release Dec 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Component: CLI Type: Enhancement New idea or feature request.
Development

Successfully merging a pull request may close this issue.

1 participant