Do not run qcheck tests on OCaml 5 #23191

Leonidas-from-XIV · 2023-01-30T15:12:35Z

Due to the changes in the RNG the amount of shrink steps and the values that it produces are different.

One could argue that this means that injecting a seed doesn't work, as the seed value to reproduce a run will differ between OCaml 4 and OCaml 5 and this could be considered a bug. I'll leave the decision up to opam-repository developers and @c-cube.

Build failure messages are part of the respective commits, since there is a number of different failures and some are very long.

Tests succeed but expect tests fails with different amount of shrink steps: #=== ERROR while compiling qcheck-alcotest.0.18 ===============================# # context 2.2.0~alpha~dev | linux/x86_64 | ocaml-base-compiler.5.0.0 | file:///home/opam/opam-repository # path ~/.opam/5.0/.opam-switch/build/qcheck-alcotest.0.18 # command ~/.opam/opam-init/hooks/sandbox.sh build dune runtest -p qcheck-alcotest -j 31 # exit-code 1 # env-file ~/.opam/log/qcheck-alcotest-7-fbdbca.env # output-file ~/.opam/log/qcheck-alcotest-7-fbdbca.out ### output ### # File "example/alcotest/output.txt.expected", line 1, characters 0-0: # /usr/bin/git --no-pager diff --no-index --color=always -u _build/default/example/alcotest/output.txt.expected _build/default/example/alcotest/output.txt # diff --git a/_build/default/example/alcotest/output.txt.expected b/_build/default/example/alcotest/output.txt # index bde0b83..c3762a8 100644 # --- a/_build/default/example/alcotest/output.txt.expected # +++ b/_build/default/example/alcotest/output.txt # @@ -11,9 +11,9 @@ Testing `my test'. # │ [FAIL] suite 1 fail_sort_id. │ # └──────────────────────────────────────────────────────────────────────────────┘ # # -test `fail_sort_id` failed on ≥ 1 cases: [1; 0] (after 20 shrink steps) # +test `fail_sort_id` failed on ≥ 1 cases: [1; 0] (after 15 shrink steps) # # -[exception] test `fail_sort_id` failed on ≥ 1 cases: [1; 0] (after 20 shrink steps) # +[exception] test `fail_sort_id` failed on ≥ 1 cases: [1; 0] (after 15 shrink steps) # # #

The tests fail because the RNG operates differently so while they act the same they fail with different error messages which makes the expect tests fail. #=== ERROR while compiling qcheck-alcotest.0.19 ===============================# # context 2.2.0~alpha~dev | linux/x86_64 | ocaml-base-compiler.5.0.0 | file:///home/opam/opam-repository # path ~/.opam/5.0/.opam-switch/build/qcheck-alcotest.0.19 # command ~/.opam/opam-init/hooks/sandbox.sh build dune runtest -p qcheck-alcotest -j 31 # exit-code 1 # env-file ~/.opam/log/qcheck-alcotest-10-fb5ffb.env # output-file ~/.opam/log/qcheck-alcotest-10-fb5ffb.out ### output ### # File "example/alcotest/output.txt.expected.64", line 1, characters 0-0: # /usr/bin/git --no-pager diff --no-index --color=always -u _build/default/example/alcotest/output.txt.expected.64 _build/default/example/alcotest/output.txt # diff --git a/_build/default/example/alcotest/output.txt.expected.64 b/_build/default/example/alcotest/output.txt # index ea706c7..23401ac 100644 # --- a/_build/default/example/alcotest/output.txt.expected.64 # +++ b/_build/default/example/alcotest/output.txt # @@ -12,18 +12,18 @@ Testing `my test'. # ┌──────────────────────────────────────────────────────────────────────────────┐ # │ [FAIL] suite 1 fail_sort_id. │ # └──────────────────────────────────────────────────────────────────────────────┘ # -test `fail_sort_id` failed on ≥ 1 cases: [1; 0] (after 11 shrink steps) # -[exception] test `fail_sort_id` failed on ≥ 1 cases: [1; 0] (after 11 shrink steps) # +test `fail_sort_id` failed on ≥ 1 cases: [1; 0] (after 16 shrink steps) # +[exception] test `fail_sort_id` failed on ≥ 1 cases: [1; 0] (after 16 shrink steps) # ────────────────────────────────────────────────────────────────────────────── # ┌──────────────────────────────────────────────────────────────────────────────┐ # │ [FAIL] suite 2 error_raise_exn. │ # └──────────────────────────────────────────────────────────────────────────────┘ # test `error_raise_exn` # raised exception `Error` # -on `0 (after 63 shrink steps)` # +on `0 (after 62 shrink steps)` # [exception] test `error_raise_exn` # raised exception `Error` # -on `0 (after 63 shrink steps)` # +on `0 (after 62 shrink steps)` # ────────────────────────────────────────────────────────────────────────────── # ┌──────────────────────────────────────────────────────────────────────────────┐ # │ [FAIL] suite 4 neg test unexpected success. │ # @@ -63,18 +63,21 @@ fail # └──────────────────────────────────────────────────────────────────────────────┘ # ~~~ Shrink ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ # Test debug_shrink successfully shrunk counter example (step 0) to: # -(3, 1) # +(2, 3) # ~~~ Shrink ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ # Test debug_shrink successfully shrunk counter example (step 1) to: # -(2, 1) # +(1, 3) # ~~~ Shrink ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ # Test debug_shrink successfully shrunk counter example (step 2) to: # -(2, 0) # +(0, 3) # ~~~ Shrink ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ # Test debug_shrink successfully shrunk counter example (step 3) to: # -(1, 0) # -law debug_shrink: 2 relevant cases (2 total) # -test `debug_shrink` failed on ≥ 1 cases: (1, 0) (after 3 shrink steps) # -[exception] test `debug_shrink` failed on ≥ 1 cases: (1, 0) (after 3 shrink steps) # +(0, 2) # +~~~ Shrink ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ # +Test debug_shrink successfully shrunk counter example (step 4) to: # +(0, 1) # +law debug_shrink: 1 relevant cases (1 total) # +test `debug_shrink` failed on ≥ 1 cases: (0, 1) (after 4 shrink steps) # +[exception] test `debug_shrink` failed on ≥ 1 cases: (0, 1) (after 4 shrink steps) # ────────────────────────────────────────────────────────────────────────────── # 6 failures! 9 tests run. - File "example/alcotest/output.txt.expected.64", line 1, characters 0-0: - /usr/bin/git --no-pager diff --no-index --color=always -u _build/default/example/alcotest/output.txt.expected.64 _build/default/example/alcotest/output.txt - diff --git a/_build/default/example/alcotest/output.txt.expected.64 b/_build/default/example/alcotest/output.txt - index ea706c7..23401ac 100644 - --- a/_build/default/example/alcotest/output.txt.expected.64 - +++ b/_build/default/example/alcotest/output.txt - @@ -12,18 +12,18 @@ Testing `my test'. - ┌──────────────────────────────────────────────────────────────────────────────┐ - │ [FAIL] suite 1 fail_sort_id. │ - └──────────────────────────────────────────────────────────────────────────────┘ - -test `fail_sort_id` failed on ≥ 1 cases: [1; 0] (after 11 shrink steps) - -[exception] test `fail_sort_id` failed on ≥ 1 cases: [1; 0] (after 11 shrink steps) - +test `fail_sort_id` failed on ≥ 1 cases: [1; 0] (after 16 shrink steps) - +[exception] test `fail_sort_id` failed on ≥ 1 cases: [1; 0] (after 16 shrink steps) - ────────────────────────────────────────────────────────────────────────────── - ┌──────────────────────────────────────────────────────────────────────────────┐ - │ [FAIL] suite 2 error_raise_exn. │ - └──────────────────────────────────────────────────────────────────────────────┘ - test `error_raise_exn` - raised exception `Error` - -on `0 (after 63 shrink steps)` - +on `0 (after 62 shrink steps)` - [exception] test `error_raise_exn` - raised exception `Error` - -on `0 (after 63 shrink steps)` - +on `0 (after 62 shrink steps)` - ────────────────────────────────────────────────────────────────────────────── - ┌──────────────────────────────────────────────────────────────────────────────┐ - │ [FAIL] suite 4 neg test unexpected success. │ - @@ -63,18 +63,21 @@ fail - └──────────────────────────────────────────────────────────────────────────────┘ - ~~~ Shrink ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Test debug_shrink successfully shrunk counter example (step 0) to: - -(3, 1) - +(2, 3) - ~~~ Shrink ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Test debug_shrink successfully shrunk counter example (step 1) to: - -(2, 1) - +(1, 3) - ~~~ Shrink ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Test debug_shrink successfully shrunk counter example (step 2) to: - -(2, 0) - +(0, 3) - ~~~ Shrink ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Test debug_shrink successfully shrunk counter example (step 3) to: - -(1, 0) - -law debug_shrink: 2 relevant cases (2 total) - -test `debug_shrink` failed on ≥ 1 cases: (1, 0) (after 3 shrink steps) - -[exception] test `debug_shrink` failed on ≥ 1 cases: (1, 0) (after 3 shrink steps) - +(0, 2) - +~~~ Shrink ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - +Test debug_shrink successfully shrunk counter example (step 4) to: - +(0, 1) - +law debug_shrink: 1 relevant cases (1 total) - +test `debug_shrink` failed on ≥ 1 cases: (0, 1) (after 4 shrink steps) - +[exception] test `debug_shrink` failed on ≥ 1 cases: (0, 1) (after 4 shrink steps) - ────────────────────────────────────────────────────────────────────────────── - 6 failures! 9 tests run.

The error message is multiple thousand lines of failing expect tests with numbers that probably changed because of the RNG. Short exerpt: #=== ERROR while compiling qcheck-core.0.19 ===================================# # context 2.2.0~alpha~dev | linux/x86_64 | ocaml-base-compiler.5.0.0 | file:///home/opam/opam-repository # path ~/.opam/5.0/.opam-switch/build/qcheck-core.0.19 # command ~/.opam/opam-init/hooks/sandbox.sh build dune runtest -p qcheck-core -j 47 # exit-code 1 # env-file ~/.opam/log/qcheck-core-7-852e05.env # output-file ~/.opam/log/qcheck-core-7-852e05.out ### output ### # (cd _build/default/test/core && ./QCheck_unit_tests.exe) # Testing `QCheck'. # This run has ID `E344FKYG'. # # [OK] Shrink 0 int. # [OK] Shrink 1 int32. # [OK] Shrink 2 int64. # [OK] Shrink 3 char. # [OK] Shrink 4 char_numeral. # [OK] Shrink 5 char_printable. # [OK] Test.check_exn 0 check_exn pass trivial. # [OK] Test.check_exn 1 check_exn pass random. # [OK] Test.check_exn 2 check_exn fail always. # [OK] Test.check_exn 3 check_exn fail random. # [OK] Test.check_exn 4 check_exn Error. # [OK] Test.make ~count 0 make with custom count. # [OK] Test.make ~count 1 make with default count. # [OK] Test.make ~count 2 make with env count. # [OK] Test.make ~long_factor 0 make with custom long_fac... # [OK] Test.make ~long_factor 1 make with default long_fa... # [OK] Test.make ~long_factor 2 make with env long_factor. # # Full test results in `~/.opam/5.0/.opam-switch/build/qcheck-core.0.19/_build/default/test/core/_build/_tests/QCheck'. # Test Successful in 0.006s. 17 tests run. # File "_build/.dune/default/test/core/dune", line 46, characters 27-45: # 46 | (names QCheck_unit_tests QCheck2_unit_tests) # ^^^^^^^^^^^^^^^^^^ # (cd _build/default/test/core && ./QCheck2_unit_tests.exe) # Testing `QCheck2'. # This run has ID `B1O7KQ46'. # # [OK] Shrink 0 int_towards. # [OK] Shrink 1 int32_towards. # [OK] Shrink 2 int64_towards. # [OK] Shrink 3 float_towards. # > [FAIL] Shrink 4 Gen.char tree. # [FAIL] Shrink 5 Gen.numeral tree. # [FAIL] Shrink 6 Gen.printable tree. # [OK] Gen 0 option with default ratio. # [OK] Gen 1 option with custom ratio. # [OK] Test.make ~count 0 make with custom count. # [OK] Test.make ~count 1 make with default count. # [OK] Test.make ~count 2 make with env count. # [OK] Test.make ~long_factor 0 make with custom long_fac... # [OK] Test.make ~long_factor 1 make with default long_fa... # [OK] Test.make ~long_factor 2 make with env long_factor. # [OK] String 0 shrinking. # [OK] Test.check_exn 0 check_exn pass trivial. # [OK] Test.check_exn 1 check_exn pass random. # [OK] Test.check_exn 2 check_exn fail always. # [OK] Test.check_exn 3 check_exn fail random. # [OK] Test.check_exn 4 check_exn Error. # # ┌──────────────────────────────────────────────────────────────────────────────┐ # │ [FAIL] Shrink 4 Gen.char tree. │ # └──────────────────────────────────────────────────────────────────────────────┘ # ASSERT 'k' on repeated failure # FAIL 'k' on repeated failure # # Expected: `['k'; 'a'; 'f'; 'h'; 'i'; 'j']' # Received: `['>'; 'a'; 'P'; 'G'; 'C'; 'A'; '@'; '?']' # # Raised at Alcotest_engine__Test.check in file "src/alcotest-engine/test.ml", line 196, characters 4-261 # Called from Alcotest_engine__Test.check' in file "src/alcotest-engine/test.ml" (inlined), line 209, characters 2-40 # Called from Dune__exe__QCheck2_unit_tests.Shrink.test_char in file "test/core/QCheck2_unit_tests.ml", line 74, characters 4-195 # Called from Alcotest_engine__Core.Make.protect_test.(fun) in file "src/alcotest-engine/core.ml", line 180, characters 17-23 # Called from Alcotest_engine__Monad.Identity.catch in file "src/alcotest-engine/monad.ml", line 24, characters 31-35

Error messages are due to different amount of shrink steps: #=== ERROR while compiling qcheck-ounit.0.19 ==================================# # context 2.2.0~alpha~dev | linux/x86_64 | ocaml-base-compiler.5.0.0 | file:///home/opam/opam-repository # path ~/.opam/5.0/.opam-switch/build/qcheck-ounit.0.19 # command ~/.opam/opam-init/hooks/sandbox.sh build dune runtest -p qcheck-ounit -j 31 # exit-code 1 # env-file ~/.opam/log/qcheck-ounit-8-2e9d56.env # output-file ~/.opam/log/qcheck-ounit-8-2e9d56.out ### output ### # File "example/ounit/output.txt.expected.64", line 1, characters 0-0: # /usr/bin/git --no-pager diff --no-index --color=always -u _build/default/example/ounit/output.txt.expected.64 _build/default/example/ounit/output.txt # diff --git a/_build/default/example/ounit/output.txt.expected.64 b/_build/default/example/ounit/output.txt # index 874cd52..2294b90 100644 # --- a/_build/default/example/ounit/output.txt.expected.64 # +++ b/_build/default/example/ounit/output.txt # @@ -17,7 +17,7 @@ Error: tests:2:error_raise_exn (in the log). # # # test `error_raise_exn` raised exception `Dune__exe__QCheck_ounit_test.Error` # -on `0 (after 63 shrink steps)` # +on `0 (after 62 shrink steps)` # # ------------------------------------------------------------------------------ # ============================================================================== # @@ -57,7 +57,7 @@ Error: tests:1:fail_sort_id (in the log). # Error: tests:1:fail_sort_id (in the code). # # # -test `fail_sort_id` failed on ≥ 1 cases: [1; 0] (after 11 shrink steps) # +test `fail_sort_id` failed on ≥ 1 cases: [1; 0] (after 16 shrink steps) # # # ------------------------------------------------------------------------------

The expect test hardcode the RNG behavior of OCaml 4 and thus lead to thousands of lines of failures. Short excerpt: #=== ERROR while compiling qcheck.0.18 ========================================# # context 2.2.0~alpha~dev | linux/x86_64 | ocaml-base-compiler.5.0.0 | file:///home/opam/opam-repository # path ~/.opam/5.0/.opam-switch/build/qcheck.0.18 # command ~/.opam/opam-init/hooks/sandbox.sh build dune runtest -p qcheck -j 47 # exit-code 1 # env-file ~/.opam/log/qcheck-7-cf6045.env # output-file ~/.opam/log/qcheck-7-cf6045.out ### output ### # File "example/output.txt.expected", line 1, characters 0-0: # /usr/bin/git --no-pager diff --no-index --color=always -u _build/default/example/output.txt.expected _build/default/example/output.txt # diff --git a/_build/default/example/output.txt.expected b/_build/default/example/output.txt # index 2bcc615..ce4682e 100644 # --- a/_build/default/example/output.txt.expected # +++ b/_build/default/example/output.txt # @@ -2,13 +2,13 @@ random seed: 1234 # # --- Failure -------------------------------------------------------------------- # # -Test should_fail_sort_id failed (18 shrink steps): # +Test should_fail_sort_id failed (12 shrink steps): # # [1; 0] # # === Error ====================================================================== # # -Test should_error_raise_exn errored on (63 shrink steps): # +Test should_error_raise_exn errored on (62 shrink steps): # # 0 # # @@ -19,59 +19,59 @@ exception Dune__exe__QCheck_runner_test.Error # # Collect results for test collect_results: # # -4: 20 cases # -3: 25 cases # -2: 17 cases # -1: 18 cases # -0: 20 cases # +4: 24 cases # +3: 20 cases # +2: 18 cases # +1: 21 cases # +0: 17 cases

raphael-proust

Apart from the minor remarks in the comments I left, this looks ok.

packages/qcheck-alcotest/qcheck-alcotest.0.19/opam

packages/qcheck-core/qcheck-core.0.19/opam

packages/qcheck-ounit/qcheck-ounit.0.19/opam

packages/qcheck/qcheck.0.19/opam

jmid · 2023-01-31T16:12:48Z

Expect test author and occasional QCheck maintainer here! 👋 PR looks good to me too!
I recently added separate expect output for OCaml 5 - but that went into version 0.20: c-cube/qcheck#262

There's now also https://github.com/ocaml/stdlib-random that we might consider switching to (this is not the first time the RNG changed)

dinosaure · 2023-02-02T10:13:45Z

That's expected that qcheck-core (0.18) fails on some platformes (like arm32 or ppc64)?

jmid · 2023-02-02T10:44:37Z

Yes:

it is expected that 0.18 fails on 32-bit architectures due to expect test failures against 64-bit .expected files. This was solved in Fix tests on 32bit arch c-cube/qcheck#250 that went into 0.19
ppc64 failures are due to missing stack overflow protection on that architecture - see 5 packages from c-cube/qcheck #21692 (comment) for more details and pointers

dinosaure · 2023-02-02T11:35:15Z

Ok, so let's merge. Thanks!

Leonidas-from-XIV added 5 commits January 30, 2023 15:37

raphael-proust reviewed Jan 31, 2023

View reviewed changes

Remove the newlines

db2d059

raphael-proust added the consider for merge label Feb 2, 2023

dinosaure added the question label Feb 2, 2023

dinosaure merged commit d85efeb into ocaml:master Feb 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Do not run qcheck tests on OCaml 5 #23191

Do not run qcheck tests on OCaml 5 #23191

Leonidas-from-XIV commented Jan 30, 2023 •

edited

Loading

raphael-proust left a comment

jmid commented Jan 31, 2023

dinosaure commented Feb 2, 2023

jmid commented Feb 2, 2023

dinosaure commented Feb 2, 2023

Do not run qcheck tests on OCaml 5 #23191

Do not run qcheck tests on OCaml 5 #23191

Conversation

Leonidas-from-XIV commented Jan 30, 2023 • edited Loading

raphael-proust left a comment

Choose a reason for hiding this comment

jmid commented Jan 31, 2023

dinosaure commented Feb 2, 2023

jmid commented Feb 2, 2023

dinosaure commented Feb 2, 2023

Leonidas-from-XIV commented Jan 30, 2023 •

edited

Loading