5 packages from c-cube/qcheck

[jmid]

This pull-request concerns:
-ppx_deriving_qcheck.0.2.0: PPX Deriver for QCheck
-qcheck.0.19: Compatibility package for qcheck
-qcheck-alcotest.0.19: Alcotest backend for qcheck
-qcheck-core.0.19: Core qcheck library
-qcheck-ounit.0.19: OUnit backend for qcheck

---

• Homepage: https://github.com/c-cube/qcheck/

---

🐫 Pull-request generated by opam-publish v2.1.0

[kit-ty-kate]

There seems to be several issues in the tests, e.g.

#=== ERROR while compiling qcheck-core.0.19 ===================================#
# context              2.1.2 | linux/x86_64 | ocaml-base-compiler.4.08.1 | pinned(https://github.com/c-cube/qcheck/archive/v0.19.tar.gz)
# path                 ~/.opam/4.08/.opam-switch/build/qcheck-core.0.19
# command              ~/.opam/opam-init/hooks/sandbox.sh build dune runtest -p qcheck-core -j 31
# exit-code            1
# env-file             ~/.opam/log/qcheck-core-1823-3b8c5d.env
# output-file          ~/.opam/log/qcheck-core-1823-3b8c5d.out
### output ###
# File "test/ppx_deriving_qcheck/deriver/dune", line 10, characters 55-61:
# 10 |  (libraries qcheck-alcotest ppxlib ppx_deriving_qcheck qcheck)
#                                                             ^^^^^^
# Error: Library "qcheck" not found.
# -> required by
#    _build/default/test/ppx_deriving_qcheck/deriver/test_textual.exe
# -> required by alias test/ppx_deriving_qcheck/deriver/runtest in
#    test/ppx_deriving_qcheck/deriver/dune:3
# File "test/ppx_deriving_qcheck/deriver/dune", line 11, characters 18-33:
# 11 |  (preprocess (pps ppxlib.metaquot ppx_deriving_qcheck)))
#                        ^^^^^^^^^^^^^^^
# Error: Library "ppxlib.metaquot" not found.
# -> required by
#    _build/default/test/ppx_deriving_qcheck/deriver/.merlin-conf/exe-test_textual-7951b6e090184e2c954c491cb5f69a2e
# -> required by
#    _build/default/test/ppx_deriving_qcheck/deriver/test_textual.exe
# -> required by alias test/ppx_deriving_qcheck/deriver/runtest in
#    test/ppx_deriving_qcheck/deriver/dune:3
# (cd _build/default/test/core && ./QCheck_unit_tests.exe)

[jmid]

Sorry about that, now trying with a proper package annotation on the ppx_deriving_qcheck tests... 🤞

[jmid]

Sorry for the CI-debugging... ppx_deriving_qcheck is starting to be in a reasonable state code-wise to share on opam - but it was clearly missing a bit packaging-wise... 😁

[kit-ty-kate]

The tests seem a bit capricious (non-deterministic)

#=== ERROR while compiling qcheck-core.0.19 ===================================#
# context              2.1.2 | linux/x86_64 | ocaml-base-compiler.4.13.1 | pinned(https://github.com/c-cube/qcheck/archive/v0.19.tar.gz)
# path                 ~/.opam/4.13/.opam-switch/build/qcheck-core.0.19
# command              ~/.opam/opam-init/hooks/sandbox.sh build dune runtest -p qcheck-core -j 31
# exit-code            1
# env-file             ~/.opam/log/qcheck-core-1815-b3f4ba.env
# output-file          ~/.opam/log/qcheck-core-1815-b3f4ba.out
### output ###
# (cd _build/default/test/core && ./QCheck_unit_tests.exe)
# Testing `QCheck'.
# This run has ID `JA4462VZ'.
# 
#   [OK]          Shrink                          0   int.
#   [OK]          Shrink                          1   int32.
#   [OK]          Shrink                          2   int64.
#   [OK]          Shrink                          3   char.
#   [OK]          Shrink                          4   char_numeral.
#   [OK]          Shrink                          5   char_printable.
#   [OK]          Test.check_exn                  0   check_exn pass trivial.
#   [OK]          Test.check_exn                  1   check_exn pass random.
#   [OK]          Test.check_exn                  2   check_exn fail always.
#   [OK]          Test.check_exn                  3   check_exn fail random.
#   [OK]          Test.check_exn                  4   check_exn Error.
#   [OK]          Test.make ~count                0   make with custom count.
#   [OK]          Test.make ~count                1   make with default count.
#   [OK]          Test.make ~count                2   make with env count.
#   [OK]          Test.make ~long_factor          0   make with custom long_fac...
#   [OK]          Test.make ~long_factor          1   make with default long_fa...
#   [OK]          Test.make ~long_factor          2   make with env long_factor.
# 
# Full test results in `~/.opam/4.13/.opam-switch/build/qcheck-core.0.19/_build/default/test/core/_build/_tests/QCheck'.
# Test Successful in 0.005s. 17 tests run.
# File "test/core/dune", line 22, characters 27-45:
# 22 |   (names QCheck_unit_tests QCheck2_unit_tests)
#                                 ^^^^^^^^^^^^^^^^^^
# (cd _build/default/test/core && ./QCheck2_unit_tests.exe)
# Testing `QCheck2'.
# This run has ID `LDZTV3C6'.
# 
#   [OK]          Shrink                          0   int_towards.
#   [OK]          Shrink                          1   int32_towards.
#   [OK]          Shrink                          2   int64_towards.
#   [OK]          Shrink                          3   float_towards.
#   [OK]          Shrink                          4   Gen.char tree.
#   [OK]          Shrink                          5   Gen.numeral tree.
#   [OK]          Shrink                          6   Gen.printable tree.
#   [OK]          Gen                             0   option with default ratio.
# > [FAIL]        Gen                             1   option with custom ratio.
#   [OK]          Test.make ~count                0   make with custom count.
#   [OK]          Test.make ~count                1   make with default count.
#   [OK]          Test.make ~count                2   make with env count.
#   [OK]          Test.make ~long_factor          0   make with custom long_fac...
#   [OK]          Test.make ~long_factor          1   make with default long_fa...
#   [OK]          Test.make ~long_factor          2   make with env long_factor.
#   [OK]          String                          0   shrinking.
#   [OK]          Test.check_exn                  0   check_exn pass trivial.
#   [OK]          Test.check_exn                  1   check_exn pass random.
#   [OK]          Test.check_exn                  2   check_exn fail always.
#   [OK]          Test.check_exn                  3   check_exn fail random.
#   [OK]          Test.check_exn                  4   check_exn Error.
# 
# ┌──────────────────────────────────────────────────────────────────────────────┐
# │ [FAIL]        Gen                             1   option with custom ratio.  │
# └──────────────────────────────────────────────────────────────────────────────┘
# ASSERT Gen.option produces around 50% of Some
# FAIL Gen.option produces around 50% of Some
# 
#    Expected: `false'
#    Received: `true'
# 
# Raised at Alcotest_engine__Test.check in file "src/alcotest-engine/test.ml", line 196, characters 4-261
# Called from Alcotest_engine__Core.Make.protect_test.(fun) in file "src/alcotest-engine/core.ml", line 180, characters 17-23
# Called from Alcotest_engine__Monad.Identity.catch in file "src/alcotest-engine/monad.ml", line 24, characters 31-35
# 
# Logs saved to `~/.opam/4.13/.opam-switch/build/qcheck-core.0.19/_build/default/test/core/_build/_tests/QCheck2/Gen.001.output'.
#  ──────────────────────────────────────────────────────────────────────────────
# 
# Full test results in `~/.opam/4.13/.opam-switch/build/qcheck-core.0.19/_build/default/test/core/_build/_tests/QCheck2'.
# 1 failure! in 0.084s. 21 tests run.

Some of them also segfault on non-x86_64 architectures:

#=== ERROR while compiling qcheck.0.19 ========================================#
# context              2.1.2 | linux/ppc64 | ocaml-base-compiler.4.14.0 | pinned(https://github.com/c-cube/qcheck/archive/v0.19.tar.gz)
# path                 ~/.opam/4.14/.opam-switch/build/qcheck.0.19
# command              ~/.opam/opam-init/hooks/sandbox.sh build dune runtest -p qcheck -j 175
# exit-code            1
# env-file             ~/.opam/log/qcheck-1963-e9863c.env
# output-file          ~/.opam/log/qcheck-1963-e9863c.out
### output ###
# File "test/core/dune", line 12, characters 9-27:
# 12 |   (names QCheck_expect_test)
#               ^^^^^^^^^^^^^^^^^^
# (cd _build/default/test/core && ./QCheck_expect_test.exe) > _build/default/test/core/QCheck_expect_test.output
# Command got signal SEGV.
# File "test/core/dune", line 17, characters 9-28:
# 17 |   (names QCheck2_expect_test)
#               ^^^^^^^^^^^^^^^^^^^
# (cd _build/default/test/core && ./QCheck2_expect_test.exe) > _build/default/test/core/QCheck2_expect_test.output
# Command got signal SEGV.

- File "test/core/dune", line 17, characters 9-28:
- 17 |   (names QCheck2_expect_test)
-               ^^^^^^^^^^^^^^^^^^^
- (cd _build/default/test/core && ./QCheck2_expect_test.exe) > _build/default/test/core/QCheck2_expect_test.output
- Command got signal SEGV.

[jmid]

The tests seem a bit capricious (non-deterministic)

Ah, that is probably to be expected for randomized tests.
I guess we could expand the bounds for the acceptable behavior - or increase the test count... 🤔

Some of them also segfault on non-x86_64 architectures:

#=== ERROR while compiling qcheck.0.19 ========================================#
# context              2.1.2 | linux/ppc64 | ocaml-base-compiler.4.14.0 | pinned(https://github.com/c-cube/qcheck/archive/v0.19.tar.gz)
# path                 ~/.opam/4.14/.opam-switch/build/qcheck.0.19
# command              ~/.opam/opam-init/hooks/sandbox.sh build dune runtest -p qcheck -j 175
# exit-code            1
# env-file             ~/.opam/log/qcheck-1963-e9863c.env
# output-file          ~/.opam/log/qcheck-1963-e9863c.out
### output ###
# File "test/core/dune", line 12, characters 9-27:
# 12 |   (names QCheck_expect_test)
#               ^^^^^^^^^^^^^^^^^^
# (cd _build/default/test/core && ./QCheck_expect_test.exe) > _build/default/test/core/QCheck_expect_test.output
# Command got signal SEGV.
# File "test/core/dune", line 17, characters 9-28:
# 17 |   (names QCheck2_expect_test)
#               ^^^^^^^^^^^^^^^^^^^
# (cd _build/default/test/core && ./QCheck2_expect_test.exe) > _build/default/test/core/QCheck2_expect_test.output
# Command got signal SEGV.

- File "test/core/dune", line 17, characters 9-28:
- 17 |   (names QCheck2_expect_test)
-               ^^^^^^^^^^^^^^^^^^^
- (cd _build/default/test/core && ./QCheck2_expect_test.exe) > _build/default/test/core/QCheck2_expect_test.output
- Command got signal SEGV.

Ouch! That's a surprise. This is supposed to be pure OCaml without any unsafe/Obj/C funny business IIRC 😬

[jmid]

Sorry for the force-push via opam publish! Happy for your help - and not intending to step over any toes... 😅 🙏
I was adding the same lower bound in a commit locally and needed it in the QCheck repo too.

[jmid]

I can see expect test mismatch on 32-bit machines.
This is a known difference caused by the 31-bit Random module having a different output distribution: c-cube/qcheck#152

[jmid]

As far as I can tell, the remaining issue is the ppc64 segfault, which I suspect to be compiler backend/runtime related 🤔
Let me know how to proceed.

[jmid]

I've now changed the test suite to also support 32-bit architectures: c-cube/qcheck#250 in the hope that this will trigger more green lights and a merge... 🤞

[mseri]

On ppc64 is still failing, should we made it unavailable?

# File "_build/.dune/default/test/core/dune", line 18, characters 0-141:
# 18 | (rule
# 19 |   (targets QCheck_expect_test.out)
# 20 |   (action
# 21 |     (with-stdout-to %{targets}
# 22 |       (run ./QCheck_expect_test.exe --no-colors -s 1234))))
# (cd _build/default/test/core && ./QCheck_expect_test.exe --no-colors -s 1234) > _build/default/test/core/QCheck_expect_test.out
# Command got signal SEGV.
# File "_build/.dune/default/test/core/dune", line 34, characters 0-143:
# 34 | (rule
# 35 |   (targets QCheck2_expect_test.out)
# 36 |   (action
# 37 |     (with-stdout-to %{targets}
# 38 |       (run ./QCheck2_expect_test.exe --no-colors -s 1234))))
# (cd _build/default/test/core && ./QCheck2_expect_test.exe --no-colors -s 1234) > _build/default/test/core/QCheck2_expect_test.out
# Command got signal SEGV.

[jmid]

On ppc64 is still failing, should we made it unavailable?

Yes, that one puzzles me... 🤔
I suspect it is a compiler bug on ppc64 - and just happens to be triggered by the latest changes.
All of those https://github.com/c-cube/qcheck/blob/master/CHANGELOG.md#019 have been

• adding or improving features (in pure OCaml)
• adding expect and unit tests
• adjusting documentation

I should be able to get access to a ppc64-machine, so there's a chance I may be able to reproduce and debug it.
I would prefer not to delay the release further however... 😬

[jmid]

I added the available constraint on everything but ppx_deriving_qcheck only for opam-ci to then find that the package depends on a combinator renaming we did in 0.19... 🙄
The latest push therefore makes qcheck 0.19 a lower bound for it.

[jmid]

I just got access to a ppc64 machine today and after some compiling and installing I can now confirm a suspicion from @olafhering: the test suite succeeds on ppc64 with a larger stack!

This machine (which I also think runs the opam-ci jobs) reports this default stack size with ulimit -a:

stack size                  (kbytes, -s) 8192

Following Olaf's suggestion of ulimit -s $((1024 * 64)) is enough to complete the test suite without blowing the stack.

I'll revert the disabling of the ppc64 target and see if I can get the default stack size on opam-ci's ppc64 machines raised...

[jmid]

TIL that ppc64 doesn't have the nice Stack_overflow protection like many other OCaml platforms: ocaml/ocaml#11415
It is therefore a bad idea(tm) to try to blow the stack and hope to catch a Stack_overflow exception in the QCheck test suite on ppc64. In the latest pushes I've therefore

• re-enabled ppc64
• commented out the offending list_equal_dupl test

So much 🤞 on this one...

[kit-ty-kate]

Thanks!