Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not run cargotest on quickcheck/proptest tests #73514

Closed
Manishearth opened this issue Jun 19, 2020 · 7 comments · Fixed by #99602
Closed

Do not run cargotest on quickcheck/proptest tests #73514

Manishearth opened this issue Jun 19, 2020 · 7 comments · Fixed by #99602
Labels
A-testsuite Area: The testsuite used to check the correctness of rustc C-bug Category: This is a bug. T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue.

Comments

@Manishearth
Copy link
Member

As part of our CI we run "cargotest" on a variety of ecosystem crates, making sure they successfully build and run tests.

Unfortunately, some of use stochastic tests like quickcheck, which means that a crate passing our CI one day may not necessarily mean it passes CI the next. Perhaps we should audit the tests being run and filter out ones like these?

See #73511 (comment) and BurntSushi/xsv#227

cc @rust-lang/release

@Manishearth Manishearth added C-bug Category: This is a bug. T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. labels Jun 19, 2020
@Mark-Simulacrum
Copy link
Member

Yeah I think we've discussed this to some extent previously within infra team as part of our "what to do about cargotest" discussions. cc @nellshamrell in particular

Not sure that we came to any conclusions though.

@jonas-schievink jonas-schievink added the A-testsuite Area: The testsuite used to check the correctness of rustc label Jun 19, 2020
@XAMPPRocky
Copy link
Member

Related: #70659

@RalfJung
Copy link
Member

This just bit me and took a lot of time as I triple-checked that after bors failing twice (!) in xsv, the same test actually works fine on my system.

@RalfJung
Copy link
Member

The xsv test suite seems to be full of quickcheck tests, so I think the only way to fix this is to remove xsv from cargotest (or to only run a short list of curated tests).

@ehuss
Copy link
Contributor

ehuss commented Jul 22, 2022

@RalfJung See #70659 where there is some discussion fo changing the cargotest set. I think the last comment implies dropping xsv.

@RalfJung
Copy link
Member

Hm, that discussion seems to have stalled though.

So to fix the immediate problem, I propose we do something like #99602.

@bors bors closed this as completed in 0549516 Jul 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-testsuite Area: The testsuite used to check the correctness of rustc C-bug Category: This is a bug. T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants