Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cargotest enhancements #70659

Open
nellshamrell opened this issue Apr 1, 2020 · 3 comments
Open

Cargotest enhancements #70659

nellshamrell opened this issue Apr 1, 2020 · 3 comments
Assignees
@nellshamrell
Labels
A-testsuite Area: The testsuite used to check the correctness of rustc C-tracking-issue Category: A tracking issue for an RFC or an unstable feature. P-low Low priority T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue.

Comments

@nellshamrell
Copy link
Contributor

nellshamrell commented Apr 1, 2020 •
edited
Loading

The Infra team recently made some decisions around the future of cargotest - and there is more to decided and implemented. This issue tracks this work.

Reference: https://paper.dropbox.com/doc/The-future-of-cargotest--AxQ6EFIbNcslFkGPv4Kbr8yhAg-KuqIf3UWJnfrfrukx8o4G
@Dylan-DPC-zz Dylan-DPC-zz added T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. C-tracking-issue Category: A tracking issue for an RFC or an unstable feature. labels Apr 1, 2020
@jonas-schievink jonas-schievink added the A-testsuite Area: The testsuite used to check the correctness of rustc label Apr 1, 2020
@nellshamrell nellshamrell self-assigned this Apr 1, 2020
@pietroalbini pietroalbini added the P-medium Medium priority label Apr 1, 2020
@nellshamrell
Copy link
Contributor Author

nellshamrell commented Apr 13, 2020 •
edited
Loading

CargoTest Summary

What is Cargotest?

  • component of rustc's CI
  • ensures a predefined suite of libraries and applications successfully compile and pass tests with the rustc and cargo binaries
  • test, called cargotest, which builds some projects on every auto CI run

Goal of Cargotest

  • decided by vote on 2020-03-25 (Discord log starts here)
  • "The goal of cargotest should be to make sure artifacts produced by CI work across a wide spectrum of projects, such as small HTTP microservices, medium-sized CLI tools, embedded (no_std) applications and big production codebases. All these kinds of projects rely on different language features, code patterns and build system capabilities, so testing a sample of each category should give us a broad coverage."

How does Cargotest work?

  • We keep a constant listing test repos in the Cargotest main.rs file
  • For each repo, cargo test...
  • clones the repo
  • runs the cargo test command on each repo

What does Cargotest test right now

Libraries

CLI apps

Big production code bases

What do we still need to decide?

  • Requirements for a project to be included in Cargotest
  • Projects we want to test with Cargotest
  • How to keep the pinned commits up to date (each repo is pinned to a specific commit in Cargotest's main.rs

Useful Links

Code Links

Discussions on this topic

@nellshamrell
Copy link
Contributor Author

Problems with Cargotest

  • we need to manually update the pinned commit of each repo we test every time the repo is update (currently pinned to a version of xsv that was committed in 2017)
  • rarely breaks
  • takes about 20 min to run in CI
  • crates like iron are outdated

@XAMPPRocky XAMPPRocky mentioned this issue Jun 20, 2020
Closed
@nellshamrell
Copy link
Contributor Author

nellshamrell commented Jul 2, 2020 •
edited by shepmaster
Loading

Requirements for a project to be included in Cargotest

  • actively maintained
  • maintainer awareness that we are testing their project with Cargotest and a willingness to help if there are spurious failures
  • no subverting the Rust stability guarantee
  • must compile on stable

Possible projects to include

(This list is based on libraries that are already tested with Cargotest (and fit the above requirements), the most commonly used libraries as found by the 2019 State or Rust Survey, and in this Discord conversation

Libraries

CLI apps

Big production code bases

Async

To Do

  • How to keep the pinned commits up to date (each repo is pinned to a specific commit in Cargotest's main.rs)

@camelid camelid added T-cargo Relevant to the cargo team, which will review and decide on the PR/issue. and removed T-cargo Relevant to the cargo team, which will review and decide on the PR/issue. labels Feb 3, 2021
@pietroalbini pietroalbini added P-low Low priority and removed P-medium Medium priority labels Apr 12, 2021
@RalfJung RalfJung mentioned this issue Jul 22, 2022
Merged
RalfJung added a commit to RalfJung/rust that referenced this issue Jul 22, 2022
@RalfJung
Rollup merge of rust-lang#99602 - RalfJung:xsv, r=Mark-Simulacrum
0549516 
cargotest: do not run quickcheck tests in xsv

Fixes rust-lang#73514
I know rust-lang#70659 discusses a larger overhaul of cargotest, but that seems to have stalled and I'd like to fix the immediate issue of PRs failing due to random test failures in xsv.

This still runs the vast majority of tests by numbers:
```
test result: ok. 394 passed; 0 failed; 0 ignored; 0 measured; 32 filtered out; finished in 1.84s
```
So the loss in test coverage is hopefully not too big.
@weihanglo weihanglo mentioned this issue Apr 30, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nellshamrell nellshamrell

Labels
A-testsuite Area: The testsuite used to check the correctness of rustc C-tracking-issue Category: A tracking issue for an RFC or an unstable feature. P-low Low priority T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@nellshamrell @jonas-schievink @pietroalbini @Dylan-DPC-zz @camelid