Tracking Issue for compiletest documentation

[jieyouxu]

Context

compiletest is the main test harness of rustc and other tools inside this repository, but there is a significant lack of documentation for it. We should write some documentation for compiletest. There are some documentation in rustc-dev-guide, but compiletest is a lot more complex than that.

Areas lacking documentation

• High-level architecture and overview
• Interaction between bootstrap <-> compiletest <-> tests
• compiletest-as-a-binary: CLI interface options and docs
• Filtering tests
• How are test modes and test suites created, registered, built and run
  • When can tests be ignored? cached test results and their invalidation: dependency stamps
  • Some test mode / test suite tests can be run in parallel / concurrently:
    • Concurrency assumptions: what if the test modifies external state?
• Test mode and test suite specific behavior, assumptions and quirks
  • UI tests: auxiliary crates and bins, compile-flags, compare modes, assumptions and conventions on stdout/stderr output, blessing
  • run-make and run-make-fulldeps tests: how the whole build pipeline is setup, concepts, how tests are built and executed, provided env vars, assumed dependencies
  • codegen tests: filecheck, revisions magic behavior
  • coverage tests: how to bless?
  • ui-fulldeps have some non-trivial behavior: e.g. Port tests/run-make-fulldeps/obtain-borrowck to ui-fulldeps #126073 (comment)
• Error handling strategy/design and diagnostics
• Interaction with CI
• Test directive collection, parsing and validation
  • Which directives can be used in which test suites?
  • What do the directives even mean/do?
• Test output normalization special syntax
  • e.g. const CWD: &str = "{{cwd}}";
• Setting current_dir when running tests

Related issues

• We should document what external dependencies we assume for our test suites #125585
• We should document more clearly env vars used between bootstrap <-> compiletest <-> test suites #125587
• Debuginfo tests don't control for ambient tools #126092

Unresolved questions

• Where do we keep compiletest docs?
  • Maybe alongside the source? Maybe a mdBook? rustc-dev-guide?
  • Ideally we would like this easy to read and reference and improve.
• How to we keep compiletest documentation update-to-date?
  • Periodic triage?

Documentation improvement history

TODO

[Zalathar]

• //@ needs-matching-clang is misleading #126180