Unify test suites' module exclusion logic

[ikonst]

Individual test suites grew to have distinct approaches for selecting or excluding modules / symbols, with distinct ad-hoc exclude lists. Rather than rely on a common set of rules, test authors have been adding to those distinct lists (having to modify the test suite to support their test case).

In this PR, we will consolidate this logic so we could have a common set of rules:

• The "tested corpus" is what's asserted on. Everything else has a supporting role but does not contribute to what's being asserted on.
• The "tested corpus" is
  • the __main__ module
  • the tested_modules: modules provided through [file ...], [outfile ...] or [outfile-re ...]
• It follows that library code, whether imported from lib-stub/ or provided through [builtins ...] or [typing ...] will not be part of the "tested corpus".
• At times we want [file ...] to also only have a supporting role and not be part of the tested corpus. In tests we used to have conventions like excluding modules starting with _. Instead, we'd have an explicit [fixture ...] block that just like [file ...] except it doesn't participate in the "tested corpus".

[ikonst]

@hauntsaninja wdyt?

[ikonst]

also @ilevkivskyi

[ilevkivskyi]

@ikonst I think this is rather where @JukkaL may have some opinions.

[ikonst]

@JukkaL do you agree with the general approach that we should have fewer one-offs on the test suite level and a more consistent approach about what's included in the asserted-on results?

[ikonst]

processed_targets is (as the comment says) for testing only, specifically for TypeCheckSuite.

I'm adding the module's name so that TypeCheckSuite could perform filtering more coherently:

 actual = [
-    t
-    for t in actual
-    if not any(t.startswith(mod) for mod in core_modules + ["mypy_extensions"])
+    target
+    for module, target in res.manager.processed_targets
+    if module in testcase.test_modules
 ]

[ikonst]

Not sure if "there's a function for that" in stdlib, but seemed easy enough to add one here. (I'm sure it misses some cases, just wanted something short and practical.)

[ikonst]

@JukkaL 👋

[ikonst]

@JukkaL could you take a look please?

[github-actions]

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉

[JukkaL]

I did a quick pass and the overall approach looks good. I don't have time for a careful review right now, which this requires since there could be some risk of test false negatives. Does anybody have bandwidth to review this PR?

[ikonst]

This PR mostly changes how the "out" data is produced. It's not touching the assertion/comparison logic. Assuming this claim is correct (please review!), I'm not seeing a great risk of test false negatives (agreed?).

[ikonst]

@ilevkivskyi @hauntsaninja would you have bandwidth to review?

As I said, in terms of assert actual == expected, it affects how the actual is made, not how == works. So as long as we're happy with how it picks what to include in the output (you can see it from the tests I had to change), then it should be alright.