./wpt tests-affected can list tests when ./wpt test-jobs doesn't list stability

[foolip]

An oddity discovered in #13936 which by the looks of it isn't accidental, but still probably worth addressing.

Among the PRs I looked at, these didn't trigger stability but did have affected tests.

#13611 and #13877 (its revert) both modified idlharness.js, but all of resources/ is excluded:

wpt/tools/ci/jobs.py

Line 17 in 3422a5d

"!resources/*",

. ./wpt tests-affected still listed 150 affected tests, however. @lukebjerring, thoughts on this?

#13733 modified payment-request/META.yml, which is excluded, but META.yml is mentioned in a lot of tests in payment-request/. This isn't really a problem, though.

An open question is whether using ./wpt test-jobs --includes stability is necessary at all, or if it's better to just call ./wpt tests-affected. An upside of ./wpt test-jobs is that it doesn't require the manifest and so is faster.

[gsnedders]

I can't find where this comes from originally (it predates the wpt tool IIRC), but it was because testharness.js caused every single testharness.js to be rerun. You can imagine that didn't work well…

[foolip]

Right, so the question is if jobs.py should just exclude resources/testharness*, instead of all of resources/.

[gsnedders]

Arguably, once we move stability on to something with more capacity, we should run everything using testharness.js for every change of it.

[jgraham]

That seems a bit academic at the moment; not only would we need more capacity we'd need some good way to figure out how many chunks to run to get the results in a reasonable amount of time (and certainly less than the timeout of the runner). Irrespective of capacity, we haven't got any good solution for that at the moment, and getting one is some way off.

[gsnedders]

@jgraham Definitely. That said, at least doing one run so regressions could be surfaced would be useful.

[gsnedders]

(Though I guess that's basically just #13263.)

[qiuzhong]

@foolip , I got this error when running ./wpt affected-tests:

$ ./wpt affected-tests
usage: wpt [-h] [--venv VENV] [--debug]
           {tc-download,run,test-jobs,branch-point,create,lint,serve,check-stability,update-expectations,manifest,install,make-hosts-file,manifest-download,files-changed,tests-affected}
           ...
wpt: error: argument command: invalid choice: 'affected-tests' (choose from u'tc-download', u'run', u'test-jobs', u'branch-point', u'create', u'lint', u'serve', u'check-stability', u'update-expectations', u'manifest', u'install', u'make-hosts-file', u'manifest-download', u'files-changed', u'tests-affected')

I thought you mean ./wpt tests-affected but typed the wrong order for this argument.

[foolip]

@qiuzhong, thank, I've updated the title and description.