Flaky reftests in Taskcluster stability jobs are not possible to debug

[foolip]

When reftests are found to be flaky in Taskcluster's stability jobs, the only information one can get from the logs is the number of passes and failures. If the problem doesn't reproduce locally there are no clues about how to proceed.

This affected #14469 and #14551 and. See also #14564.

[foolip]

@jgraham how do you think we should fix this? Even with TBPL logs, is reftest analyzer be able to extract multiple screenshots for the same test and show them?

[jgraham]

The reftest analyzer is not very clever, so probably? If we can get the data into the logs and it doesn't work I can modify it so it does.

[foolip]

@jgraham, the command is already something like ./wpt run --channel=nightly --verify --affected HEAD^ --log-tbpl-level=info --log-tbpl=- -y --no-pause --no-restart-on-unexpected --install-fonts --no-headless --install-browser firefox, with --log-tbpl=- coming from taskcluster-run.py, but it doesn't seem to have any effect. I guess that --verify overrides the output format?

[foolip]

@Hexcles OK if I assign this to you for the proximity to getting screenshots out of regular runs?

[foolip]

For bookkeeping, #14469 was merged while possibly still having a flaky reftest in Firefox.

[foolip]

@Hexcles is it still appropriate that this is assigned to you? FWIW, I haven't seen it come up again since I filed this issue, so it's not on fire :)