kola: support denylist warn or warning feature

[dustymabe]

The idea here is that we still support running the tests but bubble them up as warnings rather than hard failures with something like a warn: true knob in the denylist.

The test would still run but the overall process wouldn't error (or maybe it does error with a specific exit code; TBD) and then caller would decide how it wanted to handle the situation.

We could also combine this with snooze. So a snoozed test wouldn't run until $date and then when it did start running again it would be a warning, not an error.

[cgwalters]

It looks like coreos/fedora-coreos-config#2512 is going to add this field to every test. If that's the case...then why don't we just make warn: true the default, and have warn: false to go back to the prior behavior of not running the test at all?

Are there cases where we want warn: false even? Maybe, I'm not sure.

But it seems strange to me to just add a new thing to copy-paste into every denylisted test...

[nikita-dubrovskii]

@cgwalters

why don't we just make warn: true the default

Agree, updated PRs and set it to true by default.

[dustymabe]

It looks like coreos/fedora-coreos-config#2512 is going to add this field to every test. If that's the case...then why don't we just make warn: true the default, and have warn: false to go back to the prior behavior of not running the test at all?

IMO the behavior should be:

- pattern: fcos.internet
  tracker: https://github.com/coreos/coreos-assembler/pull/1478

behaves as it does today where it denylists the test and the test does not run.

- pattern: fcos.internet
  warn: true
  tracker: https://github.com/coreos/coreos-assembler/pull/1478

would make it so that the test runs but a test failure isn't fatal.

And this:

- pattern: fcos.internet
  warn: false
  tracker: https://github.com/coreos/coreos-assembler/pull/1478

Doesn't really make a lot of sense IMO.

Are there cases where we want warn: false even? Maybe, I'm not sure.

I don't think so (see above), but I don't know a super clear way to indicate that warn: false doesn't make sense.

But it seems strange to me to just add a new thing to copy-paste into every denylisted test...

I agree. We should only add warn: true to tests that we want to run and not have an error be fatal.

[dustymabe]

one further use case to flesh out: something like:

- pattern: fcos.internet
  snooze: 2023-08-20
  warn: true
  tracker: https://github.com/coreos/coreos-assembler/pull/1478

would NOT run the test until Aug 20th, and then when the test did start running again it would only warn. This makes dealing with snoozes easier (i.e. when the time expires our pipelines won't start failing, just warning).

[jlebon]

Isn't this fixed by #3539?

[dustymabe]

Yes. I think so.