add support for snoozing tests

[dustymabe]

Feature Request

It would be nice to be able to snooze a CI test for a period of time. In this case the failure is a known issue and is the cause of something else (like a transient infrastructure issue) or being worked on upstream.

Something like this could be really useful for our rawhide stream where we'll see failures often and need to temporarily ignore issues while we work with upstream to get them fixed.

[cgwalters]

Tangentially related I'd like to add tiers to our tests, like tier-1, tier-2 etc, probably as tags. My strawman would be "boots, core Ignition bits work (but perhaps not raid-1 e.g.), runs containers and upgrades" is tier-1.

[dustymabe]

Tangentially related I'd like to add tiers to our tests, like tier-1, tier-2 etc, probably as tags. My strawman would be "boots, core Ignition bits work (but perhaps not raid-1 e.g.), runs containers and upgrades" is tier-1.

Good idea. Mind opening a separate issue for that?

[dustymabe]

One idea for how to implement this feature:

We already have a kola-denylist.yaml. We could add an extra entry in there. Something like:

- pattern: ext.config.podman.rootless-systemd
  tracker: https://github.com/containers/buildah/issues/3071
  streams:
    - branched
    - rawhide
  snooze_until: 20210731

So this test would be snoozed until the end of july on the branched and rawhide streams.

[jlebon]

I think we should really fold the denylist stuff into kola proper before we start adding more sophistication to it (that's #1165).

[dustymabe]

I think we should really fold the denylist stuff into kola proper before we start adding more sophistication to it (that's #1165).

After a quick discussion a rough summary of the steps that would need to be taken are:

• check for kola-denylist.yaml in the source config
• parse it, accumulating the test filters there
• filter out based on the arch we're running from
• filter out based on the FCOS stream we're targeting
• leave all the other tests active
• remove denylist sugar from cmd-kola https://github.com/coreos/coreos-assembler/blob/main/src/cmd-kola#L47

then we can get back to this RFE:

• add ability to snooze a test for a period of time