New test harness

[aarnq]

What kind of PR is this?

Required: Mark one of the following that is applicable:

• kind/feature
• kind/improvement
• kind/deprecation
• kind/documentation
• kind/clean-up
• kind/bug

Optional: Mark one or more of the following that are applicable:

Important

Critical security fixes should be marked with kind/security
Breaking changes should be marked kind/admin-change or kind/dev-change depending on type

• kind/admin-change
• kind/dev-change
• kind/security
• kind/adr

What does this PR do / why do we need this PR?

This paves the way for more structured unit, integration, and end-to-end testing by using bats.

Additional information to reviewers

This has a long way to go before it can be used in the pipeline, and I want to integrate Cypress first so I know what requirements we have. I also aim to implement helpers for the current main test script and see if there is any way to generate tests from a matrix.

I've already did so for testing deployments and statefulsets, though we might want to rework it into a fetch-once-check-multiple solution as the old one did to improve the speed. Though with bats we can run things in parallel.

Cypress support is done! It is integrated with generators into the bats suite, but can be run manually as well. A support file is setup so we can add common functions if we need to, and a few config ones have been implemented.

There is also now a sort of generator for testing matrices, or at least the building blocks for doing so.

Screenshots

Checklist

• Proper commit message prefix on all commits
• Change checks:
  • The change is transparent
  • The change is disruptive
  • The change requires no migration steps
  • The change requires migration steps
• Metrics checks:
  • The metrics are still exposed and present in Grafana after the change
  • The metrics names didn't change (Grafana dashboards and Prometheus alerts are not affected)
  • The metrics names did change (Grafana dashboards and Prometheus alerts were fixed)
• Logs checks:
  • The logs do not show any errors after the change
• Network Policy checks:
  • Any changed pod is covered by Network Policies
  • The change does not cause any dropped packages in the NetworkPolicy Dashboard
• Pod Security Policy checks:
  • Any changed pod is covered by Pod Security Admission
  • Any changed pod is covered by Gatekeeper Pod Security Policies
  • The change does not cause any pods to be blocked by Pod Security Admission or Policies
• Falco checks:
  • The change does not cause any alerts to be generated by Falco

[robinAwallace]

Looks very nice 👍

Is there any plan to check if resources have the correct values?
So that thanos have the expected variables and secrets/configmaps contains the expected values?

[aarnq]

@robinAwallace that is not in the scope of this PR but in the future for sure.
I am working on methods of generating tests, and with that in place I think we could easily do loads of such tests. Because then you could just configure target cluster, resource, config key and config values and let it do the work.

[robinAwallace]

Perfect, understandably that it is not part of this PR 👍

[Xartos]

I think I would like to request a "meeting PR review" on this one. I don't think I'll be able to understand everything just by looking. Anyone else care to join? @robinAwallace @crssnd @simonklb

[robinAwallace]

As said on the retro, It looked very nice during the public PR and as such I will approve it. Good job André 👍

[simonklb]

Nice work!

I think we should consider adding the generate tests to the git repository though. I find it nice to track diffs when I update generated code. Interested to hear your take on it. (not blocking the merge btw)

[aarnq]

Nice work!

I think we should consider adding the generate tests to the git repository though. I find it nice to track diffs when I update generated code. Interested to hear your take on it. (not blocking the merge btw)

Maybe, I don't think I want to add it right now to keep the structure simple.
But if we start to implement more features into the generators (and add the output change/tracking tests) then we will definitely go in that direction.