CAPV: Release v29.0.0.

[njuettner]

Towards: giantswarm/roadmap#3710

Checklist

• Roadmap issue created
• Release uses latest stable Flatcar
• Release uses latest Kubernetes patch version

Triggering E2E tests

To trigger the E2E test for each new Release added in this PR, add a comment with the following:

/run releases-test-suites

If you want to trigger conformance tests, you can do so by adding a comment similar to the following:

/run conformance-tests PROVIDER=capa RELEASE_VERSION=29.1.0

For more details see the README.md.

[TheoBrigitte]

Can we have observability-bundle 1.7.0 as part of this release ?

[Gacko]

@vxav: Tests are failing, because the image hasn't been copied to vSphere, yet. Can you make sure the image for Flatcar 3975.2.2, Kubernetes 1.29.10 and OS Tooling 1.20.1 is present? Thank you!

[njuettner]

@vxav: Tests are failing, because the image hasn't been copied to vSphere, yet. Can you make sure the image for Flatcar 3975.2.2, Kubernetes 1.29.10 and OS Tooling 1.20.1 is present? Thank you!

Copied it 👍🏻

[Gacko]

The same observability-bundle version used in this PR is also being used in already released WC releases. They worked perfectly fine when they got released and we didn't have any issues with tests.

Now it seems like you recently introduced a change to the logging-operator which obviously affects the observability-bundle in existing WC releases and now needs to be fixed to make these releases work again.

As a customer I'd expect these releases to be tested and working, so cluster creation shouldn't break out of nowhere.

Can you please elaborate on what has been changed in logging-operator and how this affects existing WC releases? I'd expect these releases to be stable and immutable and implementing changes in an MC operator, which changes the behavior observability-bundle in existing releases, definitely breaks this contract.

[Gacko]

/run releases-test-suites TARGET_SUITES=./providers/capv/standard PREVIOUS_RELEASE=28.0.1 TARGET_RELEASES=vsphere-29.0.0

[tinkerers-ci]

releases-test-suites

Run name	pr-releases-1459-releases-test-suites9rnx2
Commit SHA	a9620c4
Result	Succeeded ✅

📋 View full results in Tekton Dashboard

Rerun trigger:
/run releases-test-suites

---

Tip

To only re-run the failed test suites you can provide a TARGET_SUITES parameter with your trigger that points to the directory path of the test suites to run, e.g. /run releases-test-suites TARGET_SUITES=./providers/capa/standard to re-run the CAPA standard test suite. This supports multiple test suites with each path separated by a comma.

Alternatively, or in addition to, you can also specify TARGET_RELEASES to trigger tests for specific releases. E.g. /run releases-test-suites TARGET_SUITES=./providers/capa/standard TARGET_RELEASES=aws-25.0.0-test.1

[njuettner]

@giantswarm/team-rocket I think we can finally move on, if you could take another look?

[Gacko]

/run releases-test-suites TARGET_SUITES=./providers/capv/upgrade PREVIOUS_RELEASE=28.0.1 TARGET_RELEASES=vsphere-29.0.0

[tinkerers-ci]

releases-test-suites

Run name	pr-releases-1459-releases-test-suitesr6lqz
Commit SHA	a9620c4
Result	Failed ❌

📋 View full results in Tekton Dashboard

Rerun trigger:
/run releases-test-suites

---

Tip

To only re-run the failed test suites you can provide a TARGET_SUITES parameter with your trigger that points to the directory path of the test suites to run, e.g. /run releases-test-suites TARGET_SUITES=./providers/capa/standard to re-run the CAPA standard test suite. This supports multiple test suites with each path separated by a comma.

Alternatively, or in addition to, you can also specify TARGET_RELEASES to trigger tests for specific releases. E.g. /run releases-test-suites TARGET_SUITES=./providers/capa/standard TARGET_RELEASES=aws-25.0.0-test.1

[njuettner]

/run releases-test-suites TARGET_SUITES=./providers/capv/upgrade PREVIOUS_RELEASE=28.0.1 TARGET_RELEASES=vsphere-29.0.0

[tinkerers-ci]

releases-test-suites

Run name	pr-releases-1459-releases-test-suites2cwrg
Commit SHA	a9620c4
Result	Failed ❌

📋 View full results in Tekton Dashboard

Rerun trigger:
/run releases-test-suites

---

Tip

To only re-run the failed test suites you can provide a TARGET_SUITES parameter with your trigger that points to the directory path of the test suites to run, e.g. /run releases-test-suites TARGET_SUITES=./providers/capa/standard to re-run the CAPA standard test suite. This supports multiple test suites with each path separated by a comma.

Alternatively, or in addition to, you can also specify TARGET_RELEASES to trigger tests for specific releases. E.g. /run releases-test-suites TARGET_SUITES=./providers/capa/standard TARGET_RELEASES=aws-25.0.0-test.1

[QuentinBisson]

/run releases-test-suites TARGET_SUITES=./providers/capv/upgrade PREVIOUS_RELEASE=28.0.1 TARGET_RELEASES=vsphere-29.0.0

[tinkerers-ci]

releases-test-suites

Run name	pr-releases-1459-releases-test-suitesg68nr
Commit SHA	a9620c4
Result	Failed ❌

📋 View full results in Tekton Dashboard

Rerun trigger:
/run releases-test-suites

---

Tip

To only re-run the failed test suites you can provide a TARGET_SUITES parameter with your trigger that points to the directory path of the test suites to run, e.g. /run releases-test-suites TARGET_SUITES=./providers/capa/standard to re-run the CAPA standard test suite. This supports multiple test suites with each path separated by a comma.

Alternatively, or in addition to, you can also specify TARGET_RELEASES to trigger tests for specific releases. E.g. /run releases-test-suites TARGET_SUITES=./providers/capa/standard TARGET_RELEASES=aws-25.0.0-test.1

[QuentinBisson]

/run releases-test-suites TARGET_SUITES=./providers/capv/upgrade PREVIOUS_RELEASE=28.0.1 TARGET_RELEASES=vsphere-29.0.0

[Gacko]

@QuentinBisson or someone else from @giantswarm/team-atlas: Can you please reply to this? It would be very helpful, even if only for documentation. Also I'd be interested in what has changed between the different runs of Releases Test Suites as I'd prefer to see them reliably fixed instead of having them pass once out of ten. 🙂

[tinkerers-ci]

releases-test-suites

Run name	pr-releases-1459-releases-test-suitesq2gvn
Commit SHA	a9620c4
Result	Succeeded ✅

📋 View full results in Tekton Dashboard

Rerun trigger:
/run releases-test-suites

---

Tip

To only re-run the failed test suites you can provide a TARGET_SUITES parameter with your trigger that points to the directory path of the test suites to run, e.g. /run releases-test-suites TARGET_SUITES=./providers/capa/standard to re-run the CAPA standard test suite. This supports multiple test suites with each path separated by a comma.

Alternatively, or in addition to, you can also specify TARGET_RELEASES to trigger tests for specific releases. E.g. /run releases-test-suites TARGET_SUITES=./providers/capa/standard TARGET_RELEASES=aws-25.0.0-test.1

[QuentinBisson]

@Gacko I'll write something on monday, I wanted to focus on fixing this first but I did not forget your message :)

[Gacko]

Ok, thank you!

I'll run the standard tests one last time and merge this PR once they pass.

/run releases-test-suites TARGET_SUITES=./providers/capv/standard TARGET_RELEASES=vsphere-29.0.0

[QuentinBisson]

There's currently a fixed branch of the logging operator on gcapeverde so tests should work 🤞🏻

[QuentinBisson]

I was going to run them again anyway 😅

[tinkerers-ci]

releases-test-suites

Run name	pr-releases-1459-releases-test-suitesp5rrg
Commit SHA	a9620c4
Result	Succeeded ✅

📋 View full results in Tekton Dashboard

Rerun trigger:
/run releases-test-suites

---

Tip

To only re-run the failed test suites you can provide a TARGET_SUITES parameter with your trigger that points to the directory path of the test suites to run, e.g. /run releases-test-suites TARGET_SUITES=./providers/capa/standard to re-run the CAPA standard test suite. This supports multiple test suites with each path separated by a comma.

Alternatively, or in addition to, you can also specify TARGET_RELEASES to trigger tests for specific releases. E.g. /run releases-test-suites TARGET_SUITES=./providers/capa/standard TARGET_RELEASES=aws-25.0.0-test.1

[QuentinBisson]

The same observability-bundle version used in this PR is also being used in already released WC releases. They worked perfectly fine when they got released and we didn't have any issues with tests.

Now it seems like you recently introduced a change to the logging-operator which obviously affects the observability-bundle in existing WC releases and now needs to be fixed to make these releases work again.

As a customer I'd expect these releases to be tested and working, so cluster creation shouldn't break out of nowhere.

Can you please elaborate on what has been changed in logging-operator and how this affects existing WC releases? I'd expect these releases to be stable and immutable and implementing changes in an MC operator, which changes the behavior observability-bundle in existing releases, definitely breaks this contract.

We are indeed configuring the observability-platform apps in releases via operators. We initialy built the logging and observability operators as a safety mechanism to be able to change some of our apps config on the fly (for prometheus-agent and so on) to counteract the lack/slowness of customer upgrades in the past because we were swarmed with lots of day and night alerts and that was unsufferable and waiting for a customer to upgrade was a no-go.

We used this mechanism to build some features on our apps as well like:

• sharding of the monitoring agent based on the metrics on prometheus and Mimir (because that would require someone managing KEDA on WCs and no one wants to)
• secret management
• dynamic enabling/disabling of loggging and monitoring at the cluster level

This used to work quite well in the past but we recently enabled a feature flag on the logging-operator that replaced promtail with alloy in the observability platform giantswarm/logging-operator#246 which caused issues last week.

This changed had been manually tested in the past but it missed that the grafana-agent application was failing to deploy (because of an issue with it's CRD management in our currently deployed release of it) which broke the cluster creation test.

In the mean time, a configuration breaking change in the alloy secret management was introduced in the observability-bundle and the change was not properly reflected in the logging-operator which caused the upgrade test to fail because the secret for alloy was not created and so alloy in CAPV 28 did not actually deploy :(

Last week, this became problematic and we are definitely sorry about all of this :(. I am opening a PM today so we can investigate how we can move forward without a lot of the operator work going on behind the hood (we need some config coming from MCs like secret to talk to loki and so on but not as much as we have today) but it should be our priority that we find something that does not break any existing releases.
Would you be up to a discussion to find out how we could integrate better?

By the way, we've been having discussions about this topic for years now and I really thought everyone was aware of it. We really need to find a better way to move forward (cc @JosephSalisbury) and that will require improvements on the release and delivery process as well :)