Compass Manager - Cutover plan for STAGE and PROD rollout

[tobiscr]

Description

To ensure a controlled rollout of the Compass Manager, this cut-over plan is used to define the required steps, their order and owner:

• Compass Manager - Add metrics to improve observability #28
• Dry-Run on STAGE completed and results reviewed, Owner: SRE / @kyma-project/framefrog
• Enabling Compass Manager on STAGE,Owner: SRE
• Review with SRE required steps for PROD rollout, Owner: SRE / @kyma-project/framefrog

Requires

#87
kyma-project/control-plane#3088

More information
Please follow the migration plan prepared for the process.

Rollout workplan:

DEV environemnt

• Compass-manager is deployed and registers SKRs in Compass when application-connector module is enabled
• Modified Provisioner is deployed
• Run migration application to Generate CompassMappings for existing provisioned and connected runtimes
• Modify Provisioner overrides to disable integration with Compass

STAGE environemnt

• Compass-manager is deployed and registers SKRs in Compass when application-connector module is enabled
• Modified Provisioner is deployed
• SRE deploys compass-manager on stage
• SRE runs migration application to Generate CompassMappings for existing provisioned and connected runtimes
• Modify Provisioner overrides to disable integration with Compass

PROD environemnt

• Compass-manager is deployed and registers SKRs in Compass when application-connector module is enabled
• Microdelivery Issue is accepted
• Missing Component Requirements are delivered
  • Backup Requirements (#backlog/5354)
  • RBAC report (#backlog/5374)
• Modified Provisioner is deployed
• Run migration application to Generate CompassMappings for existing provisioned and connected runtimes (#backlog/5345)
• Modify Provisioner overrides to disable integration with Compass
• Manual test of Application Integration with Kyma

Stretch:

• - Fully remove Compass Integration code from Provisioner Provisioner - fully remove obsolete code for runtime registration/deregistration in compass control-plane#3411

[mvshao]

Before STAGE and PROD rollout we need to migrate old runtimes that are connected to the Compass and don't have the Application Connector module enabled (creation of the Compass Manager Mapping CRs for already registered runtimes)

[tobiscr]

depends on #87

[zhoujing2022]

@koala7659 as we discussed please let me know if the tasks are done before noon of 10th.Jan. thx.

[zhoujing2022]

Compass mgr. onto STAGE: it’s postponed to 18th.Jan.2024
Compass mgr. onto PRODUCTION: it’s postponed to 25th.Jan.2024

[Sawthis]

Compass mgr. onto STAGE: it’s postponed to 15th.Mar.2024
Compass mgr. onto PRODUCTION: it’s postponed to 22nd.Mar.2024

[koala7659]

We detected a problem when after disable integration in Provisioner with Compass to KCP the newly created Kyma Runtime will be missing configuration for Compass Runtime Agent which will cause The Compass Runtime Agent to crash.

To prevent that problem we can combine Compass Manager release with Application Connector module 1.1.0 release. In such a case Compass Runtime Agent will be deployed on Kyma Runtime only when Application Connector is enabled and will be configured correctly. The Rollout plan will contain several new steps.

• Compass-manager is deployed on KCP and registers SKRs in Compass when Application Connector module is enabled
• Modified Provisioner service is deployed to KCP
• Run migration application to Generate CompassMappings for existing provisioned and connected runtimes
• Prepare Application Connector module 1.1.0 WITH COMPASS RUNTIME AGENT in Experimental channel
• Update new Kyma WITHOUT COMPASS RUNTIME AGENT only on target environment
• Upgrade Application Connector module 1.1.0 WITH COMPASS RUNTIME AGENT in fast/regular channel on target environment
• Disable reconciliation for Compass Runtime Agent only on target environment. Reconciler can be disabled.
• Modify Provisioner overrides to disable integration with Compass

This approach requires to prepare new release of Application Connector first.
The module template for fast/regular will have to be applied manually on each separate target environment without submission pipeline
It is a problem but also gives us opportunity to remove legacy reconciliator sooner than we planned.

Comment:
The migration plan above was discussed and we decided it is too risky as "big bang" approach.
Instead of this we will do the rollout in the way we described in main issue

[koala7659]

PR with overrides for Dev /kyma/management-plane-config/pull/5043

[koala7659]

@zhoujing2022
Unfortunately we must postpone STAGE delivery again.
After we disabled Compass Registration for Provisioner on KCP DEV we discovered that installation of Kyma Runtimes started to fail. We decided to revert the flags, and we must fix Provisioner now. The fix is not a big but we will need at least two days to deliver.

[koala7659]

Fix for Provisioner not creating istio-system namespace when Compass Registration is disabled kyma-project/control-plane#3415

[koala7659]

Change request issue for the process [internal github address] /kyma/backlog/issues/5345