Connectivity-proxy module template resources are sometimes not cleaned up after the module is disabled

[koala7659]

Description

During testing of release candidate of new connectivity-proxy module (version 1.0.3) we noticed an issue where after disabling the module in Busola sometimes the operator and custom resource definition were not removed from the Kyma cluster. As a result all module resources (installed by operator) were gone but the resources of the module manifest (installed by module manager) seem to stay.

This happens randomly and really not very often (less than 5% of cases) but we could reproduce it several times during last few days. It would be good to understand what is the root cause and how we could possibly avoid this problem.

Steps to reproduce

The module manifest is currently available only in experimental channel

1. Enable module,
2. wait until all resources are installed
3. disable the module
4. wait until all resources are cleaned up.

All actions (1-4) should be repeated many times (like 20). At some point after cleanup you can see that operator pod will not be terminated.

Environment Type

Managed

Environment Info

Kubernetes Version: 1.27.x
Kyma CLI Version: 2.19.0
Kyma Runtime version: 2.20.0
Landscape: Canary
Provider: AWS
Region: Europe (Frankfurt) - Canary
Global Account: 1815c4ab-caa7-4b52-87ff-65ed47424dd6
Subaccount: 59fff536-6988-4d6a-8e09-eff6cb4f3696

Attachments

[kyma-bot]

@tobiscr: The label(s) /label area/connectivity-proxy cannot be applied. These labels are supported: tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, Epic. Is this label configured under labels -> additional_labels or labels -> restricted_labels in plugin.yaml?

In response to this:

/label area/connectivity-proxy

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tobiscr]

After alignment between @koala7659 and @kyma-project/jellyfish , we agreed to treat this issue not as a blocker for the CP module rollout.

[ruanxin]

Can't reproduce this behaviour, tested on STAGE.

The logic of provision and deprovision is idempotent, can't imagine such situation that succeeded several times, but failed in the end, if it ends like this, my assumption is during the whole testing period, it is affected by some external process, however, it's hard to reproduce it purely by this description.

Close it now.