OCPBUGS-34545: Remove cloud-config and cloud-provider arguments #1696

nrb · 2024-05-31T18:59:08Z

This is effectively a backport of #1693 and #1656, combined.

The SHA used for library-go here is openshift/library-go@b8bcc87 to avoid unrelated library-go changes, and particularly the kube 1.30/go 1.22 bumps.

Signed-off-by: Nolan Brubaker <[email protected]>

The cloud provider configuration is no longer present, so this plugin will not work. Signed-off-by: Nolan Brubaker <[email protected]>

Signed-off-by: Nolan Brubaker <[email protected]>

nrb · 2024-06-03T15:03:50Z

/test ci/prow/e2e-gcp-operator-encryption-single-node

openshift-ci · 2024-06-03T15:03:56Z

@nrb: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test e2e-aws-ovn
/test e2e-aws-ovn-serial
/test e2e-aws-ovn-upgrade
/test e2e-gcp-operator
/test images
/test k8s-e2e-gcp
/test unit
/test verify
/test verify-deps

The following commands are available to trigger optional jobs:

/test e2e-aws-operator-disruptive-single-node
/test e2e-aws-ovn-single-node
/test e2e-azure-ovn
/test e2e-gcp-operator-encryption-aescbc
/test e2e-gcp-operator-encryption-aesgcm
/test e2e-gcp-operator-encryption-perf-aescbc
/test e2e-gcp-operator-encryption-perf-aesgcm
/test e2e-gcp-operator-encryption-perf-single-node
/test e2e-gcp-operator-encryption-rotation-aescbc
/test e2e-gcp-operator-encryption-rotation-aesgcm
/test e2e-gcp-operator-encryption-rotation-single-node
/test e2e-gcp-operator-encryption-single-node
/test e2e-gcp-operator-single-node
/test e2e-metal-ovn-ha-cert-rotation-shutdown-180d
/test e2e-metal-ovn-ha-cert-rotation-shutdown-360d
/test e2e-metal-ovn-ha-cert-rotation-shutdown-90d
/test e2e-metal-ovn-ha-cert-rotation-suspend-180d
/test e2e-metal-ovn-ha-cert-rotation-suspend-360d
/test e2e-metal-ovn-ha-cert-rotation-suspend-90d
/test e2e-metal-ovn-sno-cert-rotation-shutdown-180d
/test e2e-metal-ovn-sno-cert-rotation-shutdown-360d
/test e2e-metal-ovn-sno-cert-rotation-shutdown-90d
/test e2e-metal-ovn-sno-cert-rotation-suspend-180d
/test e2e-metal-ovn-sno-cert-rotation-suspend-1y
/test e2e-metal-ovn-sno-cert-rotation-suspend-2y
/test e2e-metal-ovn-sno-cert-rotation-suspend-3y
/test e2e-metal-ovn-sno-cert-rotation-suspend-90d
/test e2e-metal-single-node-live-iso
/test k8s-e2e-gcp-serial

Use /test all to run the following jobs that were automatically triggered:

pull-ci-openshift-cluster-kube-apiserver-operator-release-4.16-e2e-aws-operator-disruptive-single-node
pull-ci-openshift-cluster-kube-apiserver-operator-release-4.16-e2e-aws-ovn
pull-ci-openshift-cluster-kube-apiserver-operator-release-4.16-e2e-aws-ovn-serial
pull-ci-openshift-cluster-kube-apiserver-operator-release-4.16-e2e-aws-ovn-single-node
pull-ci-openshift-cluster-kube-apiserver-operator-release-4.16-e2e-aws-ovn-upgrade
pull-ci-openshift-cluster-kube-apiserver-operator-release-4.16-e2e-gcp-operator
pull-ci-openshift-cluster-kube-apiserver-operator-release-4.16-e2e-gcp-operator-encryption-aescbc
pull-ci-openshift-cluster-kube-apiserver-operator-release-4.16-e2e-gcp-operator-encryption-aesgcm
pull-ci-openshift-cluster-kube-apiserver-operator-release-4.16-e2e-gcp-operator-encryption-perf-aescbc
pull-ci-openshift-cluster-kube-apiserver-operator-release-4.16-e2e-gcp-operator-encryption-perf-aesgcm
pull-ci-openshift-cluster-kube-apiserver-operator-release-4.16-e2e-gcp-operator-encryption-perf-single-node
pull-ci-openshift-cluster-kube-apiserver-operator-release-4.16-e2e-gcp-operator-encryption-rotation-aescbc
pull-ci-openshift-cluster-kube-apiserver-operator-release-4.16-e2e-gcp-operator-encryption-rotation-aesgcm
pull-ci-openshift-cluster-kube-apiserver-operator-release-4.16-e2e-gcp-operator-encryption-rotation-single-node
pull-ci-openshift-cluster-kube-apiserver-operator-release-4.16-e2e-gcp-operator-encryption-single-node
pull-ci-openshift-cluster-kube-apiserver-operator-release-4.16-e2e-gcp-operator-single-node
pull-ci-openshift-cluster-kube-apiserver-operator-release-4.16-e2e-metal-single-node-live-iso
pull-ci-openshift-cluster-kube-apiserver-operator-release-4.16-images
pull-ci-openshift-cluster-kube-apiserver-operator-release-4.16-k8s-e2e-gcp
pull-ci-openshift-cluster-kube-apiserver-operator-release-4.16-k8s-e2e-gcp-serial
pull-ci-openshift-cluster-kube-apiserver-operator-release-4.16-unit
pull-ci-openshift-cluster-kube-apiserver-operator-release-4.16-verify
pull-ci-openshift-cluster-kube-apiserver-operator-release-4.16-verify-deps

In response to this:

/test ci/prow/e2e-gcp-operator-encryption-single-node

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

nrb · 2024-06-03T15:06:08Z

/test e2e-aws-operator-disruptive-single-node

nrb · 2024-06-03T15:06:23Z

/test e2e-gcp-operator-encryption-single-node

Manual backport of a04fc26 Signed-off-by: Nolan Brubaker <[email protected]>

openshift-ci · 2024-06-03T16:24:03Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: nrb
Once this PR has been reviewed and has the lgtm label, please assign vrutkovs for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: Nolan Brubaker <[email protected]>

JoelSpeed · 2024-06-10T14:27:52Z

/jira refresh

openshift-ci-robot · 2024-06-10T14:27:58Z

@JoelSpeed: This pull request references Jira Issue OCPBUGS-34545, which is invalid:

release note text must be set and not match the template OR release note type must be set to "Release Note Not Required"
expected dependent Jira Issue OCPBUGS-34544 to be in one of the following states: MODIFIED, ON_QA, VERIFIED, but it is POST instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

JoelSpeed · 2024-06-10T14:30:19Z

/retest-required

JoelSpeed · 2024-06-11T12:01:10Z

Dockerfile.rhel7

@@ -9,11 +9,11 @@ COPY --from=builder /go/src/github.com/openshift/cluster-kube-apiserver-operator
 COPY --from=builder /go/src/github.com/openshift/cluster-kube-apiserver-operator/bindata/bootkube/config /usr/share/bootkube/manifests/config/
 COPY --from=builder /go/src/github.com/openshift/cluster-kube-apiserver-operator/bindata/bootkube/manifests /usr/share/bootkube/manifests/manifests/
 COPY --from=builder /go/src/github.com/openshift/cluster-kube-apiserver-operator/bindata/bootkube/scc-manifests /usr/share/bootkube/manifests/manifests/
-COPY --from=builder /go/src/github.com/openshift/cluster-kube-apiserver-operator/vendor/github.com/openshift/api/apiserver/v1/apiserver.openshift.io_apirequestcount.yaml /usr/share/bootkube/manifests/manifests/
+COPY --from=builder /go/src/github.com/openshift/cluster-kube-apiserver-operator/vendor/github.com/openshift/api/apiserver/v1/zz_generated.featuregated-crd-manifests.yaml /usr/share/bootkube/manifests/manifests/zz_generated.featuregated-crd-manifests.yaml


This replacement isn't correct, you needed to add the path segment, not just a replace here

I updated the values based on a04fc26, but the jobs still appear to be failing. Image builds and pushes were fine.

JoelSpeed · 2024-06-11T12:01:25Z

Makefile

@@ -114,7 +114,7 @@ export TP_CMD_PATH ?=./cmd/cluster-kube-apiserver-operator

 # ensure the apirequestcounts crd is included in bindata
 APIREQUESTCOUNT_CRD_TARGET := bindata/assets/kube-apiserver/apiserver.openshift.io_apirequestcount.yaml
-APIREQUESTCOUNT_CRD_SOURCE := vendor/github.com/openshift/api/apiserver/v1/apiserver.openshift.io_apirequestcount.yaml
+APIREQUESTCOUNT_CRD_SOURCE := vendor/github.com/openshift/api/apiserver/v1/zz_generated.featuregated-crd-manifests.yaml


Also incorrect path here

Signed-off-by: Nolan Brubaker <[email protected]>

nrb · 2024-06-12T18:43:45Z

/retest

openshift-ci · 2024-06-12T21:26:00Z

@nrb: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-gcp-operator-encryption-perf-single-node	`6cf03c2`	link	false	`/test e2e-gcp-operator-encryption-perf-single-node`
ci/prow/e2e-gcp-operator-single-node	`6cf03c2`	link	false	`/test e2e-gcp-operator-single-node`
ci/prow/e2e-aws-operator-disruptive-single-node	`6cf03c2`	link	false	`/test e2e-aws-operator-disruptive-single-node`
ci/prow/e2e-aws-ovn-upgrade	`6cf03c2`	link	true	`/test e2e-aws-ovn-upgrade`
ci/prow/e2e-aws-ovn-serial	`6cf03c2`	link	true	`/test e2e-aws-ovn-serial`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

nrb · 2024-06-12T21:31:01Z

@JoelSpeed For some reason I don't understand, the kube-apiserver is still receiving the cloud-config argument in the failing tests.

Taking the e2e-aws-ovn-upgrade job as an example, the kube-apiserver log shows FLAG: --cloud-config="/etc/kubernetes/static-pod-resources/configmaps/cloud-config/cloud.conf"

The kubeapiserver-operator log shows \"cloud-config\":[\"/etc/kubernetes/static-pod-resources/configmaps/cloud-config/cloud.conf\"]

I'm not able to find any references to cloud.conf in the KASO code within this PR. The vendored library-go files have removed references to it. I do see some left over consts, but they aren't reference anywhere.

Is there somewhere else that I need to update? I think I'm missing something obvious.

JoelSpeed · 2024-06-13T09:08:03Z

@nrb I think this is an upgrade issue rather than an install time issue, looking at all of the jobs where the cluster is installed fresh, the issue does not present.

However, the operator today appears to be removing the cloud config configmap prior to rolling out updated pods, that is going to cause the existing pods to crash, and then that in turn will prevent the operator from rolling out the updated version of the pod spec.

We need to make sure the cloud config configmap is not removed until after the new pods have been rolled out.

I would have expected we already saw this with the KCMO but I can't see reference to how we fixed it 🤔

chao007 · 2024-07-02T06:08:53Z

Reproduce:
oc get pv/multizone-pv75sdv -o yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.beta.kubernetes.io/gid: "777"
    pv.kubernetes.io/bound-by-controller: "yes"
  creationTimestamp: "2024-07-02T05:22:16Z"
  finalizers:
  - kubernetes.io/pv-protection
  generateName: multizone-pv
  labels:
    topology.kubernetes.io/region: us-central1
    topology.kubernetes.io/zone: us-central1-c
  name: multizone-pv75sdv
  resourceVersion: "59982"
  uid: 54b5f58b-0539-4f5f-a33c-e8d741c639ae
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 2Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: pvc-qflhf
    namespace: e2e-multi-az-8087
    resourceVersion: "59980"
    uid: e97099f8-2bac-426f-b5f4-b93c18eb91d1
  gcePersistentDisk:
    fsType: ext3
    pdName: e2e-3c2ee91b-1f69-44db-be68-191c13ccedae
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: topology.kubernetes.io/zone
          operator: In
          values:
          - us-central1-c
        - key: topology.kubernetes.io/region
          operator: In
          values:
          - us-central1
  persistentVolumeReclaimPolicy: Retain
  volumeMode: Filesystem
status:
  lastPhaseTransitionTime: "2024-07-02T05:22:17Z"
  phase: Bound

After fix up:

oc get pv/multizone-pv967f9 -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.beta.kubernetes.io/gid: "777"
    pv.kubernetes.io/bound-by-controller: "yes"
  creationTimestamp: "2024-07-02T06:06:24Z"
  finalizers:
  - kubernetes.io/pv-protection
  generateName: multizone-pv
  name: multizone-pv967f9
  resourceVersion: "37953"
  uid: 915485a2-5263-4b3a-80d8-9391cb071616
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 2Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: pvc-2g4vh
    namespace: e2e-multi-az-6664
    resourceVersion: "37951"
    uid: ffc646a2-e671-46b5-8e05-192c6cab2ab1
  gcePersistentDisk:
    fsType: ext3
    pdName: e2e-9a53a94a-6a96-4d9c-9a50-89a93f2593c1
  persistentVolumeReclaimPolicy: Retain
  volumeMode: Filesystem
status:
  lastPhaseTransitionTime: "2024-07-02T06:06:24Z"
  phase: Bound

openshift-bot · 2024-09-30T09:00:55Z

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

nrb · 2024-10-08T15:25:06Z

/remove-lifecycle stale

elmiko

i've just started looking at this, but i am curious about the removal of the cloud-provider flag. are we still keeping this flag for when external ccms are in use?

JoelSpeed · 2024-10-14T11:30:14Z

are we still keeping this flag for when external ccms are in use?

It isn't required, there's no code in the API server that relies on cloud providers anymore, and there's no need for the API server to be aware of whether we are running an external platform or not either

elmiko · 2024-10-14T14:18:53Z

thanks Joel, i am a little paranoid based on kube-controller-manager stuff we found recently.

nrb added 2 commits May 31, 2024 14:55

Bump library-go to disable cloud provider flags

28f5dee

Signed-off-by: Nolan Brubaker <[email protected]>

Disable volume admission plugin

73e8e26

The cloud provider configuration is no longer present, so this plugin will not work. Signed-off-by: Nolan Brubaker <[email protected]>

openshift-ci bot requested review from deads2k and p0lyn0mial May 31, 2024 19:08

nrb added 2 commits May 31, 2024 15:47

Remove unused api server arguments

ec9df73

Signed-off-by: Nolan Brubaker <[email protected]>

Fix set typing

49a2eb9

Signed-off-by: Nolan Brubaker <[email protected]>

jsafrane mentioned this pull request Jun 3, 2024

4.16: OCPBUGS-34545: UPSTREAM: <carry>: Disable e2e tests that need PersistentVolumeLabel openshift/kubernetes#1985

Merged

Update kube-apiserver CRD manifest location

84fdbef

Manual backport of a04fc26 Signed-off-by: Nolan Brubaker <[email protected]>

Add feature gate package

0fb4f54

Signed-off-by: Nolan Brubaker <[email protected]>

nrb force-pushed the OCPBUGS-34545 branch from 7b02454 to 0fb4f54 Compare June 3, 2024 17:06

nrb added 2 commits June 3, 2024 13:29

Copy API server config from new location

c61d558

Signed-off-by: Nolan Brubaker <[email protected]>

Vendor in CRD YAML packages

4227063

Signed-off-by: Nolan Brubaker <[email protected]>

openshift-ci-robot added jira/severity-moderate Referenced Jira bug's severity is moderate for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. labels Jun 10, 2024

openshift-ci-robot added the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label Jun 10, 2024

JoelSpeed reviewed Jun 11, 2024

View reviewed changes

Fix path to apirequestcount CRD

6cf03c2

Signed-off-by: Nolan Brubaker <[email protected]>

nrb force-pushed the OCPBUGS-34545 branch from 311dd7e to 6cf03c2 Compare June 12, 2024 15:35

This was referenced Jun 19, 2024

[release-4.16] OCPBUGS-34545: UPSTREAM: <carry>: Disable PersistentVolumeLabel by default openshift/kubernetes#1993

Merged

[release-4.16] OCPBUGS-34545: Disable PersistentVolumeLabel admission plugin openshift/hypershift#4241

Merged

openshift-ci bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 30, 2024

openshift-ci bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 8, 2024

elmiko reviewed Oct 10, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OCPBUGS-34545: Remove cloud-config and cloud-provider arguments #1696

OCPBUGS-34545: Remove cloud-config and cloud-provider arguments #1696

nrb commented May 31, 2024

nrb commented Jun 3, 2024

openshift-ci bot commented Jun 3, 2024

nrb commented Jun 3, 2024

nrb commented Jun 3, 2024

openshift-ci bot commented Jun 3, 2024

JoelSpeed commented Jun 10, 2024

openshift-ci-robot commented Jun 10, 2024

JoelSpeed commented Jun 10, 2024

JoelSpeed Jun 11, 2024

nrb Jun 12, 2024

JoelSpeed Jun 11, 2024

nrb commented Jun 12, 2024

openshift-ci bot commented Jun 12, 2024

nrb commented Jun 12, 2024

JoelSpeed commented Jun 13, 2024

chao007 commented Jul 2, 2024

openshift-bot commented Sep 30, 2024

nrb commented Oct 8, 2024

elmiko left a comment

JoelSpeed commented Oct 14, 2024

elmiko commented Oct 14, 2024

OCPBUGS-34545: Remove cloud-config and cloud-provider arguments #1696

Are you sure you want to change the base?

OCPBUGS-34545: Remove cloud-config and cloud-provider arguments #1696

Conversation

nrb commented May 31, 2024

nrb commented Jun 3, 2024

openshift-ci bot commented Jun 3, 2024

nrb commented Jun 3, 2024

nrb commented Jun 3, 2024

openshift-ci bot commented Jun 3, 2024

JoelSpeed commented Jun 10, 2024

openshift-ci-robot commented Jun 10, 2024

JoelSpeed commented Jun 10, 2024

JoelSpeed Jun 11, 2024

Choose a reason for hiding this comment

nrb Jun 12, 2024

Choose a reason for hiding this comment

JoelSpeed Jun 11, 2024

Choose a reason for hiding this comment

nrb commented Jun 12, 2024

openshift-ci bot commented Jun 12, 2024

nrb commented Jun 12, 2024

JoelSpeed commented Jun 13, 2024

chao007 commented Jul 2, 2024

openshift-bot commented Sep 30, 2024

nrb commented Oct 8, 2024

elmiko left a comment

Choose a reason for hiding this comment

JoelSpeed commented Oct 14, 2024

elmiko commented Oct 14, 2024