OCP 4.11 Upgrade Main Epic

[tbaker1313]

Describe the epic
Top level epic to track all OCP 4.11 upgrade work separate of quarters or sprints. Sub tasks may end up in multiple epics for better tracking and grouping.

Additional context
https://docs.openshift.com/container-platform/4.11/release_notes/ocp-4-11-release-notes.html

Notable items in release notes:
Changes that impact Developers:

• In the Web Console in the developer perspective, you can add your GitHub repository containing pipelines to the OpenShift Container Platform cluster. You can now run pipelines and tasks from your GitHub repository on the cluster when relevant Git events, such as push or pull requests are triggered.
• With this update, you can create a customized pipeline using your own set of curated tasks. You can search, install, and upgrade your tasks directly from the developer console.
• The OpenShift Container Platform web console now supports the dark mode theme.
• Pod Disruption Budgets can now be configured from the Web Console
• LegacyServiceAccountTokenNoAutoGeneration is on by default. As a result, when creating new service accounts (SA), a service account token secret is no longer automatically generated. If you need a service account token secret, you must manually use the TokenRequest API to request bound service account tokens or create a service account token secret.
• There's a known issue with api changes in the oc client for 4.11 . In the past the 4.10 oc client would automatically fix errors like using v1 for the API definition. Using non-groupfied API resources has been removed from the 4.11 oc client and will no longer automatically fix these issues.
• There's a change in how pods and volumes work in regards to SCC, some pods may get a new annotation which can prevent things from running. More info is https://access.redhat.com/articles/6973044 - opened RH case 03397754

Bug that impacts Mac OS users:

• The OpenShift CLI (oc) for OpenShift Container Platform 4.11 does not work properly on macOS due to a change in error handling of untrusted certificates in Go 1.18 libraries. Due to this change, oc login and other oc commands can fail with a certificate is not trusted error without proceeding further when running on macOS. Until the error handling is properly fixed in Go 1.18 (tracked by Go issue #52010, the workaround is to use the OpenShift Container Platform 4.10 oc CLI instead.

Changes that impact admins:

• Previously, any arguments provided to the toolbox command were ignored when the command was first invoked. This fix updates the toolbox script to initiate the podman container create command followed by the podman start and podman exec commands. It also modifies the script to handle multiple arguments and whitespaces as an array. As a result, the arguments passed to the toolbox command are executed every time as expected.
• The haproxy.router.openshift.io/balance variable, which sets the router load-balancing algorithm, now defaults to the value random instead of leastconn.
• Alertmanager to 0.24.0
• kube-state-metrics to 2.5.0
• Prometheus to 2.36.2
• Prometheus operator to 0.57.0
• Thanos to 0.26.0
• Create, browse, and manage PromQL queries more easily in the web console
• The Machine Config Operator (MCO) now updates the affected nodes alphabetically by zone, based on the topology.kubernetes.io/zone label. If a zone has more than one node, the oldest nodes are updated first. For nodes that do not use zones, such as in bare metal deployments, the nodes are upgraded by age, with the oldest nodes updated first. Previously, the MCO did not consider zones or node age.
• Enhanced notification for paused Machine Config Pools upon certificate renewal
• If you have Operator projects that were previously created or maintained with Operator SDK 1.16.0, update your projects to keep compatibility with Operator SDK 1.22.0.

Security related changes:

• Audit logs now include OAuth server audit events
• Pod security admission is now enabled on OpenShift Container Platform. Pod admission is enforced by both pod security and security context constraints (SCC) admissions. Pod security admission runs globally with privileged enforcement and restricted audit logging and API warnings. See https://kubernetes.io/docs/concepts/security/pod-security-admission/

Proposed Implementation Schedule
This schedule is subject to change until we have completed the first LAB upgrade to OPC 4.11. Currently we do feel that the timelines for PROD clusters have enough lee-way to not likely need extension.

• First LAB cluster is estimated to be completed prior to December 15th 2022.
• Second LAB cluster is estimated to be completed the week prior to commencing SILVER.
• Proposed RFC change window for SILVER upgrade would be January 22nd-February 3rd 2023.
• Proposed RFC change window for GOLD upgrade would be February 5th-10th 2023.
• Proposed RFC change window for GOLDDR upgrade would be February 12th-17th 2023.

Some lee-way between doing the upgrades for the two LAB clusters accounts for some staffing level changes due to vacations, and also allows us to reach out to Matt Robson one last time to make sure there's not a new 4.11.x release about to drop that we absolutely want to include, and thus need to quickly go through LAB upgrades again.

Definition of done

• All clusters upgraded to OCP 4.11 and all post-upgrade tasks completed.

[wmhutchison]

Per discovery today, something that was supposed to have been removed in OCP 4.9 did not in fact do so until OCP 4.11. Original link to this removal here: https://docs.openshift.com/container-platform/4.9/release_notes/ocp-4-9-release-notes.html#ocp-4-9-apiversion-v1-no-group

[wmhutchison]

updated estimated timelines since we're past the defined dates already. Can update further after meeting this week to finalize things.

[tbaker1313]

This is being closed as a duplicate of #3310