Manifests WG and Kubeflow 1.6 release

[annajung]

Hello @kubeflow/wg-manifests-leads

The 1.6 release team will use this tracking issue to coordinate the integration of the Manifests with the Kubeflow 1.6 release.

Heads up that the feature freeze is scheduled for Wednesday, Jun 1st, 2022. For the full release schedule, refer to the 1.6 release timeline.

(note: links are pending kubeflow/community#558 and kubeflow/community#560)

To prepare for the 1.6 release, please provide answers to the following questions:

1. What version of Manifests would you like to include for the 1.6 release?
2. Could you provide a branch/tag for this version? (note: It doesn't have to be final)
3. Can you share a link to all open / work in progress issues that you're expecting to include in the 1.6?
4. If you're using the following dependencies, could you provide the versions you'll be supporting for each dependency?

• Knative
• Istio
• Kubernetes
• Cert-Manager
• Kustomize

The release team WG liaison for Manifests WG will be @kimwnasptd @DomFleischmann @DnPlas nand you can keep track of high-level release updates in the 1.6 tracking issue.

[kimwnasptd]

Thanks for opening the issue @annajung! We are still evaluating these versions, but I'll be back very soon with a definitive answer.

What we are currently targeting is:

1. Knative 0.23.2
2. For Istio we are evaluating 1.13, since the upgrade path seems simple enough https://istio.io/latest/news/releases/1.13.x/announcing-1.13/upgrade-notes/ https://istio.io/latest/news/releases/1.12.x/announcing-1.12/upgrade-notes/, but we need to thoroughly test this
3. For K8s we want to start looking into 1.23, for the common components like Istio, Knative and Cert-Manager
4. For Kustomize we are still trying to get to 4+, with main blocker being Upgrade to use Kustomize v4+ #1797 (comment)

[juliusvonkohout]

I have two important things for 1.6

1. Update minio and metadata-envoy networkpolicy #2205 - updating the networkpolicies
2. feat: upgrade oidc-authservice #2150 - new authentication methods in oidc-authservice

[annajung]

Due to a blocker that was identified last week, we are delaying the release for 2 weeks, with the possibility of another extension. The new feature freeze date is Wednesday, Jun 15th, 2022. If another extension is needed, another announcement will be made to notify the community.

• Tracking issue: Alternative solution to removal of test on optional-test-infra testing#1006
• Email announcement: https://groups.google.com/u/2/g/kubeflow-discuss
• New schedule PR: Update release timeline community#561

[annajung]

After checking with all WGs, we are delaying the release for 2 more weeks. The new feature freeze date is Wednesday, Jun 29th, 2022.

Email announcement: https://groups.google.com/g/kubeflow-discuss/c/I4l97HvrGEA/m/227aCe_mCgAJ
New schedule PR: kubeflow/community#562

[kimwnasptd]

@annajung @kubeflow/release-team we are running a little bit behind for syncing with the manifests. Specifically these are the pending items for the RC0:

1. Syncing the manifests of the different repos
   • I don't have the RC tag for Notebooks yet Notebook WG and Kubeflow 1.6 release #2199 (comment)
   • I don't have the RC tag for Katib, but I see that @johnugeorge has this under control and will most probably be ready tomorrow Release v0.14.0 tracking  katib#1908
   • I don't know which tag to use for KFP
   • I'm going to use KServe 0.8, unless we have any further news cc @mstopa @pvaneck
2. @NickLoukas and I are preparing some basic PRs for applying all KF manifests in a KinD 1.22 cluster. This is almost done and will be up first thing tomorrow Test that each component can be applied in a K8s 1.22 cluster #2230
3. Update Istio manifests to 1.14 common: Add Istio 1.14.1 manifests #2233 Kubeflow 1.6 Dependency Versions #2207 (comment)
4. Update Knative to 1.4 Update Knative 1.4 #2231

I don't think that I'll manage to make the above today, but they are definitely manageable to do first thing tomorrow

[annajung]

Thanks for the update @kimwnasptd!

Looks like both Notebook and Katib will have their tags by tomorrow

@kubeflow/wg-pipeline-leads @zijianjoy, can you provide us with an update on the tag to use?

[annajung]

Hi @kubeflow/wg-pipeline-leads @zijianjoy @james-jwu pipeline leads, after various attempts to clarify the version for KF 1.6, the release team hasn't received a response yet on what tag to use for 1.6 since previously mentioned tag v2.0.0 is not available.

With the feature freeze being yesterday, the release team will go ahead with the latest stable version available for pipelines as of June 30th, which is 1.8.2.

Please reach out to us if you have any concerns! thank you!

[zijianjoy]

There are so many threads going on and I remembered I have communicated that we are going to use KFP v2.0.0 a while back. So reconfirming the version we are going to use for KFP: We are going to use KFP 2.0.0-alpha.2 in Kubeflow 1.6 release: https://github.com/kubeflow/pipelines/releases/tag/2.0.0-alpha.2.

[annajung]

Thanks @zijianjoy, we were under the same understanding until this Monday. During the release team meeting this week, it was brought to our attention that 1.8.x is being considered for the KF 1.6 release and that it was discussed in the last pipeline WG meeting. Therefore, we wanted to verify with the Pipeline WG team again in case there was a change in decision with the release delays. There must have been some miscommunication that happened between the two groups that caused all this confusion at the last min.

I agree that there are too many communications happening in multiple places with different people. We need to improve this process for both WG leads and for the release team members. I'm hoping we can discuss this more during the release meeting and find a better way to get the correct versions into the manifest repo before the feature freeze.

We also weren't sure if you wanted us to use the pre-release 2.0.0-alpha.2 or provide a new alpha tag, so thanks for confirming which tag to include in the release!

[surajkota]

@zijianjoy As discussed in last community meeting, 2.0.0.alpha2 was cut before 1.8.2 and hence does not include the necessary cert-manager overlays. If pipeline want to use 2.x, can you please provide new alpha tag to avoid regression?

[annajung]

Hi @zijianjoy manifest test is using cert manager overlays and would require revert for 2.0.0-alpha2. In reverting, it would break AWS again and we like to hear from you in regards to pipeline support for cert manager overlays for 2.0.0.

Could you provide a new tag for us to use for 1.6 with support for cert manager overlays? Or are there concerns and should we revert the test back?

[zijianjoy]

I have raised the request to make new alpha to @chensun, but he is out-of-office for the coming week. Once he is back, we can make a new release.

[annajung]

Currently, with the buffer time that we have already scheduled between the feature freeze and manifest testing, I believe the release team has some flexibility to wait for the new KFP release until the start of the manifest testing on July 13th.

If a new KFP release is not available by then, the release team will work with Manifest WG to revert the manifest testing to support 2.0.0-alpha2 to cut a 1.6 RC for the manifest testing phase. Once the new KFP release is available, we’ll cut a new RC then. This is to make sure that we still move forward with the release without delaying other deadlines.

@zijianjoy @chensun hopefully you can provide us with a new KFP release by Wed July 13th! :)

[zijianjoy]

cc @chensun : Are we able to cut a new 2.0.0 alpha version for KFP backend by July 13th?

[chensun]

@annajung We'll target the next alpha release on or before 7/13.

[kimwnasptd]

@annajung Knative 1.4 has a minimum version requirement of K8s 1.22 https://github.com/knative/serving/releases/tag/knative-v1.4.0

Considering the fact that this will be the first KF release with full K8s 1.22 support, I propose to use Knative 1.2 instead. This version should be compatible with both K8s 1.21 and 1.22 to allow a transition period for end users.

@pvaneck @yuzisun is there going to be an issue with using Knative 1.2 and KServe 0.8?

[chensun]

@annajung We'll target the next alpha release on or before 7/13.

@annajung We released KFP 2.0.0-alpha.3 today: https://github.com/kubeflow/pipelines/releases/tag/2.0.0-alpha.3

[annajung]

Thanks @chensun!

PR to sync 2.0.0-alpha.3 to manifest is available #2250. Once PR is reviewed and merged, hoping @kimwnasptd can help create the first 1.6 RC!

[pvaneck]

@pvaneck @yuzisun is there going to be an issue with using Knative 1.2 and KServe 0.8?

@kimwnasptd I think Knative 1.2 should also be fine.

[johnugeorge]

Katib v0.14.0 release is created from release-0.14 branch. @kimwnasptd Please sync this branch before the next release

Ref: kubeflow/katib#1908 (comment)

[johnugeorge]

Training operator v1.5.0 is out from v1.5-branch

Ref: kubeflow/training-operator#1622 (comment)

[annajung]

With KF 1.6 released, closing the issue - 1.6.1 related work is tracked at #2194