Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Numerous kubeflow manifests can't deploy on Kubernetes 1.16 #375

Closed
jbrette opened this issue Sep 22, 2019 · 30 comments · Fixed by #1287
Closed

Numerous kubeflow manifests can't deploy on Kubernetes 1.16 #375

jbrette opened this issue Sep 22, 2019 · 30 comments · Fixed by #1287

Comments

@jbrette
Copy link
Contributor

jbrette commented Sep 22, 2019

Old declaration of key objects such as DaemonSet, StatefulSet and Deployment can't be deployed on Kubernetes 1.16 anymore if they are using the old apiVersion such as extensions/v1beta or the apps/v1beta

  • Deployments are supposed to use apps/v1 since kubernetes 1.9. => Need updates.
  • Ingress will using extensions/xxx will be rejected in kubernetes 1.20 => Need updates.
  • CRD definition just started to use v1 in K8s. To ensure they still works => Unchanged.

K8s 1.16 rejects Deployment for which the selector has not been set => Needs updates.

@jbrette jbrette changed the title Numerous kubeflow manifest can't deploy on Kubernetes 1.16 Numerous kubeflow manifests can't deploy on Kubernetes 1.16 Sep 22, 2019
@jbrette
Copy link
Contributor Author

jbrette commented Sep 22, 2019

ran

kfctl init master
kfctl generate

For each directory in the kustomize directory, run kubectl apply -k followed by kubectl delete -k

########################################################
# INIT                                                 #
########################################################
namespace/kubeflow created
customresourcedefinition.apiextensions.k8s.io/applications.app.k8s.io created
customresourcedefinition.apiextensions.k8s.io/pytorchjobs.kubeflow.org created
customresourcedefinition.apiextensions.k8s.io/adapters.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/apikeys.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/attributemanifests.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/authorizations.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/bypasses.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/certificates.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/challenges.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/checknothings.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/circonuses.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/cloudwatches.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/clusterrbacconfigs.rbac.istio.io created
customresourcedefinition.apiextensions.k8s.io/deniers.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/destinationrules.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/dogstatsds.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/edges.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/envoyfilters.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/fluentds.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/gateways.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/handlers.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/httpapispecbindings.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/httpapispecs.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/instances.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/issuers.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/kubernetesenvs.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/kuberneteses.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/listcheckers.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/listentries.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/logentries.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/memquotas.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/meshpolicies.authentication.istio.io created
customresourcedefinition.apiextensions.k8s.io/metrics.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/noops.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/opas.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/orders.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/policies.authentication.istio.io created
customresourcedefinition.apiextensions.k8s.io/prometheuses.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/quotas.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/quotaspecbindings.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/quotaspecs.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/rbacconfigs.rbac.istio.io created
customresourcedefinition.apiextensions.k8s.io/rbacs.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/redisquotas.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/reportnothings.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/rules.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/serviceentries.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/servicerolebindings.rbac.istio.io created
customresourcedefinition.apiextensions.k8s.io/serviceroles.rbac.istio.io created
customresourcedefinition.apiextensions.k8s.io/sidecars.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/signalfxs.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/solarwindses.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/stackdrivers.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/statsds.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/stdios.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/templates.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/tracespans.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/virtualservices.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/zipkins.config.istio.io created
########################################################
# ------------------
#  api-service
# ------------------
serviceaccount/ml-pipeline created
role.rbac.authorization.k8s.io/ml-pipeline created
rolebinding.rbac.authorization.k8s.io/ml-pipeline created
service/ml-pipeline created
error: unable to recognize "kustomize/api-service/": no matches for kind "Deployment" in version "apps/v1beta2"
# -------
serviceaccount "ml-pipeline" deleted
role.rbac.authorization.k8s.io "ml-pipeline" deleted
rolebinding.rbac.authorization.k8s.io "ml-pipeline" deleted
service "ml-pipeline" deleted
error: unable to recognize "kustomize/api-service/": no matches for kind "Deployment" in version "apps/v1beta2"
# ------------------
#  application
# ------------------
serviceaccount/application-controller-service-account created
clusterrole.rbac.authorization.k8s.io/application-controller-cluster-role created
clusterrolebinding.rbac.authorization.k8s.io/application-controller-cluster-role-binding created
configmap/application-controller-parameters created
service/application-controller-service created
statefulset.apps/application-controller-stateful-set created
application.app.k8s.io/kubeflow created
# -------
serviceaccount "application-controller-service-account" deleted
clusterrole.rbac.authorization.k8s.io "application-controller-cluster-role" deleted
clusterrolebinding.rbac.authorization.k8s.io "application-controller-cluster-role-binding" deleted
configmap "application-controller-parameters" deleted
service "application-controller-service" deleted
statefulset.apps "application-controller-stateful-set" deleted
application.app.k8s.io "kubeflow" deleted
# ------------------
#  argo
# ------------------
customresourcedefinition.apiextensions.k8s.io/workflows.argoproj.io created
serviceaccount/argo-ui created
serviceaccount/argo created
clusterrole.rbac.authorization.k8s.io/argo-ui created
clusterrole.rbac.authorization.k8s.io/argo created
clusterrolebinding.rbac.authorization.k8s.io/argo-ui created
clusterrolebinding.rbac.authorization.k8s.io/argo created
configmap/workflow-controller-configmap created
configmap/workflow-controller-parameters created
service/argo-ui created
virtualservice.networking.istio.io/argo-ui created
unable to recognize "kustomize/argo/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/argo/": no matches for kind "Deployment" in version "extensions/v1beta1"
# -------
customresourcedefinition.apiextensions.k8s.io "workflows.argoproj.io" deleted
serviceaccount "argo-ui" deleted
serviceaccount "argo" deleted
clusterrole.rbac.authorization.k8s.io "argo-ui" deleted
clusterrole.rbac.authorization.k8s.io "argo" deleted
clusterrolebinding.rbac.authorization.k8s.io "argo-ui" deleted
clusterrolebinding.rbac.authorization.k8s.io "argo" deleted
configmap "workflow-controller-configmap" deleted
configmap "workflow-controller-parameters" deleted
service "argo-ui" deleted
virtualservice.networking.istio.io "argo-ui" deleted
unable to recognize "kustomize/argo/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/argo/": no matches for kind "Deployment" in version "extensions/v1beta1"
# ------------------
#  bootstrap
# ------------------
serviceaccount/admission-webhook-bootstrap-service-account created
clusterrole.rbac.authorization.k8s.io/admission-webhook-bootstrap-cluster-role created
clusterrolebinding.rbac.authorization.k8s.io/admission-webhook-bootstrap-cluster-role-binding created
configmap/admission-webhook-bootstrap-config-map created
statefulset.apps/admission-webhook-bootstrap-stateful-set created
# -------
serviceaccount "admission-webhook-bootstrap-service-account" deleted
clusterrole.rbac.authorization.k8s.io "admission-webhook-bootstrap-cluster-role" deleted
clusterrolebinding.rbac.authorization.k8s.io "admission-webhook-bootstrap-cluster-role-binding" deleted
configmap "admission-webhook-bootstrap-config-map" deleted
statefulset.apps "admission-webhook-bootstrap-stateful-set" deleted
# ------------------
#  centraldashboard
# ------------------
serviceaccount/centraldashboard created
role.rbac.authorization.k8s.io/centraldashboard created
clusterrole.rbac.authorization.k8s.io/centraldashboard created
rolebinding.rbac.authorization.k8s.io/centraldashboard created
clusterrolebinding.rbac.authorization.k8s.io/centraldashboard created
configmap/parameters created
service/centraldashboard created
application.app.k8s.io/centraldashboard created
virtualservice.networking.istio.io/centraldashboard created
error: unable to recognize "kustomize/centraldashboard/": no matches for kind "Deployment" in version "extensions/v1beta1"
# -------
serviceaccount "centraldashboard" deleted
role.rbac.authorization.k8s.io "centraldashboard" deleted
clusterrole.rbac.authorization.k8s.io "centraldashboard" deleted
rolebinding.rbac.authorization.k8s.io "centraldashboard" deleted
clusterrolebinding.rbac.authorization.k8s.io "centraldashboard" deleted
configmap "parameters" deleted
service "centraldashboard" deleted
application.app.k8s.io "centraldashboard" deleted
virtualservice.networking.istio.io "centraldashboard" deleted
error: unable to recognize "kustomize/centraldashboard/": no matches for kind "Deployment" in version "extensions/v1beta1"
# ------------------
#  istio
# ------------------
configmap/istio-parameters-9m7m4fhdb8 created
gateway.networking.istio.io/kubeflow-gateway created
serviceentry.networking.istio.io/google-api-entry created
serviceentry.networking.istio.io/google-storage-api-entry created
virtualservice.networking.istio.io/google-api-vs created
virtualservice.networking.istio.io/google-storage-api-vs created
virtualservice.networking.istio.io/grafana-vs created
clusterrbacconfig.rbac.istio.io/default created
# -------
configmap "istio-parameters-9m7m4fhdb8" deleted
gateway.networking.istio.io "kubeflow-gateway" deleted
serviceentry.networking.istio.io "google-api-entry" deleted
serviceentry.networking.istio.io "google-storage-api-entry" deleted
virtualservice.networking.istio.io "google-api-vs" deleted
virtualservice.networking.istio.io "google-storage-api-vs" deleted
virtualservice.networking.istio.io "grafana-vs" deleted
clusterrbacconfig.rbac.istio.io "default" deleted
# ------------------
#  istio-install
# ------------------
namespace/istio-system created
mutatingwebhookconfiguration.admissionregistration.k8s.io/istio-sidecar-injector created
serviceaccount/istio-citadel-service-account created
serviceaccount/istio-cleanup-secrets-service-account created
serviceaccount/istio-egressgateway-service-account created
serviceaccount/istio-galley-service-account created
serviceaccount/istio-grafana-post-install-account created
serviceaccount/istio-ingressgateway-service-account created
serviceaccount/istio-mixer-service-account created
serviceaccount/istio-multi created
serviceaccount/istio-pilot-service-account created
serviceaccount/istio-security-post-install-account created
serviceaccount/istio-sidecar-injector-service-account created
serviceaccount/kiali-service-account created
serviceaccount/prometheus created
role.rbac.authorization.k8s.io/istio-ingressgateway-sds created
clusterrole.rbac.authorization.k8s.io/istio-citadel-istio-system created
clusterrole.rbac.authorization.k8s.io/istio-cleanup-secrets-istio-system created
clusterrole.rbac.authorization.k8s.io/istio-egressgateway-istio-system created
clusterrole.rbac.authorization.k8s.io/istio-galley-istio-system created
clusterrole.rbac.authorization.k8s.io/istio-grafana-post-install-istio-system created
clusterrole.rbac.authorization.k8s.io/istio-ingressgateway-istio-system created
clusterrole.rbac.authorization.k8s.io/istio-mixer-istio-system created
clusterrole.rbac.authorization.k8s.io/istio-pilot-istio-system created
clusterrole.rbac.authorization.k8s.io/istio-reader created
clusterrole.rbac.authorization.k8s.io/istio-sidecar-injector-istio-system created
clusterrole.rbac.authorization.k8s.io/kiali-viewer created
clusterrole.rbac.authorization.k8s.io/kiali created
clusterrole.rbac.authorization.k8s.io/prometheus-istio-system created
clusterrole.rbac.authorization.k8s.io/istio-security-post-install-istio-system created
rolebinding.rbac.authorization.k8s.io/istio-ingressgateway-sds created
clusterrolebinding.rbac.authorization.k8s.io/istio-citadel-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-cleanup-secrets-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-egressgateway-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-galley-admin-role-binding-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-grafana-post-install-role-binding-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-ingressgateway-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-kiali-admin-role-binding-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-mixer-admin-role-binding-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-multi created
clusterrolebinding.rbac.authorization.k8s.io/istio-pilot-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-sidecar-injector-admin-role-binding-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-security-post-install-role-binding-istio-system created
configmap/istio-galley-configuration created
configmap/istio-grafana-configuration-dashboards-galley-dashboard created
configmap/istio-grafana-configuration-dashboards-istio-mesh-dashboard created
configmap/istio-grafana-configuration-dashboards-istio-performance-dashboard created
configmap/istio-grafana-configuration-dashboards-istio-service-dashboard created
configmap/istio-grafana-configuration-dashboards-istio-workload-dashboard created
configmap/istio-grafana-configuration-dashboards-mixer-dashboard created
configmap/istio-grafana-configuration-dashboards-pilot-dashboard created
configmap/istio-grafana-custom-resources created
configmap/istio-grafana created
configmap/istio-security-custom-resources created
configmap/istio-sidecar-injector created
configmap/istio created
configmap/kiali created
configmap/prometheus created
secret/kiali created
service/grafana created
service/istio-citadel created
service/istio-egressgateway created
service/istio-galley created
service/istio-ingressgateway created
service/istio-pilot created
service/istio-policy created
service/istio-sidecar-injector created
service/istio-telemetry created
service/jaeger-agent created
service/jaeger-collector created
service/jaeger-query created
service/kiali created
service/prometheus created
service/tracing created
service/zipkin created
poddisruptionbudget.policy/istio-egressgateway created
poddisruptionbudget.policy/istio-galley created
poddisruptionbudget.policy/istio-ingressgateway created
poddisruptionbudget.policy/istio-pilot created
poddisruptionbudget.policy/istio-policy created
poddisruptionbudget.policy/istio-telemetry created
horizontalpodautoscaler.autoscaling/istio-egressgateway created
horizontalpodautoscaler.autoscaling/istio-ingressgateway created
horizontalpodautoscaler.autoscaling/istio-pilot created
horizontalpodautoscaler.autoscaling/istio-policy created
horizontalpodautoscaler.autoscaling/istio-telemetry created
job.batch/istio-cleanup-secrets-1.1.6 created
job.batch/istio-grafana-post-install-1.1.6 created
job.batch/istio-security-post-install-1.1.6 created
attributemanifest.config.istio.io/istioproxy created
attributemanifest.config.istio.io/kubernetes created
handler.config.istio.io/kubernetesenv created
handler.config.istio.io/prometheus created
handler.config.istio.io/stdio created
kubernetes.config.istio.io/attributes created
logentry.config.istio.io/accesslog created
logentry.config.istio.io/tcpaccesslog created
metric.config.istio.io/requestcount created
metric.config.istio.io/requestduration created
metric.config.istio.io/requestsize created
metric.config.istio.io/responsesize created
metric.config.istio.io/tcpbytereceived created
metric.config.istio.io/tcpbytesent created
metric.config.istio.io/tcpconnectionsclosed created
metric.config.istio.io/tcpconnectionsopened created
rule.config.istio.io/kubeattrgenrulerule created
rule.config.istio.io/promhttp created
rule.config.istio.io/promtcpconnectionclosed created
rule.config.istio.io/promtcpconnectionopen created
rule.config.istio.io/promtcp created
rule.config.istio.io/stdiotcp created
rule.config.istio.io/stdio created
rule.config.istio.io/tcpkubeattrgenrulerule created
destinationrule.networking.istio.io/istio-policy created
destinationrule.networking.istio.io/istio-telemetry created
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
# -------
namespace "istio-system" deleted
mutatingwebhookconfiguration.admissionregistration.k8s.io "istio-sidecar-injector" deleted
serviceaccount "istio-citadel-service-account" deleted
serviceaccount "istio-cleanup-secrets-service-account" deleted
serviceaccount "istio-egressgateway-service-account" deleted
serviceaccount "istio-galley-service-account" deleted
serviceaccount "istio-grafana-post-install-account" deleted
serviceaccount "istio-ingressgateway-service-account" deleted
serviceaccount "istio-mixer-service-account" deleted
serviceaccount "istio-multi" deleted
serviceaccount "istio-pilot-service-account" deleted
serviceaccount "istio-security-post-install-account" deleted
serviceaccount "istio-sidecar-injector-service-account" deleted
serviceaccount "kiali-service-account" deleted
serviceaccount "prometheus" deleted
role.rbac.authorization.k8s.io "istio-ingressgateway-sds" deleted
clusterrole.rbac.authorization.k8s.io "istio-citadel-istio-system" deleted
clusterrole.rbac.authorization.k8s.io "istio-cleanup-secrets-istio-system" deleted
clusterrole.rbac.authorization.k8s.io "istio-egressgateway-istio-system" deleted
clusterrole.rbac.authorization.k8s.io "istio-galley-istio-system" deleted
clusterrole.rbac.authorization.k8s.io "istio-grafana-post-install-istio-system" deleted
clusterrole.rbac.authorization.k8s.io "istio-ingressgateway-istio-system" deleted
clusterrole.rbac.authorization.k8s.io "istio-mixer-istio-system" deleted
clusterrole.rbac.authorization.k8s.io "istio-pilot-istio-system" deleted
clusterrole.rbac.authorization.k8s.io "istio-reader" deleted
clusterrole.rbac.authorization.k8s.io "istio-sidecar-injector-istio-system" deleted
clusterrole.rbac.authorization.k8s.io "kiali-viewer" deleted
clusterrole.rbac.authorization.k8s.io "kiali" deleted
clusterrole.rbac.authorization.k8s.io "prometheus-istio-system" deleted
clusterrole.rbac.authorization.k8s.io "istio-security-post-install-istio-system" deleted
rolebinding.rbac.authorization.k8s.io "istio-ingressgateway-sds" deleted
clusterrolebinding.rbac.authorization.k8s.io "istio-citadel-istio-system" deleted
clusterrolebinding.rbac.authorization.k8s.io "istio-cleanup-secrets-istio-system" deleted
clusterrolebinding.rbac.authorization.k8s.io "istio-egressgateway-istio-system" deleted
clusterrolebinding.rbac.authorization.k8s.io "istio-galley-admin-role-binding-istio-system" deleted
clusterrolebinding.rbac.authorization.k8s.io "istio-grafana-post-install-role-binding-istio-system" deleted
clusterrolebinding.rbac.authorization.k8s.io "istio-ingressgateway-istio-system" deleted
clusterrolebinding.rbac.authorization.k8s.io "istio-kiali-admin-role-binding-istio-system" deleted
clusterrolebinding.rbac.authorization.k8s.io "istio-mixer-admin-role-binding-istio-system" deleted
clusterrolebinding.rbac.authorization.k8s.io "istio-multi" deleted
clusterrolebinding.rbac.authorization.k8s.io "istio-pilot-istio-system" deleted
clusterrolebinding.rbac.authorization.k8s.io "istio-sidecar-injector-admin-role-binding-istio-system" deleted
clusterrolebinding.rbac.authorization.k8s.io "prometheus-istio-system" deleted
clusterrolebinding.rbac.authorization.k8s.io "istio-security-post-install-role-binding-istio-system" deleted
configmap "istio-galley-configuration" deleted
configmap "istio-grafana-configuration-dashboards-galley-dashboard" deleted
configmap "istio-grafana-configuration-dashboards-istio-mesh-dashboard" deleted
configmap "istio-grafana-configuration-dashboards-istio-performance-dashboard" deleted
configmap "istio-grafana-configuration-dashboards-istio-service-dashboard" deleted
configmap "istio-grafana-configuration-dashboards-istio-workload-dashboard" deleted
configmap "istio-grafana-configuration-dashboards-mixer-dashboard" deleted
configmap "istio-grafana-configuration-dashboards-pilot-dashboard" deleted
configmap "istio-grafana-custom-resources" deleted
configmap "istio-grafana" deleted
configmap "istio-security-custom-resources" deleted
configmap "istio-sidecar-injector" deleted
configmap "istio" deleted
configmap "kiali" deleted
configmap "prometheus" deleted
secret "kiali" deleted
service "grafana" deleted
service "istio-citadel" deleted
service "istio-egressgateway" deleted
service "istio-galley" deleted
service "istio-ingressgateway" deleted
service "istio-pilot" deleted
service "istio-policy" deleted
service "istio-sidecar-injector" deleted
service "istio-telemetry" deleted
service "jaeger-agent" deleted
service "jaeger-collector" deleted
service "jaeger-query" deleted
service "kiali" deleted
service "prometheus" deleted
service "tracing" deleted
service "zipkin" deleted
poddisruptionbudget.policy "istio-egressgateway" deleted
poddisruptionbudget.policy "istio-galley" deleted
poddisruptionbudget.policy "istio-ingressgateway" deleted
poddisruptionbudget.policy "istio-pilot" deleted
poddisruptionbudget.policy "istio-policy" deleted
poddisruptionbudget.policy "istio-telemetry" deleted
horizontalpodautoscaler.autoscaling "istio-egressgateway" deleted
horizontalpodautoscaler.autoscaling "istio-ingressgateway" deleted
horizontalpodautoscaler.autoscaling "istio-pilot" deleted
horizontalpodautoscaler.autoscaling "istio-policy" deleted
horizontalpodautoscaler.autoscaling "istio-telemetry" deleted
job.batch "istio-cleanup-secrets-1.1.6" deleted
job.batch "istio-grafana-post-install-1.1.6" deleted
job.batch "istio-security-post-install-1.1.6" deleted
attributemanifest.config.istio.io "istioproxy" deleted
attributemanifest.config.istio.io "kubernetes" deleted
handler.config.istio.io "kubernetesenv" deleted
handler.config.istio.io "prometheus" deleted
handler.config.istio.io "stdio" deleted
kubernetes.config.istio.io "attributes" deleted
logentry.config.istio.io "accesslog" deleted
logentry.config.istio.io "tcpaccesslog" deleted
metric.config.istio.io "requestcount" deleted
metric.config.istio.io "requestduration" deleted
metric.config.istio.io "requestsize" deleted
metric.config.istio.io "responsesize" deleted
metric.config.istio.io "tcpbytereceived" deleted
metric.config.istio.io "tcpbytesent" deleted
metric.config.istio.io "tcpconnectionsclosed" deleted
metric.config.istio.io "tcpconnectionsopened" deleted
rule.config.istio.io "kubeattrgenrulerule" deleted
rule.config.istio.io "promhttp" deleted
rule.config.istio.io "promtcpconnectionclosed" deleted
rule.config.istio.io "promtcpconnectionopen" deleted
rule.config.istio.io "promtcp" deleted
rule.config.istio.io "stdiotcp" deleted
rule.config.istio.io "stdio" deleted
rule.config.istio.io "tcpkubeattrgenrulerule" deleted
destinationrule.networking.istio.io "istio-policy" deleted
destinationrule.networking.istio.io "istio-telemetry" deleted
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/istio-install/": no matches for kind "Deployment" in version "extensions/v1beta1"
# ------------------
#  jupyter-web-app
# ------------------
serviceaccount/jupyter-web-app-service-account created
role.rbac.authorization.k8s.io/jupyter-web-app-jupyter-notebook-role created
clusterrole.rbac.authorization.k8s.io/jupyter-web-app-cluster-role created
rolebinding.rbac.authorization.k8s.io/jupyter-web-app-jupyter-notebook-role-binding created
clusterrolebinding.rbac.authorization.k8s.io/jupyter-web-app-cluster-role-binding created
configmap/jupyter-web-app-config created
configmap/jupyter-web-app-parameters created
service/jupyter-web-app-service created
deployment.apps/jupyter-web-app-deployment created
application.app.k8s.io/jupyter-web-app created
virtualservice.networking.istio.io/jupyter-web-app created
# -------
serviceaccount "jupyter-web-app-service-account" deleted
role.rbac.authorization.k8s.io "jupyter-web-app-jupyter-notebook-role" deleted
clusterrole.rbac.authorization.k8s.io "jupyter-web-app-cluster-role" deleted
rolebinding.rbac.authorization.k8s.io "jupyter-web-app-jupyter-notebook-role-binding" deleted
clusterrolebinding.rbac.authorization.k8s.io "jupyter-web-app-cluster-role-binding" deleted
configmap "jupyter-web-app-config" deleted
configmap "jupyter-web-app-parameters" deleted
service "jupyter-web-app-service" deleted
deployment.apps "jupyter-web-app-deployment" deleted
application.app.k8s.io "jupyter-web-app" deleted
virtualservice.networking.istio.io "jupyter-web-app" deleted
# ------------------
#  katib-controller
# ------------------
customresourcedefinition.apiextensions.k8s.io/experiments.kubeflow.org created
customresourcedefinition.apiextensions.k8s.io/trials.kubeflow.org created
serviceaccount/katib-controller created
clusterrole.rbac.authorization.k8s.io/katib-controller created
clusterrolebinding.rbac.authorization.k8s.io/katib-controller created
configmap/trial-template created
secret/katib-controller created
service/katib-controller created
deployment.apps/katib-controller created
# -------
customresourcedefinition.apiextensions.k8s.io "experiments.kubeflow.org" deleted
customresourcedefinition.apiextensions.k8s.io "trials.kubeflow.org" deleted
serviceaccount "katib-controller" deleted
clusterrole.rbac.authorization.k8s.io "katib-controller" deleted
clusterrolebinding.rbac.authorization.k8s.io "katib-controller" deleted
configmap "trial-template" deleted
secret "katib-controller" deleted
service "katib-controller" deleted
deployment.apps "katib-controller" deleted
# ------------------
#  katib-db
# ------------------
secret/katib-db-secrets created
service/katib-db created
persistentvolumeclaim/katib-mysql created
error: unable to recognize "kustomize/katib-db/": no matches for kind "Deployment" in version "extensions/v1beta1"
# -------
secret "katib-db-secrets" deleted
service "katib-db" deleted
persistentvolumeclaim "katib-mysql" deleted
error: unable to recognize "kustomize/katib-db/": no matches for kind "Deployment" in version "extensions/v1beta1"
# ------------------
#  katib-manager
# ------------------
service/katib-manager-rest created
service/katib-manager created
unable to recognize "kustomize/katib-manager/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/katib-manager/": no matches for kind "Deployment" in version "extensions/v1beta1"
# -------
service "katib-manager-rest" deleted
service "katib-manager" deleted
unable to recognize "kustomize/katib-manager/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/katib-manager/": no matches for kind "Deployment" in version "extensions/v1beta1"
# ------------------
#  katib-ui
# ------------------
serviceaccount/katib-ui created
clusterrole.rbac.authorization.k8s.io/katib-ui created
clusterrolebinding.rbac.authorization.k8s.io/katib-ui created
configmap/katib-parameters created
service/katib-ui created
virtualservice.networking.istio.io/katib-ui created
error: unable to recognize "kustomize/katib-ui/": no matches for kind "Deployment" in version "extensions/v1beta1"
# -------
serviceaccount "katib-ui" deleted
clusterrole.rbac.authorization.k8s.io "katib-ui" deleted
clusterrolebinding.rbac.authorization.k8s.io "katib-ui" deleted
configmap "katib-parameters" deleted
service "katib-ui" deleted
virtualservice.networking.istio.io "katib-ui" deleted
error: unable to recognize "kustomize/katib-ui/": no matches for kind "Deployment" in version "extensions/v1beta1"
# ------------------
#  metacontroller
# ------------------
customresourcedefinition.apiextensions.k8s.io/compositecontrollers.metacontroller.k8s.io created
customresourcedefinition.apiextensions.k8s.io/controllerrevisions.metacontroller.k8s.io created
customresourcedefinition.apiextensions.k8s.io/decoratorcontrollers.metacontroller.k8s.io created
serviceaccount/meta-controller-service created
clusterrolebinding.rbac.authorization.k8s.io/meta-controller-cluster-role-binding created
error: unable to recognize "kustomize/metacontroller/": no matches for kind "StatefulSet" in version "apps/v1beta2"
# -------
customresourcedefinition.apiextensions.k8s.io "compositecontrollers.metacontroller.k8s.io" deleted
customresourcedefinition.apiextensions.k8s.io "controllerrevisions.metacontroller.k8s.io" deleted
customresourcedefinition.apiextensions.k8s.io "decoratorcontrollers.metacontroller.k8s.io" deleted
serviceaccount "meta-controller-service" deleted
clusterrolebinding.rbac.authorization.k8s.io "meta-controller-cluster-role-binding" deleted
error: unable to recognize "kustomize/metacontroller/": no matches for kind "StatefulSet" in version "apps/v1beta2"
# ------------------
#  metadata
# ------------------
serviceaccount/metadata-ui created
role.rbac.authorization.k8s.io/metadata-ui created
rolebinding.rbac.authorization.k8s.io/metadata-ui created
configmap/metadata-ui-parameters-b6c8ghff7c created
secret/metadata-db-secrets created
service/metadata-db created
service/metadata-service created
service/metadata-ui created
virtualservice.networking.istio.io/metadata-ui created
persistentvolumeclaim/metadata-mysql created
unable to recognize "kustomize/metadata/": no matches for kind "Deployment" in version "apps/v1beta2"
unable to recognize "kustomize/metadata/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/metadata/": no matches for kind "Deployment" in version "extensions/v1beta1"
# -------
serviceaccount "metadata-ui" deleted
role.rbac.authorization.k8s.io "metadata-ui" deleted
rolebinding.rbac.authorization.k8s.io "metadata-ui" deleted
configmap "metadata-ui-parameters-b6c8ghff7c" deleted
secret "metadata-db-secrets" deleted
service "metadata-db" deleted
service "metadata-service" deleted
service "metadata-ui" deleted
virtualservice.networking.istio.io "metadata-ui" deleted
persistentvolumeclaim "metadata-mysql" deleted
unable to recognize "kustomize/metadata/": no matches for kind "Deployment" in version "apps/v1beta2"
unable to recognize "kustomize/metadata/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/metadata/": no matches for kind "Deployment" in version "extensions/v1beta1"
# ------------------
#  metrics-collector
# ------------------
serviceaccount/metrics-collector created
clusterrole.rbac.authorization.k8s.io/metrics-collector created
clusterrolebinding.rbac.authorization.k8s.io/metrics-collector created
configmap/metrics-collector-template created
# -------
serviceaccount "metrics-collector" deleted
clusterrole.rbac.authorization.k8s.io "metrics-collector" deleted
clusterrolebinding.rbac.authorization.k8s.io "metrics-collector" deleted
configmap "metrics-collector-template" deleted
# ------------------
#  minio
# ------------------
configmap/pipeline-minio-parameters created
secret/mlpipeline-minio-artifact created
service/minio-service created
persistentvolume/minio-pv created
persistentvolumeclaim/minio-pv-claim created
error: unable to recognize "kustomize/minio/": no matches for kind "Deployment" in version "apps/v1beta1"
# -------
configmap "pipeline-minio-parameters" deleted
secret "mlpipeline-minio-artifact" deleted
service "minio-service" deleted
persistentvolume "minio-pv" deleted
persistentvolumeclaim "minio-pv-claim" deleted
error: unable to recognize "kustomize/minio/": no matches for kind "Deployment" in version "apps/v1beta1"
# ------------------
#  mysql
# ------------------
configmap/pipeline-mysql-parameters created
service/mysql created
persistentvolume/mysql-pv created
persistentvolumeclaim/mysql-pv-claim created
error: unable to recognize "kustomize/mysql/": no matches for kind "Deployment" in version "apps/v1beta2"
# -------
configmap "pipeline-mysql-parameters" deleted
service "mysql" deleted
persistentvolume "mysql-pv" deleted
persistentvolumeclaim "mysql-pv-claim" deleted
error: unable to recognize "kustomize/mysql/": no matches for kind "Deployment" in version "apps/v1beta2"
# ------------------
#  notebook-controller
# ------------------
customresourcedefinition.apiextensions.k8s.io/notebooks.kubeflow.org created
serviceaccount/notebook-controller-service-account created
clusterrole.rbac.authorization.k8s.io/notebook-controller-role created
clusterrolebinding.rbac.authorization.k8s.io/notebook-controller-role-binding created
configmap/notebook-controller-parameters created
service/notebook-controller-service created
application.app.k8s.io/notebook-controller created
error: unable to recognize "kustomize/notebook-controller/": no matches for kind "Deployment" in version "apps/v1beta1"
# -------
customresourcedefinition.apiextensions.k8s.io "notebooks.kubeflow.org" deleted
serviceaccount "notebook-controller-service-account" deleted
clusterrole.rbac.authorization.k8s.io "notebook-controller-role" deleted
clusterrolebinding.rbac.authorization.k8s.io "notebook-controller-role-binding" deleted
configmap "notebook-controller-parameters" deleted
service "notebook-controller-service" deleted
application.app.k8s.io "notebook-controller" deleted
error: unable to recognize "kustomize/notebook-controller/": no matches for kind "Deployment" in version "apps/v1beta1"
# ------------------
#  persistent-agent
# ------------------
serviceaccount/ml-pipeline-persistenceagent created
clusterrole.rbac.authorization.k8s.io/ml-pipeline-persistenceagent created
clusterrolebinding.rbac.authorization.k8s.io/ml-pipeline-persistenceagent created
error: unable to recognize "kustomize/persistent-agent/": no matches for kind "Deployment" in version "apps/v1beta2"
# -------
serviceaccount "ml-pipeline-persistenceagent" deleted
clusterrole.rbac.authorization.k8s.io "ml-pipeline-persistenceagent" deleted
clusterrolebinding.rbac.authorization.k8s.io "ml-pipeline-persistenceagent" deleted
error: unable to recognize "kustomize/persistent-agent/": no matches for kind "Deployment" in version "apps/v1beta2"
# ------------------
#  pipelines-runner
# ------------------
serviceaccount/pipeline-runner created
clusterrole.rbac.authorization.k8s.io/pipeline-runner created
clusterrolebinding.rbac.authorization.k8s.io/pipeline-runner created
# -------
serviceaccount "pipeline-runner" deleted
clusterrole.rbac.authorization.k8s.io "pipeline-runner" deleted
clusterrolebinding.rbac.authorization.k8s.io "pipeline-runner" deleted
# ------------------
#  pipelines-ui
# ------------------
serviceaccount/ml-pipeline-ui created
role.rbac.authorization.k8s.io/ml-pipeline-ui created
rolebinding.rbac.authorization.k8s.io/ml-pipeline-ui created
configmap/ui-parameters-hb792fcf5d created
service/ml-pipeline-tensorboard-ui created
service/ml-pipeline-ui created
virtualservice.networking.istio.io/ml-pipeline-tensorboard-ui created
virtualservice.networking.istio.io/ml-pipeline-ui created
error: unable to recognize "kustomize/pipelines-ui/": no matches for kind "Deployment" in version "apps/v1beta2"
# -------
serviceaccount "ml-pipeline-ui" deleted
role.rbac.authorization.k8s.io "ml-pipeline-ui" deleted
rolebinding.rbac.authorization.k8s.io "ml-pipeline-ui" deleted
configmap "ui-parameters-hb792fcf5d" deleted
service "ml-pipeline-tensorboard-ui" deleted
service "ml-pipeline-ui" deleted
virtualservice.networking.istio.io "ml-pipeline-tensorboard-ui" deleted
virtualservice.networking.istio.io "ml-pipeline-ui" deleted
error: unable to recognize "kustomize/pipelines-ui/": no matches for kind "Deployment" in version "apps/v1beta2"
# ------------------
#  pipelines-viewer
# ------------------
customresourcedefinition.apiextensions.k8s.io/viewers.kubeflow.org created
serviceaccount/ml-pipeline-viewer-crd-service-account created
clusterrole.rbac.authorization.k8s.io/ml-pipeline-viewer-controller-role created
clusterrolebinding.rbac.authorization.k8s.io/ml-pipeline-viewer-crd-role-binding created
error: unable to recognize "kustomize/pipelines-viewer/": no matches for kind "Deployment" in version "apps/v1beta2"
# -------
customresourcedefinition.apiextensions.k8s.io "viewers.kubeflow.org" deleted
serviceaccount "ml-pipeline-viewer-crd-service-account" deleted
clusterrole.rbac.authorization.k8s.io "ml-pipeline-viewer-controller-role" deleted
clusterrolebinding.rbac.authorization.k8s.io "ml-pipeline-viewer-crd-role-binding" deleted
error: unable to recognize "kustomize/pipelines-viewer/": no matches for kind "Deployment" in version "apps/v1beta2"
# ------------------
#  pytorch-operator
# ------------------
serviceaccount/pytorch-operator created
clusterrole.rbac.authorization.k8s.io/pytorch-operator created
clusterrolebinding.rbac.authorization.k8s.io/pytorch-operator created
configmap/pytorch-operator-config created
service/pytorch-operator created
application.app.k8s.io/pytorch-operator created
error: unable to recognize "kustomize/pytorch-operator/": no matches for kind "Deployment" in version "extensions/v1beta1"
# -------
serviceaccount "pytorch-operator" deleted
clusterrole.rbac.authorization.k8s.io "pytorch-operator" deleted
clusterrolebinding.rbac.authorization.k8s.io "pytorch-operator" deleted
configmap "pytorch-operator-config" deleted
service "pytorch-operator" deleted
application.app.k8s.io "pytorch-operator" deleted
error: unable to recognize "kustomize/pytorch-operator/": no matches for kind "Deployment" in version "extensions/v1beta1"
# ------------------
#  scheduledworkflow
# ------------------
customresourcedefinition.apiextensions.k8s.io/scheduledworkflows.kubeflow.org created
serviceaccount/ml-pipeline-scheduledworkflow created
role.rbac.authorization.k8s.io/ml-pipeline-scheduledworkflow created
clusterrolebinding.rbac.authorization.k8s.io/ml-pipeline-scheduledworkflow created
error: unable to recognize "kustomize/scheduledworkflow/": no matches for kind "Deployment" in version "apps/v1beta2"
# -------
customresourcedefinition.apiextensions.k8s.io "scheduledworkflows.kubeflow.org" deleted
serviceaccount "ml-pipeline-scheduledworkflow" deleted
role.rbac.authorization.k8s.io "ml-pipeline-scheduledworkflow" deleted
clusterrolebinding.rbac.authorization.k8s.io "ml-pipeline-scheduledworkflow" deleted
error: unable to recognize "kustomize/scheduledworkflow/": no matches for kind "Deployment" in version "apps/v1beta2"
# ------------------
#  spartakus
# ------------------
serviceaccount/spartakus created
clusterrole.rbac.authorization.k8s.io/spartakus created
clusterrolebinding.rbac.authorization.k8s.io/spartakus created
configmap/spartakus-parameters created
error: unable to recognize "kustomize/spartakus/": no matches for kind "Deployment" in version "extensions/v1beta1"
# -------
serviceaccount "spartakus" deleted
clusterrole.rbac.authorization.k8s.io "spartakus" deleted
clusterrolebinding.rbac.authorization.k8s.io "spartakus" deleted
configmap "spartakus-parameters" deleted
error: unable to recognize "kustomize/spartakus/": no matches for kind "Deployment" in version "extensions/v1beta1"
# ------------------
#  suggestion
# ------------------
service/katib-suggestion-bayesianoptimization created
service/katib-suggestion-grid created
service/katib-suggestion-hyperband created
service/katib-suggestion-nasrl created
service/katib-suggestion-random created
unable to recognize "kustomize/suggestion/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/suggestion/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/suggestion/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/suggestion/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/suggestion/": no matches for kind "Deployment" in version "extensions/v1beta1"
# -------
service "katib-suggestion-bayesianoptimization" deleted
service "katib-suggestion-grid" deleted
service "katib-suggestion-hyperband" deleted
service "katib-suggestion-nasrl" deleted
service "katib-suggestion-random" deleted
unable to recognize "kustomize/suggestion/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/suggestion/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/suggestion/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/suggestion/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/suggestion/": no matches for kind "Deployment" in version "extensions/v1beta1"
# ------------------
#  tensorboard
# ------------------
configmap/parameters-dgd4h256h5 created
service/tensorboard created
virtualservice.networking.istio.io/tensorboard created
error: unable to recognize "kustomize/tensorboard/": no matches for kind "Deployment" in version "apps/v1beta1"
# -------
configmap "parameters-dgd4h256h5" deleted
service "tensorboard" deleted
virtualservice.networking.istio.io "tensorboard" deleted
error: unable to recognize "kustomize/tensorboard/": no matches for kind "Deployment" in version "apps/v1beta1"
# ------------------
#  tf-job-operator
# ------------------
customresourcedefinition.apiextensions.k8s.io/tfjobs.kubeflow.org created
serviceaccount/tf-job-dashboard created
serviceaccount/tf-job-operator created
clusterrole.rbac.authorization.k8s.io/tf-job-dashboard created
clusterrole.rbac.authorization.k8s.io/tf-job-operator created
clusterrolebinding.rbac.authorization.k8s.io/tf-job-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/tf-job-operator created
configmap/parameters-dgd4h256h5 created
configmap/tf-job-operator-config created
service/tf-job-dashboard created
service/tf-job-operator created
virtualservice.networking.istio.io/tf-job-dashboard created
unable to recognize "kustomize/tf-job-operator/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/tf-job-operator/": no matches for kind "Deployment" in version "extensions/v1beta1"
# -------
customresourcedefinition.apiextensions.k8s.io "tfjobs.kubeflow.org" deleted
serviceaccount "tf-job-dashboard" deleted
serviceaccount "tf-job-operator" deleted
clusterrole.rbac.authorization.k8s.io "tf-job-dashboard" deleted
clusterrole.rbac.authorization.k8s.io "tf-job-operator" deleted
clusterrolebinding.rbac.authorization.k8s.io "tf-job-dashboard" deleted
clusterrolebinding.rbac.authorization.k8s.io "tf-job-operator" deleted
configmap "parameters-dgd4h256h5" deleted
configmap "tf-job-operator-config" deleted
service "tf-job-dashboard" deleted
service "tf-job-operator" deleted
virtualservice.networking.istio.io "tf-job-dashboard" deleted
unable to recognize "kustomize/tf-job-operator/": no matches for kind "Deployment" in version "extensions/v1beta1"
unable to recognize "kustomize/tf-job-operator/": no matches for kind "Deployment" in version "extensions/v1beta1"
# ------------------
#  webhook
# ------------------
customresourcedefinition.apiextensions.k8s.io/poddefaults.kubeflow.org created
mutatingwebhookconfiguration.admissionregistration.k8s.io/admission-webhook-mutating-webhook-configuration created
serviceaccount/admission-webhook-service-account created
clusterrole.rbac.authorization.k8s.io/admission-webhook-cluster-role created
clusterrolebinding.rbac.authorization.k8s.io/admission-webhook-cluster-role-binding created
configmap/admission-webhook-admission-webhook-parameters created
service/admission-webhook-service created
error: unable to recognize "kustomize/webhook/": no matches for kind "Deployment" in version "extensions/v1beta1"
# -------
customresourcedefinition.apiextensions.k8s.io "poddefaults.kubeflow.org" deleted
mutatingwebhookconfiguration.admissionregistration.k8s.io "admission-webhook-mutating-webhook-configuration" deleted
serviceaccount "admission-webhook-service-account" deleted
clusterrole.rbac.authorization.k8s.io "admission-webhook-cluster-role" deleted
clusterrolebinding.rbac.authorization.k8s.io "admission-webhook-cluster-role-binding" deleted
configmap "admission-webhook-admission-webhook-parameters" deleted
service "admission-webhook-service" deleted
error: unable to recognize "kustomize/webhook/": no matches for kind "Deployment" in version "extensions/v1beta1"
########################################################
# DELETE                                               #
########################################################
customresourcedefinition.apiextensions.k8s.io "adapters.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "apikeys.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "attributemanifests.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "authorizations.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "bypasses.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "certificates.certmanager.k8s.io" deleted
customresourcedefinition.apiextensions.k8s.io "challenges.certmanager.k8s.io" deleted
customresourcedefinition.apiextensions.k8s.io "checknothings.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "circonuses.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "cloudwatches.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "clusterissuers.certmanager.k8s.io" deleted
customresourcedefinition.apiextensions.k8s.io "clusterrbacconfigs.rbac.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "deniers.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "destinationrules.networking.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "dogstatsds.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "edges.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "envoyfilters.networking.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "fluentds.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "gateways.networking.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "handlers.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "httpapispecbindings.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "httpapispecs.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "instances.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "issuers.certmanager.k8s.io" deleted
customresourcedefinition.apiextensions.k8s.io "kubernetesenvs.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "kuberneteses.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "listcheckers.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "listentries.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "logentries.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "memquotas.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "meshpolicies.authentication.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "metrics.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "noops.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "opas.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "orders.certmanager.k8s.io" deleted
customresourcedefinition.apiextensions.k8s.io "policies.authentication.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "prometheuses.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "quotas.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "quotaspecbindings.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "quotaspecs.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "rbacconfigs.rbac.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "rbacs.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "redisquotas.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "reportnothings.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "rules.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "serviceentries.networking.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "servicerolebindings.rbac.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "serviceroles.rbac.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "sidecars.networking.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "signalfxs.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "solarwindses.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "stackdrivers.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "statsds.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "stdios.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "templates.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "tracespans.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "virtualservices.networking.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "zipkins.config.istio.io" deleted
customresourcedefinition.apiextensions.k8s.io "applications.app.k8s.io" deleted
customresourcedefinition.apiextensions.k8s.io "pytorchjobs.kubeflow.org" deleted
namespace "kubeflow" deleted
########################################################

@jbrette
Copy link
Contributor Author

jbrette commented Sep 24, 2019

Changes to apiVersion statefulset, daemonset and deployment have been made.
Ingress have been left to extensions but roles have been adjusted.

@jlewi
Copy link
Contributor

jlewi commented Oct 24, 2019

@jbrette it looks like we are still getting reports that Kubeflow doesn't work on 1.16
Looks like the deployments might be an issue

See: kubeflow/kubeflow#4386 for example.

Can we create an E2E test to verify that Kubeflow is working on 1.16.

It doesn't look like GKE supports 1.16 yet so we either need to

  1. Wait for GKE to support 1.16
  2. Start to build an E2E test that uses something like kops to deploy Kubernetes

@jlewi
Copy link
Contributor

jlewi commented Nov 4, 2019

I think we should treat 1.16 support as P0 for Kubeflow 1.0.
1.16 was released in September (https://github.com/kubernetes/sig-release/tree/master/releases/release-1.16).

And Kubernetes 1.17 is slated for December
https://github.com/kubernetes/sig-release/tree/master/releases/release-1.17

So by January 2020 (ETA for Kubeflow 1.0) it will be approximately 4 months old and 1.17 will likely be here.

kubeflow/kubeflow#3496 is tracking setting up an E2E test that would hopefully allow us to test on Kubernetes 1.16.

@AndriiNeverov
Copy link

Getting a number of errors with Seldon CRD on Kubernetes 1.16.3. Noticed some differences to https://github.com/SeldonIO/seldon-core/blob/master/helm-charts/seldon-core-operator/templates/customresourcedefinition_seldondeployments.machinelearning.seldon.io.yaml:

                      endpoint:
                        properties:
                          service_host:
                            type: string
                          service_port:
                            type: integer
                          type:
                            enum:
                            - REST
                            - GRPC
                            type: string

vs

endpoint:
                        properties:
                          service_host:
                            type: string
                          service_port:
                            format: int32
                            type: integer
                          type:
                            type: string
                        type: object

WARN[0010] Encountered error during apply: (kubeflow.error): Code 500 with message: Apply.Run Error error when creating "/tmp/kout829193174": CustomResourceDefinition.apiextensions.k8s.io "seldondeployments.machinelearning.seldon.io" is invalid:

[spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[explainer].properties[endpoint].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[graph].properties[children].items.properties[children].items.properties[children].items.type: Required value: must not be empty for specified array items,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[graph].properties[children].items.properties[children].items.properties[endpoint].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[graph].properties[children].items.properties[children].items.type: Required value: must not be empty for specified array items,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[graph].properties[children].items.properties[endpoint].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[graph].properties[children].items.type: Required value: must not be empty for specified array items,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[graph].properties[endpoint].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[graph].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[svcOrchSpec].properties[env].items.properties[valueFrom].properties[configMapKeyRef].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[svcOrchSpec].properties[env].items.properties[valueFrom].properties[fieldRef].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[svcOrchSpec].properties[env].items.properties[valueFrom].properties[resourceFieldRef].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[svcOrchSpec].properties[env].items.properties[valueFrom].properties[secretKeyRef].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[svcOrchSpec].properties[env].items.properties[valueFrom].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[svcOrchSpec].properties[env].items.type: Required value: must not be empty for specified array items,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[svcOrchSpec].properties[resources].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.type: Required value: must not be empty for specified array items,

spec.validation.openAPIV3Schema.properties[spec].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.type: Required value: must not be empty at the root] filename="kustomize/kustomize.go:183"

@ukclivecox
Copy link
Contributor

Getting a number of errors with Seldon CRD on Kubernetes 1.16.3. Noticed some differences to https://github.com/SeldonIO/seldon-core/blob/master/helm-charts/seldon-core-operator/templates/customresourcedefinition_seldondeployments.machinelearning.seldon.io.yaml:

                      endpoint:
                        properties:
                          service_host:
                            type: string
                          service_port:
                            type: integer
                          type:
                            enum:
                            - REST
                            - GRPC
                            type: string

vs

endpoint:
                        properties:
                          service_host:
                            type: string
                          service_port:
                            format: int32
                            type: integer
                          type:
                            type: string
                        type: object

WARN[0010] Encountered error during apply: (kubeflow.error): Code 500 with message: Apply.Run Error error when creating "/tmp/kout829193174": CustomResourceDefinition.apiextensions.k8s.io "seldondeployments.machinelearning.seldon.io" is invalid:

[spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[explainer].properties[endpoint].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[graph].properties[children].items.properties[children].items.properties[children].items.type: Required value: must not be empty for specified array items,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[graph].properties[children].items.properties[children].items.properties[endpoint].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[graph].properties[children].items.properties[children].items.type: Required value: must not be empty for specified array items,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[graph].properties[children].items.properties[endpoint].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[graph].properties[children].items.type: Required value: must not be empty for specified array items,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[graph].properties[endpoint].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[graph].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[svcOrchSpec].properties[env].items.properties[valueFrom].properties[configMapKeyRef].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[svcOrchSpec].properties[env].items.properties[valueFrom].properties[fieldRef].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[svcOrchSpec].properties[env].items.properties[valueFrom].properties[resourceFieldRef].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[svcOrchSpec].properties[env].items.properties[valueFrom].properties[secretKeyRef].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[svcOrchSpec].properties[env].items.properties[valueFrom].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[svcOrchSpec].properties[env].items.type: Required value: must not be empty for specified array items,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.properties[svcOrchSpec].properties[resources].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.properties[spec].properties[predictors].items.type: Required value: must not be empty for specified array items,

spec.validation.openAPIV3Schema.properties[spec].type: Required value: must not be empty for specified object fields,

spec.validation.openAPIV3Schema.type: Required value: must not be empty at the root] filename="kustomize/kustomize.go:183"

@AndriiNeverov Can you try installing the latest Seldon as discussed at https://docs.seldon.io/projects/seldon-core/en/latest/workflow/install.html

This should work on 1.16.2.

We will update the kubeflow Manifests for Seldon to match the latest.

@AndriiNeverov
Copy link

@cliveseldon Followed the Kustomize section against v0.5.1, installed, adjusted ISTIO_ vars to match Kubeflow, created a deployment and posted a request with no issues. Please update the Manifests. Thanks!

@jlewi
Copy link
Contributor

jlewi commented Dec 16, 2019

GKE 1.16.0 is now available via the rapid channel.
https://cloud.google.com/kubernetes-engine/docs/release-notes-rapid

So it should now be possible to begin testing against 1.16 using GKE.

Anyone interested in picking this up?

#676 should unblock being able to test against 1.16 with GKE.

@jlewi
Copy link
Contributor

jlewi commented Jan 6, 2020

@jbrette or anyone else interested in trying to test this out with 1.16?

@MnrGreg
Copy link

MnrGreg commented Feb 14, 2020

I was using https://github.com/kubeflow/manifests/raw/master/kfdef/kfctl_k8s_istio.yaml

and pointing repos to:

  repos:
  - name: manifests
    uri: https://github.com/kubeflow/manifests/archive/v1.0-rc.0.tar.gz

since using https://github.com/kubeflow/manifests/raw/master/kfdef/kfctl_k8s_istio.v1.0.0.yaml
with

  repos:
  - name: manifests
    uri: https://github.com/kubeflow/manifests/archive/v1.0-branch.tar.gz

I haven't seen the Seldon CRD issue. kfctl apply goes though fine.

@ukclivecox
Copy link
Contributor

@MnrGreg
The rc.0.tar had the old version of Seldon from 0.7 release as the updates were not yet cherrypicked to the release branch

@yanniszark
Copy link
Contributor

@MnrGreg did you manage to deploy any of the Kubeflow configurations successfully on K8s 1.16?

@MnrGreg
Copy link

MnrGreg commented Mar 5, 2020

@yanniszark yes, with kfctl_k8s_istio.v1.0.0.yaml we were able to create all CRDs and deployments on k8s 1.16.2


Kubernetes 1.16.2
containerd/cri
PhotonOS 3R2
nfs-client-provisioner v3.1.0-k8s1.11

@saschagrunert
Copy link
Contributor

saschagrunert commented Mar 6, 2020

@yanniszark yes, with kfctl_k8s_istio.v1.0.0.yaml we were able to create all CRDs and deployments on k8s 1.16.2

Kubernetes 1.16.2
containerd/cri
PhotonOS 3R2
nfs-client-provisioner v3.1.0-k8s1.11

Can confirm that, works on the latest k8s beta as well (v1.18.0-beta.1)

@Kyrremann
Copy link

This may be the wrong thread, but I'm trying to run Kubeflow on on-prem v1.16, and we're having trouble with the cert-manager. The webhook returns 403 when the api-server tries to communicate with it.

failing or missing response from https://10.254.80.38:443/apis/webhook.cert-manager.io/v1beta1: bad status from https://10.254.80.38:443/apis/webhook.cert-manager.io/v1beta1: 403

I've made an issue over at cert-manager/cert-manager#2678.

@jlewi
Copy link
Contributor

jlewi commented Mar 15, 2020

@Kyrremann yeah that looks like a different issue then this issue. If there is something specific to kubeflow regarding cert-manager/cert-manager#2678 then I would suggest opening a new issue.

@animeshsingh
Copy link
Contributor

@jlewi looks like based on thread, except for Seldon most of the components are running on 1.16 - is there some formalization for testing et all you are looking at?

@ukclivecox
Copy link
Contributor

@animeshsingh Seldon should run on 1.16

@vpavlin
Copy link
Member

vpavlin commented Mar 23, 2020

Yeah, Seldon in master works fine on OpenShift 4.3 (K8S 1.16), it just takes longer to successfully sttart due to slow ceert-manager.

@jlewi
Copy link
Contributor

jlewi commented Mar 31, 2020

@animeshsingh yes. We need CI. We should be able to look at one or more of our test grids and have confidence that everything is working.
https://k8s-testgrid.appspot.com/sig-big-data

This probably means we need to create a GKE deployment running on 1.16 and run our tests on that.

Alternatively, if we didn't want to use GKE someone would have to invest in CI for deploying some otherway; e.g. using kops.

@yanniszark
Copy link
Contributor

Can we close this issue now that users have confirmed Kubeflow to be working on 1.16?

@jlewi
Copy link
Contributor

jlewi commented May 4, 2020

Ideally we'd have a test to verify this. Is anyone willing to work on such a test?

@jlewi
Copy link
Contributor

jlewi commented May 18, 2020

Can we add unittests that verify that we don't have any manifests that violate the restrictions in this issue? So all the issues @jbrette mentioned in his original comment?

I added some validation tests here:
https://github.com/kubeflow/manifests/blob/master/tests/validate_resources_test.go

It should be easy to copy those tests to perform additional validations; e.g. for version.

@jlewi
Copy link
Contributor

jlewi commented Jun 15, 2020

Downgrading to P2 and removing from 1.1 since no one is working on the unittests to verify we don't have any incompatible resources.

jlewi pushed a commit to jlewi/manifests that referenced this issue Jun 19, 2020
* ingress is now networking.k8s.io/v1beta1 this was introduced in
  in 1.8 so there shouldn't be any blocker to upgrading our ingresses now
  https://kubernetes.io/blog/2019/07/18/api-deprecations-in-1-16/

  * Most of the ingresses had already been updated

* Deployments are now apps/v1 not extensions/v1beta1

* Delete gcp/basic-auth-ingress rather than updating it.

* Delete kfdef/generic/istio rather than update it

  * It doesn't look like this has been modified since it was copied over
    from the kubeflow/kubeflow repo in 2019. ISTIO configs should be in /istio

* Add a test for deprecated Kubernetes resources fixes kubeflow#375
jlewi pushed a commit to jlewi/manifests that referenced this issue Jun 19, 2020
* ingress is now networking.k8s.io/v1beta1 this was introduced in
  in 1.8 so there shouldn't be any blocker to upgrading our ingresses now
  https://kubernetes.io/blog/2019/07/18/api-deprecations-in-1-16/

  * Most of the ingresses had already been updated

* Deployments are now apps/v1 not extensions/v1beta1

* Delete gcp/basic-auth-ingress rather than updating it.

* Delete kfdef/generic/istio rather than update it

  * It doesn't look like this has been modified since it was copied over
    from the kubeflow/kubeflow repo in 2019. ISTIO configs should be in /istio

* Add a test for deprecated Kubernetes resources fixes kubeflow#375

Regenerate tests.
k8s-ci-robot pushed a commit that referenced this issue Jun 20, 2020
* ingress is now networking.k8s.io/v1beta1 this was introduced in
  in 1.8 so there shouldn't be any blocker to upgrading our ingresses now
  https://kubernetes.io/blog/2019/07/18/api-deprecations-in-1-16/

  * Most of the ingresses had already been updated

* Deployments are now apps/v1 not extensions/v1beta1

* Delete gcp/basic-auth-ingress rather than updating it.

* Delete kfdef/generic/istio rather than update it

  * It doesn't look like this has been modified since it was copied over
    from the kubeflow/kubeflow repo in 2019. ISTIO configs should be in /istio

* Add a test for deprecated Kubernetes resources fixes #375

Regenerate tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.