diff --git a/README.md b/README.md index 972a383198..52352cefa2 100644 --- a/README.md +++ b/README.md @@ -328,6 +328,7 @@ Below are some of the core components together with link to the logs that provid * [Payload Logging with ELK ](https://docs.seldon.io/projects/seldon-core/en/latest/analytics/logging.html) * [Distributed Tracing with Jaeger ](https://docs.seldon.io/projects/seldon-core/en/latest/graph/distributed-tracing.html) * [Replica Scaling ](https://docs.seldon.io/projects/seldon-core/en/latest/graph/scaling.html) +* [Budgeting Disruptions](https://docs.seldon.io/projects/seldon-core/en/latest/graph/disruption-budgets.html) * [Custom Inference Servers](https://docs.seldon.io/projects/seldon-core/en/latest/servers/custom.html) ### Advanced Inference diff --git a/doc/source/examples/notebooks.rst b/doc/source/examples/notebooks.rst index 5e15caf280..c08cb24d99 100644 --- a/doc/source/examples/notebooks.rst +++ b/doc/source/examples/notebooks.rst @@ -107,7 +107,6 @@ MLOps: Scaling and Monitoring and Observability CI / CD with Jenkins Classic CI / CD with Jenkins X Replica control - Production Configurations and Integrations ------------------------------------------ @@ -121,6 +120,7 @@ Production Configurations and Integrations Deploy Multiple Seldon Core Operators Protocol Examples Custom Protobuf Data Example + Disruption Budgets Example Complex Graph Examples ---------------------- diff --git a/doc/source/graph/disruption-budgets.md b/doc/source/graph/disruption-budgets.md new file mode 100644 index 0000000000..b49ed9f128 --- /dev/null +++ b/doc/source/graph/disruption-budgets.md @@ -0,0 +1,39 @@ +# Budgeting Disruptions + +High availability is an important aspect in running production systems. +To this end, you can add Pod Disruption Budget Specifications to the Pod Template Specifications you create. +Depending on how you want your application to handle disruptions, you can define your disruption budget accordingly. + +An example Seldon Deployment with disruption budgets defined can be seen below: + +```yaml +apiVersion: machinelearning.seldon.io/v1 +kind: SeldonDeployment +metadata: + name: seldon-model +spec: + name: test-deployment + replicas: 2 + predictors: + - componentSpecs: + - pdbSpec: + minAvailable: 90% + spec: + containers: + - image: seldonio/mock_classifier_rest:1.3 + imagePullPolicy: IfNotPresent + name: classifier + resources: + requests: + cpu: '0.5' + terminationGracePeriodSeconds: 1 + graph: + children: [] + endpoint: + type: REST + name: classifier + type: MODEL + name: example +``` + +This example ensures that our serving capacity does not decrease by more than 10%. diff --git a/doc/source/index.rst b/doc/source/index.rst index 4c37a4711e..b1cf71b558 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -110,6 +110,7 @@ Documentation Index Payload Logging with ELK Distributed Tracing with Jaeger Replica Scaling + Budgeting Disruptions Custom Inference Servers .. toctree:: @@ -137,18 +138,6 @@ Documentation Index Istio Ingress OpenShift -.. toctree:: - :maxdepth: 1 - :caption: Production - - Supported API Protocols - CI/CD MLOps at Scale - Metrics with Prometheus - Payload Logging with ELK - Distributed Tracing with Jaeger - Replica Scaling - Custom Inference Servers - .. toctree:: :maxdepth: 1 :caption: Streaming and Batch Processing diff --git a/examples/models/disruption_budgets/model_with_patched_pdb.yaml b/examples/models/disruption_budgets/model_with_patched_pdb.yaml new file mode 100644 index 0000000000..c754823d18 --- /dev/null +++ b/examples/models/disruption_budgets/model_with_patched_pdb.yaml @@ -0,0 +1,27 @@ +apiVersion: machinelearning.seldon.io/v1 +kind: SeldonDeployment +metadata: + name: seldon-model +spec: + name: test-deployment + replicas: 2 + predictors: + - componentSpecs: + - pdbSpec: + maxUnavailable: 1 + spec: + containers: + - image: seldonio/mock_classifier_rest:1.3 + imagePullPolicy: IfNotPresent + name: classifier + resources: + requests: + cpu: '0.5' + terminationGracePeriodSeconds: 1 + graph: + children: [] + endpoint: + type: REST + name: classifier + type: MODEL + name: example diff --git a/examples/models/disruption_budgets/model_with_pdb.yaml b/examples/models/disruption_budgets/model_with_pdb.yaml new file mode 100644 index 0000000000..c958dc3f6e --- /dev/null +++ b/examples/models/disruption_budgets/model_with_pdb.yaml @@ -0,0 +1,27 @@ +apiVersion: machinelearning.seldon.io/v1 +kind: SeldonDeployment +metadata: + name: seldon-model +spec: + name: test-deployment + replicas: 2 + predictors: + - componentSpecs: + - pdbSpec: + maxUnavailable: 2 + spec: + containers: + - image: seldonio/mock_classifier_rest:1.3 + imagePullPolicy: IfNotPresent + name: classifier + resources: + requests: + cpu: '0.5' + terminationGracePeriodSeconds: 1 + graph: + children: [] + endpoint: + type: REST + name: classifier + type: MODEL + name: example diff --git a/examples/models/disruption_budgets/pdbs_example.ipynb b/examples/models/disruption_budgets/pdbs_example.ipynb new file mode 100644 index 0000000000..7e9a6440f2 --- /dev/null +++ b/examples/models/disruption_budgets/pdbs_example.ipynb @@ -0,0 +1,249 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Defining Disruption Budgets for Seldon Deployments" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Prerequisites\n", + " \n", + "* A kubernetes cluster with kubectl configured\n", + "* pygmentize" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Setup Seldon Core\n", + "\n", + "Use the setup notebook to [Setup Cluster](../../../notebooks/seldon_core_setup.ipynb#Setup-Cluster) with [Ambassador Ingress](../../../notebooks/seldon_core_setup.ipynb#Ambassador) and [Install Seldon Core](../../seldon_core_setup.ipynb#Install-Seldon-Core). Instructions [also online](./seldon_core_setup.html)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!kubectl create namespace seldon" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!kubectl config set-context $(kubectl config current-context) --namespace=seldon" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Create model with Pod Disruption Budget\n", + "\n", + "To create a model with a Pod Disruption Budget, it is first important to understand how you would like your application to respond to [voluntary disruptions](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/#voluntary-and-involuntary-disruptions). Depending on the type of disruption budgeting your application needs, you will either define either of the following:\n", + "\n", + "* `minAvailable` which is a description of the number of pods from that set that must still be available after the eviction, even in the absence of the evicted pod. `minAvailable` can be either an absolute number or a percentage.\n", + "* `maxUnavailable` which is a description of the number of pods from that set that can be unavailable after the eviction. It can be either an absolute number or a percentage.\n", + "\n", + "The full SeldonDeployment spec is shown below." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!pygmentize model_with_pdb.yaml" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!kubectl apply -f model_with_pdb.yaml" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=seldon-model -o jsonpath='{.items[0].metadata.name}')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Validate Disruption Budget Configuration" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import json\n", + "\n", + "def getPdbConfig():\n", + " dp=!kubectl get pdb seldon-model-example-0-classifier -o json\n", + " dp=json.loads(\"\".join(dp))\n", + " return dp[\"spec\"][\"maxUnavailable\"]\n", + " \n", + "assert getPdbConfig() == 2" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!kubectl get pods,deployments,pdb" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Update Disruption Budget and Validate Change\n", + "\n", + "Next, we'll update the maximum number of unavailable pods and check that the PDB is properly updated to match." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!pygmentize model_with_patched_pdb.yaml" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!kubectl apply -f model_with_patched_pdb.yaml" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=seldon-model -o jsonpath='{.items[0].metadata.name}')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "assert getPdbConfig() == 1" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Clean Up" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!kubectl get pods,deployments,pdb" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!kubectl delete -f model_with_patched_pdb.yaml" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "anaconda-cloud": {}, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.7" + }, + "varInspector": { + "cols": { + "lenName": 16, + "lenType": 16, + "lenVar": 40 + }, + "kernels_config": { + "python": { + "delete_cmd_postfix": "", + "delete_cmd_prefix": "del ", + "library": "var_list.py", + "varRefreshCmd": "print(var_dic_list())" + }, + "r": { + "delete_cmd_postfix": ") ", + "delete_cmd_prefix": "rm(", + "library": "var_list.r", + "varRefreshCmd": "cat(var_dic_list()) " + } + }, + "types_to_exclude": [ + "module", + "function", + "builtin_function_or_method", + "instance", + "_Feature" + ], + "window_display": false + } + }, + "nbformat": 4, + "nbformat_minor": 1 +} diff --git a/examples/models/disruption_budgets/pdbs_example.py b/examples/models/disruption_budgets/pdbs_example.py new file mode 100755 index 0000000000..f64b5e9cb2 --- /dev/null +++ b/examples/models/disruption_budgets/pdbs_example.py @@ -0,0 +1,185 @@ +#!/usr/bin/env ipython +import time +#!/usr/bin/env python +# coding: utf-8 + +# # Defining Disruption Budgets for Seldon Deployments + +# ## Prerequisites +# +# * A kubernetes cluster with kubectl configured +# * pygmentize + +# ## Setup Seldon Core +# +# Use the setup notebook to [Setup Cluster](../../../notebooks/seldon_core_setup.ipynb#Setup-Cluster) with [Ambassador Ingress](../../../notebooks/seldon_core_setup.ipynb#Ambassador) and [Install Seldon Core](../../seldon_core_setup.ipynb#Install-Seldon-Core). Instructions [also online](./seldon_core_setup.html). + + + +# In[ ]: + + +get_ipython().system('kubectl create namespace seldon') + + + + + + +# In[ ]: + + +get_ipython().system('kubectl config set-context $(kubectl config current-context) --namespace=seldon') + + + + +# ## Create model with Pod Disruption Budget +# +# To create a model with a Pod Disruption Budget, it is first important to understand how you would like your application to respond to [voluntary disruptions](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/#voluntary-and-involuntary-disruptions). Depending on the type of disruption budgeting your application needs, you will either define either of the following: +# +# * `minAvailable` which is a description of the number of pods from that set that must still be available after the eviction, even in the absence of the evicted pod. `minAvailable` can be either an absolute number or a percentage. +# * `maxUnavailable` which is a description of the number of pods from that set that can be unavailable after the eviction. It can be either an absolute number or a percentage. +# +# The full SeldonDeployment spec is shown below. + + + +# In[ ]: + + +get_ipython().system('pygmentize model_with_pdb.yaml') + + + + + + +# In[ ]: + + +get_ipython().system('kubectl apply -f model_with_pdb.yaml') + + + + + +time.sleep(10) + +# In[ ]: + + +get_ipython().system("kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=seldon-model -o jsonpath='{.items[0].metadata.name}')") + + +time.sleep(5) + + +# ## Validate Disruption Budget Configuration + + + +# In[ ]: + + +import json + +def getPdbConfig(): + dp = get_ipython().getoutput('kubectl get pdb seldon-model-example-0-classifier -o json') + dp=json.loads("".join(dp)) + return dp["spec"]["maxUnavailable"] + +assert getPdbConfig() == 2 + + + + + + +# In[ ]: + + +get_ipython().system('kubectl get pods,deployments,pdb') + + + + +# ## Update Disruption Budget and Validate Change +# +# Next, we'll update the maximum number of unavailable pods and check that the PDB is properly updated to match. + + + +# In[ ]: + + +get_ipython().system('pygmentize model_with_patched_pdb.yaml') + + + + + + +# In[ ]: + + +get_ipython().system('kubectl apply -f model_with_patched_pdb.yaml') + + + + + +time.sleep(10) + +# In[ ]: + + +get_ipython().system("kubectl rollout status deploy/$(kubectl get deploy -l seldon-deployment-id=seldon-model -o jsonpath='{.items[0].metadata.name}')") + + +time.sleep(5) + + + + +# In[ ]: + + +assert getPdbConfig() == 1 + + + + +# ## Clean Up + + + +# In[ ]: + + +get_ipython().system('kubectl get pods,deployments,pdb') + + + + + +time.sleep(10) + +# In[ ]: + + +get_ipython().system('kubectl delete -f model_with_patched_pdb.yaml') + + +time.sleep(5) + + + + +# In[ ]: + + + + + + diff --git a/helm-charts/seldon-core-operator/templates/clusterrole_seldon-manager-role.yaml b/helm-charts/seldon-core-operator/templates/clusterrole_seldon-manager-role.yaml index bfd894f694..267eed2a00 100644 --- a/helm-charts/seldon-core-operator/templates/clusterrole_seldon-manager-role.yaml +++ b/helm-charts/seldon-core-operator/templates/clusterrole_seldon-manager-role.yaml @@ -166,6 +166,26 @@ rules: - get - patch - update +- apiGroups: + - policy + resources: + - poddisruptionbudgets + verbs: + - create + - delete + - get + - list + - patch + - update + - watch +- apiGroups: + - policy + resources: + - poddisruptionbudgets/status + verbs: + - get + - patch + - update - apiGroups: - v1 resources: diff --git a/helm-charts/seldon-core-operator/templates/customresourcedefinition_seldondeployments.machinelearning.seldon.io.yaml b/helm-charts/seldon-core-operator/templates/customresourcedefinition_seldondeployments.machinelearning.seldon.io.yaml index a5d0044fbd..1e06453d2f 100644 --- a/helm-charts/seldon-core-operator/templates/customresourcedefinition_seldondeployments.machinelearning.seldon.io.yaml +++ b/helm-charts/seldon-core-operator/templates/customresourcedefinition_seldondeployments.machinelearning.seldon.io.yaml @@ -409,6 +409,21 @@ spec: type: object metadata: type: object + pdbSpec: + properties: + maxUnavailable: + anyOf: + - type: integer + - type: string + description: An eviction is allowed if at most "maxUnavailable" pods in the deployment corresponding to a componentSpec are unavailable after the eviction, i.e. even in absence of the evicted pod. For example, one can prevent all voluntary evictions by specifying 0. MaxUnavailable and MinAvailable are mutually exclusive. + x-kubernetes-int-or-string: true + minAvailable: + anyOf: + - type: integer + - type: string + description: An eviction is allowed if at least "minAvailable" pods in the deployment corresponding to a componentSpec will still be available after the eviction, i.e. even in the absence of the evicted pod. So for example you can prevent all voluntary evictions by specifying "100%". + x-kubernetes-int-or-string: true + type: object replicas: format: int32 type: integer diff --git a/helm-charts/seldon-core-operator/templates/customresourcedefinition_v1_seldondeployments.machinelearning.seldon.io.yaml b/helm-charts/seldon-core-operator/templates/customresourcedefinition_v1_seldondeployments.machinelearning.seldon.io.yaml index 2047872a9c..1d185c2b41 100644 --- a/helm-charts/seldon-core-operator/templates/customresourcedefinition_v1_seldondeployments.machinelearning.seldon.io.yaml +++ b/helm-charts/seldon-core-operator/templates/customresourcedefinition_v1_seldondeployments.machinelearning.seldon.io.yaml @@ -401,6 +401,21 @@ spec: type: object metadata: type: object + pdbSpec: + properties: + maxUnavailable: + anyOf: + - type: integer + - type: string + description: An eviction is allowed if at most "maxUnavailable" pods in the deployment corresponding to a componentSpec are unavailable after the eviction, i.e. even in absence of the evicted pod. For example, one can prevent all voluntary evictions by specifying 0. MaxUnavailable and MinAvailable are mutually exclusive. + x-kubernetes-int-or-string: true + minAvailable: + anyOf: + - type: integer + - type: string + description: An eviction is allowed if at least "minAvailable" pods in the deployment corresponding to a componentSpec will still be available after the eviction, i.e. even in the absence of the evicted pod. So for example you can prevent all voluntary evictions by specifying "100%". + x-kubernetes-int-or-string: true + type: object replicas: format: int32 type: integer @@ -5863,6 +5878,21 @@ spec: type: object metadata: type: object + pdbSpec: + properties: + maxUnavailable: + anyOf: + - type: integer + - type: string + description: An eviction is allowed if at most "maxUnavailable" pods in the deployment corresponding to a componentSpec are unavailable after the eviction, i.e. even in absence of the evicted pod. For example, one can prevent all voluntary evictions by specifying 0. MaxUnavailable and MinAvailable are mutually exclusive. + x-kubernetes-int-or-string: true + minAvailable: + anyOf: + - type: integer + - type: string + description: An eviction is allowed if at least "minAvailable" pods in the deployment corresponding to a componentSpec will still be available after the eviction, i.e. even in the absence of the evicted pod. So for example you can prevent all voluntary evictions by specifying "100%". + x-kubernetes-int-or-string: true + type: object replicas: format: int32 type: integer @@ -11325,6 +11355,21 @@ spec: type: object metadata: type: object + pdbSpec: + properties: + maxUnavailable: + anyOf: + - type: integer + - type: string + description: An eviction is allowed if at most "maxUnavailable" pods in the deployment corresponding to a componentSpec are unavailable after the eviction, i.e. even in absence of the evicted pod. For example, one can prevent all voluntary evictions by specifying 0. MaxUnavailable and MinAvailable are mutually exclusive. + x-kubernetes-int-or-string: true + minAvailable: + anyOf: + - type: integer + - type: string + description: An eviction is allowed if at least "minAvailable" pods in the deployment corresponding to a componentSpec will still be available after the eviction, i.e. even in the absence of the evicted pod. So for example you can prevent all voluntary evictions by specifying "100%". + x-kubernetes-int-or-string: true + type: object replicas: format: int32 type: integer diff --git a/operator/apis/machinelearning.seldon.io/v1/seldondeployment_types.go b/operator/apis/machinelearning.seldon.io/v1/seldondeployment_types.go index 53ea1ac35b..b6dd20de7b 100644 --- a/operator/apis/machinelearning.seldon.io/v1/seldondeployment_types.go +++ b/operator/apis/machinelearning.seldon.io/v1/seldondeployment_types.go @@ -26,6 +26,7 @@ import ( autoscalingv2beta2 "k8s.io/api/autoscaling/v2beta1" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/util/intstr" ) const ( @@ -291,6 +292,7 @@ type SeldonPodSpec struct { HpaSpec *SeldonHpaSpec `json:"hpaSpec,omitempty" protobuf:"bytes,3,opt,name=hpaSpec"` Replicas *int32 `json:"replicas,omitempty" protobuf:"bytes,4,opt,name=replicas"` KedaSpec *SeldonScaledObjectSpec `json:"kedaSpec,omitempty" protobuf:"bytes,5,opt,name=kedaSpec"` + PdbSpec *SeldonPdbSpec `json:"pdbSpec,omitempty" protobuf:"bytes,6,opt,name=pdbSpec"` } // SeldonScaledObjectSpec is the spec for a KEDA ScaledObject resource @@ -314,6 +316,23 @@ type SeldonHpaSpec struct { Metrics []autoscalingv2beta2.MetricSpec `json:"metrics,omitempty" protobuf:"bytes,3,opt,name=metrics"` } +type SeldonPdbSpec struct { + // An eviction is allowed if at least "minAvailable" pods in the deployment + // corresponding to a componentSpec will still be available after the eviction, i.e. even in the + // absence of the evicted pod. So for example you can prevent all voluntary + // evictions by specifying "100%". + // +optional + MinAvailable *intstr.IntOrString `json:"minAvailable,omitempty" protobuf:"bytes,1,opt,name=minAvailable"` + + // An eviction is allowed if at most "maxUnavailable" pods in the deployment + // corresponding to a componentSpec are unavailable after the eviction, i.e. even in absence of + // the evicted pod. For example, one can prevent all voluntary evictions + // by specifying 0. + // MaxUnavailable and MinAvailable are mutually exclusive. + // +optional + MaxUnavailable *intstr.IntOrString `json:"maxUnavailable,omitempty" protobuf:"bytes,2,opt,name=maxUnavailable"` +} + type PredictiveUnitType string const ( diff --git a/operator/apis/machinelearning.seldon.io/v1/zz_generated.deepcopy.go b/operator/apis/machinelearning.seldon.io/v1/zz_generated.deepcopy.go index de019ce80d..4efa65a4db 100644 --- a/operator/apis/machinelearning.seldon.io/v1/zz_generated.deepcopy.go +++ b/operator/apis/machinelearning.seldon.io/v1/zz_generated.deepcopy.go @@ -25,6 +25,7 @@ import ( "k8s.io/api/autoscaling/v2beta1" corev1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/util/intstr" ) // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. @@ -481,6 +482,31 @@ func (in *SeldonHpaSpec) DeepCopy() *SeldonHpaSpec { return out } +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *SeldonPdbSpec) DeepCopyInto(out *SeldonPdbSpec) { + *out = *in + if in.MinAvailable != nil { + in, out := &in.MinAvailable, &out.MinAvailable + *out = new(intstr.IntOrString) + **out = **in + } + if in.MaxUnavailable != nil { + in, out := &in.MaxUnavailable, &out.MaxUnavailable + *out = new(intstr.IntOrString) + **out = **in + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new SeldonPdbSpec. +func (in *SeldonPdbSpec) DeepCopy() *SeldonPdbSpec { + if in == nil { + return nil + } + out := new(SeldonPdbSpec) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *SeldonPodSpec) DeepCopyInto(out *SeldonPodSpec) { *out = *in @@ -501,6 +527,11 @@ func (in *SeldonPodSpec) DeepCopyInto(out *SeldonPodSpec) { *out = new(SeldonScaledObjectSpec) (*in).DeepCopyInto(*out) } + if in.PdbSpec != nil { + in, out := &in.PdbSpec, &out.PdbSpec + *out = new(SeldonPdbSpec) + (*in).DeepCopyInto(*out) + } } // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new SeldonPodSpec. diff --git a/operator/config/crd/bases/machinelearning.seldon.io_seldondeployments.yaml b/operator/config/crd/bases/machinelearning.seldon.io_seldondeployments.yaml index 7ef4bc699a..6e77a9416a 100644 --- a/operator/config/crd/bases/machinelearning.seldon.io_seldondeployments.yaml +++ b/operator/config/crd/bases/machinelearning.seldon.io_seldondeployments.yaml @@ -4,7 +4,7 @@ apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: annotations: - controller-gen.kubebuilder.io/version: v0.2.4 + controller-gen.kubebuilder.io/version: v0.2.5 creationTimestamp: null name: seldondeployments.machinelearning.seldon.io spec: @@ -139,15 +139,23 @@ spec: type: object type: object targetAverageValue: + anyOf: + - type: integer + - type: string description: targetAverageValue is the target per-pod value of global metric (as a quantity). Mutually exclusive with TargetValue. - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true targetValue: + anyOf: + - type: integer + - type: string description: targetValue is the target value of the metric (as a quantity). Mutually exclusive with TargetAverageValue. - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true required: - metricName type: object @@ -157,10 +165,14 @@ spec: on an Ingress object). properties: averageValue: + anyOf: + - type: integer + - type: string description: averageValue is the target value of the average of the metric across all relevant pods (as a quantity) - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true metricName: description: metricName is the name of the metric in question. @@ -242,9 +254,13 @@ spec: - name type: object targetValue: + anyOf: + - type: integer + - type: string description: targetValue is the target value of the metric (as a quantity). - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true required: - metricName - target @@ -319,10 +335,14 @@ spec: type: object type: object targetAverageValue: + anyOf: + - type: integer + - type: string description: targetAverageValue is the target value of the average of the metric across all relevant pods (as a quantity) - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true required: - metricName - targetAverageValue @@ -349,12 +369,16 @@ spec: format: int32 type: integer targetAverageValue: + anyOf: + - type: integer + - type: string description: targetAverageValue is the target value of the average of the resource metric across all relevant pods, as a raw value (instead of as a percentage of the request), similar to the "pods" metric source type. - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true required: - name type: object @@ -561,19 +585,27 @@ spec: format: int32 type: integer averageValue: + anyOf: + - type: integer + - type: string description: averageValue is the target value of the average of the metric across all relevant pods (as a quantity) - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true type: description: type represents whether the metric type is Utilization, Value, or AverageValue type: string value: + anyOf: + - type: integer + - type: string description: value is the target value of the metric (as a quantity). - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true required: - type type: object @@ -631,6 +663,31 @@ spec: type: object metadata: type: object + pdbSpec: + properties: + maxUnavailable: + anyOf: + - type: integer + - type: string + description: An eviction is allowed if at most "maxUnavailable" + pods in the deployment corresponding to a componentSpec + are unavailable after the eviction, i.e. even in absence + of the evicted pod. For example, one can prevent all + voluntary evictions by specifying 0. MaxUnavailable + and MinAvailable are mutually exclusive. + x-kubernetes-int-or-string: true + minAvailable: + anyOf: + - type: integer + - type: string + description: An eviction is allowed if at least "minAvailable" + pods in the deployment corresponding to a componentSpec + will still be available after the eviction, i.e. even + in the absence of the evicted pod. So for example + you can prevent all voluntary evictions by specifying + "100%". + x-kubernetes-int-or-string: true + type: object replicas: format: int32 type: integer @@ -1465,10 +1522,14 @@ spec: vars' type: string divisor: + anyOf: + - type: integer + - type: string description: Specifies the output format of the exposed resources, defaults to "1" - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true resource: description: 'Required: resource to select' @@ -1972,6 +2033,10 @@ spec: - containerPort type: object type: array + x-kubernetes-list-map-keys: + - containerPort + - protocol + x-kubernetes-list-type: map readinessProbe: description: 'Periodic probe of container service readiness. Container will be removed from service @@ -2108,14 +2173,22 @@ spec: properties: limits: additionalProperties: - type: string + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true description: 'Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/' type: object requests: additionalProperties: - type: string + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true description: 'Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it @@ -2722,10 +2795,14 @@ spec: vars' type: string divisor: + anyOf: + - type: integer + - type: string description: Specifies the output format of the exposed resources, defaults to "1" - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true resource: description: 'Required: resource to select' @@ -3352,14 +3429,22 @@ spec: properties: limits: additionalProperties: - type: string + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true description: 'Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/' type: object requests: additionalProperties: - type: string + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true description: 'Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it @@ -3965,10 +4050,14 @@ spec: vars' type: string divisor: + anyOf: + - type: integer + - type: string description: Specifies the output format of the exposed resources, defaults to "1" - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true resource: description: 'Required: resource to select' @@ -4472,6 +4561,10 @@ spec: - containerPort type: object type: array + x-kubernetes-list-map-keys: + - containerPort + - protocol + x-kubernetes-list-type: map readinessProbe: description: 'Periodic probe of container service readiness. Container will be removed from service @@ -4608,14 +4701,22 @@ spec: properties: limits: additionalProperties: - type: string + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true description: 'Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/' type: object requests: additionalProperties: - type: string + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true description: 'Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it @@ -5048,7 +5149,11 @@ spec: type: object overhead: additionalProperties: - type: string + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true description: 'Overhead represents the resource overhead associated with running a pod for a given RuntimeClass. This field will be autopopulated at admission time @@ -5466,6 +5571,10 @@ spec: - whenUnsatisfiable type: object type: array + x-kubernetes-list-map-keys: + - topologyKey + - whenUnsatisfiable + x-kubernetes-list-type: map volumes: description: 'List of volumes that can be mounted by containers belonging to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes' @@ -5847,10 +5956,14 @@ spec: vars' type: string divisor: + anyOf: + - type: integer + - type: string description: Specifies the output format of the exposed resources, defaults to "1" - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true resource: description: 'Required: resource to select' @@ -5876,6 +5989,9 @@ spec: or Memory. More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir' type: string sizeLimit: + anyOf: + - type: integer + - type: string description: 'Total amount of local storage required for this EmptyDir volume. The size limit is also applicable for memory medium. @@ -5885,7 +6001,8 @@ spec: of all containers in a pod. The default is nil which means that the limit is undefined. More info: http://kubernetes.io/docs/user-guide/volumes#emptydir' - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true type: object fc: description: FC represents a Fibre Channel resource @@ -6418,11 +6535,15 @@ spec: optional for env vars' type: string divisor: + anyOf: + - type: integer + - type: string description: Specifies the output format of the exposed resources, defaults to "1" - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true resource: description: 'Required: resource to select' @@ -6856,13 +6977,21 @@ spec: properties: limits: additionalProperties: - type: string + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true description: 'Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/' type: object requests: additionalProperties: - type: string + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true description: 'Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise @@ -6979,10 +7108,14 @@ spec: volumes, optional for env vars' type: string divisor: + anyOf: + - type: integer + - type: string description: Specifies the output format of the exposed resources, defaults to "1" - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true resource: description: 'Required: resource to select' type: string @@ -7447,6 +7580,10 @@ spec: - containerPort type: object type: array + x-kubernetes-list-map-keys: + - containerPort + - protocol + x-kubernetes-list-type: map readinessProbe: description: 'Periodic probe of container service readiness. Container will be removed from service endpoints if @@ -7572,13 +7709,21 @@ spec: properties: limits: additionalProperties: - type: string + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true description: 'Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/' type: object requests: additionalProperties: - type: string + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true description: 'Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is @@ -8128,9 +8273,13 @@ spec: optional for env vars' type: string divisor: + anyOf: + - type: integer + - type: string description: Specifies the output format of the exposed resources, defaults to "1" - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true resource: description: 'Required: resource to select' type: string @@ -8172,13 +8321,21 @@ spec: properties: limits: additionalProperties: - type: string + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true description: 'Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/' type: object requests: additionalProperties: - type: string + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true description: 'Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly diff --git a/operator/config/crd_v1/bases/machinelearning.seldon.io_seldondeployments.yaml b/operator/config/crd_v1/bases/machinelearning.seldon.io_seldondeployments.yaml index 61aebb928a..9c66146466 100644 --- a/operator/config/crd_v1/bases/machinelearning.seldon.io_seldondeployments.yaml +++ b/operator/config/crd_v1/bases/machinelearning.seldon.io_seldondeployments.yaml @@ -642,6 +642,31 @@ spec: type: object metadata: type: object + pdbSpec: + properties: + maxUnavailable: + anyOf: + - type: integer + - type: string + description: An eviction is allowed if at most "maxUnavailable" + pods in the deployment corresponding to a componentSpec + are unavailable after the eviction, i.e. even in + absence of the evicted pod. For example, one can + prevent all voluntary evictions by specifying 0. + MaxUnavailable and MinAvailable are mutually exclusive. + x-kubernetes-int-or-string: true + minAvailable: + anyOf: + - type: integer + - type: string + description: An eviction is allowed if at least "minAvailable" + pods in the deployment corresponding to a componentSpec + will still be available after the eviction, i.e. + even in the absence of the evicted pod. So for + example you can prevent all voluntary evictions + by specifying "100%". + x-kubernetes-int-or-string: true + type: object replicas: format: int32 type: integer @@ -9117,6 +9142,31 @@ spec: type: object metadata: type: object + pdbSpec: + properties: + maxUnavailable: + anyOf: + - type: integer + - type: string + description: An eviction is allowed if at most "maxUnavailable" + pods in the deployment corresponding to a componentSpec + are unavailable after the eviction, i.e. even in + absence of the evicted pod. For example, one can + prevent all voluntary evictions by specifying 0. + MaxUnavailable and MinAvailable are mutually exclusive. + x-kubernetes-int-or-string: true + minAvailable: + anyOf: + - type: integer + - type: string + description: An eviction is allowed if at least "minAvailable" + pods in the deployment corresponding to a componentSpec + will still be available after the eviction, i.e. + even in the absence of the evicted pod. So for + example you can prevent all voluntary evictions + by specifying "100%". + x-kubernetes-int-or-string: true + type: object replicas: format: int32 type: integer @@ -17592,6 +17642,31 @@ spec: type: object metadata: type: object + pdbSpec: + properties: + maxUnavailable: + anyOf: + - type: integer + - type: string + description: An eviction is allowed if at most "maxUnavailable" + pods in the deployment corresponding to a componentSpec + are unavailable after the eviction, i.e. even in + absence of the evicted pod. For example, one can + prevent all voluntary evictions by specifying 0. + MaxUnavailable and MinAvailable are mutually exclusive. + x-kubernetes-int-or-string: true + minAvailable: + anyOf: + - type: integer + - type: string + description: An eviction is allowed if at least "minAvailable" + pods in the deployment corresponding to a componentSpec + will still be available after the eviction, i.e. + even in the absence of the evicted pod. So for + example you can prevent all voluntary evictions + by specifying "100%". + x-kubernetes-int-or-string: true + type: object replicas: format: int32 type: integer diff --git a/operator/config/rbac/role.yaml b/operator/config/rbac/role.yaml index ec05e51aab..3e7fa4c33c 100644 --- a/operator/config/rbac/role.yaml +++ b/operator/config/rbac/role.yaml @@ -161,6 +161,26 @@ rules: - get - patch - update +- apiGroups: + - policy + resources: + - poddisruptionbudgets + verbs: + - create + - delete + - get + - list + - patch + - update + - watch +- apiGroups: + - policy + resources: + - poddisruptionbudgets/status + verbs: + - get + - patch + - update - apiGroups: - v1 resources: diff --git a/operator/constants/constants.go b/operator/constants/constants.go index ba5e608acd..9da712520b 100644 --- a/operator/constants/constants.go +++ b/operator/constants/constants.go @@ -58,6 +58,9 @@ const ( EventsCreateScaledObject = "CreateScaledObject" EventsUpdateScaledObject = "UpdateScaledObject" EventsDeleteScaledObject = "DeleteScaledObject" + EventsCreatePDB = "CreatePDB" + EventsUpdatePDB = "UpdatePDB" + EventsDeletePDB = "DeletePDB" EventsCreateDeployment = "CreateDeployment" EventsUpdateDeployment = "UpdateDeployment" EventsDeleteDeployment = "DeleteDeployment" diff --git a/operator/controllers/seldondeployment_controller.go b/operator/controllers/seldondeployment_controller.go index 9aa038531a..1340302b71 100644 --- a/operator/controllers/seldondeployment_controller.go +++ b/operator/controllers/seldondeployment_controller.go @@ -53,6 +53,7 @@ import ( appsv1 "k8s.io/api/apps/v1" autoscaling "k8s.io/api/autoscaling/v2beta1" corev1 "k8s.io/api/core/v1" + policy "k8s.io/api/policy/v1beta1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" ) @@ -98,6 +99,7 @@ type components struct { services []*corev1.Service hpas []*autoscaling.HorizontalPodAutoscaler kedaScaledObjects []*kedav1alpha1.ScaledObject + pdbs []*policy.PodDisruptionBudget virtualServices []*istio.VirtualService destinationRules []*istio.DestinationRule defaultDeploymentName string @@ -180,6 +182,24 @@ func createHpa(podSpec *machinelearningv1.SeldonPodSpec, deploymentName string, return &hpa } +func createPdb(podSpec *machinelearningv1.SeldonPodSpec, deploymentName string, seldonId string, namespace string) *policy.PodDisruptionBudget { + pdb := policy.PodDisruptionBudget{ + ObjectMeta: metav1.ObjectMeta{ + Name: deploymentName, + Namespace: namespace, + Labels: map[string]string{machinelearningv1.Label_seldon_id: seldonId}, + }, + Spec: policy.PodDisruptionBudgetSpec{ + MinAvailable: podSpec.PdbSpec.MinAvailable, + MaxUnavailable: podSpec.PdbSpec.MaxUnavailable, + Selector: &metav1.LabelSelector{ + MatchLabels: map[string]string{machinelearningv1.Label_seldon_id: seldonId}, + }, + }, + } + return &pdb +} + // Create istio virtual service and destination rule. // Creates routes for each predictor with traffic weight split func createIstioResources(mlDep *machinelearningv1.SeldonDeployment, @@ -498,6 +518,11 @@ func (r *SeldonDeploymentReconciler) createComponents(ctx context.Context, mlDep } } + // Add PDB if needed + if cSpec.PdbSpec != nil { + c.pdbs = append(c.pdbs, createPdb(cSpec, depName, seldonId, namespace)) + } + // create services for each container for k := 0; k < len(cSpec.Spec.Containers); k++ { var con *corev1.Container @@ -1330,6 +1355,86 @@ func (r *SeldonDeploymentReconciler) createHpas(components *components, instance return ready, nil } +// Create Services specified in components. +func (r *SeldonDeploymentReconciler) createPdbs(components *components, instance *machinelearningv1.SeldonDeployment, log logr.Logger) (bool, error) { + ready := true + pdbSet := make(map[string]bool) + for _, pdb := range components.pdbs { + if err := ctrl.SetControllerReference(instance, pdb, r.Scheme); err != nil { + return ready, err + } + pdbSet[pdb.Name] = true + found := &policy.PodDisruptionBudget{} + err := r.Get(context.TODO(), types.NamespacedName{Name: pdb.Name, Namespace: pdb.Namespace}, found) + if err != nil && errors.IsNotFound(err) { + ready = false + log.Info("Creating PDB", "namespace", pdb.Namespace, "name", pdb.Name) + err = r.Create(context.TODO(), pdb) + if err != nil { + return ready, err + } + r.Recorder.Eventf(instance, corev1.EventTypeNormal, constants.EventsCreatePDB, "Created PodDisruptionBudget %q", pdb.GetName()) + } else if err != nil { + return ready, err + } else { + // Update the found object and write the result back if there are any changes + if !equality.Semantic.DeepEqual(pdb.Spec, found.Spec) { + + desiredPdb := found.DeepCopy() + found.Spec = pdb.Spec + + log.Info("Updating PDB", "namespace", pdb.Namespace, "name", pdb.Name) + err = r.Update(context.TODO(), found) + if err != nil { + return ready, err + } + + // Check if what came back from server modulo the defaults applied by k8s is the same or not + if !equality.Semantic.DeepEqual(desiredPdb.Spec, found.Spec) { + ready = false + r.Recorder.Eventf(instance, corev1.EventTypeNormal, constants.EventsUpdatePDB, "Updated HorizontalPodAutoscaler %q", pdb.GetName()) + //For debugging we will show the difference + diff, err := kmp.SafeDiff(desiredPdb.Spec, found.Spec) + if err != nil { + log.Error(err, "Failed to diff") + } else { + log.Info(fmt.Sprintf("Difference in PDBs: %v", diff)) + } + } else { + log.Info("The PDBs are the same - api server defaults ignored") + } + + } else { + log.Info("Found identical PDB", "namespace", found.Namespace, "name", found.Name, "status", found.Status) + } + } + + } + + // For all Deployments check if any PDBs exist and they are not required + for _, deploy := range components.deployments { + if _, ok := pdbSet[deploy.Name]; !ok { + found := &policy.PodDisruptionBudget{} + err := r.Get(context.TODO(), types.NamespacedName{Name: deploy.Name, Namespace: deploy.Namespace}, found) + if err != nil { + if !errors.IsNotFound(err) { + return false, err + } + // Do nothing + } else { + // Delete PDB + log.Info("Deleting pdb", "name", deploy.Name) + err := r.Delete(context.TODO(), found, client.PropagationPolicy(metav1.DeletePropagationForeground)) + if err != nil { + return ready, err + } + } + } + } + + return ready, nil +} + func jsonEquals(a, b interface{}) (bool, error) { b1, err := json.Marshal(a) if err != nil { @@ -1513,6 +1618,24 @@ func (r *SeldonDeploymentReconciler) completeServiceCreation(instance *machinele } r.Recorder.Eventf(instance, corev1.EventTypeNormal, constants.EventsDeleteHPA, "Deleted HorizontalPodAutoscaler %q", foundHpa.GetName()) } + + // Delete any dangling PDBs + foundPdb := &policy.PodDisruptionBudget{} + err = r.Get(context.TODO(), types.NamespacedName{Name: found.Name, Namespace: found.Namespace}, foundPdb) + if err != nil { + if !errors.IsNotFound(err) { + return err + } + // Do nothing + } else { + // Delete PDB that should not exist + log.Info("Deleting pdb for removed predictor", "name", foundPdb.Name) + err := r.Delete(context.TODO(), foundPdb, client.PropagationPolicy(metav1.DeletePropagationForeground)) + if err != nil { + return err + } + r.Recorder.Eventf(instance, corev1.EventTypeNormal, constants.EventsDeletePDB, "Deleted PodDisruptionBudget %q", foundPdb.GetName()) + } } } if remaining == 0 { @@ -1554,6 +1677,8 @@ func (r *SeldonDeploymentReconciler) completeServiceCreation(instance *machinele // +kubebuilder:rbac:groups=autoscaling,resources=horizontalpodautoscalers/status,verbs=get;update;patch // +kubebuilder:rbac:groups=keda.sh/v1alpha1,resources=ScaledObject,verbs=get;list;watch;create;update;patch;delete // +kubebuilder:rbac:groups=keda.sh/v1alpha1,resources=ScaledObject/status,verbs=get;update;patch +// +kubebuilder:rbac:groups=policy,resources=poddisruptionbudgets,verbs=get;list;watch;create;update;patch;delete +// +kubebuilder:rbac:groups=policy,resources=poddisruptionbudgets/status,verbs=get;update;patch // +kubebuilder:rbac:groups=machinelearning.seldon.io,resources=seldondeployments,verbs=get;list;watch;create;update;patch;delete // +kubebuilder:rbac:groups=machinelearning.seldon.io,resources=seldondeployments/status,verbs=get;update;patch // +kubebuilder:rbac:groups=machinelearning.seldon.io,resources=seldondeployments/finalizers,verbs=get;update;patch @@ -1650,6 +1775,13 @@ func (r *SeldonDeploymentReconciler) Reconcile(req ctrl.Request) (ctrl.Result, e } } + pdbsReady, err := r.createPdbs(components, instance, log) + if err != nil { + r.Recorder.Eventf(instance, corev1.EventTypeWarning, constants.EventsInternalError, err.Error()) + r.updateStatusForError(instance, err, log) + return ctrl.Result{}, err + } + deploymentsReady, err := r.createDeployments(components, instance, log) if err != nil { r.Recorder.Eventf(instance, corev1.EventTypeWarning, constants.EventsInternalError, err.Error()) @@ -1666,7 +1798,7 @@ func (r *SeldonDeploymentReconciler) Reconcile(req ctrl.Request) (ctrl.Result, e } } - if deploymentsReady && servicesReady && hpasReady && (!withKedaSupport || kedaScaledObjectsReady) { + if deploymentsReady && servicesReady && hpasReady && pdbsReady && (!withKedaSupport || kedaScaledObjectsReady) { instance.Status.State = machinelearningv1.StatusStateAvailable instance.Status.Description = "" } else { diff --git a/operator/controllers/seldondeployment_controller_test.go b/operator/controllers/seldondeployment_controller_test.go index 1532f92610..c3544c7e51 100644 --- a/operator/controllers/seldondeployment_controller_test.go +++ b/operator/controllers/seldondeployment_controller_test.go @@ -32,6 +32,7 @@ import ( v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/intstr" "k8s.io/apimachinery/pkg/util/rand" ) @@ -527,6 +528,129 @@ var _ = Describe("Create a Seldon Deployment with hpa", func() { }) }) +var _ = Describe("Create a Seldon Deployment with pdb", func() { + const timeout = time.Second * 30 + const interval = time.Second * 1 + namespaceName := rand.String(10) + It("should create a resources", func() { + Expect(k8sClient).NotTo(BeNil()) + var modelType = machinelearningv1.MODEL + key := types.NamespacedName{ + Name: "dep", + Namespace: namespaceName, + } + maxUnavailable := intstr.FromInt(1) + instance := &machinelearningv1.SeldonDeployment{ + ObjectMeta: metav1.ObjectMeta{ + Name: key.Name, + Namespace: key.Namespace, + }, + Spec: machinelearningv1.SeldonDeploymentSpec{ + Name: "mydep", + Predictors: []machinelearningv1.PredictorSpec{ + { + Name: "p1", + ComponentSpecs: []*machinelearningv1.SeldonPodSpec{ + { + Spec: v1.PodSpec{ + Containers: []v1.Container{ + { + Image: "seldonio/mock_classifier:1.0", + Name: "classifier", + }, + }, + }, + PdbSpec: &machinelearningv1.SeldonPdbSpec{ + MinAvailable: nil, + MaxUnavailable: &maxUnavailable, + }, + }, + }, + Graph: machinelearningv1.PredictiveUnit{ + Name: "classifier", + Type: &modelType, + }, + }, + }, + }, + } + + //Create namespace + namespace := &v1.Namespace{ + ObjectMeta: metav1.ObjectMeta{ + Name: namespaceName, + }, + } + Expect(k8sClient.Create(context.Background(), namespace)).Should(Succeed()) + + // Run Defaulter + instance.Default() + + Expect(k8sClient.Create(context.Background(), instance)).Should(Succeed()) + //time.Sleep(time.Second * 5) + + fetched := &machinelearningv1.SeldonDeployment{} + Eventually(func() error { + err := k8sClient.Get(context.Background(), key, fetched) + return err + }, timeout, interval).Should(BeNil()) + Expect(fetched.Name).Should(Equal("dep")) + + // Check deployment created + depKey := types.NamespacedName{ + Name: machinelearningv1.GetDeploymentName(instance, instance.Spec.Predictors[0], instance.Spec.Predictors[0].ComponentSpecs[0], 0), + Namespace: namespaceName, + } + depFetched := &appsv1.Deployment{} + Eventually(func() error { + err := k8sClient.Get(context.Background(), depKey, depFetched) + return err + }, timeout, interval).Should(BeNil()) + Expect(len(depFetched.Spec.Template.Spec.Containers)).Should(Equal(2)) + + //Check svc created + svcKey := types.NamespacedName{ + Name: machinelearningv1.GetContainerServiceName("dep", instance.Spec.Predictors[0], &instance.Spec.Predictors[0].ComponentSpecs[0].Spec.Containers[0]), + Namespace: namespaceName, + } + svcFetched := &v1.Service{} + Eventually(func() error { + err := k8sClient.Get(context.Background(), svcKey, svcFetched) + return err + }, timeout, interval).Should(BeNil()) + + //Check pdb created + pdbKey := types.NamespacedName{ + Name: machinelearningv1.GetContainerServiceName("dep", instance.Spec.Predictors[0], &instance.Spec.Predictors[0].ComponentSpecs[0].Spec.Containers[0]), + Namespace: namespaceName, + } + pdbFetched := &v1.Service{} + Eventually(func() error { + err := k8sClient.Get(context.Background(), pdbKey, pdbFetched) + return err + }, timeout, interval).Should(BeNil()) + + // Check events created + serviceCreatedEvents := 0 + deploymentsCreatedEvents := 0 + evts, err := clientset.CoreV1().Events(namespaceName).Search(scheme, fetched) + Expect(err).To(BeNil()) + for _, evt := range evts.Items { + if evt.Reason == constants.EventsCreateService { + serviceCreatedEvents = serviceCreatedEvents + 1 + } else if evt.Reason == constants.EventsCreateDeployment { + deploymentsCreatedEvents = deploymentsCreatedEvents + 1 + } + } + + Expect(serviceCreatedEvents).To(Equal(2)) + Expect(deploymentsCreatedEvents).To(Equal(1)) + + Expect(k8sClient.Delete(context.Background(), instance)).Should(Succeed()) + + }) +}) + var _ = Describe("Create a Seldon Deployment and then a new one", func() { const timeout = time.Second * 30 const interval = time.Second * 1 diff --git a/operator/testing/machinelearning.seldon.io_seldondeployments.yaml b/operator/testing/machinelearning.seldon.io_seldondeployments.yaml index b43ede3e5c..d53a8a8974 100644 --- a/operator/testing/machinelearning.seldon.io_seldondeployments.yaml +++ b/operator/testing/machinelearning.seldon.io_seldondeployments.yaml @@ -3,7 +3,7 @@ kind: CustomResourceDefinition metadata: annotations: cert-manager.io/inject-ca-from: seldon-system/seldon-serving-cert - controller-gen.kubebuilder.io/version: v0.2.4 + controller-gen.kubebuilder.io/version: v0.2.5 labels: app: seldon app.kubernetes.io/instance: seldon1 @@ -107,11 +107,19 @@ spec: type: object type: object targetAverageValue: + anyOf: + - type: integer + - type: string description: targetAverageValue is the target per-pod value of global metric (as a quantity). Mutually exclusive with TargetValue. - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true targetValue: + anyOf: + - type: integer + - type: string description: targetValue is the target value of the metric (as a quantity). Mutually exclusive with TargetAverageValue. - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true required: - metricName type: object @@ -119,8 +127,12 @@ spec: description: object refers to a metric describing a single kubernetes object (for example, hits-per-second on an Ingress object). properties: averageValue: + anyOf: + - type: integer + - type: string description: averageValue is the target value of the average of the metric across all relevant pods (as a quantity) - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true metricName: description: metricName is the name of the metric in question. type: string @@ -171,8 +183,12 @@ spec: - name type: object targetValue: + anyOf: + - type: integer + - type: string description: targetValue is the target value of the metric (as a quantity). - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true required: - metricName - target @@ -215,8 +231,12 @@ spec: type: object type: object targetAverageValue: + anyOf: + - type: integer + - type: string description: targetAverageValue is the target value of the average of the metric across all relevant pods (as a quantity) - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true required: - metricName - targetAverageValue @@ -232,8 +252,12 @@ spec: format: int32 type: integer targetAverageValue: + anyOf: + - type: integer + - type: string description: targetAverageValue is the target value of the average of the resource metric across all relevant pods, as a raw value (instead of as a percentage of the request), similar to the "pods" metric source type. - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true required: - name type: object @@ -344,14 +368,22 @@ spec: format: int32 type: integer averageValue: + anyOf: + - type: integer + - type: string description: averageValue is the target value of the average of the metric across all relevant pods (as a quantity) - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true type: description: type represents whether the metric type is Utilization, Value, or AverageValue type: string value: + anyOf: + - type: integer + - type: string description: value is the target value of the metric (as a quantity). - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true required: - type type: object @@ -406,6 +438,21 @@ spec: type: object metadata: type: object + pdbSpec: + properties: + maxUnavailable: + anyOf: + - type: integer + - type: string + description: An eviction is allowed if at most "maxUnavailable" pods in the deployment corresponding to a componentSpec are unavailable after the eviction, i.e. even in absence of the evicted pod. For example, one can prevent all voluntary evictions by specifying 0. MaxUnavailable and MinAvailable are mutually exclusive. + x-kubernetes-int-or-string: true + minAvailable: + anyOf: + - type: integer + - type: string + description: An eviction is allowed if at least "minAvailable" pods in the deployment corresponding to a componentSpec will still be available after the eviction, i.e. even in the absence of the evicted pod. So for example you can prevent all voluntary evictions by specifying "100%". + x-kubernetes-int-or-string: true + type: object replicas: format: int32 type: integer @@ -823,8 +870,12 @@ spec: description: 'Container name: required for volumes, optional for env vars' type: string divisor: + anyOf: + - type: integer + - type: string description: Specifies the output format of the exposed resources, defaults to "1" - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true resource: description: 'Required: resource to select' type: string @@ -1115,8 +1166,13 @@ spec: type: string required: - containerPort + - protocol type: object type: array + x-kubernetes-list-map-keys: + - containerPort + - protocol + x-kubernetes-list-type: map readinessProbe: description: 'Periodic probe of container service readiness. Container will be removed from service endpoints if the probe fails. Cannot be updated. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' properties: @@ -1201,12 +1257,20 @@ spec: properties: limits: additionalProperties: - type: string + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true description: 'Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/' type: object requests: additionalProperties: - type: string + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true description: 'Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/' type: object type: object @@ -1528,8 +1592,12 @@ spec: description: 'Container name: required for volumes, optional for env vars' type: string divisor: + anyOf: + - type: integer + - type: string description: Specifies the output format of the exposed resources, defaults to "1" - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true resource: description: 'Required: resource to select' type: string @@ -1930,12 +1998,20 @@ spec: properties: limits: additionalProperties: - type: string + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true description: 'Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/' type: object requests: additionalProperties: - type: string + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true description: 'Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/' type: object type: object @@ -2265,8 +2341,12 @@ spec: description: 'Container name: required for volumes, optional for env vars' type: string divisor: + anyOf: + - type: integer + - type: string description: Specifies the output format of the exposed resources, defaults to "1" - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true resource: description: 'Required: resource to select' type: string @@ -2557,8 +2637,13 @@ spec: type: string required: - containerPort + - protocol type: object type: array + x-kubernetes-list-map-keys: + - containerPort + - protocol + x-kubernetes-list-type: map readinessProbe: description: 'Periodic probe of container service readiness. Container will be removed from service endpoints if the probe fails. Cannot be updated. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' properties: @@ -2643,12 +2728,20 @@ spec: properties: limits: additionalProperties: - type: string + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true description: 'Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/' type: object requests: additionalProperties: - type: string + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true description: 'Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/' type: object type: object @@ -2885,7 +2978,11 @@ spec: type: object overhead: additionalProperties: - type: string + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true description: 'Overhead represents the resource overhead associated with running a pod for a given RuntimeClass. This field will be autopopulated at admission time by the RuntimeClass admission controller. If the RuntimeClass admission controller is enabled, overhead must not be set in Pod create requests. The RuntimeClass admission controller will reject Pod create requests which have the overhead already set. If RuntimeClass is configured and selected in the PodSpec, Overhead will be set to the value defined in the corresponding RuntimeClass, otherwise it will remain unset and treated as zero. More info: https://git.k8s.io/enhancements/keps/sig-node/20190226-pod-overhead.md This field is alpha-level as of Kubernetes v1.16, and is only honored by servers that enable the PodOverhead feature.' type: object preemptionPolicy: @@ -3082,6 +3179,10 @@ spec: - whenUnsatisfiable type: object type: array + x-kubernetes-list-map-keys: + - topologyKey + - whenUnsatisfiable + x-kubernetes-list-type: map volumes: description: 'List of volumes that can be mounted by containers belonging to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes' items: @@ -3097,12 +3198,20 @@ spec: properties: limits: additionalProperties: - type: string + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true description: 'Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/' type: object requests: additionalProperties: - type: string + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true description: 'Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/' type: object type: object @@ -3173,8 +3282,12 @@ spec: description: 'Container name: required for volumes, optional for env vars' type: string divisor: + anyOf: + - type: integer + - type: string description: Specifies the output format of the exposed resources, defaults to "1" - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true resource: description: 'Required: resource to select' type: string @@ -3465,8 +3578,13 @@ spec: type: string required: - containerPort + - protocol type: object type: array + x-kubernetes-list-map-keys: + - containerPort + - protocol + x-kubernetes-list-type: map readinessProbe: description: 'Periodic probe of container service readiness. Container will be removed from service endpoints if the probe fails. Cannot be updated. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes' properties: @@ -3551,12 +3669,20 @@ spec: properties: limits: additionalProperties: - type: string + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true description: 'Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/' type: object requests: additionalProperties: - type: string + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true description: 'Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/' type: object type: object @@ -4130,8 +4256,12 @@ spec: description: 'Container name: required for volumes, optional for env vars' type: string divisor: + anyOf: + - type: integer + - type: string description: Specifies the output format of the exposed resources, defaults to "1" - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true resource: description: 'Required: resource to select' type: string @@ -4166,12 +4296,20 @@ spec: properties: limits: additionalProperties: - type: string + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true description: 'Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/' type: object requests: additionalProperties: - type: string + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true description: 'Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. More info: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/' type: object type: object diff --git a/testing/scripts/test_notebooks.py b/testing/scripts/test_notebooks.py index d7ae31deda..11c242626c 100644 --- a/testing/scripts/test_notebooks.py +++ b/testing/scripts/test_notebooks.py @@ -188,3 +188,8 @@ def test_upgrade(self): except: run("make install_seldon", shell=True, check=False) raise + + def test_disruption_budgets(self): + create_and_run_script( + "../../examples/models/disruption_budgets", "pdbs_example" + )