diff --git a/.gitignore b/.gitignore index 058da7654e..a0f71eb1e4 100644 --- a/.gitignore +++ b/.gitignore @@ -18,6 +18,7 @@ release/ __pycache__/ /charts/flyte/charts /charts/flyte/Chart.lock +/charts/flyte-core/Chart.lock **/*.bak- bin/- bin/ diff --git a/charts/flyte-core/.helmignore b/charts/flyte-core/.helmignore new file mode 100644 index 0000000000..0e8a0eb36f --- /dev/null +++ b/charts/flyte-core/.helmignore @@ -0,0 +1,23 @@ +# Patterns to ignore when building packages. +# This supports shell glob matching, relative path matching, and +# negation (prefixed with !). Only one pattern per line. +.DS_Store +# Common VCS dirs +.git/ +.gitignore +.bzr/ +.bzrignore +.hg/ +.hgignore +.svn/ +# Common backup files +*.swp +*.bak +*.tmp +*.orig +*~ +# Various IDEs +.project +.idea/ +*.tmproj +.vscode/ diff --git a/charts/flyte-core/Chart.yaml b/charts/flyte-core/Chart.yaml new file mode 100644 index 0000000000..17044e89b1 --- /dev/null +++ b/charts/flyte-core/Chart.yaml @@ -0,0 +1,5 @@ +apiVersion: v2 +name: flyte-core +description: A Helm chart for Flyte +type: application +version: v0.1.10 # VERSION diff --git a/charts/flyte-core/README.md b/charts/flyte-core/README.md new file mode 100644 index 0000000000..ba7ba370eb --- /dev/null +++ b/charts/flyte-core/README.md @@ -0,0 +1,168 @@ +# flyte-core + +![Version: v0.1.10](https://img.shields.io/badge/Version-v0.1.10-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) + +A Helm chart for Flyte + +### Flyte INSTALLATION: +- [Install helm 3](https://helm.sh/docs/intro/install/) +- Fetch chart dependencies `` +- Install Flyte: + +```bash +helm repo add flyte https://flyteorg.github.io/flyte +helm install -n flyte -f values-eks.yaml --create-namespace flyte flyte/flyte-core +``` + +Customize your installation by changing settings in `values-eks.yaml`. +You can use the helm diff plugin to review any value changes you've made to your values: + +```bash +helm plugin install https://github.com/databus23/helm-diff +helm diff upgrade -f values-eks.yaml flyte flyte/flyte-core +``` + +Then apply your changes: +```bash +helm upgrade -f values-eks.yaml flyte flyte/flyte-core +``` + +Install ingress controller (By default Flyte helm chart have contour ingress resource) +```bash +helm repo add bitnami https://charts.bitnami.com/bitnami +helm install gateway bitnami/contour -n flyte +``` + +#### Alternative: Generate raw kubernetes yaml with helm template +- `helm template --name-template=flyte-eks . -n flyte -f values-eks.yaml > flyte_generated_eks.yaml` +- Deploy the manifest `kubectl apply -f flyte_generated_eks.yaml` + +- When all pods are running - run end2end tests: `kubectl apply -f ../end2end/tests/endtoend.yaml` +- Get flyte host `minikube service contour -n heptio-contour --url`. And then visit `http:///console` + +### CONFIGURATION NOTES: +- The docker images, their tags and other default parameters are configured in `values.yaml` file. +- Each Flyte installation type should have separate `values-*.yaml` file: for sandbox, EKS and etc. The configuration in `values.yaml` and the choosen config `values-*.yaml` are merged when generating the deployment manifest. +- The configuration in `values-sandbox.yaml` is ready for installation in minikube. But `values-eks.yaml` should be edited before installation: s3 bucket, RDS hosts, iam roles, secrets and etc need to be modified. + +## Values + +| Key | Type | Default | Description | +|-----|------|---------|-------------| +| cluster_resource_manager | object | `{"config":{"cluster_resources":{"customData":[{"production":[{"projectQuotaCpu":{"value":"5"}},{"projectQuotaMemory":{"value":"4000Mi"}},{"defaultIamRole":{"value":""}}]},{"staging":[{"projectQuotaCpu":{"value":"2"}},{"projectQuotaMemory":{"value":"3000Mi"}},{"defaultIamRole":{"value":""}}]},{"development":[{"projectQuotaCpu":{"value":"4"}},{"projectQuotaMemory":{"value":"3000Mi"}},{"defaultIamRole":{"value":""}}]}],"refresh":"5m","refreshInterval":"5m","templatePath":"/etc/flyte/clusterresource/templates"}},"enabled":true,"templates":[{"key":"aa_namespace","value":"apiVersion: v1\nkind: Namespace\nmetadata:\n name: {{ namespace }}\nspec:\n finalizers:\n - kubernetes\n"},{"key":"ab_project_resource_quota","value":"apiVersion: v1\nkind: ResourceQuota\nmetadata:\n name: project-quota\n namespace: {{ namespace }}\nspec:\n hard:\n limits.cpu: {{ projectQuotaCpu }}\n limits.memory: {{ projectQuotaMemory }}\n"}]}` | Configuration for the Cluster resource manager component. This is an optional component, that enables automatic cluster configuration. This is useful to set default quotas, manage namespaces etc that map to a project/domain | +| cluster_resource_manager.config | object | `{"cluster_resources":{"customData":[{"production":[{"projectQuotaCpu":{"value":"5"}},{"projectQuotaMemory":{"value":"4000Mi"}},{"defaultIamRole":{"value":""}}]},{"staging":[{"projectQuotaCpu":{"value":"2"}},{"projectQuotaMemory":{"value":"3000Mi"}},{"defaultIamRole":{"value":""}}]},{"development":[{"projectQuotaCpu":{"value":"4"}},{"projectQuotaMemory":{"value":"3000Mi"}},{"defaultIamRole":{"value":""}}]}],"refresh":"5m","refreshInterval":"5m","templatePath":"/etc/flyte/clusterresource/templates"}}` | Configmap for ClusterResource parameters | +| cluster_resource_manager.config.cluster_resources | object | `{"customData":[{"production":[{"projectQuotaCpu":{"value":"5"}},{"projectQuotaMemory":{"value":"4000Mi"}},{"defaultIamRole":{"value":""}}]},{"staging":[{"projectQuotaCpu":{"value":"2"}},{"projectQuotaMemory":{"value":"3000Mi"}},{"defaultIamRole":{"value":""}}]},{"development":[{"projectQuotaCpu":{"value":"4"}},{"projectQuotaMemory":{"value":"3000Mi"}},{"defaultIamRole":{"value":""}}]}],"refresh":"5m","refreshInterval":"5m","templatePath":"/etc/flyte/clusterresource/templates"}` | ClusterResource parameters Refer to the [structure](https://pkg.go.dev/github.com/lyft/flyteadmin@v0.3.37/pkg/runtime/interfaces#ClusterResourceConfig) to customize. | +| cluster_resource_manager.enabled | bool | `true` | Enables the Cluster resource manager component | +| cluster_resource_manager.templates | list | `[{"key":"aa_namespace","value":"apiVersion: v1\nkind: Namespace\nmetadata:\n name: {{ namespace }}\nspec:\n finalizers:\n - kubernetes\n"},{"key":"ab_project_resource_quota","value":"apiVersion: v1\nkind: ResourceQuota\nmetadata:\n name: project-quota\n namespace: {{ namespace }}\nspec:\n hard:\n limits.cpu: {{ projectQuotaCpu }}\n limits.memory: {{ projectQuotaMemory }}\n"}]` | Resource templates that should be applied | +| cluster_resource_manager.templates[0] | object | `{"key":"aa_namespace","value":"apiVersion: v1\nkind: Namespace\nmetadata:\n name: {{ namespace }}\nspec:\n finalizers:\n - kubernetes\n"}` | Template for namespaces resources | +| common.databaseSecret.name | string | `"db-pass"` | Specify name of K8s Secret which contains Database password. Leave it empty if you don't need this Secret | +| common.databaseSecret.secretManifest | object | `{}` | Specify your Secret (with sensitive data) or pseudo-manifest (without sensitive data). See https://github.com/godaddy/kubernetes-external-secrets | +| common.flyteNamespaceTemplate.enabled | bool | `false` | - Enable or disable creating Flyte namespace in template. Enable when using helm as template-engine only. Disable when using `helm install ...`. | +| common.ingress.albSSLRedirect | bool | `false` | - albSSLRedirect adds a special route for ssl redirect. Only useful in combination with the AWS LoadBalancer Controller. | +| common.ingress.annotations | object | `{}` | - Ingress annotations applied to both HTTP and GRPC ingresses. | +| common.ingress.enabled | bool | `true` | - Enable or disable creating Ingress for Flyte. Relevant to disable when using e.g. Istio as ingress controller. | +| common.ingress.separateGrpcIngress | bool | `false` | - separateGrpcIngress puts GRPC routes into a separate ingress if true. Required for certain ingress controllers like nginx. | +| common.ingress.separateGrpcIngressAnnotations | object | `{"nginx.ingress.kubernetes.io/backend-protocol":"GRPC"}` | - Extra Ingress annotations applied only to the GRPC ingress. Only makes sense if `separateGrpcIngress` is enabled. | +| common.ingress.tls | object | `{"enabled":false}` | - Ingress hostname host: | +| common.ingress.webpackHMR | bool | `false` | - Enable or disable HMR route to flyteconsole. This is useful only for frontend development. | +| configmap | object | `{"admin":{"admin":{"clientId":"flytepropeller","clientSecretLocation":"/etc/secrets/client_secret","endpoint":"flyteadmin:81","insecure":true},"event":{"capacity":1000,"rate":500,"type":"admin"}},"adminServer":{"auth":{"appAuth":{"thirdPartyConfig":{"flyteClient":{"clientId":"flytectl","redirectUri":"http://localhost:53593/callback","scopes":["offline","all"]}}},"authorizedUris":["https://localhost:30081","http://flyteadmin:80","http://flyteadmin.flyte.svc.cluster.local:80"],"userAuth":{"openId":{"baseUrl":"https://accounts.google.com","clientId":"657465813211-6eog7ek7li5k7i7fvgv2921075063hpe.apps.googleusercontent.com","scopes":["profile","openid"]}}},"flyteadmin":{"eventVersion":1,"metadataStoragePrefix":["metadata","admin"],"metricsScope":"flyte:","profilerPort":10254,"roleNameKey":"iam.amazonaws.com/role","testing":{"host":"http://flyteadmin"}},"server":{"grpcPort":8089,"httpPort":8088,"security":{"allowCors":true,"allowedHeaders":["Content-Type","flyte-authorization"],"allowedOrigins":["*"],"secure":false,"useAuth":false}}},"catalog":{"catalog-cache":{"endpoint":"datacatalog:89","insecure":true,"type":"datacatalog"}},"console":{"BASE_URL":"/console","CONFIG_DIR":"/etc/flyte/config","DISABLE_AUTH":"1"},"copilot":{"plugins":{"k8s":{"co-pilot":{"image":"cr.flyte.org/lyft/flyteplugins/flytecopilot:dc4bdbd61cac88a39a5ff43e40f026bdbc2c78a2","name":"flyte-copilot-","start-timeout":"30s"}}}},"core":{"propeller":{"downstream-eval-duration":"30s","enable-admin-launcher":true,"leader-election":{"enabled":true,"lease-duration":"15s","lock-config-map":{"name":"propeller-leader","namespace":"flyte"},"renew-deadline":"10s","retry-period":"2s"},"limit-namespace":"all","max-workflow-retries":30,"metadata-prefix":"metadata/propeller","metrics-prefix":"flyte","prof-port":10254,"queue":{"batch-size":-1,"batching-interval":"2s","queue":{"base-delay":"5s","capacity":1000,"max-delay":"120s","rate":100,"type":"maxof"},"sub-queue":{"capacity":100,"rate":10,"type":"bucket"},"type":"batch"},"rawoutput-prefix":"s3://my-s3-bucket/","workers":4,"workflow-reeval-duration":"30s"},"webhook":{"certDir":"/etc/webhook/certs","serviceName":"flyte-pod-webhook"}},"datacatalogServer":{"application":{"grpcPort":8089,"grpcServerReflection":true,"httpPort":8080},"datacatalog":{"metrics-scope":"datacatalog","profiler-port":10254,"storage-prefix":"metadata/datacatalog"}},"domain":{"domains":[{"id":"development","name":"development"},{"id":"staging","name":"staging"},{"id":"production","name":"production"}]},"enabled_plugins":{"tasks":{"task-plugins":{"default-for-task-types":{"container":"container","container_array":"k8s-array","sidecar":"sidecar"},"enabled-plugins":["container","sidecar","k8s-array"]}}},"k8s":{"plugins":{"k8s":{"default-cpus":"100m","default-env-vars":[],"default-memory":"100Mi"}}},"logger":{"logger":{"level":4,"show-source":true}},"remoteData":{"remoteData":{"region":"us-east-1","scheme":"local","signedUrls":{"durationMinutes":3}}},"resource_manager":{"propeller":{"resourcemanager":{"redis":{"hostKey":"mypassword","hostPath":"redis-resource-manager:6379"},"resourceMaxQuota":10000,"type":null}}},"task_logs":{"plugins":{"logs":{"cloudwatch-enabled":false,"kubernetes-enabled":false}}},"task_resource_defaults":{"task_resources":{"defaults":{"cpu":"100m","memory":"100Mi","storage":"5Mi"},"limits":{"cpu":2,"gpu":1,"memory":"1Gi","storage":"20Mi"}}}}` | ------------------------------------------------------------------ Specializing your deployment using configuration ------------------------------------------------------------------- CONFIGMAPS SETTINGS | +| configmap.admin | object | `{"admin":{"clientId":"flytepropeller","clientSecretLocation":"/etc/secrets/client_secret","endpoint":"flyteadmin:81","insecure":true},"event":{"capacity":1000,"rate":500,"type":"admin"}}` | Admin Client configuration [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/subworkflow/launchplan#AdminConfig) | +| configmap.adminServer | object | `{"auth":{"appAuth":{"thirdPartyConfig":{"flyteClient":{"clientId":"flytectl","redirectUri":"http://localhost:53593/callback","scopes":["offline","all"]}}},"authorizedUris":["https://localhost:30081","http://flyteadmin:80","http://flyteadmin.flyte.svc.cluster.local:80"],"userAuth":{"openId":{"baseUrl":"https://accounts.google.com","clientId":"657465813211-6eog7ek7li5k7i7fvgv2921075063hpe.apps.googleusercontent.com","scopes":["profile","openid"]}}},"flyteadmin":{"eventVersion":1,"metadataStoragePrefix":["metadata","admin"],"metricsScope":"flyte:","profilerPort":10254,"roleNameKey":"iam.amazonaws.com/role","testing":{"host":"http://flyteadmin"}},"server":{"grpcPort":8089,"httpPort":8088,"security":{"allowCors":true,"allowedHeaders":["Content-Type","flyte-authorization"],"allowedOrigins":["*"],"secure":false,"useAuth":false}}}` | FlyteAdmin server configuration | +| configmap.adminServer.auth | object | `{"appAuth":{"thirdPartyConfig":{"flyteClient":{"clientId":"flytectl","redirectUri":"http://localhost:53593/callback","scopes":["offline","all"]}}},"authorizedUris":["https://localhost:30081","http://flyteadmin:80","http://flyteadmin.flyte.svc.cluster.local:80"],"userAuth":{"openId":{"baseUrl":"https://accounts.google.com","clientId":"657465813211-6eog7ek7li5k7i7fvgv2921075063hpe.apps.googleusercontent.com","scopes":["profile","openid"]}}}` | Authentication configuration | +| configmap.adminServer.server.security.secure | bool | `false` | Controls whether to serve requests over SSL/TLS. | +| configmap.adminServer.server.security.useAuth | bool | `false` | Controls whether to enforce authentication. Follow the guide in https://docs.flyte.org/ on how to setup authentication. | +| configmap.catalog | object | `{"catalog-cache":{"endpoint":"datacatalog:89","insecure":true,"type":"datacatalog"}}` | Catalog Client configuration [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/catalog#Config) Additional advanced Catalog configuration [here](https://pkg.go.dev/github.com/lyft/flyteplugins/go/tasks/pluginmachinery/catalog#Config) | +| configmap.console | object | `{"BASE_URL":"/console","CONFIG_DIR":"/etc/flyte/config","DISABLE_AUTH":"1"}` | Configuration for Flyte console UI | +| configmap.copilot | object | `{"plugins":{"k8s":{"co-pilot":{"image":"cr.flyte.org/lyft/flyteplugins/flytecopilot:dc4bdbd61cac88a39a5ff43e40f026bdbc2c78a2","name":"flyte-copilot-","start-timeout":"30s"}}}}` | Copilot configuration | +| configmap.copilot.plugins.k8s.co-pilot | object | `{"image":"cr.flyte.org/lyft/flyteplugins/flytecopilot:dc4bdbd61cac88a39a5ff43e40f026bdbc2c78a2","name":"flyte-copilot-","start-timeout":"30s"}` | Structure documented [here](https://pkg.go.dev/github.com/lyft/flyteplugins@v0.5.28/go/tasks/pluginmachinery/flytek8s/config#FlyteCoPilotConfig) | +| configmap.core | object | `{"propeller":{"downstream-eval-duration":"30s","enable-admin-launcher":true,"leader-election":{"enabled":true,"lease-duration":"15s","lock-config-map":{"name":"propeller-leader","namespace":"flyte"},"renew-deadline":"10s","retry-period":"2s"},"limit-namespace":"all","max-workflow-retries":30,"metadata-prefix":"metadata/propeller","metrics-prefix":"flyte","prof-port":10254,"queue":{"batch-size":-1,"batching-interval":"2s","queue":{"base-delay":"5s","capacity":1000,"max-delay":"120s","rate":100,"type":"maxof"},"sub-queue":{"capacity":100,"rate":10,"type":"bucket"},"type":"batch"},"rawoutput-prefix":"s3://my-s3-bucket/","workers":4,"workflow-reeval-duration":"30s"},"webhook":{"certDir":"/etc/webhook/certs","serviceName":"flyte-pod-webhook"}}` | Core propeller configuration | +| configmap.core.propeller | object | `{"downstream-eval-duration":"30s","enable-admin-launcher":true,"leader-election":{"enabled":true,"lease-duration":"15s","lock-config-map":{"name":"propeller-leader","namespace":"flyte"},"renew-deadline":"10s","retry-period":"2s"},"limit-namespace":"all","max-workflow-retries":30,"metadata-prefix":"metadata/propeller","metrics-prefix":"flyte","prof-port":10254,"queue":{"batch-size":-1,"batching-interval":"2s","queue":{"base-delay":"5s","capacity":1000,"max-delay":"120s","rate":100,"type":"maxof"},"sub-queue":{"capacity":100,"rate":10,"type":"bucket"},"type":"batch"},"rawoutput-prefix":"s3://my-s3-bucket/","workers":4,"workflow-reeval-duration":"30s"}` | follows the structure specified [here](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/config). | +| configmap.datacatalogServer | object | `{"application":{"grpcPort":8089,"grpcServerReflection":true,"httpPort":8080},"datacatalog":{"metrics-scope":"datacatalog","profiler-port":10254,"storage-prefix":"metadata/datacatalog"}}` | Datacatalog server config | +| configmap.domain | object | `{"domains":[{"id":"development","name":"development"},{"id":"staging","name":"staging"},{"id":"production","name":"production"}]}` | Domains configuration for Flyte projects. This enables the specified number of domains across all projects in Flyte. | +| configmap.enabled_plugins.tasks | object | `{"task-plugins":{"default-for-task-types":{"container":"container","container_array":"k8s-array","sidecar":"sidecar"},"enabled-plugins":["container","sidecar","k8s-array"]}}` | Tasks specific configuration [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#GetConfig) | +| configmap.enabled_plugins.tasks.task-plugins | object | `{"default-for-task-types":{"container":"container","container_array":"k8s-array","sidecar":"sidecar"},"enabled-plugins":["container","sidecar","k8s-array"]}` | Plugins configuration, [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#TaskPluginConfig) | +| configmap.enabled_plugins.tasks.task-plugins.enabled-plugins | list | `["container","sidecar","k8s-array"]` | [Enabled Plugins](https://pkg.go.dev/github.com/lyft/flyteplugins/go/tasks/config#Config). Enable sagemaker*, athena if you install the backend plugins | +| configmap.k8s | object | `{"plugins":{"k8s":{"default-cpus":"100m","default-env-vars":[],"default-memory":"100Mi"}}}` | Kubernetes specific Flyte configuration | +| configmap.k8s.plugins.k8s | object | `{"default-cpus":"100m","default-env-vars":[],"default-memory":"100Mi"}` | Configuration section for all K8s specific plugins [Configuration structure](https://pkg.go.dev/github.com/lyft/flyteplugins/go/tasks/pluginmachinery/flytek8s/config) | +| configmap.logger | object | `{"logger":{"level":4,"show-source":true}}` | Logger configuration | +| configmap.resource_manager | object | `{"propeller":{"resourcemanager":{"redis":{"hostKey":"mypassword","hostPath":"redis-resource-manager:6379"},"resourceMaxQuota":10000,"type":null}}}` | Resource manager configuration | +| configmap.resource_manager.propeller | object | `{"resourcemanager":{"redis":{"hostKey":"mypassword","hostPath":"redis-resource-manager:6379"},"resourceMaxQuota":10000,"type":null}}` | resource manager configuration | +| configmap.task_logs | object | `{"plugins":{"logs":{"cloudwatch-enabled":false,"kubernetes-enabled":false}}}` | Section that configures how the Task logs are displayed on the UI. This has to be changed based on your actual logging provider. Refer to [structure](https://pkg.go.dev/github.com/lyft/flyteplugins/go/tasks/logs#LogConfig) to understand how to configure various logging engines | +| configmap.task_logs.plugins.logs.cloudwatch-enabled | bool | `false` | One option is to enable cloudwatch logging for EKS, update the region and log group accordingly | +| configmap.task_resource_defaults | object | `{"task_resources":{"defaults":{"cpu":"100m","memory":"100Mi","storage":"5Mi"},"limits":{"cpu":2,"gpu":1,"memory":"1Gi","storage":"20Mi"}}}` | Task default resources configuration Refer to the full [structure](https://pkg.go.dev/github.com/lyft/flyteadmin@v0.3.37/pkg/runtime/interfaces#TaskResourceConfiguration). | +| configmap.task_resource_defaults.task_resources | object | `{"defaults":{"cpu":"100m","memory":"100Mi","storage":"5Mi"},"limits":{"cpu":2,"gpu":1,"memory":"1Gi","storage":"20Mi"}}` | Task default resources parameters | +| datacatalog | object | `{"affinity":{},"configPath":"/etc/datacatalog/config/*.yaml","image":{"pullPolicy":"IfNotPresent","repository":"cr.flyte.org/flyteorg/datacatalog","tag":"v0.3.6"},"nodeSelector":{},"podAnnotations":{},"replicaCount":1,"resources":{"limits":{"cpu":"500m","ephemeral-storage":"100Mi","memory":"500Mi"},"requests":{"cpu":"10m","ephemeral-storage":"50Mi","memory":"50Mi"}},"service":{"annotations":{"projectcontour.io/upstream-protocol.h2c":"grpc"},"type":"NodePort"},"serviceAccount":{"annotations":{},"create":true,"imagePullSecrets":{}},"tolerations":[]}` | DATACATALOG SETTINGS | +| datacatalog.affinity | object | `{}` | affinity for Datacatalog deployment | +| datacatalog.configPath | string | `"/etc/datacatalog/config/*.yaml"` | Default regex string for searching configuration files | +| datacatalog.image.repository | string | `"cr.flyte.org/flyteorg/datacatalog"` | Docker image for Datacatalog deployment | +| datacatalog.nodeSelector | object | `{}` | nodeSelector for Datacatalog deployment | +| datacatalog.podAnnotations | object | `{}` | Annotations for Datacatalog pods | +| datacatalog.replicaCount | int | `1` | Replicas count for Datacatalog deployment | +| datacatalog.resources | object | `{"limits":{"cpu":"500m","ephemeral-storage":"100Mi","memory":"500Mi"},"requests":{"cpu":"10m","ephemeral-storage":"50Mi","memory":"50Mi"}}` | Default resources requests and limits for Datacatalog deployment | +| datacatalog.service | object | `{"annotations":{"projectcontour.io/upstream-protocol.h2c":"grpc"},"type":"NodePort"}` | Service settings for Datacatalog | +| datacatalog.serviceAccount | object | `{"annotations":{},"create":true,"imagePullSecrets":{}}` | Configuration for service accounts for Datacatalog | +| datacatalog.serviceAccount.annotations | object | `{}` | Annotations for ServiceAccount attached to Datacatalog pods | +| datacatalog.serviceAccount.create | bool | `true` | Should a service account be created for Datacatalog | +| datacatalog.serviceAccount.imagePullSecrets | object | `{}` | ImapgePullSecrets to automatically assign to the service account | +| datacatalog.tolerations | list | `[]` | tolerations for Datacatalog deployment | +| db.database.host | string | `"postgres"` | username: postgres | +| flyteadmin.affinity | object | `{}` | affinity for Flyteadmin deployment | +| flyteadmin.configPath | string | `"/etc/flyte/config/*.yaml"` | Default regex string for searching configuration files | +| flyteadmin.image.pullPolicy | string | `"IfNotPresent"` | | +| flyteadmin.image.repository | string | `"cr.flyte.org/flyteorg/flyteadmin"` | Docker image for Flyteadmin deployment | +| flyteadmin.image.tag | string | `"v0.6.16"` | | +| flyteadmin.initialProjects | list | `["flytesnacks","flytetester","flyteexamples"]` | Initial projects to create | +| flyteadmin.nodeSelector | object | `{}` | nodeSelector for Flyteadmin deployment | +| flyteadmin.podAnnotations | object | `{}` | Annotations for Flyteadmin pods | +| flyteadmin.replicaCount | int | `1` | Replicas count for Flyteadmin deployment | +| flyteadmin.resources | object | `{"limits":{"cpu":"250m","ephemeral-storage":"100Mi","memory":"500Mi"},"requests":{"cpu":"10m","ephemeral-storage":"50Mi","memory":"50Mi"}}` | Default resources requests and limits for Flyteadmin deployment | +| flyteadmin.secrets | object | `{}` | | +| flyteadmin.service | object | `{"annotations":{"projectcontour.io/upstream-protocol.h2c":"grpc"},"loadBalancerSourceRanges":[],"type":"ClusterIP"}` | Service settings for Flyteadmin | +| flyteadmin.serviceAccount | object | `{"annotations":{},"create":true,"imagePullSecrets":{}}` | Configuration for service accounts for FlyteAdmin | +| flyteadmin.serviceAccount.annotations | object | `{}` | Annotations for ServiceAccount attached to Flyteadmin pods | +| flyteadmin.serviceAccount.create | bool | `true` | Should a service account be created for flyteadmin | +| flyteadmin.serviceAccount.imagePullSecrets | object | `{}` | ImapgePullSecrets to automatically assign to the service account | +| flyteadmin.tolerations | list | `[]` | tolerations for Flyteadmin deployment | +| flyteconsole | object | `{"affinity":{},"image":{"pullPolicy":"IfNotPresent","repository":"cr.flyte.org/flyteorg/flyteconsole","tag":"v0.20.1"},"nodeSelector":{},"podAnnotations":{},"replicaCount":1,"resources":{"limits":{"cpu":"500m","memory":"250Mi"},"requests":{"cpu":"10m","memory":"50Mi"}},"service":{"annotations":{},"type":"ClusterIP"},"tolerations":[]}` | FLYTECONSOLE SETTINGS | +| flyteconsole.affinity | object | `{}` | affinity for Flyteconsole deployment | +| flyteconsole.image.repository | string | `"cr.flyte.org/flyteorg/flyteconsole"` | Docker image for Flyteconsole deployment | +| flyteconsole.nodeSelector | object | `{}` | nodeSelector for Flyteconsole deployment | +| flyteconsole.podAnnotations | object | `{}` | Annotations for Flyteconsole pods | +| flyteconsole.replicaCount | int | `1` | Replicas count for Flyteconsole deployment | +| flyteconsole.resources | object | `{"limits":{"cpu":"500m","memory":"250Mi"},"requests":{"cpu":"10m","memory":"50Mi"}}` | Default resources requests and limits for Flyteconsole deployment | +| flyteconsole.service | object | `{"annotations":{},"type":"ClusterIP"}` | Service settings for Flyteconsole | +| flyteconsole.tolerations | list | `[]` | tolerations for Flyteconsole deployment | +| flytepropeller | object | `{"affinity":{},"cacheSizeMbs":0,"configPath":"/etc/flyte/config/*.yaml","image":{"pullPolicy":"IfNotPresent","repository":"cr.flyte.org/flyteorg/flytepropeller","tag":"v0.13.4"},"nodeSelector":{},"podAnnotations":{},"replicaCount":1,"resources":{"limits":{"cpu":"200m","ephemeral-storage":"100Mi","memory":"200Mi"},"requests":{"cpu":"10m","ephemeral-storage":"50Mi","memory":"50Mi"}},"serviceAccount":{"annotations":{},"create":true,"imagePullSecrets":{}},"tolerations":[]}` | FLYTEPROPELLER SETTINGS | +| flytepropeller.affinity | object | `{}` | affinity for Flytepropeller deployment | +| flytepropeller.configPath | string | `"/etc/flyte/config/*.yaml"` | Default regex string for searching configuration files | +| flytepropeller.image.repository | string | `"cr.flyte.org/flyteorg/flytepropeller"` | Docker image for Flytepropeller deployment | +| flytepropeller.nodeSelector | object | `{}` | nodeSelector for Flytepropeller deployment | +| flytepropeller.podAnnotations | object | `{}` | Annotations for Flytepropeller pods | +| flytepropeller.replicaCount | int | `1` | Replicas count for Flytepropeller deployment | +| flytepropeller.resources | object | `{"limits":{"cpu":"200m","ephemeral-storage":"100Mi","memory":"200Mi"},"requests":{"cpu":"10m","ephemeral-storage":"50Mi","memory":"50Mi"}}` | Default resources requests and limits for Flytepropeller deployment | +| flytepropeller.serviceAccount | object | `{"annotations":{},"create":true,"imagePullSecrets":{}}` | Configuration for service accounts for FlytePropeller | +| flytepropeller.serviceAccount.annotations | object | `{}` | Annotations for ServiceAccount attached to FlytePropeller pods | +| flytepropeller.serviceAccount.create | bool | `true` | Should a service account be created for FlytePropeller | +| flytepropeller.serviceAccount.imagePullSecrets | object | `{}` | ImapgePullSecrets to automatically assign to the service account | +| flytepropeller.tolerations | list | `[]` | tolerations for Flytepropeller deployment | +| sagemaker | object | `{"enabled":false,"plugin_config":{"plugins":{"sagemaker":{"region":"","roleArn":""}}}}` | --------------- -- Training on AWS Sagemaker using AWS Sagemaker operator. To actually install the operator, please follow instructions [here](https://github.com/aws/amazon-sagemaker-operator-for-k8s/tree/master/hack/charts/installer/rolebased) Use the config section here to just enable sagemaker plugin in Flyte, after you have installed the operator using the information | +| sparkoperator | object | `{"enabled":false,"plugin_config":{"plugins":{"spark":{"spark-config-default":[{"spark.hadoop.fs.s3a.aws.credentials.provider":"com.amazonaws.auth.DefaultAWSCredentialsProviderChain"},{"spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version":"2"},{"spark.kubernetes.allocation.batch.size":"50"},{"spark.hadoop.fs.s3a.acl.default":"BucketOwnerFullControl"},{"spark.hadoop.fs.s3n.impl":"org.apache.hadoop.fs.s3a.S3AFileSystem"},{"spark.hadoop.fs.AbstractFileSystem.s3n.impl":"org.apache.hadoop.fs.s3a.S3A"},{"spark.hadoop.fs.s3.impl":"org.apache.hadoop.fs.s3a.S3AFileSystem"},{"spark.hadoop.fs.AbstractFileSystem.s3.impl":"org.apache.hadoop.fs.s3a.S3A"},{"spark.hadoop.fs.s3a.impl":"org.apache.hadoop.fs.s3a.S3AFileSystem"},{"spark.hadoop.fs.AbstractFileSystem.s3a.impl":"org.apache.hadoop.fs.s3a.S3A"},{"spark.hadoop.fs.s3a.multipart.threshold":"536870912"},{"spark.blacklist.enabled":"true"},{"spark.blacklist.timeout":"5m"},{"spark.task.maxfailures":"8"}]}}}}` | ------------------------------------------------------ Optional Plugins -------------------------------------------------------- -- Optional: Spark Plugin using the Spark Operator | +| sparkoperator.enabled | bool | `false` | - enable or disable Sparkoperator deployment installation | +| sparkoperator.plugin_config | object | `{"plugins":{"spark":{"spark-config-default":[{"spark.hadoop.fs.s3a.aws.credentials.provider":"com.amazonaws.auth.DefaultAWSCredentialsProviderChain"},{"spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version":"2"},{"spark.kubernetes.allocation.batch.size":"50"},{"spark.hadoop.fs.s3a.acl.default":"BucketOwnerFullControl"},{"spark.hadoop.fs.s3n.impl":"org.apache.hadoop.fs.s3a.S3AFileSystem"},{"spark.hadoop.fs.AbstractFileSystem.s3n.impl":"org.apache.hadoop.fs.s3a.S3A"},{"spark.hadoop.fs.s3.impl":"org.apache.hadoop.fs.s3a.S3AFileSystem"},{"spark.hadoop.fs.AbstractFileSystem.s3.impl":"org.apache.hadoop.fs.s3a.S3A"},{"spark.hadoop.fs.s3a.impl":"org.apache.hadoop.fs.s3a.S3AFileSystem"},{"spark.hadoop.fs.AbstractFileSystem.s3a.impl":"org.apache.hadoop.fs.s3a.S3A"},{"spark.hadoop.fs.s3a.multipart.threshold":"536870912"},{"spark.blacklist.enabled":"true"},{"spark.blacklist.timeout":"5m"},{"spark.task.maxfailures":"8"}]}}}` | Spark plugin configuration | +| sparkoperator.plugin_config.plugins.spark.spark-config-default | list | `[{"spark.hadoop.fs.s3a.aws.credentials.provider":"com.amazonaws.auth.DefaultAWSCredentialsProviderChain"},{"spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version":"2"},{"spark.kubernetes.allocation.batch.size":"50"},{"spark.hadoop.fs.s3a.acl.default":"BucketOwnerFullControl"},{"spark.hadoop.fs.s3n.impl":"org.apache.hadoop.fs.s3a.S3AFileSystem"},{"spark.hadoop.fs.AbstractFileSystem.s3n.impl":"org.apache.hadoop.fs.s3a.S3A"},{"spark.hadoop.fs.s3.impl":"org.apache.hadoop.fs.s3a.S3AFileSystem"},{"spark.hadoop.fs.AbstractFileSystem.s3.impl":"org.apache.hadoop.fs.s3a.S3A"},{"spark.hadoop.fs.s3a.impl":"org.apache.hadoop.fs.s3a.S3AFileSystem"},{"spark.hadoop.fs.AbstractFileSystem.s3a.impl":"org.apache.hadoop.fs.s3a.S3A"},{"spark.hadoop.fs.s3a.multipart.threshold":"536870912"},{"spark.blacklist.enabled":"true"},{"spark.blacklist.timeout":"5m"},{"spark.task.maxfailures":"8"}]` | Spark default configuration | +| sparkoperator.plugin_config.plugins.spark.spark-config-default[0] | object | `{"spark.hadoop.fs.s3a.aws.credentials.provider":"com.amazonaws.auth.DefaultAWSCredentialsProviderChain"}` | it can use the serviceAccount based IAM role or ec2 metadata based. This is more in line with how AWS works | +| storage | object | `{"bucketName":"my-s3-bucket","custom":{},"gcs":null,"s3":{"region":"us-east-1"},"type":"sandbox"}` | --------------------------------------------------- Core dependencies that should be configured for Flyte to work on any platform Specifically 2 - Storage (s3, gcs etc), Production RDBMS - Aurora, CloudSQL etc ------------------------------------------------------ STORAGE SETTINGS | +| storage.bucketName | string | `"my-s3-bucket"` | bucketName defines the storage bucket flyte will use. Required for all types except for sandbox. | +| storage.custom | object | `{}` | GCP project ID. Required for storage type gcs. projectId: -- Settings for storage type custom. See https://github:com/graymeta/stow for supported storage providers/settings. | +| storage.gcs | string | `nil` | settings for storage type gcs | +| storage.s3 | object | `{"region":"us-east-1"}` | settings for storage type s3 | +| storage.type | string | `"sandbox"` | Sets the storage type. Supported values are sandbox, s3, gcs and custom. | +| webhook | object | `{"enabled":true,"service":{"annotations":{"projectcontour.io/upstream-protocol.h2c":"grpc"},"type":"ClusterIP"},"serviceAccount":{"annotations":{},"create":true,"imagePullSecrets":{}}}` | WEBHOOK SETTINGS | +| webhook.enabled | bool | `true` | enable or disable secrets webhook | +| webhook.service | object | `{"annotations":{"projectcontour.io/upstream-protocol.h2c":"grpc"},"type":"ClusterIP"}` | Service settings for the webhook | +| webhook.serviceAccount | object | `{"annotations":{},"create":true,"imagePullSecrets":{}}` | Configuration for service accounts for the webhook | +| webhook.serviceAccount.annotations | object | `{}` | Annotations for ServiceAccount attached to the webhook | +| webhook.serviceAccount.create | bool | `true` | Should a service account be created for the webhook | +| webhook.serviceAccount.imagePullSecrets | object | `{}` | ImapgePullSecrets to automatically assign to the service account | +| workflow_notifications | object | `{"config":{},"enabled":false}` | **Optional Component** Workflow notifications module is an optional dependency. Flyte uses cloud native pub-sub systems to notify users of various events in their workflows | +| workflow_scheduler | object | `{"config":{},"enabled":false}` | -------------------------------------------------------------- Optional Modules Flyte built extensions that enable various additional features in Flyte. All these features are optional, but are critical to run certain features ------------------------------------------------------------------------ -- **Optional Component** Flyte uses a cloud hosted Cron scheduler to run workflows on a schedule. The following module is optional. Without, this module, you will not have scheduled launchplans / workflows. Docs: https://docs.flyte.org/en/latest/howto/enable_and_use_schedules.html#setting-up-scheduled-workflows | diff --git a/charts/flyte-core/README.md.gotmpl b/charts/flyte-core/README.md.gotmpl new file mode 100644 index 0000000000..4976a6a39c --- /dev/null +++ b/charts/flyte-core/README.md.gotmpl @@ -0,0 +1,58 @@ +{{ template "chart.header" . }} +{{ template "chart.deprecationWarning" . }} + +{{ template "chart.versionBadge" . }}{{ template "chart.typeBadge" . }}{{ template "chart.appVersionBadge" . }} + +{{ template "chart.description" . }} + +{{ template "chart.homepageLine" . }} + +{{ template "chart.maintainersSection" . }} + +{{ template "chart.sourcesSection" . }} + +{{ template "chart.requirementsSection" . }} + +### Flyte INSTALLATION: +- [Install helm 3](https://helm.sh/docs/intro/install/) +- Fetch chart dependencies `` +- Install Flyte: + +```bash +helm repo add flyte https://flyteorg.github.io/flyte +helm install -n flyte -f values-eks.yaml --create-namespace flyte flyte/flyte-core +``` + +Customize your installation by changing settings in `values-eks.yaml`. +You can use the helm diff plugin to review any value changes you've made to your values: + +```bash +helm plugin install https://github.com/databus23/helm-diff +helm diff upgrade -f values-eks.yaml flyte flyte/flyte-core +``` + +Then apply your changes: +```bash +helm upgrade -f values-eks.yaml flyte flyte/flyte-core +``` + +Install ingress controller (By default Flyte helm chart have contour ingress resource) +```bash +helm repo add bitnami https://charts.bitnami.com/bitnami +helm install gateway bitnami/contour -n flyte +``` + +#### Alternative: Generate raw kubernetes yaml with helm template +- `helm template --name-template=flyte-eks . -n flyte -f values-eks.yaml > flyte_generated_eks.yaml` +- Deploy the manifest `kubectl apply -f flyte_generated_eks.yaml` + + +- When all pods are running - run end2end tests: `kubectl apply -f ../end2end/tests/endtoend.yaml` +- Get flyte host `minikube service contour -n heptio-contour --url`. And then visit `http:///console` + +### CONFIGURATION NOTES: +- The docker images, their tags and other default parameters are configured in `values.yaml` file. +- Each Flyte installation type should have separate `values-*.yaml` file: for sandbox, EKS and etc. The configuration in `values.yaml` and the choosen config `values-*.yaml` are merged when generating the deployment manifest. +- The configuration in `values-sandbox.yaml` is ready for installation in minikube. But `values-eks.yaml` should be edited before installation: s3 bucket, RDS hosts, iam roles, secrets and etc need to be modified. + +{{ template "chart.valuesSection" . }} diff --git a/charts/flyte-core/templates/_helpers.tpl b/charts/flyte-core/templates/_helpers.tpl new file mode 100755 index 0000000000..3ba6f3182a --- /dev/null +++ b/charts/flyte-core/templates/_helpers.tpl @@ -0,0 +1,149 @@ +{{/* vim: set filetype=mustache: */}} + +{{- define "flyte.name" -}} +{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}} +{{- end -}} + +{{- define "flyte.chart" -}} +{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}} +{{- end -}} + +{{- define "flyte.namespace" -}} +{{- default .Release.Namespace .Values.forceNamespace | trunc 63 | trimSuffix "-" -}} +{{- end -}} + + +{{- define "flyteadmin.name" -}} +flyteadmin +{{- end -}} + +{{- define "flyteadmin.selectorLabels" -}} +app.kubernetes.io/name: {{ template "flyteadmin.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end -}} + +{{- define "flyteadmin.labels" -}} +{{ include "flyteadmin.selectorLabels" . }} +helm.sh/chart: {{ include "flyte.chart" . }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +{{- end -}} + + +{{- define "datacatalog.name" -}} +datacatalog +{{- end -}} + +{{- define "datacatalog.selectorLabels" -}} +app.kubernetes.io/name: {{ template "datacatalog.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end -}} + +{{- define "datacatalog.labels" -}} +{{ include "datacatalog.selectorLabels" . }} +helm.sh/chart: {{ include "flyte.chart" . }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +{{- end -}} + + +{{- define "flytepropeller.name" -}} +flytepropeller +{{- end -}} + +{{- define "flytepropeller.selectorLabels" -}} +app.kubernetes.io/name: {{ template "flytepropeller.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end -}} + +{{- define "flytepropeller.labels" -}} +{{ include "flytepropeller.selectorLabels" . }} +helm.sh/chart: {{ include "flyte.chart" . }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +{{- end -}} + +{{- define "flyte-pod-webhook.name" -}} +flyte-pod-webhook +{{- end -}} + + +{{- define "flyteconsole.name" -}} +flyteconsole +{{- end -}} + +{{- define "flyteconsole.selectorLabels" -}} +app.kubernetes.io/name: {{ template "flyteconsole.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end -}} + +{{- define "flyteconsole.labels" -}} +{{ include "flyteconsole.selectorLabels" . }} +helm.sh/chart: {{ include "flyte.chart" . }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +{{- end -}} + +# Optional blocks for secret mount + +{{- define "databaseSecret.volume" -}} +{{- with .Values.common.databaseSecret.name -}} +- name: {{ . }} + secret: + secretName: {{ . }} +{{- end }} +{{- end }} + +{{- define "databaseSecret.volumeMount" -}} +{{- with .Values.common.databaseSecret.name -}} +- mountPath: /etc/db + name: {{ . }} +{{- end }} +{{- end }} + +{{- define "storage.base" -}} +storage: +{{- if eq .Values.storage.type "s3" }} + type: s3 + container: {{ .Values.storage.bucketName | quote }} + connection: + auth-type: iam + region: {{ .Values.storage.s3.region }} +{{- else if eq .Values.storage.type "gcs" }} + type: stow + stow: + kind: google + config: + json: "" + project_id: {{ .Values.storage.gcs.projectId }} + scopes: https://www.googleapis.com/auth/devstorage.read_write + container: {{ .Values.storage.bucketName | quote }} +{{- else if eq .Values.storage.type "sandbox" }} + type: minio + container: {{ .Values.storage.bucketName | quote }} + connection: + access-key: minio + auth-type: accesskey + secret-key: miniostorage + disable-ssl: true + endpoint: http://minio.{{ .Release.Namespace }}.svc.cluster.local:9000 + region: us-east-1 +{{- else if eq .Values.storage.type "custom" }} +{{- with .Values.storage.custom -}} + {{ toYaml . | nindent 2 }} +{{- end }} +{{- end }} +{{- end }} + +{{- define "storage" -}} +{{ include "storage.base" .}} + limits: + maxDownloadMBs: 10 +{{- end }} + +{{- define "copilot.config" -}} +kind: ConfigMap +apiVersion: v1 +metadata: + name: flyte-data-config + namespace: {{`{{ namespace }}`}} +data: + config.yaml: | {{ tpl (include "storage.base" .) $ | nindent 4 }} + enable-multicontainer: true +{{- end }} diff --git a/charts/flyte-core/templates/admin/cluster_resource_configmap.yaml b/charts/flyte-core/templates/admin/cluster_resource_configmap.yaml new file mode 100644 index 0000000000..f5f62f58b5 --- /dev/null +++ b/charts/flyte-core/templates/admin/cluster_resource_configmap.yaml @@ -0,0 +1,13 @@ +{{- if .Values.cluster_resource_manager.enabled }} +apiVersion: v1 +kind: ConfigMap +metadata: + name: clusterresource-template + namespace: {{ template "flyte.namespace" . }} + labels: {{ include "flyteadmin.labels" . | nindent 4 }} +data: +{{- range .Values.cluster_resource_manager.templates }} + {{ .key }}.yaml: | {{ .value | nindent 4 }} +{{- end }} + zz_copilot_config.yaml: | {{ include "copilot.config" . | nindent 4 }} +{{- end }} diff --git a/charts/flyte-core/templates/admin/configmap.yaml b/charts/flyte-core/templates/admin/configmap.yaml new file mode 100644 index 0000000000..78701983d7 --- /dev/null +++ b/charts/flyte-core/templates/admin/configmap.yaml @@ -0,0 +1,41 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: flyte-admin-config + namespace: {{ template "flyte.namespace" . }} + labels: {{ include "flyteadmin.labels" . | nindent 4 }} +data: +{{- with .Values.db }} + db.yaml: | {{ tpl (toYaml .) $ | nindent 4 }} +{{- end }} +{{- with .Values.configmap.domain }} + domain.yaml: | {{ toYaml . | nindent 4 }} +{{- end }} +{{- with .Values.configmap.logger }} + logger.yaml: | {{ toYaml . | nindent 4 }} +{{- end }} +{{- with .Values.configmap.adminServer }} + server.yaml: | {{ toYaml . | nindent 4 }} +{{- end }} +{{- with .Values.configmap.remoteData }} + remoteData.yaml: | {{ toYaml . | nindent 4 }} +{{- end }} + storage.yaml: | {{ tpl (include "storage" .) $ | nindent 4 }} +{{- with .Values.configmap.task_resource_defaults }} + task_resource_defaults.yaml: | {{ toYaml . | nindent 4 }} +{{- end }} +{{- if .Values.workflow_notifications.enabled }} +{{- with .Values.workflow_notifications.config }} + notifications.yaml: | {{ tpl (toYaml .) $ | nindent 4 }} +{{- end }} +{{- end }} +{{- if .Values.cluster_resource_manager.enabled }} +{{- with .Values.cluster_resource_manager.config }} + cluster_resources.yaml: | {{ tpl (toYaml .) $ | nindent 4 }} +{{- end }} +{{- end }} +{{- if .Values.workflow_scheduler.enabled }} +{{- with .Values.workflow_scheduler.config }} + scheduler.yaml: | {{ tpl (toYaml .) $ | nindent 4 }} +{{- end }} +{{- end }} diff --git a/charts/flyte-core/templates/admin/cronjob.yaml b/charts/flyte-core/templates/admin/cronjob.yaml new file mode 100644 index 0000000000..0c13789fba --- /dev/null +++ b/charts/flyte-core/templates/admin/cronjob.yaml @@ -0,0 +1,42 @@ +{{- if .Values.cluster_resource_manager.enabled }} +apiVersion: batch/v1beta1 +kind: CronJob +metadata: + name: syncresources + namespace: {{ template "flyte.namespace" . }} + labels: {{ include "flyteadmin.labels" . | nindent 4 }} +spec: + schedule: '*/1 * * * *' + jobTemplate: + spec: + template: + {{- with .Values.flyteadmin.podAnnotations }} + metadata: + annotations: {{ toYaml . | nindent 12 }} + {{- end }} + spec: + containers: + - command: + - flyteadmin + - --config + - {{ .Values.flyteadmin.configPath }} + - clusterresource + - sync + image: "{{ .Values.flyteadmin.image.repository }}:{{ .Values.flyteadmin.image.tag }}" + imagePullPolicy: "{{ .Values.flyteadmin.image.pullPolicy }}" + name: sync-cluster-resources + volumeMounts: {{- include "databaseSecret.volumeMount" . | nindent 12 }} + - mountPath: /etc/flyte/clusterresource/templates + name: resource-templates + - mountPath: /etc/flyte/config + name: config-volume + restartPolicy: OnFailure + serviceAccountName: flyteadmin + volumes: {{- include "databaseSecret.volume" . | nindent 10 }} + - configMap: + name: clusterresource-template + name: resource-templates + - configMap: + name: flyte-admin-config + name: config-volume + {{- end }} \ No newline at end of file diff --git a/charts/flyte-core/templates/admin/deployment.yaml b/charts/flyte-core/templates/admin/deployment.yaml new file mode 100755 index 0000000000..2ef8868abc --- /dev/null +++ b/charts/flyte-core/templates/admin/deployment.yaml @@ -0,0 +1,148 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ template "flyteadmin.name" . }} + namespace: {{ template "flyte.namespace" . }} + labels: {{ include "flyteadmin.labels" . | nindent 4 }} +spec: + replicas: {{ .Values.flyteadmin.replicaCount }} + selector: + matchLabels: {{ include "flyteadmin.selectorLabels" . | nindent 6 }} + template: + metadata: + annotations: + configChecksum: {{ include (print .Template.BasePath "/admin/configmap.yaml") . | sha256sum | trunc 63 | quote }} + {{- with .Values.flyteadmin.podAnnotations }} + {{- toYaml . | nindent 8 }} + {{- end }} + labels: {{ include "flyteadmin.labels" . | nindent 8 }} + spec: + initContainers: + - command: + - flyteadmin + - --config + - {{ .Values.flyteadmin.configPath }} + - migrate + - run + image: "{{ .Values.flyteadmin.image.repository }}:{{ .Values.flyteadmin.image.tag }}" + imagePullPolicy: "{{ .Values.flyteadmin.image.pullPolicy }}" + name: run-migrations + volumeMounts: {{- include "databaseSecret.volumeMount" . | nindent 10 }} + - mountPath: /etc/flyte/config + name: config-volume + {{- if .Values.flyteadmin.initialProjects }} + - command: + - flyteadmin + - --config + - {{ .Values.flyteadmin.configPath }} + - migrate + - seed-projects + {{- range .Values.flyteadmin.initialProjects }} + - {{ . }} + {{- end }} + image: "{{ .Values.flyteadmin.image.repository }}:{{ .Values.flyteadmin.image.tag }}" + imagePullPolicy: "{{ .Values.flyteadmin.image.pullPolicy }}" + name: seed-projects + volumeMounts: {{- include "databaseSecret.volumeMount" . | nindent 10 }} + - mountPath: /etc/flyte/config + name: config-volume + {{- end }} + {{- if .Values.cluster_resource_manager.enabled }} + - command: + - flyteadmin + - --config + - {{ .Values.flyteadmin.configPath }} + - clusterresource + - sync + image: "{{ .Values.flyteadmin.image.repository }}:{{ .Values.flyteadmin.image.tag }}" + imagePullPolicy: "{{ .Values.flyteadmin.image.pullPolicy }}" + name: sync-cluster-resources + volumeMounts: {{- include "databaseSecret.volumeMount" . | nindent 10 }} + - mountPath: /etc/flyte/clusterresource/templates + name: resource-templates + - mountPath: /etc/flyte/config + name: config-volume + {{- end }} + - name: generate-secrets + image: "{{ .Values.flyteadmin.image.repository }}:{{ .Values.flyteadmin.image.tag }}" + imagePullPolicy: "{{ .Values.flyteadmin.image.pullPolicy }}" + command: ["/bin/sh", "-c"] + args: + [ + "flyteadmin --config={{ .Values.flyteadmin.configPath }} secrets init --localPath /etc/secrets/auth && flyteadmin --config=/etc/flyte/config/*.yaml secrets create --name flyte-admin-secrets --fromPath /etc/secrets/auth", + ] + volumeMounts: + - name: config-volume + mountPath: /etc/flyte/config + env: + - name: POD_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace + containers: + - command: + - flyteadmin + - --config + - {{ .Values.flyteadmin.configPath }} + {{- with .Values.flyteadmin.extraArgs }} + {{- toYaml . | nindent 8 }} + {{- end }} + - serve + image: "{{ .Values.flyteadmin.image.repository }}:{{ .Values.flyteadmin.image.tag }}" + imagePullPolicy: "{{ .Values.flyteadmin.image.pullPolicy }}" + name: flyteadmin + ports: + - containerPort: 8088 + - containerPort: 8089 + resources: {{ toYaml .Values.flyteadmin.resources | nindent 10 }} + volumeMounts: {{- include "databaseSecret.volumeMount" . | nindent 8 }} + - mountPath: /srv/flyte + name: shared-data + - mountPath: /etc/flyte/config + name: config-volume + - name: auth + mountPath: /etc/secrets/ + - command: + - sh + - -c + - ln -s /usr/share/nginx/html /usr/share/nginx/html/openapi && sh /usr/local/bin/docker-run.sh + env: + - name: PAGE_TITLE + value: Flyte Admin OpenAPI + - name: SPEC_URL + value: /api/v1/openapi + - name: PORT + value: "8087" + image: docker.io/redocly/redoc + imagePullPolicy: IfNotPresent + name: redoc + ports: + - containerPort: 8087 + resources: + limits: + cpu: "0.1" + memory: 200Mi + serviceAccountName: {{ template "flyteadmin.name" . }} + volumes: {{- include "databaseSecret.volume" . | nindent 8 }} + - emptyDir: {} + name: shared-data + - configMap: + name: flyte-admin-config + name: config-volume + {{- if .Values.cluster_resource_manager.enabled }} + - configMap: + name: clusterresource-template + name: resource-templates + {{- end }} + - name: auth + secret: + secretName: flyte-admin-secrets + {{- with .Values.flyteadmin.nodeSelector }} + nodeSelector: {{ toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.flyteadmin.affinity }} + affinity: {{ toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.flyteadmin.tolerations }} + tolerations: {{ toYaml . | nindent 8 }} + {{- end }} diff --git a/charts/flyte-core/templates/admin/rbac.yaml b/charts/flyte-core/templates/admin/rbac.yaml new file mode 100644 index 0000000000..2ed6307578 --- /dev/null +++ b/charts/flyte-core/templates/admin/rbac.yaml @@ -0,0 +1,57 @@ + +--- +{{- if .Values.flyteadmin.serviceAccount.create }} +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ template "flyteadmin.name" . }} + namespace: {{ template "flyte.namespace" . }} + labels: {{ include "flyteadmin.labels" . | nindent 4 }} + {{- with .Values.flyteadmin.serviceAccount.annotations }} + annotations: {{ tpl (toYaml .) $ | nindent 4 }} + {{- end }} +{{- with .Values.flyteadmin.serviceAccount.imagePullSecrets }} +imagePullSecrets: {{ toYaml . | nindent 2 }} +{{- end }} +{{- end }} + +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: {{ template "flyteadmin.name" . }} + labels: {{ include "flyteadmin.labels" . | nindent 4 }} +rules: +- apiGroups: + - "" + - flyte.lyft.com + - rbac.authorization.k8s.io + resources: + - configmaps + - flyteworkflows + - namespaces + - pods + - resourcequotas + - roles + - rolebindings + - secrets + - services + - serviceaccounts + - spark-role + verbs: + - '*' + +--- +apiVersion: rbac.authorization.k8s.io/v1beta1 +kind: ClusterRoleBinding +metadata: + name: {{ template "flyteadmin.name" . }}-binding + labels: {{ include "flyteadmin.labels" . | nindent 4 }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: {{ template "flyteadmin.name" . }} +subjects: +- kind: ServiceAccount + name: {{ template "flyteadmin.name" . }} + namespace: {{ template "flyte.namespace" . }} diff --git a/charts/flyte-core/templates/admin/secret-auth.yaml b/charts/flyte-core/templates/admin/secret-auth.yaml new file mode 100644 index 0000000000..7e5a29467c --- /dev/null +++ b/charts/flyte-core/templates/admin/secret-auth.yaml @@ -0,0 +1,10 @@ +apiVersion: v1 +kind: Secret +metadata: + name: flyte-admin-secrets + namespace: {{ template "flyte.namespace" . }} +type: Opaque +stringData: +{{- with .Values.flyteadmin.secrets -}} +{{ toYaml . | nindent 2 }} +{{- end }} diff --git a/charts/flyte-core/templates/admin/service.yaml b/charts/flyte-core/templates/admin/service.yaml new file mode 100644 index 0000000000..07132a45d4 --- /dev/null +++ b/charts/flyte-core/templates/admin/service.yaml @@ -0,0 +1,34 @@ +apiVersion: v1 +kind: Service +metadata: + name: {{ template "flyteadmin.name" . }} + namespace: {{ template "flyte.namespace" . }} + labels: {{ include "flyteadmin.labels" . | nindent 4 }} + {{- with .Values.flyteadmin.service.annotations }} + annotations: {{ toYaml . | nindent 4 }} + {{- end }} +spec: + {{- with .Values.flyteadmin.service.type}} + type: {{ . }} + {{- end }} + {{- with .Values.flyteadmin.service.loadBalancerSourceRanges }} + loadBalancerSourceRanges: + {{ . }} + {{- end }} + ports: + - name: http + port: 80 + protocol: TCP + targetPort: 8088 + - name: grpc + port: 81 + protocol: TCP + targetPort: 8089 + - name: redoc + protocol: TCP + port: 87 + targetPort: 8087 + - name: http-metrics + protocol: TCP + port: 10254 + selector: {{ include "flyteadmin.selectorLabels" . | nindent 4 }} diff --git a/charts/flyte-core/templates/common/ingress.yaml b/charts/flyte-core/templates/common/ingress.yaml new file mode 100644 index 0000000000..c4b4a817ba --- /dev/null +++ b/charts/flyte-core/templates/common/ingress.yaml @@ -0,0 +1,222 @@ +{{- define "grpcRoutes" -}} +# NOTE: Port 81 in flyteadmin is the GRPC server port for FlyteAdmin. +- path: /flyteidl.service.AdminService + pathType: ImplementationSpecific + backend: + serviceName: flyteadmin + servicePort: 81 +- path: /flyteidl.service.AdminService/* + pathType: ImplementationSpecific + backend: + serviceName: flyteadmin + servicePort: 81 +- path: /flyteidl.service.AuthMetadataService + pathType: ImplementationSpecific + backend: + serviceName: flyteadmin + servicePort: 81 +- path: /flyteidl.service.AuthMetadataService/* + pathType: ImplementationSpecific + backend: + serviceName: flyteadmin + servicePort: 81 +- path: /flyteidl.service.IdentityService + pathType: ImplementationSpecific + backend: + serviceName: flyteadmin + servicePort: 81 +- path: /flyteidl.service.IdentityService/* + pathType: ImplementationSpecific + backend: + serviceName: flyteadmin + servicePort: 81 +{{- end }} +{{- if .Values.common.ingress.enabled }} +apiVersion: networking.k8s.io/v1beta1 +kind: Ingress +metadata: + name: {{ template "flyte.name" . }} + namespace: {{ template "flyte.namespace" . }} + {{- with .Values.common.ingress.annotations }} + annotations: {{ tpl (toYaml .) $ | nindent 4}} + {{- end }} +spec: + rules: + - http: + paths: + {{- if .Values.common.ingress.albSSLRedirect }} + - path: /* + pathType: ImplementationSpecific + backend: + serviceName: ssl-redirect + servicePort: use-annotation + {{- end }} + # This is useful only for frontend development + {{- if .Values.common.ingress.webpackHMR }} + - path: /__webpack_hmr + pathType: ImplementationSpecific + backend: + serviceName: flyteconsole + servicePort: 80 + {{- end }} + # Port 87 in FlyteAdmin maps to the redoc container. + - path: /openapi + pathType: ImplementationSpecific + backend: + serviceName: flyteadmin + servicePort: 87 + # NOTE: If you change this, you must update the BASE_URL value in flyteconsole.yaml + - path: /console + pathType: ImplementationSpecific + backend: + serviceName: flyteconsole + servicePort: 80 + - path: /console/* + pathType: ImplementationSpecific + backend: + serviceName: flyteconsole + servicePort: 80 + - path: /api + pathType: ImplementationSpecific + backend: + serviceName: flyteadmin + servicePort: 80 + - path: /api/* + pathType: ImplementationSpecific + backend: + serviceName: flyteadmin + servicePort: 80 + - path: /healthcheck + pathType: ImplementationSpecific + backend: + serviceName: flyteadmin + servicePort: 80 + - path: /v1/* + pathType: ImplementationSpecific + backend: + serviceName: flyteadmin + servicePort: 80 + # Port 87 in FlyteAdmin maps to the redoc container. + - path: /openapi/* + pathType: ImplementationSpecific + backend: + serviceName: flyteadmin + servicePort: 80 + - path: /.well-known + pathType: ImplementationSpecific + backend: + serviceName: flyteadmin + servicePort: 80 + - path: /.well-known/* + pathType: ImplementationSpecific + backend: + serviceName: flyteadmin + servicePort: 80 + - path: /login + pathType: ImplementationSpecific + backend: + serviceName: flyteadmin + servicePort: 80 + - path: /login/* + pathType: ImplementationSpecific + backend: + serviceName: flyteadmin + servicePort: 80 + - path: /logout + pathType: ImplementationSpecific + backend: + serviceName: flyteadmin + servicePort: 80 + - path: /logout/* + pathType: ImplementationSpecific + backend: + serviceName: flyteadmin + servicePort: 80 + - path: /callback + pathType: ImplementationSpecific + backend: + serviceName: flyteadmin + servicePort: 80 + - path: /callback/* + pathType: ImplementationSpecific + backend: + serviceName: flyteadmin + servicePort: 80 + - path: /me + pathType: ImplementationSpecific + backend: + serviceName: flyteadmin + servicePort: 80 + - path: /config + pathType: ImplementationSpecific + backend: + serviceName: flyteadmin + servicePort: 80 + - path: /config/* + pathType: ImplementationSpecific + backend: + serviceName: flyteadmin + servicePort: 80 + - path: /oauth2 + pathType: ImplementationSpecific + backend: + serviceName: flyteadmin + servicePort: 80 + - path: /oauth2/* + pathType: ImplementationSpecific + backend: + serviceName: flyteadmin + servicePort: 80 + {{- if not .Values.common.ingress.separateGrpcIngress }} + {{- include "grpcRoutes" . | nindent 10 -}} + {{- end }} + {{- with .Values.common.ingress.host }} + host: {{ . }} + {{- end }} + {{- if .Values.common.ingress.tls.enabled }} + tls: + - secretName: {{ .Release.Name }}-flyte-tls + hosts: + - {{ .Values.common.ingress.host }} + {{ end }} + + +{{- if .Values.common.ingress.separateGrpcIngress }} +# Certain ingress controllers like nginx cannot serve HTTP 1 and GRPC with a single ingress because GRPC can only +# enabled on the ingress object, not on backend services (GRPC annotation is set on the ingress, not on the services). +--- +apiVersion: networking.k8s.io/v1beta1 +kind: Ingress +metadata: + name: {{ template "flyte.name" . }}-grpc + namespace: {{ template "flyte.namespace" . }} + annotations: + {{- with .Values.common.ingress.annotations }} + {{- tpl (toYaml .) $ | nindent 4}} + {{- end }} + {{- with .Values.common.ingress.separateGrpcIngressAnnotations }} + {{- toYaml . | nindent 4}} + {{- end }} +spec: + rules: + - http: + paths: + # {{- if .Values.common.ingress.albSSLRedirect }} + # - backend: + # serviceName: ssl-redirect + # servicePort: use-annotation + # path: /* + # pathType: ImplementationSpecific + # {{- end }} + {{- include "grpcRoutes" . | nindent 10 -}} + {{- with .Values.common.ingress.host }} + host: {{ . }} + {{- end }} + {{- if .Values.common.ingress.tls.enabled }} + tls: + - secretName: {{ .Release.Name }}-flyte-tls + hosts: + - {{ .Values.common.ingress.host }} + {{ end }} +{{- end }} +{{- end }} diff --git a/charts/flyte-core/templates/common/namespace.yaml b/charts/flyte-core/templates/common/namespace.yaml new file mode 100644 index 0000000000..7e4f26d461 --- /dev/null +++ b/charts/flyte-core/templates/common/namespace.yaml @@ -0,0 +1,9 @@ +{{- if .Values.common.flyteNamespaceTemplate.enabled }} +apiVersion: v1 +kind: Namespace +metadata: + name: {{ template "flyte.namespace" . }} +spec: + finalizers: + - kubernetes +{{- end }} diff --git a/charts/flyte-core/templates/common/secret.yaml b/charts/flyte-core/templates/common/secret.yaml new file mode 100644 index 0000000000..d01088ac1f --- /dev/null +++ b/charts/flyte-core/templates/common/secret.yaml @@ -0,0 +1,3 @@ +{{- with .Values.common.databaseSecret.secretManifest }} +{{ tpl (toYaml .) $ }} +{{- end }} diff --git a/charts/flyte-core/templates/console/configmap.yaml b/charts/flyte-core/templates/console/configmap.yaml new file mode 100644 index 0000000000..084091b781 --- /dev/null +++ b/charts/flyte-core/templates/console/configmap.yaml @@ -0,0 +1,7 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: flyte-console-config + namespace: {{ template "flyte.namespace" . }} + labels: {{ include "flyteconsole.labels" . | nindent 4 }} +data: {{ toYaml .Values.configmap.console | nindent 2 }} diff --git a/charts/flyte-core/templates/console/deployment.yaml b/charts/flyte-core/templates/console/deployment.yaml new file mode 100644 index 0000000000..298fb2661e --- /dev/null +++ b/charts/flyte-core/templates/console/deployment.yaml @@ -0,0 +1,44 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ template "flyteconsole.name" . }} + namespace: {{ template "flyte.namespace" . }} + labels: {{ include "flyteconsole.labels" . | nindent 4 }} +spec: + replicas: {{ .Values.flyteconsole.replicaCount }} + selector: + matchLabels: {{ include "flyteconsole.selectorLabels" . | nindent 6 }} + template: + metadata: + annotations: + configChecksum: {{ include (print .Template.BasePath "/console/configmap.yaml") . | sha256sum | trunc 63 | quote }} + {{- with .Values.flyteconsole.podAnnotations }} + {{- toYaml . | nindent 8 }} + {{- end }} + labels: {{ include "flyteconsole.labels" . | nindent 8 }} + spec: + containers: + - image: "{{ .Values.flyteconsole.image.repository }}:{{ .Values.flyteconsole.image.tag }}" + imagePullPolicy: "{{ .Values.flyteconsole.image.pullPolicy }}" + name: flyteconsole + envFrom: + - configMapRef: + name: flyte-console-config + ports: + - containerPort: 8080 + resources: {{ toYaml .Values.flyteconsole.resources | nindent 10 }} + volumeMounts: + - mountPath: /srv/flyte + name: shared-data + volumes: + - emptyDir: {} + name: shared-data + {{- with .Values.flyteconsole.nodeSelector }} + nodeSelector: {{ toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.flyteconsole.affinity }} + affinity: {{ toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.flyteconsole.tolerations }} + tolerations: {{ toYaml . | nindent 8 }} + {{- end }} \ No newline at end of file diff --git a/charts/flyte-core/templates/console/service.yaml b/charts/flyte-core/templates/console/service.yaml new file mode 100644 index 0000000000..c007243455 --- /dev/null +++ b/charts/flyte-core/templates/console/service.yaml @@ -0,0 +1,18 @@ +apiVersion: v1 +kind: Service +metadata: + name: {{ template "flyteconsole.name" . }} + namespace: {{ template "flyte.namespace" . }} + labels: {{ include "flyteconsole.labels" . | nindent 4 }} + {{- with .Values.flyteconsole.service.annotations }} + annotations: {{ toYaml . | nindent 4 }} + {{- end }} +spec: + {{- with .Values.flyteconsole.service.type}} + type: {{ . }} + {{- end }} + ports: + - port: 80 + protocol: TCP + targetPort: 8080 + selector: {{ include "flyteconsole.selectorLabels" . | nindent 4 }} diff --git a/charts/flyte-core/templates/datacatalog/configmap.yaml b/charts/flyte-core/templates/datacatalog/configmap.yaml new file mode 100644 index 0000000000..bd579158bb --- /dev/null +++ b/charts/flyte-core/templates/datacatalog/configmap.yaml @@ -0,0 +1,18 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: datacatalog-config + namespace: {{ template "flyte.namespace" . }} + labels: {{ include "datacatalog.labels" . | nindent 4 }} +data: +{{- with .Values.db }} + db.yaml: | {{ tpl (toYaml .) $ | nindent 4 }} +{{- end }} +{{- with .Values.configmap.logger }} + logger.yaml: | {{ toYaml . | nindent 4 }} +{{- end }} +{{- with .Values.configmap.datacatalogServer }} + server.yaml: | {{ toYaml . | nindent 4 }} +{{- end }} + storage.yaml: | {{ tpl (include "storage" .) $ | nindent 4 }} + diff --git a/charts/flyte-core/templates/datacatalog/deployment.yaml b/charts/flyte-core/templates/datacatalog/deployment.yaml new file mode 100644 index 0000000000..1e3e2b6c6a --- /dev/null +++ b/charts/flyte-core/templates/datacatalog/deployment.yaml @@ -0,0 +1,67 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ template "datacatalog.name" . }} + namespace: {{ template "flyte.namespace" . }} + labels: {{ include "datacatalog.labels" . | nindent 4 }} +spec: + replicas: {{ .Values.datacatalog.replicaCount }} + selector: + matchLabels: {{ include "datacatalog.selectorLabels" . | nindent 6 }} + template: + metadata: + annotations: + configChecksum: {{ include (print .Template.BasePath "/datacatalog/configmap.yaml") . | sha256sum | trunc 63 | quote }} + {{- with .Values.datacatalog.podAnnotations }} + {{- toYaml . | nindent 8 }} + {{- end }} + labels: {{ include "datacatalog.labels" . | nindent 8 }} + spec: + initContainers: + - command: + - datacatalog + - --config + - {{ .Values.datacatalog.configPath }} + - migrate + - run + image: "{{ .Values.datacatalog.image.repository }}:{{ .Values.datacatalog.image.tag }}" + imagePullPolicy: "{{ .Values.datacatalog.image.pullPolicy }}" + name: run-migrations + volumeMounts: {{- include "databaseSecret.volumeMount" . | nindent 8 }} + - mountPath: /etc/datacatalog/config + name: config-volume + containers: + - command: + - datacatalog + - --config + - {{ .Values.datacatalog.configPath }} + {{- with .Values.datacatalog.extraArgs }} + {{- toYaml . | nindent 8 }} + {{- end }} + - serve + image: "{{ .Values.datacatalog.image.repository }}:{{ .Values.datacatalog.image.tag }}" + imagePullPolicy: "{{ .Values.datacatalog.image.pullPolicy }}" + name: datacatalog + ports: + - containerPort: 8088 + - containerPort: 8089 + resources: {{ toYaml .Values.datacatalog.resources | nindent 10 }} + volumeMounts: {{- include "databaseSecret.volumeMount" . | nindent 8 }} + - mountPath: /etc/datacatalog/config + name: config-volume + serviceAccountName: {{ template "datacatalog.name" . }} + volumes: {{- include "databaseSecret.volume" . | nindent 6 }} + - emptyDir: {} + name: shared-data + - configMap: + name: datacatalog-config + name: config-volume + {{- with .Values.datacatalog.nodeSelector }} + nodeSelector: {{ toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.datacatalog.affinity }} + affinity: {{ toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.datacatalog.tolerations }} + tolerations: {{ toYaml . | nindent 8 }} + {{- end }} diff --git a/charts/flyte-core/templates/datacatalog/rbac.yaml b/charts/flyte-core/templates/datacatalog/rbac.yaml new file mode 100644 index 0000000000..f068b8242c --- /dev/null +++ b/charts/flyte-core/templates/datacatalog/rbac.yaml @@ -0,0 +1,16 @@ + +--- +{{- if .Values.datacatalog.serviceAccount.create }} +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ template "datacatalog.name" . }} + namespace: {{ template "flyte.namespace" . }} + labels: {{ include "datacatalog.labels" . | nindent 4 }} + {{- with .Values.datacatalog.serviceAccount.annotations }} + annotations: {{ tpl (toYaml .) $ | nindent 4 }} + {{- end }} +{{- with .Values.datacatalog.serviceAccount.imagePullSecrets }} +imagePullSecrets: {{ toYaml . | nindent 2 }} +{{- end }} +{{- end }} \ No newline at end of file diff --git a/charts/flyte-core/templates/datacatalog/service.yaml b/charts/flyte-core/templates/datacatalog/service.yaml new file mode 100644 index 0000000000..07958d2f34 --- /dev/null +++ b/charts/flyte-core/templates/datacatalog/service.yaml @@ -0,0 +1,27 @@ +apiVersion: v1 +kind: Service +metadata: + name: {{ template "datacatalog.name" . }} + namespace: {{ template "flyte.namespace" . }} + labels: {{ include "datacatalog.labels" . | nindent 4 }} + {{- with .Values.datacatalog.service.annotations }} + annotations: {{ toYaml . | nindent 4 }} + {{- end }} +spec: + {{- with .Values.datacatalog.service.type}} + type: {{ . }} + {{- end }} + ports: + - name: grpc-2 + port: 8089 + protocol: TCP + targetPort: 8089 + - name: http + port: 88 + protocol: TCP + targetPort: 8088 + - name: grpc + port: 89 + protocol: TCP + targetPort: 8089 + selector: {{ include "datacatalog.selectorLabels" . | nindent 4 }} diff --git a/charts/flyte-core/templates/propeller/configmap.yaml b/charts/flyte-core/templates/propeller/configmap.yaml new file mode 100644 index 0000000000..3c3e1b3f9d --- /dev/null +++ b/charts/flyte-core/templates/propeller/configmap.yaml @@ -0,0 +1,50 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: flyte-propeller-config + namespace: {{ template "flyte.namespace" . }} + labels: {{ include "flyteadmin.labels" . | nindent 4 }} +data: +{{- with .Values.configmap.admin }} + admin.yaml: | {{ toYaml . | nindent 4 }} +{{- end }} +{{- with .Values.configmap.catalog }} + catalog.yaml: | {{ toYaml . | nindent 4 }} +{{- end }} +{{- with .Values.configmap.catalog_cache }} + catalog_cache.yaml: | {{ toYaml . | nindent 4 }} +{{- end }} +{{- with .Values.configmap.copilot }} + copilot.yaml: | {{ toYaml . | nindent 4 }} +{{- end }} +{{- with .Values.configmap.core }} + core.yaml: | {{ tpl (toYaml .) $ | nindent 4 }} +{{- end }} +{{- with .Values.configmap.enabled_plugins }} + enabled_plugins.yaml: | {{ toYaml . | nindent 4 }} +{{- end }} +{{- with .Values.configmap.k8s }} + k8s.yaml: | {{ toYaml . | nindent 4 }} +{{- end }} +{{- with .Values.configmap.logger }} + logger.yaml: | {{ toYaml . | nindent 4 }} +{{- end }} +{{- with .Values.configmap.qubole }} + qubole.yaml: | {{ toYaml . | nindent 4 }} +{{- end }} +{{- with .Values.configmap.resource_manager }} + resource_manager.yaml: | {{ toYaml . | nindent 4 }} +{{- end }} +{{- if .Values.sparkoperator.enabled }} +{{- with .Values.sparkoperator.plugin_config }} + spark.yaml: | {{ toYaml . | nindent 4 }} +{{- end }} +{{- end }} + storage.yaml: | {{ tpl (include "storage" .) $ | nindent 4 }} + cache.yaml: | + cache: + max_size_mbs: {{ .Values.flytepropeller.cacheSizeMbs }} + target_gc_percent: 70 +{{- with .Values.configmap.task_logs }} + task_logs.yaml: | {{ tpl (toYaml .) $ | nindent 4 }} +{{- end }} diff --git a/charts/flyte-core/templates/propeller/crds/flyteworkflow.yaml b/charts/flyte-core/templates/propeller/crds/flyteworkflow.yaml new file mode 100644 index 0000000000..1095feca6b --- /dev/null +++ b/charts/flyte-core/templates/propeller/crds/flyteworkflow.yaml @@ -0,0 +1,14 @@ +apiVersion: apiextensions.k8s.io/v1beta1 +kind: CustomResourceDefinition +metadata: + name: flyteworkflows.flyte.lyft.com +spec: + group: flyte.lyft.com + names: + kind: FlyteWorkflow + plural: flyteworkflows + shortNames: + - fly + singular: flyteworkflow + scope: Namespaced + version: v1alpha1 diff --git a/charts/flyte-core/templates/propeller/deployment.yaml b/charts/flyte-core/templates/propeller/deployment.yaml new file mode 100644 index 0000000000..7b5ae7e63a --- /dev/null +++ b/charts/flyte-core/templates/propeller/deployment.yaml @@ -0,0 +1,60 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ template "flytepropeller.name" . }} + namespace: {{ template "flyte.namespace" . }} + labels: {{ include "flytepropeller.labels" . | nindent 4 }} +spec: + replicas: {{ .Values.flytepropeller.replicaCount }} + selector: + matchLabels: {{ include "flytepropeller.selectorLabels" . | nindent 6 }} + template: + metadata: + annotations: + configChecksum: {{ include (print .Template.BasePath "/propeller/configmap.yaml") . | sha256sum | trunc 63 | quote }} + {{- with .Values.flytepropeller.podAnnotations }} + {{- toYaml . | nindent 8 }} + {{- end }} + labels: {{ include "flytepropeller.labels" . | nindent 8 }} + spec: + containers: + - command: + - flytepropeller + - --config + - {{ .Values.flytepropeller.configPath }} + {{- with .Values.flytepropeller.extraArgs }} + {{- toYaml . | nindent 8 }} + {{- end }} + env: + - name: POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + image: "{{ .Values.flytepropeller.image.repository }}:{{ .Values.flytepropeller.image.tag }}" + imagePullPolicy: "{{ .Values.flytepropeller.image.pullPolicy }}" + name: flytepropeller + ports: + - containerPort: 10254 + resources: {{ toYaml .Values.flytepropeller.resources | nindent 10 }} + volumeMounts: + - name: config-volume + mountPath: /etc/flyte/config + - name: auth + mountPath: /etc/secrets/ + serviceAccountName: {{ template "flytepropeller.name" . }} + volumes: + - configMap: + name: flyte-propeller-config + name: config-volume + - name: auth + secret: + secretName: flyte-propeller-auth + {{- with .Values.flytepropeller.nodeSelector }} + nodeSelector: {{ toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.flytepropeller.affinity }} + affinity: {{ toYaml . | nindent 8 }} + {{- end }} + {{- with .Values.flytepropeller.tolerations }} + tolerations: {{ toYaml . | nindent 8 }} + {{- end }} diff --git a/charts/flyte-core/templates/propeller/rbac.yaml b/charts/flyte-core/templates/propeller/rbac.yaml new file mode 100644 index 0000000000..4b1f42b591 --- /dev/null +++ b/charts/flyte-core/templates/propeller/rbac.yaml @@ -0,0 +1,98 @@ + +--- +{{- if .Values.flytepropeller.serviceAccount.create }} +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ template "flytepropeller.name" . }} + namespace: {{ template "flyte.namespace" . }} + labels: {{ include "flytepropeller.labels" . | nindent 4 }} + {{- with .Values.flytepropeller.serviceAccount.annotations }} + annotations: {{ tpl (toYaml .) $ | nindent 4 }} + {{- end}} +{{- with .Values.flytepropeller.serviceAccount.imagePullSecrets }} +imagePullSecrets: {{ toYaml . | nindent 2 }} +{{- end }} +{{- end }} +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: {{ template "flytepropeller.name" . }} + labels: {{ include "flytepropeller.labels" . | nindent 4 }} +rules: +# Allow RO access to PODS +- apiGroups: + - "" + resources: + - pods + verbs: + - get + - list + - watch +# Allow Event recording access +- apiGroups: + - "" + resources: + - events + verbs: + - create + - update + - delete + - patch +# Allow Access All plugin objects +- apiGroups: + - '*' + resources: + - '*' + verbs: + - get + - list + - watch + - create + - update + - delete + - patch +# Allow Access to CRD +- apiGroups: + - apiextensions.k8s.io + resources: + - customresourcedefinitions + verbs: + - get + - list + - watch + - create + - delete + - update +# Allow Access to all resources under flyte.lyft.com +- apiGroups: + - flyte.lyft.com + resources: + - flyteworkflows + - flyteworkflows/finalizers + verbs: + - get + - list + - watch + - create + - update + - delete + - patch + - post + - deletecollection + +--- +apiVersion: rbac.authorization.k8s.io/v1beta1 +kind: ClusterRoleBinding +metadata: + name: {{ template "flytepropeller.name" . }} + labels: {{ include "flytepropeller.labels" . | nindent 4 }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: {{ template "flytepropeller.name" . }} +subjects: +- kind: ServiceAccount + name: {{ template "flytepropeller.name" . }} + namespace: {{ template "flyte.namespace" . }} diff --git a/charts/flyte-core/templates/propeller/secret-auth.yaml b/charts/flyte-core/templates/propeller/secret-auth.yaml new file mode 100644 index 0000000000..0430727057 --- /dev/null +++ b/charts/flyte-core/templates/propeller/secret-auth.yaml @@ -0,0 +1,8 @@ +apiVersion: v1 +kind: Secret +metadata: + name: flyte-propeller-auth + namespace: {{ template "flyte.namespace" . }} +type: Opaque +stringData: + client_secret: foobar diff --git a/charts/flyte-core/templates/propeller/webhook.yaml b/charts/flyte-core/templates/propeller/webhook.yaml new file mode 100644 index 0000000000..b0e018b527 --- /dev/null +++ b/charts/flyte-core/templates/propeller/webhook.yaml @@ -0,0 +1,158 @@ +{{- if .Values.webhook.enabled }} +# Create an empty secret that the first propeller pod will populate +apiVersion: v1 +kind: Secret +metadata: + name: flyte-pod-webhook + namespace: {{ template "flyte.namespace" . }} +type: Opaque +--- +# Create the actual deployment +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ template "flyte-pod-webhook.name" . }} + namespace: {{ template "flyte.namespace" . }} + labels: + app: {{ template "flyte-pod-webhook.name" . }} +spec: + selector: + matchLabels: + app: {{ template "flyte-pod-webhook.name" . }} + template: + metadata: + labels: + app: {{ template "flyte-pod-webhook.name" . }} + app.kubernetes.io/name: {{ template "flyte-pod-webhook.name" . }} + app.kubernetes.io/version: {{ .Values.flytepropeller.image.tag }} + annotations: + configChecksum: {{ include (print .Template.BasePath "/propeller/configmap.yaml") . | sha256sum | trunc 63 | quote }} + {{- with .Values.flyteadmin.podAnnotations }} + {{- toYaml . | nindent 8 }} + {{- end }} + spec: + serviceAccountName: {{ template "flyte-pod-webhook.name" . }} +{{- if .Values.webhook.enabled }} + initContainers: + - name: generate-secrets + image: "{{ .Values.flytepropeller.image.repository }}:{{ .Values.flytepropeller.image.tag }}" + imagePullPolicy: "{{ .Values.flytepropeller.image.pullPolicy }}" + command: + - flytepropeller + args: + - webhook + - init-certs + - --config + - /etc/flyte/config/*.yaml + env: + - name: POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + - name: POD_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace + volumeMounts: + - name: config-volume + mountPath: /etc/flyte/config +{{- end }} + containers: + - name: webhook + image: "{{ .Values.flytepropeller.image.repository }}:{{ .Values.flytepropeller.image.tag }}" + imagePullPolicy: "{{ .Values.flytepropeller.image.pullPolicy }}" + command: + - flytepropeller + args: + - webhook + - --config + - /etc/flyte/config/*.yaml + env: + - name: POD_NAME + valueFrom: + fieldRef: + fieldPath: metadata.name + - name: POD_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace + volumeMounts: + - name: config-volume + mountPath: /etc/flyte/config + readOnly: true + - name: webhook-certs + mountPath: /etc/webhook/certs + readOnly: true + volumes: + - name: config-volume + configMap: + name: flyte-propeller-config + - name: webhook-certs + secret: + secretName: flyte-pod-webhook +--- +# Service +apiVersion: v1 +kind: Service +metadata: + name: {{ template "flyte-pod-webhook.name" . }} + namespace: {{ template "flyte.namespace" . }} + {{- with .Values.webhook.service.annotations }} + annotations: {{ toYaml . | nindent 4 }} + {{- end }} +spec: + selector: + app: {{ template "flyte-pod-webhook.name" . }} + ports: + - name: https + protocol: TCP + port: 443 + targetPort: 9443 +--- +# Create a ClusterRole for the webhook +# https://kubernetes.io/docs/admin/authorization/rbac/ +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: {{ template "flyte-pod-webhook.name" . }} + namespace: {{ template "flyte.namespace" . }} +rules: + - apiGroups: + - "*" + resources: + - mutatingwebhookconfigurations + - secrets + - pods + verbs: + - get + - create + - update + - patch +--- +{{- if .Values.webhook.serviceAccount.create }} +# Create a Service Account for webhook +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ template "flyte-pod-webhook.name" . }} + namespace: {{ template "flyte.namespace" . }} + {{- with .Values.webhook.serviceAccount.annotations }} + annotations: {{ toYaml . | nindent 4 }} + {{- end}} +{{- end }} +--- +# Create a binding from Role -> ServiceAccount +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: {{ template "flyte-pod-webhook.name" . }} + namespace: {{ template "flyte.namespace" . }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: {{ template "flyte-pod-webhook.name" . }} +subjects: + - kind: ServiceAccount + name: {{ template "flyte-pod-webhook.name" . }} + namespace: {{ template "flyte.namespace" . }} +{{- end }} \ No newline at end of file diff --git a/charts/flyte-core/values-eks.yaml b/charts/flyte-core/values-eks.yaml new file mode 100644 index 0000000000..95c5d5ff56 --- /dev/null +++ b/charts/flyte-core/values-eks.yaml @@ -0,0 +1,514 @@ +userSettings: + accountNumber: + accountRegion: + certificateArn: + dbPassword: + rdsHost: + bucketName: + logGroup: + +# +# FLYTEADMIN +# + +flyteadmin: + replicaCount: 2 + # -- IAM role for SA: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html + serviceAccount: + # -- If the service account is created by you, make this false, else a new service account will be created and the iam-role-flyte will be added + # you can change the name of this role + create: true + annotations: + eks.amazonaws.com/role-arn: arn:aws:iam::{{ .Values.userSettings.accountNumber }}:role/iam-role-flyte + + resources: + limits: + ephemeral-storage: 200Mi + requests: + cpu: 50m + ephemeral-storage: 200Mi + memory: 200Mi + + service: + annotations: + service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "600" + # TODO alter domain + external-dns.alpha.kubernetes.io/hostname: "flyte.example.com" + type: LoadBalancer + loadBalancerSourceRanges: + # TODO change source ip range if desired + - 0.0.0.0/0 + + affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchLabels: + app.kubernetes.io/name: flyteadmin + topologyKey: kubernetes.io/hostname + +# +# DATACATALOG +# + +datacatalog: + replicaCount: 2 + serviceAccount: + # -- If the service account is created by you, make this false + create: true + annotations: + eks.amazonaws.com/role-arn: arn:aws:iam::{{ .Values.userSettings.accountNumber }}:role/iam-role-flyte + resources: + limits: + cpu: 500m + ephemeral-storage: 200Mi + requests: + cpu: 50m + ephemeral-storage: 200Mi + memory: 200Mi + service: + annotations: + service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "600" + type: LoadBalancer + loadBalancerSourceRanges: + # TODO change source ip range if desired + - 0.0.0.0/0 + affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchLabels: + app.kubernetes.io/name: datacatalog + topologyKey: kubernetes.io/hostname + +# +# FLYTEPROPELLER +# + +flytepropeller: + replicaCount: 2 + serviceAccount: + # -- If the service account is created by you, make this false + create: true + annotations: + eks.amazonaws.com/role-arn: arn:aws:iam::{{ .Values.userSettings.accountNumber }}:role/iam-role-flyte + resources: + limits: + cpu: 1 + ephemeral-storage: 1Gi + memory: 2Gi + requests: + cpu: 1 + ephemeral-storage: 1Gi + memory: 2Gi + cacheSizeMbs: 1024 + affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchLabels: + app.kubernetes.io/name: flytepropeller + topologyKey: kubernetes.io/hostname + +# +# FLYTECONSOLE +# + +flyteconsole: + replicaCount: 2 + resources: + limits: + cpu: 250m + service: + annotations: + service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "600" + # TODO alter domain + external-dns.alpha.kubernetes.io/hostname: "flyte.example.com" + type: LoadBalancer + loadBalancerSourceRanges: + # TODO change source ip range if desired + - 0.0.0.0/0 + affinity: + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchLabels: + app.kubernetes.io/name: flyteconsole + topologyKey: kubernetes.io/hostname + +# +# REDIS +# + +redis: + resources: + requests: + cpu: 100m + memory: 250Mi + +# +# POSTGRES +# + +postgres: + enabled: false + +# +# MINIO +# + +minio: + enabled: false + +# +# CONTOUR +# + +contour: + enabled: false + +# +# COMMON +# + +common: + ingress: + albSSLRedirect: true + separateGrpcIngress: true + annotations: + # -- aws-load-balancer-controller v2.1 or higher is required - https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.1/ + # For EKS if using [ALB](https://kubernetes-sigs.github.io/aws-load-balancer-controller/guide/ingress/annotations/), these annotations are set + kubernetes.io/ingress.class: alb + alb.ingress.kubernetes.io/tags: service_instance=production + alb.ingress.kubernetes.io/scheme: internet-facing + # -- This is the certificate arn of the cert imported in AWS certificate manager. + alb.ingress.kubernetes.io/certificate-arn: "{{ .Values.userSettings.certificateArn }}" + alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]' + alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}' + # -- Instruct ALB Controller to not create multiple load balancers (and hence maintain a single endpoint for both GRPC and Http) + alb.ingress.kubernetes.io/group.name: flyte + separateGrpcIngressAnnotations: + alb.ingress.kubernetes.io/backend-protocol-version: HTTP2 + databaseSecret: + name: db-pass + secretManifest: + # -- Leave it empty if your secret already exists + # Else you can create your own secret object. You can use Kubernetes secrets, else you can configure external secrets + # For external secrets please install Necessary dependencies, like, of your choice + # - https://github.com/hashicorp/vault + # - https://github.com/godaddy/kubernetes-external-secrets + apiVersion: v1 + kind: Secret + metadata: + name: db-pass + type: Opaque + stringData: + # -- If using plain text you can provide the password here + pass.txt: "{{ .Values.userSettings.dbPassword }}" + +# ----------------------------------------------------- +# Core dependencies that should be configured for Flyte to work on any platform +# Specifically 2 - Storage (s3, gcs etc), Production RDBMS - Aurora, CloudSQL etc +# ------------------------------------------------------ +# +# STORAGE SETTINGS +# + +storage: + # -- Sets the storage type. Supported values are sandbox, s3, gcs and custom. + type: s3 + # -- bucketName defines the storage bucket flyte will use. Required for all types except for sandbox. + bucketName: "{{ .Values.userSettings.bucketName }}" + s3: + region: "{{ .Values.userSettings.accountRegion }}" + +db: + database: + port: 5432 + # -- Create a user called flyteadmin + username: flyteadmin + host: "{{ .Values.userSettings.rdsHost }}" + # -- Create a DB called flyteadmin (OR change the name here) + dbname: flyteadmin + passwordPath: /etc/db/pass.txt + +# +# CONFIGMAPS +# + +configmap: + adminServer: + server: + httpPort: 8088 + grpcPort: 8089 + security: + secure: false + useAuth: false + allowCors: true + allowedOrigins: + # Accepting all domains for Sandbox installation + - "*" + allowedHeaders: + - "Content-Type" + + task_resource_defaults: + task_resources: + defaults: + cpu: 1000m + memory: 1000Mi + storage: 1000Mi + limits: + storage: 2000Mi + + core: + propeller: + rawoutput-prefix: "s3://{{ .Values.userSettings.bucketName }}/" + workers: 40 + gc-interval: 12h + max-workflow-retries: 50 + kube-client-config: + qps: 100 + burst: 25 + timeout: 30s + queue: + sub-queue: + type: bucket + rate: 100 + capacity: 1000 + + enabled_plugins: + # -- Tasks specific configuration [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#GetConfig) + tasks: + # -- Plugins configuration, [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#TaskPluginConfig) + task-plugins: + # -- [Enabled Plugins](https://pkg.go.dev/github.com/lyft/flyteplugins/go/tasks/config#Config). Enable sagemaker*, athena if you install the backend + # plugins + enabled-plugins: + - container + - sidecar + - spark + - k8s-array + - pytorch + - athena + # - sagemaker_hyperparameter_tuning + # - sagemaker_custom_training + # - sagemaker_training + default-for-task-types: + container: container + sidecar: sidecar + spark: spark + container_array: k8s-array + pytorch: pytorch + hive: athena + # sagemaker_custom_training_task: sagemaker_custom_training + # sagemaker_custom_training_job_task: sagemaker_custom_training + + logger: + logger: + level: 5 + + # -- Section that configures how the Task logs are displayed on the UI. This has to be changed based on your actual logging provider. + # Refer to [structure](https://pkg.go.dev/github.com/lyft/flyteplugins/go/tasks/logs#LogConfig) to understand how to configure various + # logging engines + task_logs: + plugins: + logs: + kubernetes-enabled: false + # -- One option is to enable cloudwatch logging for EKS, update the region and log group accordingly + # You can even disable this + cloudwatch-enabled: true + # -- region where logs are hosted + cloudwatch-region: "{{ .Values.userSettings.accountRegion }}" + # -- cloudwatch log-group + cloudwatch-log-group: "{{ .Values.userSettings.logGroup }}" + +# ---------------------------------------------------------------- +# Optional Modules +# Flyte built extensions that enable various additional features in Flyte. +# All these features are optional, but are critical to run certain features +# ------------------------------------------------------------------------ + +# -- # Flyte uses a cloud hosted Cron scheduler to run workflows on a schedule. The following module is optional. Without, +# this module, you will not have scheduled launchplans/workflows. +workflow_scheduler: + enabled: false + config: + scheduler: + # -- This is configured to use Cloudwatch schedules as explained [here](https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/Create-CloudWatch-Events-Scheduled-Rule.html) + eventScheduler: + scheme: aws + region: "{{ .Values.userSettings.accountRegion }}" + scheduleRole: "arn:aws:iam::{{ .Values.userSettings.accountNumber }}:role/flyte_cron_scheduler_role" + targetName: "arn:aws:sqs:{{ .Values.userSettings.accountRegion }}:{{ .Values.userSettings.accountNumber }}:flyte-helm-test-cron-scheduler-queue" + scheduleNamePrefix: flyte + workflowExecutor: + scheme: aws + region: "{{ .Values.userSettings.accountRegion }}" + scheduleQueueName: flyte-helm-test-cron-scheduler-queue + accountId: "{{ .Values.userSettings.accountNumber }}" + reconnectAttempts: 10 + reconnectDelaySeconds: 30 + +# -- +# Workflow notifications module is an optional dependency. Flyte uses cloud native pub-sub systems to notify users of +# various events in their workflows +workflow_notifications: + enabled: false + config: + notifications: + type: aws + region: "{{ .Values.userSettings.accountRegion }}" + publisher: + topicName: "arn:aws:sns:{{ .Values.userSettings.accountRegion }}:{{ .Values.userSettings.accountNumber }}:flyte-notifications-topic" + processor: + queueName: flyte-notifications-queue + accountId: "{{ .Values.userSettings.accountNumber }}" + emailer: + subject: "Flyte: {{ project }}/{{ domain }}/{{ launch_plan.name }} has '{{ phase }}'" + sender: "flyte@example.com" + body: | + Execution {{ workflow.project }}/{{ workflow.domain }}/{{ workflow.name }}/{{ name }} has {{ phase }}. + Details: https://flyte.example.com/console/projects/{{ project }}/domains/{{ domain }}/executions/{{ name }}. + {{ error }} + +# -- Configuration for the Cluster resource manager component. This is an optional component, that enables automatic +# cluster configuration. This is useful to set default quotas, manage namespaces etc that map to a project/domain +cluster_resource_manager: + # -- Enables the Cluster resource manager component + enabled: true + config: + cluster_resources: + customData: + - production: + - projectQuotaCpu: + value: "5" + - projectQuotaMemory: + value: "4000Mi" + - defaultIamRole: + value: "arn:aws:iam::{{ .Values.userSettings.accountNumber }}:role/flyte-user-role" + - staging: + - projectQuotaCpu: + value: "2" + - projectQuotaMemory: + value: "3000Mi" + - defaultIamRole: + value: "arn:aws:iam::{{ .Values.userSettings.accountNumber }}:role/flyte-user-role" + - development: + - projectQuotaCpu: + value: "4" + - projectQuotaMemory: + value: "3000Mi" + - defaultIamRole: + value: "arn:aws:iam::{{ .Values.userSettings.accountNumber }}:role/flyte-user-role" + + templates: + # -- Template for namespaces resources + - key: aa_namespace + value: | + apiVersion: v1 + kind: Namespace + metadata: + name: {{ namespace }} + spec: + finalizers: + - kubernetes + + # -- Patch default service account + - key: aab_default_service_account + value: | + apiVersion: v1 + kind: ServiceAccount + metadata: + name: default + namespace: {{ namespace }} + annotations: + eks.amazonaws.com/role-arn: {{ defaultIamRole }} + + - key: ab_project_resource_quota + value: | + apiVersion: v1 + kind: ResourceQuota + metadata: + name: project-quota + namespace: {{ namespace }} + spec: + hard: + limits.cpu: {{ projectQuotaCpu }} + limits.memory: {{ projectQuotaMemory }} + + # -- Template for spark role + - key: ad_spark_role + value: | + apiVersion: rbac.authorization.k8s.io/v1beta1 + kind: Role + metadata: + name: spark-role + namespace: {{ namespace }} + rules: + - apiGroups: + - "" + resources: + - pods + verbs: + - '*' + - apiGroups: + - "" + resources: + - services + verbs: + - '*' + - apiGroups: + - "" + resources: + - configmaps + verbs: + - '*' + + # -- Template for spark SA + - key: ae_spark_service_account + value: | + apiVersion: v1 + kind: ServiceAccount + metadata: + name: spark + namespace: {{ namespace }} + annotations: + eks.amazonaws.com/role-arn: {{ defaultIamRole }} + + # -- Template for spark role binding + - key: af_spark_role_binding + value: | + apiVersion: rbac.authorization.k8s.io/v1beta1 + kind: RoleBinding + metadata: + name: spark-role-binding + namespace: {{ namespace }} + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: spark-role + subjects: + - kind: ServiceAccount + name: spark + namespace: {{ namespace }} + +# +# SPARKOPERATOR +# + +sparkoperator: + enabled: true + +# ----------------- +# -- Training on AWS Sagemaker using AWS Sagemaker operator. To actually install the operator, please follow instructions [here](https://github.com/aws/amazon-sagemaker-operator-for-k8s/tree/master/hack/charts/installer/rolebased) +# Use the config section here to just enable sagemaker plugin in Flyte, after you have installed the operator using the information +sagemaker: + enabled: false +# plugin_config: +# plugins: +# sagemaker: +# roleArn: arn:aws:iam::{{ .Values.userSettings.accountNumber }}:role/ +# region: {{ .Values.userSettings.accountRegion }} +# --------------------------- +# Flink jobs using the Flink Operator +# diff --git a/charts/flyte-core/values-gcp.yaml b/charts/flyte-core/values-gcp.yaml new file mode 100644 index 0000000000..7fa40b2a3f --- /dev/null +++ b/charts/flyte-core/values-gcp.yaml @@ -0,0 +1,48 @@ +# +# COMMON +# + +common: + databaseSecret: {} + ingress: {} + +# ----------------------------------------------------- +# Core dependencies that should be configured for Flyte to work on any platform +# Specifically 2 - Storage (s3, gcs etc), Production RDBMS - Aurora, CloudSQL etc +# ------------------------------------------------------ +# +# STORAGE SETTINGS +# + +storage: + # -- Sets the storage type. Supported values are sandbox, s3, gcs and custom. + type: gcs + # -- bucketName defines the storage bucket flyte will use. Required for all types except for sandbox. + bucketName: + gcs: + projectId: + +# +# CONFIGMAPS +# + +configmap: + remoteData: + remoteData: + scheme: "gcp" + + tasks: + max-plugin-phase-versions: 1000000 + task-plugins: + enabled-plugins: + - container + - sidecar + - spark + - k8s-array + - pytorch + default-for-task-types: + container: container + sidecar: sidecar + spark: spark + container_array: k8s-array + pytorch: pytorch diff --git a/charts/flyte-core/values.yaml b/charts/flyte-core/values.yaml new file mode 100755 index 0000000000..ad6a65b04b --- /dev/null +++ b/charts/flyte-core/values.yaml @@ -0,0 +1,634 @@ + +# --------------------------------------------------------------------- +# Core System settings +# This section consists of Core components of Flyte and their deployment +# settings. This includes FlyteAdmin service, Datacatalog, FlytePropeller and +# Flyteconsole +# --------------------------------------------------------------------- + +# +# FLYTEADMIN SETTINGS +# + +flyteadmin: + # -- Replicas count for Flyteadmin deployment + replicaCount: 1 + image: + # -- Docker image for Flyteadmin deployment + repository: cr.flyte.org/flyteorg/flyteadmin + tag: v0.6.16 # FLYTEADMIN_TAG + pullPolicy: IfNotPresent + # -- Default resources requests and limits for Flyteadmin deployment + resources: + limits: + cpu: 250m + ephemeral-storage: 100Mi + memory: 500Mi + requests: + cpu: 10m + ephemeral-storage: 50Mi + memory: 50Mi + # -- Default regex string for searching configuration files + configPath: /etc/flyte/config/*.yaml + # -- Initial projects to create + initialProjects: + - flytesnacks + - flytetester + - flyteexamples + # -- Service settings for Flyteadmin + service: + annotations: + projectcontour.io/upstream-protocol.h2c: grpc + type: ClusterIP + loadBalancerSourceRanges: [] + # -- Configuration for service accounts for FlyteAdmin + serviceAccount: + # -- Should a service account be created for flyteadmin + create: true + # -- Annotations for ServiceAccount attached to Flyteadmin pods + annotations: {} + # -- ImapgePullSecrets to automatically assign to the service account + imagePullSecrets: {} + # -- Annotations for Flyteadmin pods + podAnnotations: {} + # -- nodeSelector for Flyteadmin deployment + nodeSelector: {} + # -- tolerations for Flyteadmin deployment + tolerations: [] + # -- affinity for Flyteadmin deployment + affinity: {} + secrets: {} + +# +# DATACATALOG SETTINGS +# + +datacatalog: + # -- Replicas count for Datacatalog deployment + replicaCount: 1 + image: + # -- Docker image for Datacatalog deployment + repository: cr.flyte.org/flyteorg/datacatalog + tag: v0.3.6 # DATACATALOG_TAG + pullPolicy: IfNotPresent + # -- Default resources requests and limits for Datacatalog deployment + resources: + limits: + cpu: 500m + ephemeral-storage: 100Mi + memory: 500Mi + requests: + cpu: 10m + ephemeral-storage: 50Mi + memory: 50Mi + # -- Default regex string for searching configuration files + configPath: /etc/datacatalog/config/*.yaml + # -- Service settings for Datacatalog + service: + annotations: + projectcontour.io/upstream-protocol.h2c: grpc + type: NodePort + # -- Configuration for service accounts for Datacatalog + serviceAccount: + # -- Should a service account be created for Datacatalog + create: true + # -- Annotations for ServiceAccount attached to Datacatalog pods + annotations: {} + # -- ImapgePullSecrets to automatically assign to the service account + imagePullSecrets: {} + # -- Annotations for Datacatalog pods + podAnnotations: {} + # -- nodeSelector for Datacatalog deployment + nodeSelector: {} + # -- tolerations for Datacatalog deployment + tolerations: [] + # -- affinity for Datacatalog deployment + affinity: {} + +# +# FLYTEPROPELLER SETTINGS +# + +flytepropeller: + # -- Replicas count for Flytepropeller deployment + replicaCount: 1 + image: + # -- Docker image for Flytepropeller deployment + repository: cr.flyte.org/flyteorg/flytepropeller + tag: v0.13.4 # FLYTEPROPELLER_TAG + pullPolicy: IfNotPresent + # -- Default resources requests and limits for Flytepropeller deployment + resources: + limits: + cpu: 200m + ephemeral-storage: 100Mi + memory: 200Mi + requests: + cpu: 10m + ephemeral-storage: 50Mi + memory: 50Mi + cacheSizeMbs: 0 + # -- Default regex string for searching configuration files + configPath: /etc/flyte/config/*.yaml + + # -- Configuration for service accounts for FlytePropeller + serviceAccount: + # -- Should a service account be created for FlytePropeller + create: true + # -- Annotations for ServiceAccount attached to FlytePropeller pods + annotations: {} + # -- ImapgePullSecrets to automatically assign to the service account + imagePullSecrets: {} + # -- Annotations for Flytepropeller pods + podAnnotations: {} + # -- nodeSelector for Flytepropeller deployment + nodeSelector: {} + # -- tolerations for Flytepropeller deployment + tolerations: [] + # -- affinity for Flytepropeller deployment + affinity: {} + +# +# FLYTECONSOLE SETTINGS +# + +flyteconsole: + # -- Replicas count for Flyteconsole deployment + replicaCount: 1 + image: + # -- Docker image for Flyteconsole deployment + repository: cr.flyte.org/flyteorg/flyteconsole + tag: v0.20.1 # FLYTECONSOLE_TAG + pullPolicy: IfNotPresent + # -- Default resources requests and limits for Flyteconsole deployment + resources: + limits: + cpu: 500m + memory: 250Mi + requests: + cpu: 10m + memory: 50Mi + # -- Service settings for Flyteconsole + service: + annotations: {} + type: ClusterIP + # -- Annotations for Flyteconsole pods + podAnnotations: {} + # -- nodeSelector for Flyteconsole deployment + nodeSelector: {} + # -- tolerations for Flyteconsole deployment + tolerations: [] + # -- affinity for Flyteconsole deployment + affinity: {} + +# +# WEBHOOK SETTINGS +# + +webhook: + # -- enable or disable secrets webhook + enabled: true + # -- Configuration for service accounts for the webhook + serviceAccount: + # -- Should a service account be created for the webhook + create: true + # -- Annotations for ServiceAccount attached to the webhook + annotations: {} + # -- ImapgePullSecrets to automatically assign to the service account + imagePullSecrets: {} + # -- Service settings for the webhook + service: + annotations: + projectcontour.io/upstream-protocol.h2c: grpc + type: ClusterIP + + +common: + databaseSecret: + # -- Specify name of K8s Secret which contains Database password. Leave it empty if you don't need this Secret + name: "db-pass" + # -- Specify your Secret (with sensitive data) or pseudo-manifest (without sensitive data). See https://github.com/godaddy/kubernetes-external-secrets + secretManifest: {} + ingress: + # --- Enable or disable creating Ingress for Flyte. Relevant to disable when using e.g. Istio as ingress controller. + enabled: true + # --- Enable or disable HMR route to flyteconsole. This is useful only for frontend development. + webpackHMR: false + # --- separateGrpcIngress puts GRPC routes into a separate ingress if true. Required for certain ingress controllers like nginx. + separateGrpcIngress: false + # --- Extra Ingress annotations applied only to the GRPC ingress. Only makes sense if `separateGrpcIngress` is enabled. + separateGrpcIngressAnnotations: + nginx.ingress.kubernetes.io/backend-protocol: "GRPC" + # --- Ingress annotations applied to both HTTP and GRPC ingresses. + annotations: {} + # --- albSSLRedirect adds a special route for ssl redirect. Only useful in combination with the AWS LoadBalancer Controller. + albSSLRedirect: false + # --- Ingress hostname + # host: + tls: + enabled: false + flyteNamespaceTemplate: + # --- Enable or disable creating Flyte namespace in template. Enable when using helm as template-engine only. Disable when using `helm install ...`. + enabled: false + +# ----------------------------------------------------- +# Core dependencies that should be configured for Flyte to work on any platform +# Specifically 2 - Storage (s3, gcs etc), Production RDBMS - Aurora, CloudSQL etc +# ------------------------------------------------------ +# +# STORAGE SETTINGS +# + +storage: + # -- Sets the storage type. Supported values are sandbox, s3, gcs and custom. + type: sandbox + # -- bucketName defines the storage bucket flyte will use. Required for all types except for sandbox. + bucketName: my-s3-bucket + # -- settings for storage type s3 + s3: + region: us-east-1 + # -- settings for storage type gcs + gcs: + # -- GCP project ID. Required for storage type gcs. + # projectId: + # -- Settings for storage type custom. See https://github:com/graymeta/stow for supported storage providers/settings. + custom: {} + +# Database configuration +db: + database: + # port: 5432 + # username: postgres + host: postgres + # dbname: flyte + # passwordPath: /etc/db/pass.txt + # options: + +# -------------------------------------------------------------------- +# Specializing your deployment using configuration +# ------------------------------------------------------------------- +# +# CONFIGMAPS SETTINGS +# + +configmap: + # -- Configuration for Flyte console UI + console: + BASE_URL: /console + CONFIG_DIR: /etc/flyte/config + DISABLE_AUTH: "1" + + # -- Domains configuration for Flyte projects. This enables the specified number of domains across all projects in Flyte. + domain: + domains: + - id: development + name: development + - id: staging + name: staging + - id: production + name: production + + # -- FlyteAdmin server configuration + adminServer: + # Refer to the [server config](https://pkg.go.dev/github.com/lyft/flyteadmin@v0.3.37/pkg/config#ServerConfig). + server: + httpPort: 8088 + grpcPort: 8089 + security: + # -- Controls whether to serve requests over SSL/TLS. + secure: false + # -- Controls whether to enforce authentication. Follow the guide in https://docs.flyte.org/ on how to setup authentication. + useAuth: false + allowCors: true + allowedOrigins: + # Accepting all domains for Sandbox installation + - "*" + allowedHeaders: + - "Content-Type" + - "flyte-authorization" + # Refer to the full [structure](https://pkg.go.dev/github.com/lyft/flyteadmin@v0.3.37/pkg/runtime/interfaces#ApplicationConfig) for documentation. + flyteadmin: + roleNameKey: "iam.amazonaws.com/role" + profilerPort: 10254 + metricsScope: "flyte:" + metadataStoragePrefix: + - "metadata" + - "admin" + eventVersion: 1 + testing: + host: http://flyteadmin + # -- Authentication configuration + auth: + authorizedUris: + # This should point at your public http Uri. + - https://localhost:30081 + # This will be used by internal services in the same namespace as flyteadmin + - http://flyteadmin:80 + # This will be used by internal services in the same cluster but different namespaces + - http://flyteadmin.flyte.svc.cluster.local:80 + + # Controls app authentication config + appAuth: + thirdPartyConfig: + flyteClient: + clientId: flytectl + redirectUri: http://localhost:53593/callback + scopes: + - offline + - all + # Controls user authentication + userAuth: + openId: + baseUrl: https://accounts.google.com + scopes: + - profile + - openid + clientId: 657465813211-6eog7ek7li5k7i7fvgv2921075063hpe.apps.googleusercontent.com + + # -- Datacatalog server config + datacatalogServer: + datacatalog: + storage-prefix: metadata/datacatalog + metrics-scope: datacatalog + profiler-port: 10254 + application: + grpcPort: 8089 + httpPort: 8080 + grpcServerReflection: true + + # -- Task default resources configuration + # Refer to the full [structure](https://pkg.go.dev/github.com/lyft/flyteadmin@v0.3.37/pkg/runtime/interfaces#TaskResourceConfiguration). + task_resource_defaults: + # -- Task default resources parameters + task_resources: + defaults: + cpu: 100m + memory: 100Mi + storage: 5Mi + limits: + cpu: 2 + memory: 1Gi + storage: 20Mi + gpu: 1 + + # -- Admin Client configuration [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/subworkflow/launchplan#AdminConfig) + admin: + event: + type: admin + rate: 500 + capacity: 1000 + admin: + endpoint: flyteadmin:81 + insecure: true + clientId: flytepropeller + clientSecretLocation: /etc/secrets/client_secret + + # -- Catalog Client configuration [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/catalog#Config) + # Additional advanced Catalog configuration [here](https://pkg.go.dev/github.com/lyft/flyteplugins/go/tasks/pluginmachinery/catalog#Config) + catalog: + catalog-cache: + endpoint: datacatalog:89 + type: datacatalog + insecure: true + + # -- Copilot configuration + copilot: + plugins: + k8s: + # -- Structure documented [here](https://pkg.go.dev/github.com/lyft/flyteplugins@v0.5.28/go/tasks/pluginmachinery/flytek8s/config#FlyteCoPilotConfig) + co-pilot: + name: flyte-copilot- + image: cr.flyte.org/lyft/flyteplugins/flytecopilot:dc4bdbd61cac88a39a5ff43e40f026bdbc2c78a2 + start-timeout: 30s + + # -- Core propeller configuration + core: + # -- follows the structure specified [here](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/config). + propeller: + rawoutput-prefix: s3://my-s3-bucket/ + metadata-prefix: metadata/propeller + workers: 4 + max-workflow-retries: 30 + workflow-reeval-duration: 30s + downstream-eval-duration: 30s + limit-namespace: "all" + prof-port: 10254 + metrics-prefix: flyte + enable-admin-launcher: true + leader-election: + lock-config-map: + name: propeller-leader + namespace: flyte + enabled: true + lease-duration: 15s + renew-deadline: 10s + retry-period: 2s + queue: + type: batch + batching-interval: 2s + batch-size: -1 + queue: + type: maxof + rate: 100 + capacity: 1000 + base-delay: 5s + max-delay: 120s + sub-queue: + type: bucket + rate: 10 + capacity: 100 + webhook: + certDir: /etc/webhook/certs + serviceName: flyte-pod-webhook + # -- For Workflow store use configuration [here](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/workflowstore#Config) + # + + enabled_plugins: + # -- Tasks specific configuration [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#GetConfig) + tasks: + # -- Plugins configuration, [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#TaskPluginConfig) + task-plugins: + # -- [Enabled Plugins](https://pkg.go.dev/github.com/lyft/flyteplugins/go/tasks/config#Config). Enable sagemaker*, athena if you install the backend + # plugins + enabled-plugins: + - container + - sidecar + - k8s-array + default-for-task-types: + container: container + sidecar: sidecar + container_array: k8s-array + + # -- Kubernetes specific Flyte configuration + k8s: + plugins: + # -- Configuration section for all K8s specific plugins [Configuration structure](https://pkg.go.dev/github.com/lyft/flyteplugins/go/tasks/pluginmachinery/flytek8s/config) + k8s: + default-env-vars: [] + # DEFAULT_ENV_VAR: VALUE + default-cpus: 100m + default-memory: 100Mi + + # -- Logger configuration + logger: + logger: + show-source: true + level: 4 + + remoteData: + remoteData: + region: "us-east-1" + scheme: "local" + signedUrls: + durationMinutes: 3 + + # -- Resource manager configuration + resource_manager: + # -- resource manager configuration + propeller: + resourcemanager: + type: + + resourceMaxQuota: 10000 + redis: + hostPath: redis-resource-manager:6379 + hostKey: mypassword + + # -- Section that configures how the Task logs are displayed on the UI. This has to be changed based on your actual logging provider. + # Refer to [structure](https://pkg.go.dev/github.com/lyft/flyteplugins/go/tasks/logs#LogConfig) to understand how to configure various + # logging engines + task_logs: + plugins: + logs: + kubernetes-enabled: false + # -- One option is to enable cloudwatch logging for EKS, update the region and log group accordingly + cloudwatch-enabled: false + +# ---------------------------------------------------------------- +# Optional Modules +# Flyte built extensions that enable various additional features in Flyte. +# All these features are optional, but are critical to run certain features +# ------------------------------------------------------------------------ + +# -- **Optional Component** +# Flyte uses a cloud hosted Cron scheduler to run workflows on a schedule. The following module is optional. Without, +# this module, you will not have scheduled launchplans / workflows. +# Docs: https://docs.flyte.org/en/latest/howto/enable_and_use_schedules.html#setting-up-scheduled-workflows +workflow_scheduler: + enabled: false + config: {} + +# -- **Optional Component** +# Workflow notifications module is an optional dependency. Flyte uses cloud native pub-sub systems to notify users of +# various events in their workflows +workflow_notifications: + enabled: false + config: {} + + + +# -- Configuration for the Cluster resource manager component. This is an optional component, that enables automatic +# cluster configuration. This is useful to set default quotas, manage namespaces etc that map to a project/domain +cluster_resource_manager: + # -- Enables the Cluster resource manager component + enabled: true + # -- Configmap for ClusterResource parameters + config: + # -- ClusterResource parameters + # Refer to the [structure](https://pkg.go.dev/github.com/lyft/flyteadmin@v0.3.37/pkg/runtime/interfaces#ClusterResourceConfig) to customize. + cluster_resources: + refreshInterval: 5m + templatePath: "/etc/flyte/clusterresource/templates" + customData: + - production: + - projectQuotaCpu: + value: "5" + - projectQuotaMemory: + value: "4000Mi" + - defaultIamRole: + value: "" + - staging: + - projectQuotaCpu: + value: "2" + - projectQuotaMemory: + value: "3000Mi" + - defaultIamRole: + value: "" + - development: + - projectQuotaCpu: + value: "4" + - projectQuotaMemory: + value: "3000Mi" + - defaultIamRole: + value: "" + refresh: 5m + + # -- Resource templates that should be applied + templates: + # -- Template for namespaces resources + - key: aa_namespace + value: | + apiVersion: v1 + kind: Namespace + metadata: + name: {{ namespace }} + spec: + finalizers: + - kubernetes + - key: ab_project_resource_quota + value: | + apiVersion: v1 + kind: ResourceQuota + metadata: + name: project-quota + namespace: {{ namespace }} + spec: + hard: + limits.cpu: {{ projectQuotaCpu }} + limits.memory: {{ projectQuotaMemory }} + +# -------------------------------------------------------- +# Optional Plugins +# -------------------------------------------------------- + +# -- Optional: Spark Plugin using the Spark Operator +sparkoperator: + # --- enable or disable Sparkoperator deployment installation + enabled: false + # -- Spark plugin configuration + plugin_config: + plugins: + spark: + # -- Spark default configuration + spark-config-default: + # We override the default credentials chain provider for Hadoop so that + # it can use the serviceAccount based IAM role or ec2 metadata based. + # This is more in line with how AWS works + - spark.hadoop.fs.s3a.aws.credentials.provider: "com.amazonaws.auth.DefaultAWSCredentialsProviderChain" + - spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version: "2" + - spark.kubernetes.allocation.batch.size: "50" + - spark.hadoop.fs.s3a.acl.default: "BucketOwnerFullControl" + - spark.hadoop.fs.s3n.impl: "org.apache.hadoop.fs.s3a.S3AFileSystem" + - spark.hadoop.fs.AbstractFileSystem.s3n.impl: "org.apache.hadoop.fs.s3a.S3A" + - spark.hadoop.fs.s3.impl: "org.apache.hadoop.fs.s3a.S3AFileSystem" + - spark.hadoop.fs.AbstractFileSystem.s3.impl: "org.apache.hadoop.fs.s3a.S3A" + - spark.hadoop.fs.s3a.impl: "org.apache.hadoop.fs.s3a.S3AFileSystem" + - spark.hadoop.fs.AbstractFileSystem.s3a.impl: "org.apache.hadoop.fs.s3a.S3A" + - spark.hadoop.fs.s3a.multipart.threshold: "536870912" + - spark.blacklist.enabled: "true" + - spark.blacklist.timeout: "5m" + - spark.task.maxfailures: "8" + + +# ----------------- +# -- Training on AWS Sagemaker using AWS Sagemaker operator. To actually install the operator, please follow instructions [here](https://github.com/aws/amazon-sagemaker-operator-for-k8s/tree/master/hack/charts/installer/rolebased) +# Use the config section here to just enable sagemaker plugin in Flyte, after you have installed the operator using the information +sagemaker: + enabled: false + plugin_config: + plugins: + sagemaker: + roleArn: + region: + + diff --git a/charts/flyte/README.md b/charts/flyte/README.md index 45dd6c144c..f2f35053bf 100644 --- a/charts/flyte/README.md +++ b/charts/flyte/README.md @@ -12,32 +12,38 @@ A Helm chart for Flyte | https://googlecloudplatform.github.io/spark-on-k8s-operator | sparkoperator(spark-operator) | 1.0.6 | | https://kubernetes.github.io/dashboard/ | kubernetes-dashboard | 4.0.2 | -### SANDBOX INSTALLATION: +### Flyte INSTALLATION: - [Install helm 3](https://helm.sh/docs/intro/install/) -- Fetch chart dependencies `helm dep up` -- Install Flyte sandbox: +- Fetch chart dependencies `` +- Install Flyte: ```bash -cd helm -helm install -n flyte -f values-sandbox.yaml --create-namespace flyte . +helm repo add flyte https://flyteorg.github.io/flyte +helm install -n flyte -f values-eks.yaml --create-namespace flyte flyte/flyte-core ``` -Customize your installation by changing settings in `values-sandbox.yaml`. +Customize your installation by changing settings in `values-eks.yaml`. You can use the helm diff plugin to review any value changes you've made to your values: ```bash helm plugin install https://github.com/databus23/helm-diff -helm diff upgrade -f values-sandbox.yaml flyte . +helm diff upgrade -f values-eks.yaml flyte flyte/flyte-core ``` Then apply your changes: ```bash -helm upgrade -f values-sandbox.yaml flyte . +helm upgrade -f values-eks.yaml flyte flyte/flyte-core +``` + +Install ingress controller (By default Flyte helm chart have contour ingress resource) +```bash +helm repo add bitnami https://charts.bitnami.com/bitnami +helm install gateway bitnami/contour -n flyte ``` #### Alternative: Generate raw kubernetes yaml with helm template -- `helm template --name-template=flyte-sandbox . -n flyte -f values-sandbox.yaml > flyte_generated_sandbox.yaml` -- Deploy the manifest `kubectl apply -f flyte_generated_sandbox.yaml` +- `helm template --name-template=flyte-eks . -n flyte -f values-eks.yaml > flyte_generated_eks.yaml` +- Deploy the manifest `kubectl apply -f flyte_generated_eks.yaml` - When all pods are running - run end2end tests: `kubectl apply -f ../end2end/tests/endtoend.yaml` - Get flyte host `minikube service contour -n heptio-contour --url`. And then visit `http:///console` diff --git a/script/generate_helm.sh b/script/generate_helm.sh index 923232d9e0..f30414b423 100644 --- a/script/generate_helm.sh +++ b/script/generate_helm.sh @@ -22,6 +22,7 @@ then fi ${GOPATH:-~/go}/bin/helm-docs -t ${DIR}/../charts/flyte/README.md.gotmpl ${DIR}/../charts/flyte/ +${GOPATH:-~/go}/bin/helm-docs -t ${DIR}/../charts/flyte-core/README.md.gotmpl ${DIR}/../charts/flyte-core/ # This section is used by GitHub workflow to ensure that the generation step was run if [ -n "$DELTA_CHECK" ]; then diff --git a/script/prepare_artifacts.sh b/script/prepare_artifacts.sh index 7afeb13fad..e0c1dd48c4 100644 --- a/script/prepare_artifacts.sh +++ b/script/prepare_artifacts.sh @@ -16,4 +16,8 @@ done grep -rlZ "version:[^P]*# VERSION" ./charts/flyte/Chart.yaml | xargs -0 sed -i "s/version:[^P]*# VERSION/version: ${VERSION} # VERSION/g" sed "s/v0.1.10/${VERSION}/g" ./charts/flyte/README.md > temp.txt && mv temp.txt ./charts/flyte/README.md + +grep -rlZ "version:[^P]*# VERSION" ./charts/flyte-core/Chart.yaml | xargs -0 sed -i "s/version:[^P]*# VERSION/version: ${VERSION} # VERSION/g" +sed "s/v0.1.10/${VERSION}/g" ./charts/flyte-core/README.md > temp.txt && mv temp.txt ./charts/flyte-core/README.md + helm dep update ./charts/flyte