forked from flyteorg/flyte
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Update Flyte components (flyteorg#1927)
* Update Flyte Components Signed-off-by: Flyte-Bot <[email protected]> * Add changelog Signed-off-by: Haytham Abuelfutuh <[email protected]> * Bump flytetools SHA to 125f6b21c3ff9336d9fb98a3e35fc7c6469d8d1c Signed-off-by: Eduardo Apolinario <[email protected]> Co-authored-by: EngHabu <[email protected]> Co-authored-by: Haytham Abuelfutuh <[email protected]> Co-authored-by: Eduardo Apolinario <[email protected]>
- Loading branch information
1 parent
e948ac0
commit 992c466
Showing
21 changed files
with
380 additions
and
226 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,62 @@ | ||
# 0.18.2 Release ChangeLog | ||
|
||
[Closed Issues](https://github.com/flyteorg/flyte/issues?q=is%3Aissue+milestone%3A0.18.2+is%3Aclosed) | ||
|
||
## UX | ||
* Added advanced options to launch form | ||
* Added support for all tasks-types (task execution view) | ||
* Replaced execution id's with node id's on execution list-view | ||
* Fixed bug with some properties not being repopulated on relaunch | ||
* minor fixes | ||
|
||
### FlyteKit | ||
See the flytekit [0.25.0 release notes](https://github.com/flyteorg/flytekit/releases/tag/v0.25.0) for the full list of changes. Here are some of the highlights: | ||
|
||
* Improved support for tasks that [run shell scripts](https://github.com/flyteorg/flytekit/pull/747) | ||
* Support for more types in dataclasses: | ||
* [enums](https://github.com/flyteorg/flytekit/pull/753) | ||
* [FlyteFile](https://github.com/flyteorg/flytekit/pull/725) and [FlyteSchema](https://github.com/flyteorg/flytekit/pull/722) | ||
* flyteremote improvements, including: | ||
* [Access to raw inputs and outputs](https://github.com/flyteorg/flytekit/pull/675) | ||
* [Ability to serialize tasks containing arbitrary images](https://github.com/flyteorg/flytekit/pull/733) | ||
* [Improved UX for navigation of subworkflows and launchplans](https://github.com/flyteorg/flytekit/pull/751) | ||
* [Support for FlytePickle](https://github.com/flyteorg/flytekit/pull/764) | ||
|
||
## System | ||
* Various stability fixes. | ||
* New docker image tags! | ||
|
||
In addition to component-specific versions released from each of the flyte repositories (e.g. flytepropeller:v0.16.5), new images will be re-tagged and pushed that match the flyte release version (e.g. the upcoming flytepropeller-release:v0.18.2). This makes it easier to make sure all your deployments are on the same version to ensure best compatibility. | ||
* Helm changes | ||
* [flyte-core](https://artifacthub.io/packages/helm/flyte/flyte-core) helm chart has reached release preview and can be leveraged to install your cloud(AWS/GCP) deployments of flyte. | ||
* Going forward flyte-core will install flyte native scheduler, For AWS backword compatibility you need to define `workflow_schedule.type` to `aws`. (https://github.com/flyteorg/flyte/pull/1896) | ||
* [flyte](https://artifacthub.io/packages/helm/flyte/flyte) helm chart has been refactored to depend on flyte-core helm chart and install additional dependencies to continue to provide a sandboxed installation of flyte. | ||
|
||
**Migration Notes** | ||
|
||
As part of this move, ``flyte`` helm chart is becoming the canonical sandbox cluster. It comes with all external resources needed to fully standup a Flyte cluster. If you have previously been using this chart to deploy flyte on your cloud providers, there will be changes you need to do to migrate: | ||
* If you have your own ``myvalues.yaml``, you will need to add another nesting level under ``flyte:`` for the sections that are now managed through ``flyte-core``. For example: | ||
```yaml | ||
configmaps: | ||
... | ||
flyteadmin: | ||
... | ||
minio: | ||
... | ||
countour: | ||
... | ||
``` | ||
to: | ||
```yaml | ||
flyte: | ||
configmaps: | ||
... | ||
flyteadmin: | ||
... | ||
minio: | ||
... | ||
countour: | ||
... | ||
``` | ||
* Alternatively, if you do not have any dependency on external flyte depdencies, you can keep your ``myvalues.yaml`` and switch to using ``flyte-core`` helm chart directly with no changes. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -73,8 +73,8 @@ helm install gateway bitnami/contour -n flyte | |
| configmap.adminServer.server.security.useAuth | bool | `false` | Controls whether to enforce authentication. Follow the guide in https://docs.flyte.org/ on how to setup authentication. | | ||
| configmap.catalog | object | `{"catalog-cache":{"endpoint":"datacatalog:89","insecure":true,"type":"datacatalog"}}` | Catalog Client configuration [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/catalog#Config) Additional advanced Catalog configuration [here](https://pkg.go.dev/github.com/lyft/flyteplugins/go/tasks/pluginmachinery/catalog#Config) | | ||
| configmap.console | object | `{"BASE_URL":"/console","CONFIG_DIR":"/etc/flyte/config","DISABLE_AUTH":"1"}` | Configuration for Flyte console UI | | ||
| configmap.copilot | object | `{"plugins":{"k8s":{"co-pilot":{"image":"cr.flyte.org/flyteorg/flytecopilot:v0.0.22","name":"flyte-copilot-","start-timeout":"30s"}}}}` | Copilot configuration | | ||
| configmap.copilot.plugins.k8s.co-pilot | object | `{"image":"cr.flyte.org/flyteorg/flytecopilot:v0.0.22","name":"flyte-copilot-","start-timeout":"30s"}` | Structure documented [here](https://pkg.go.dev/github.com/lyft/[email protected]/go/tasks/pluginmachinery/flytek8s/config#FlyteCoPilotConfig) | | ||
| configmap.copilot | object | `{"plugins":{"k8s":{"co-pilot":{"image":"cr.flyte.org/flyteorg/flytecopilot:v0.0.24","name":"flyte-copilot-","start-timeout":"30s"}}}}` | Copilot configuration | | ||
| configmap.copilot.plugins.k8s.co-pilot | object | `{"image":"cr.flyte.org/flyteorg/flytecopilot:v0.0.24","name":"flyte-copilot-","start-timeout":"30s"}` | Structure documented [here](https://pkg.go.dev/github.com/lyft/[email protected]/go/tasks/pluginmachinery/flytek8s/config#FlyteCoPilotConfig) | | ||
| configmap.core | object | `{"propeller":{"downstream-eval-duration":"30s","enable-admin-launcher":true,"leader-election":{"enabled":true,"lease-duration":"15s","lock-config-map":{"name":"propeller-leader","namespace":"flyte"},"renew-deadline":"10s","retry-period":"2s"},"limit-namespace":"all","max-workflow-retries":30,"metadata-prefix":"metadata/propeller","metrics-prefix":"flyte","prof-port":10254,"queue":{"batch-size":-1,"batching-interval":"2s","queue":{"base-delay":"5s","capacity":1000,"max-delay":"120s","rate":100,"type":"maxof"},"sub-queue":{"capacity":100,"rate":10,"type":"bucket"},"type":"batch"},"rawoutput-prefix":"s3://my-s3-bucket/","workers":4,"workflow-reeval-duration":"30s"},"webhook":{"certDir":"/etc/webhook/certs","serviceName":"flyte-pod-webhook"}}` | Core propeller configuration | | ||
| configmap.core.propeller | object | `{"downstream-eval-duration":"30s","enable-admin-launcher":true,"leader-election":{"enabled":true,"lease-duration":"15s","lock-config-map":{"name":"propeller-leader","namespace":"flyte"},"renew-deadline":"10s","retry-period":"2s"},"limit-namespace":"all","max-workflow-retries":30,"metadata-prefix":"metadata/propeller","metrics-prefix":"flyte","prof-port":10254,"queue":{"batch-size":-1,"batching-interval":"2s","queue":{"base-delay":"5s","capacity":1000,"max-delay":"120s","rate":100,"type":"maxof"},"sub-queue":{"capacity":100,"rate":10,"type":"bucket"},"type":"batch"},"rawoutput-prefix":"s3://my-s3-bucket/","workers":4,"workflow-reeval-duration":"30s"}` | follows the structure specified [here](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/config). | | ||
| configmap.datacatalogServer | object | `{"application":{"grpcPort":8089,"grpcServerReflection":true,"httpPort":8080},"datacatalog":{"metrics-scope":"datacatalog","profiler-port":10254,"storage-prefix":"metadata/datacatalog"}}` | Datacatalog server config | | ||
|
@@ -96,12 +96,12 @@ helm install gateway bitnami/contour -n flyte | |
| configmap.task_logs.plugins.logs.cloudwatch-enabled | bool | `false` | One option is to enable cloudwatch logging for EKS, update the region and log group accordingly | | ||
| configmap.task_resource_defaults | object | `{"task_resources":{"defaults":{"cpu":"100m","memory":"100Mi","storage":"5Mi"},"limits":{"cpu":2,"gpu":1,"memory":"1Gi","storage":"20Mi"}}}` | Task default resources configuration Refer to the full [structure](https://pkg.go.dev/github.com/lyft/[email protected]/pkg/runtime/interfaces#TaskResourceConfiguration). | | ||
| configmap.task_resource_defaults.task_resources | object | `{"defaults":{"cpu":"100m","memory":"100Mi","storage":"5Mi"},"limits":{"cpu":2,"gpu":1,"memory":"1Gi","storage":"20Mi"}}` | Task default resources parameters | | ||
| datacatalog | object | `{"affinity":{},"configPath":"/etc/datacatalog/config/*.yaml","enabled":true,"image":{"pullPolicy":"IfNotPresent","repository":"cr.flyte.org/flyteorg/datacatalog","tag":"v0.3.16"},"nodeSelector":{},"podAnnotations":{},"replicaCount":1,"resources":{"limits":{"cpu":"500m","ephemeral-storage":"100Mi","memory":"500Mi"},"requests":{"cpu":"10m","ephemeral-storage":"50Mi","memory":"50Mi"}},"service":{"annotations":{"projectcontour.io/upstream-protocol.h2c":"grpc"},"type":"NodePort"},"serviceAccount":{"annotations":{},"create":true,"imagePullSecrets":{}},"tolerations":[]}` | DATACATALOG SETTINGS | | ||
| datacatalog | object | `{"affinity":{},"configPath":"/etc/datacatalog/config/*.yaml","enabled":true,"image":{"pullPolicy":"IfNotPresent","repository":"cr.flyte.org/flyteorg/datacatalog","tag":"v0.3.19"},"nodeSelector":{},"podAnnotations":{},"replicaCount":1,"resources":{"limits":{"cpu":"500m","ephemeral-storage":"100Mi","memory":"500Mi"},"requests":{"cpu":"10m","ephemeral-storage":"50Mi","memory":"50Mi"}},"service":{"annotations":{"projectcontour.io/upstream-protocol.h2c":"grpc"},"type":"NodePort"},"serviceAccount":{"annotations":{},"create":true,"imagePullSecrets":{}},"tolerations":[]}` | DATACATALOG SETTINGS | | ||
| datacatalog.affinity | object | `{}` | affinity for Datacatalog deployment | | ||
| datacatalog.configPath | string | `"/etc/datacatalog/config/*.yaml"` | Default regex string for searching configuration files | | ||
| datacatalog.image.pullPolicy | string | `"IfNotPresent"` | Docker image pull policy | | ||
| datacatalog.image.repository | string | `"cr.flyte.org/flyteorg/datacatalog"` | Docker image for Datacatalog deployment | | ||
| datacatalog.image.tag | string | `"v0.3.16"` | Docker image tag | | ||
| datacatalog.image.tag | string | `"v0.3.19"` | Docker image tag | | ||
| datacatalog.nodeSelector | object | `{}` | nodeSelector for Datacatalog deployment | | ||
| datacatalog.podAnnotations | object | `{}` | Annotations for Datacatalog pods | | ||
| datacatalog.replicaCount | int | `1` | Replicas count for Datacatalog deployment | | ||
|
@@ -129,7 +129,7 @@ helm install gateway bitnami/contour -n flyte | |
| flyteadmin.enabled | bool | `true` | | | ||
| flyteadmin.image.pullPolicy | string | `"IfNotPresent"` | | | ||
| flyteadmin.image.repository | string | `"cr.flyte.org/flyteorg/flyteadmin"` | Docker image for Flyteadmin deployment | | ||
| flyteadmin.image.tag | string | `"v0.6.61"` | | | ||
| flyteadmin.image.tag | string | `"v0.6.64"` | | | ||
| flyteadmin.initialProjects | list | `["flytesnacks","flytetester","flyteexamples"]` | Initial projects to create | | ||
| flyteadmin.nodeSelector | object | `{}` | nodeSelector for Flyteadmin deployment | | ||
| flyteadmin.podAnnotations | object | `{}` | Annotations for Flyteadmin pods | | ||
|
@@ -142,7 +142,7 @@ helm install gateway bitnami/contour -n flyte | |
| flyteadmin.serviceAccount.create | bool | `true` | Should a service account be created for flyteadmin | | ||
| flyteadmin.serviceAccount.imagePullSecrets | object | `{}` | ImapgePullSecrets to automatically assign to the service account | | ||
| flyteadmin.tolerations | list | `[]` | tolerations for Flyteadmin deployment | | ||
| flyteconsole | object | `{"affinity":{},"enabled":true,"image":{"pullPolicy":"IfNotPresent","repository":"cr.flyte.org/flyteorg/flyteconsole","tag":"v0.30.0"},"nodeSelector":{},"podAnnotations":{},"replicaCount":1,"resources":{"limits":{"cpu":"500m","memory":"250Mi"},"requests":{"cpu":"10m","memory":"50Mi"}},"service":{"annotations":{},"type":"ClusterIP"},"tolerations":[]}` | FLYTECONSOLE SETTINGS | | ||
| flyteconsole | object | `{"affinity":{},"enabled":true,"image":{"pullPolicy":"IfNotPresent","repository":"cr.flyte.org/flyteorg/flyteconsole","tag":"v0.34.0"},"nodeSelector":{},"podAnnotations":{},"replicaCount":1,"resources":{"limits":{"cpu":"500m","memory":"250Mi"},"requests":{"cpu":"10m","memory":"50Mi"}},"service":{"annotations":{},"type":"ClusterIP"},"tolerations":[]}` | FLYTECONSOLE SETTINGS | | ||
| flyteconsole.affinity | object | `{}` | affinity for Flyteconsole deployment | | ||
| flyteconsole.image.repository | string | `"cr.flyte.org/flyteorg/flyteconsole"` | Docker image for Flyteconsole deployment | | ||
| flyteconsole.nodeSelector | object | `{}` | nodeSelector for Flyteconsole deployment | | ||
|
@@ -151,7 +151,7 @@ helm install gateway bitnami/contour -n flyte | |
| flyteconsole.resources | object | `{"limits":{"cpu":"500m","memory":"250Mi"},"requests":{"cpu":"10m","memory":"50Mi"}}` | Default resources requests and limits for Flyteconsole deployment | | ||
| flyteconsole.service | object | `{"annotations":{},"type":"ClusterIP"}` | Service settings for Flyteconsole | | ||
| flyteconsole.tolerations | list | `[]` | tolerations for Flyteconsole deployment | | ||
| flytepropeller | object | `{"affinity":{},"cacheSizeMbs":0,"configPath":"/etc/flyte/config/*.yaml","enabled":true,"image":{"pullPolicy":"IfNotPresent","repository":"cr.flyte.org/flyteorg/flytepropeller","tag":"v0.15.17"},"nodeSelector":{},"podAnnotations":{},"replicaCount":1,"resources":{"limits":{"cpu":"200m","ephemeral-storage":"100Mi","memory":"200Mi"},"requests":{"cpu":"10m","ephemeral-storage":"50Mi","memory":"50Mi"}},"serviceAccount":{"annotations":{},"create":true,"imagePullSecrets":{}},"tolerations":[]}` | FLYTEPROPELLER SETTINGS | | ||
| flytepropeller | object | `{"affinity":{},"cacheSizeMbs":0,"configPath":"/etc/flyte/config/*.yaml","enabled":true,"image":{"pullPolicy":"IfNotPresent","repository":"cr.flyte.org/flyteorg/flytepropeller","tag":"v0.16.4"},"nodeSelector":{},"podAnnotations":{},"replicaCount":1,"resources":{"limits":{"cpu":"200m","ephemeral-storage":"100Mi","memory":"200Mi"},"requests":{"cpu":"10m","ephemeral-storage":"50Mi","memory":"50Mi"}},"serviceAccount":{"annotations":{},"create":true,"imagePullSecrets":{}},"tolerations":[]}` | FLYTEPROPELLER SETTINGS | | ||
| flytepropeller.affinity | object | `{}` | affinity for Flytepropeller deployment | | ||
| flytepropeller.configPath | string | `"/etc/flyte/config/*.yaml"` | Default regex string for searching configuration files | | ||
| flytepropeller.image.repository | string | `"cr.flyte.org/flyteorg/flytepropeller"` | Docker image for Flytepropeller deployment | | ||
|
@@ -164,12 +164,12 @@ helm install gateway bitnami/contour -n flyte | |
| flytepropeller.serviceAccount.create | bool | `true` | Should a service account be created for FlytePropeller | | ||
| flytepropeller.serviceAccount.imagePullSecrets | object | `{}` | ImapgePullSecrets to automatically assign to the service account | | ||
| flytepropeller.tolerations | list | `[]` | tolerations for Flytepropeller deployment | | ||
| flytescheduler | object | `{"affinity":{},"configPath":"/etc/flyte/config/*.yaml","image":{"pullPolicy":"IfNotPresent","repository":"cr.flyte.org/flyteorg/flytescheduler","tag":"v0.6.61"},"nodeSelector":{},"podAnnotations":{},"resources":{"limits":{"cpu":"250m","ephemeral-storage":"100Mi","memory":"500Mi"},"requests":{"cpu":"10m","ephemeral-storage":"50Mi","memory":"50Mi"}},"secrets":{},"serviceAccount":{"annotations":{},"create":true,"imagePullSecrets":{}},"tolerations":[]}` | FLYTESCHEDULER SETTINGS | | ||
| flytescheduler | object | `{"affinity":{},"configPath":"/etc/flyte/config/*.yaml","image":{"pullPolicy":"IfNotPresent","repository":"cr.flyte.org/flyteorg/flytescheduler","tag":"v0.6.64"},"nodeSelector":{},"podAnnotations":{},"resources":{"limits":{"cpu":"250m","ephemeral-storage":"100Mi","memory":"500Mi"},"requests":{"cpu":"10m","ephemeral-storage":"50Mi","memory":"50Mi"}},"secrets":{},"serviceAccount":{"annotations":{},"create":true,"imagePullSecrets":{}},"tolerations":[]}` | FLYTESCHEDULER SETTINGS | | ||
| flytescheduler.affinity | object | `{}` | affinity for Flytescheduler deployment | | ||
| flytescheduler.configPath | string | `"/etc/flyte/config/*.yaml"` | Default regex string for searching configuration files | | ||
| flytescheduler.image.pullPolicy | string | `"IfNotPresent"` | Docker image pull policy | | ||
| flytescheduler.image.repository | string | `"cr.flyte.org/flyteorg/flytescheduler"` | Docker image for Flytescheduler deployment | | ||
| flytescheduler.image.tag | string | `"v0.6.61"` | Docker image tag | | ||
| flytescheduler.image.tag | string | `"v0.6.64"` | Docker image tag | | ||
| flytescheduler.nodeSelector | object | `{}` | nodeSelector for Flytescheduler deployment | | ||
| flytescheduler.podAnnotations | object | `{}` | Annotations for Flytescheduler pods | | ||
| flytescheduler.resources | object | `{"limits":{"cpu":"250m","ephemeral-storage":"100Mi","memory":"500Mi"},"requests":{"cpu":"10m","ephemeral-storage":"50Mi","memory":"50Mi"}}` | Default resources requests and limits for Flytescheduler deployment | | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -16,7 +16,7 @@ flyteadmin: | |
image: | ||
# -- Docker image for Flyteadmin deployment | ||
repository: cr.flyte.org/flyteorg/flyteadmin # FLYTEADMIN_IMAGE | ||
tag: v0.6.61 # FLYTEADMIN_TAG | ||
tag: v0.6.64 # FLYTEADMIN_TAG | ||
pullPolicy: IfNotPresent | ||
# -- Default resources requests and limits for Flyteadmin deployment | ||
resources: | ||
|
@@ -72,7 +72,7 @@ flytescheduler: | |
# -- Docker image for Flytescheduler deployment | ||
repository: cr.flyte.org/flyteorg/flytescheduler # FLYTESCHEDULER_IMAGE | ||
# -- Docker image tag | ||
tag: v0.6.61 # FLYTESCHEDULER_TAG | ||
tag: v0.6.64 # FLYTESCHEDULER_TAG | ||
# -- Docker image pull policy | ||
pullPolicy: IfNotPresent | ||
# -- Default resources requests and limits for Flytescheduler deployment | ||
|
@@ -118,7 +118,7 @@ datacatalog: | |
# -- Docker image for Datacatalog deployment | ||
repository: cr.flyte.org/flyteorg/datacatalog # DATACATALOG_IMAGE | ||
# -- Docker image tag | ||
tag: v0.3.16 # DATACATALOG_TAG | ||
tag: v0.3.19 # DATACATALOG_TAG | ||
# -- Docker image pull policy | ||
pullPolicy: IfNotPresent | ||
# -- Default resources requests and limits for Datacatalog deployment | ||
|
@@ -166,7 +166,7 @@ flytepropeller: | |
image: | ||
# -- Docker image for Flytepropeller deployment | ||
repository: cr.flyte.org/flyteorg/flytepropeller # FLYTEPROPELLER_IMAGE | ||
tag: v0.15.17 # FLYTEPROPELLER_TAG | ||
tag: v0.16.4 # FLYTEPROPELLER_TAG | ||
pullPolicy: IfNotPresent | ||
# -- Default resources requests and limits for Flytepropeller deployment | ||
resources: | ||
|
@@ -210,7 +210,7 @@ flyteconsole: | |
image: | ||
# -- Docker image for Flyteconsole deployment | ||
repository: cr.flyte.org/flyteorg/flyteconsole # FLYTECONSOLE_IMAGE | ||
tag: v0.30.0 # FLYTECONSOLE_TAG | ||
tag: v0.34.0 # FLYTECONSOLE_TAG | ||
pullPolicy: IfNotPresent | ||
# -- Default resources requests and limits for Flyteconsole deployment | ||
resources: | ||
|
@@ -476,7 +476,7 @@ configmap: | |
# -- Structure documented [here](https://pkg.go.dev/github.com/lyft/[email protected]/go/tasks/pluginmachinery/flytek8s/config#FlyteCoPilotConfig) | ||
co-pilot: | ||
name: flyte-copilot- | ||
image: cr.flyte.org/flyteorg/flytecopilot:v0.0.22 # FLYTECOPILOT_IMAGE | ||
image: cr.flyte.org/flyteorg/flytecopilot:v0.0.24 # FLYTECOPILOT_IMAGE | ||
start-timeout: 30s | ||
|
||
# -- Core propeller configuration | ||
|
Oops, something went wrong.