Update sandbox deployment to ensure my-s3-bucket is created before registration

Add init container that will pre-create buckets in minio (#905)

* Add init container that will pre-create buckets in minio

Signed-off-by: Jeev B <[email protected]>

* move minio tag override to overlay

Signed-off-by: Jeev B <[email protected]>

Milestone 0.12.0

General

• Added CoPilot configuration to sandbox manifest
• Updated and streamlined documentation navigation and themes. Better content organization coming soon!

Performance

• Improved execution performance including:
  • Reduced cache lookups
  • Improved GetWorkflowExecution performance
  • Capped max number of nodes in each propeller round
  • Misc. propeller performance tweaks
• TaskTemplate offloading

Housekeeping

• Migrated Datacatalog protobuf definitions to flyteidl [thanks @tnsetting]
• Upgraded stow version used in flytestdlib
• Moved off lyft kubernetes forks and onto official kubernetes library dependencies
• Revamped pod tasks to use official kubernetes python client library for defining PodSpecs

Events

• Richer event metadata for task executions
• Better merging of custom info across task events

Bug fixes

• Resolved non-backwards protobuf role changes that prevented launching single task executions [thanks @kanterov]
• Better handling of large workflows and errors in flytepropeller

Flytekit (Python)

• Access to secrets
• Bug fixes around the 0.16 release.
  • Use original FlyteFile/FlyteDirectory
  • Fix serialization of pod specs in pod plugin [thanks @jeevb]
  • Accept auth role arg in single task execution
  • Fixed task resolver in map task
  • Requests and limits added to ContainerTask [thanks @migueltol22]

Update sandbox images to allow overriding api port

Allow K3s K8s Api Port to be overriden (#812)

Signed-off-by: Haytham Abuelfutuh <[email protected]>

Flyte v0.11.0

Flyte Platform

• New to flyte? https://start.flyte.org takes you through first run experience. (Thanks to @jeevb)
• Grafana templates for monitoring Flyte System and User Workflows.
• Extend Flyte docs.
• FlyteIdl Docs are published! You can learn about the core language that makes it all work.
• Additional knob for fine tuning flyte propeller performance that speeds up executions drastically.
• OidC support for Google Idp (And other OidC compliant Idps)
• Various stabilization bugs.

Flytekit

Since v0.16.0a2, the last flytekit milestone release, all effort has been towards stabilizing the new API. Please see the individual releases for detailed information. The highlights are

• Serialization/registration processes have been firmed up and utilities to ease that process introduced (not having to build a container to serialize for instance).
• Plugins structure revamped (eventually we'll move to a separate new repo entirely)
• User-facing imports have been organized into three top-level subpackages (flytekit, flytekit.testing, and flytekit.extend)
• Retries added to read-only Admin calls in client
• Lots of cleanup and additions to the cookbook and documentation generally.
• Bug fixes.

Include Kubernetes Dashboard in Sandbox

Add kubernetes dashboard to sandbox overlay (#789)

* Add kubernetes dashboard to sandbox overlay

Signed-off-by: Jeev B <[email protected]>

* Remove task logs from headless kustomize module

Signed-off-by: Jeev B <[email protected]>

* Update first run doc to expose 30082 for kubernetes dashboard

Signed-off-by: Jeev B <[email protected]>

* use kubernetes-template-url for specifying log URL in sandbox overlay

Signed-off-by: Jeev B <[email protected]>

* Update flytepropeller to v0.6.1 to leverage `kubernetes-template-uri` config

Signed-off-by: Jeev B <[email protected]>

Release containerd & dind docker images

v0.11.0b1

Fix push image condition (#767)

Test tag creation

v0.11.0b0

Fix push image condition

Flyte v0.10.0 Milestone release

Flytekit

• Alpha release of Flytekit (v0.16.0a2). Please check the release for the detailed changelog.
• ^Alpha A revamp of the flytekit examples can be found here.

FlyteConsole

• Show all executions across project and domain
• Yielding launch plans from dynamic tasks will now show links for executions in the UI
• Ability to pass role/service account for single task executions in the launch form
• If auth is enabled, Console will automatically attempt login, saving you a click
• Bug and stability fixes

Flyte Platform

• SparkOperator v1beta2 support
• Bug Fixes

Flyte v0.9.0 Milestone release

Flytekit

• Fast register - code only changes no longer need a rebuild of the container to run (read the docs!)
• Project archive/activate command in flyte-cli
• Bug fixes and misc improvements

Please take a look at the newest alpha release of Flytekit (v0.16.0a0) which introduces native Python based typing, and a new way of workflow authoring that enables local workflow execution.
For examples of the native typed SDK usage refer to: Cookbook Examples

FlyteConsole

• Better Single task launch support
• Launch form with more types supported
• Bug and stability fixes

Flyte Platform

• SparkOperator v1beta2 support
• Archive/Unarchive projects
• Bug Fixes

v0.8.0: September month-end release - "foundation"

Flyte v0.8.0

Core Platform

• Metadata available for every task execution (First one: if the task was executed on interruptible nodes, useful when running on spot instances)
• Support for Cron schedules with offset (only on supported schedulers - e.g. Styx)
• Plugin overrides using Admin (docs coming soon)
• Custom model support in Sagemaker for single node and distributed training

Console

• Full inputs and outputs from Execution data
• Timestamp rendering always in UTC

Flytekit

• Dynamic overridable Spark configuration for spark tasks
• New Tensorflow task
• Formally removed python 2.x support. Supports only 3.6+
• Refactor - getting ready for enhanced auto-typed flytekit