Skip to content

Flytekit v0.22.0
Compare
Choose a tag to compare
@wild-endeavor wild-endeavor released this 31 Aug 23:49
· 1560 commits to master since this release
v0.22.0
204b13f
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

Flytekit v0.22.0

Note: A backwards incompatible change was made to the IDL this release. To mitigate this, we've released patch versions of flytekit going back to 0.15.x.

The major changes this release are

  • Access to durable blob stores (AWS/GCS/etc) are now pluggable.
  • Local task execution has been updated to also trigger the type engine.
  • Tasks that have cache=True should now be cached when running locally as well (this was actually released as part of v0.21.4 but has been updated for this release).

Data Persistence and Automatic Plugin Loading

When a flytekit task runs, it will download and upload files to the Flyte blob store to access inputs and upload outputs. Also, flytekit is configured to offload larger data types to the blob store as well (see FlyteFile, FlyteDirectory, and FlyteSchema for more information). With #559, this mechanism is now pluggable. See the module for the interface data persistence plugins are expected to implement.

Persistence Plugins

  • AWS/GCS/HTTP/local disk plugins have been ported over to the new style.
  • An FSSpec plugin has been added that implements the FS Spec library.

Plugin Loading

Lastly, a new mechanism was introduced as part of #559 that enables the automatic loading of certain flytekit plugins. Basically, if you add

entry_points={"flytekit.plugins": ["<plugin_name>=flytekitplugins.<plugin_name>"]},

to your setup.py, if a user has the plugin pip-installed, then flytekit will automatically load it. See FS Spec plugin for an example

Local Task Execution Alignment

Local task execution that invoked the task directly (i.e. not called through a local workflow execution) previously did not trigger the type engine. This led to inconsistent behavior and has now been rectified.

You may need to update unit tests to support this change. Keep in mind though that this code change only affects local task execution - live execution on a Flyte backed is unaffected.

See the issue and corresponding PR for additional details and examples of unit tests that may need to be updated.

Local Caching

Caching the result of tasks was previously not available in local executions, this was inconsistent with the expected behavior found in remote Flyte and has now been fixed.

The result of the execution of cacheable tasks is written to a local file under the ~/.flyte/local-cache/ directory and persists across multiple workflow and task executions. A similar semantics as the remote Flyte case applies, i.e. task outputs are cached by cache version, task signature and parameters.

The local cache can be emptied by running pyflyte local-cache clear.

Refer to https://docs.flyte.org/projects/cookbook/en/latest/auto/core/flyte_basics/task_cache.html?caching for more details on how caching works.

Please join the upcoming OSS meeting on September 7th for a demo on local caching.

Other Minor Changes

FlyteRemote

Minor fixes and updates to FlyteRemote this release include

  • Ability to list recent executions.
  • Ability to list recent tasks.
  • Fetched launch plans with settings were previously being ignored by FlyteRemote. This has been fixed.

Blob-type objects

FlyteFile and FlyteDirectory now have a download() function to explicitly trigger downloading.

Assets 2
Loading