From c1a6799fb3d40164ae4036849a566300571bc8d9 Mon Sep 17 00:00:00 2001 From: Ketan Umare <16888709+kumare3@users.noreply.github.com> Date: Thu, 21 Oct 2021 10:10:43 -0700 Subject: [PATCH] Better docs on how to read flytekit code (#714) --- README.md | 47 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) diff --git a/README.md b/README.md index df823aaf39..942132dddb 100644 --- a/README.md +++ b/README.md @@ -68,6 +68,53 @@ def my_workflow(x: int, y: int) -> int: ## 📖 How to Contribute to Flytekit You can find the detailed contribution guide [here](https://docs.flyte.org/projects/flytekit/en/latest/contributing.html). Plugins' contribution guide is included as well. +## Code Guide +The first version of the flytekit library was written circa 2017, before mypy typing was mainstream, and +targeted Python 2. That legacy code will be fully deprecated and removed in 2022 but because there are still +users of flytekit that rely on that legacy api, you'll see 2 separate and distinct code paths within this repo. +Users and contributors should ignore the legacy sections. Below is a listing of the most important packages that +comprise the new API: + +- `flytekit/core` + This holds all the core functionality of the new API. +- `flytekit/types` + We bundle some special types like `FlyteFile, FlyteSchema etc` by default here. +- `flytekit/extend` + This is the future home of extension points, and currently serves as the raw documentation for extensions. +- `flytekit/extras` + This contains code that we want bundled with flytekit but not everyone may find useful (for example AWS and GCP + specific logic). +- `flytekit/remote` + This implements the interface to interact with the Flyte service. Think of the code here as the Python-object version of Console. +- `flytekit/testing` + is the future home for testing functionality like `mock` etc, and currently serves as documentation. + All test extensions should be imported from here. +- `flytekit/models` + Protobuf generated Python code is not terribly user-friendly, so we improve upon those `flyteidl` classes here. +- `plugins` + is the source of all plugins +- `flytekit/bin/entrypoint.py` + The run time entrypoint for flytekit. When a task kicks off, this is where the click command goes. +- `flytekit/clis` + This is the home for the clis. +- `flytekit/configuration` + This holds all the configuration objects, but dependency on configuration should be carefully considered as it + makes compiled Flyte tasks and workflows less portable (i.e. if you run `pyflyte package` can someone else use + those serialized objects). + +Most of the other folders are for legacy Flytekit, support for which will be dropped in early 2022. For the most part, +please ignore the following folders: + +- `flytekit/plugins` +- `flytekit/common` + (the `translator.py` file is an exception) +- `flytekit/engines` +- `flytekit/interfaces` +- `flytekit/sdk` +- `flytekit/type_engines` + +Please also see the [overview section](https://docs.flyte.org/projects/flytekit/en/latest/design/index.html) of the formal flytekit documentation for more information. + ## 🐞 File an Issue Refer to the [issues](https://docs.flyte.org/en/latest/community/contribute.html#issues) section in the contribution guide if you'd like to file an issue.