diff --git a/cookbook/deployment/README.rst b/cookbook/deployment/README.rst index e69de29bb2..80c9e17afb 100644 --- a/cookbook/deployment/README.rst +++ b/cookbook/deployment/README.rst @@ -0,0 +1,7 @@ +.. _deployment_workflow: + +Workflow Deployment +------------------- + +Once you've finished iterating on your tasks and workflows locally, you can serialize and register them to a Flyte +backend. This section covers how to configure and deploy your workflows for your particular needs. diff --git a/cookbook/deployment/cluster/README.rst b/cookbook/deployment/cluster/README.rst deleted file mode 100644 index 532a118f89..0000000000 --- a/cookbook/deployment/cluster/README.rst +++ /dev/null @@ -1,8 +0,0 @@ -.. _deployment_cluster: - -Cluster Deployment -------------------- - -As the complexity of your user base grows, you may find yourself tweaking resource assignments based on specific -projects, domains, and workflows. This section of the user guide walks through how and in what ways you can -configure your Flyte deployment for production. diff --git a/cookbook/deployment/cluster/__init__.py b/cookbook/deployment/cluster/__init__.py deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/cookbook/deployment/cluster/access_cloud_resources.py b/cookbook/deployment/cluster/access_cloud_resources.py deleted file mode 100644 index 6c657a9d3e..0000000000 --- a/cookbook/deployment/cluster/access_cloud_resources.py +++ /dev/null @@ -1,25 +0,0 @@ -""" -Accessing Cloud Resources -------------------------- - -.. TODO: add intro paragraph here - -********************************** -Kubernetes ServiceAccount Examples -********************************** - -Configure project-wide Kubernetes ServiceAccounts by adding the following to your config: - -.. code:: python - - [auth] - kubernetes_service_account=my-kube-service-acct - - -Alternatively, pass the role as an argument to ``create_launch_plan``: - -.. code:: python - - my_lp = MyWorkflow.create_launch_plan(kubernetes_service_account='my-kube-service-acct') - -""" diff --git a/cookbook/deployment/cluster/auth_migration.py b/cookbook/deployment/cluster/auth_migration.py deleted file mode 100644 index 9e8ad18bbf..0000000000 --- a/cookbook/deployment/cluster/auth_migration.py +++ /dev/null @@ -1,164 +0,0 @@ -""" -.. _migrating-auth-config: - -#################################### -Migrating Your Authentication Config -#################################### - -Using Okta as an example, you would have previously seen something like the following: - -Okta Config changes -=================== - -* An Application (OpenID Connect Web) for Flyte Admin itself (e.g. **0oal5rch46pVhCGF45d6**). -* An Application (OpenID Native app) for Flyte-cli/flytectl (e.g. **0oal62nxuD6OSFSRq5d6**). - These two applications would be assigned to the relevant users. -* An Application (Web) for Flyte Propeller (e.g. **0abc5rch46pVhCGF9876**). - This application would either use the default Authorization server, or you would create a new one. - -Admin Config Changes -==================== - -.. code-block:: yaml - - server: - # ... other settings - security: - secure: false - useAuth: true - allowCors: true - allowedOrigins: - - "*" - allowedHeaders: - - "Content-Type" - oauth: - baseUrl: https://dev-62129345.okta.com/oauth2/default/ - scopes: - - profile - - openid - - email - claims: - iss: https://dev-62129345.okta.com/oauth2/default - aud: 0oal5rch46pVhCGF45d6 - clientId: 0oal5rch46pVhCGF45d6 - clientSecretFile: "/Users/ytong/etc/secrets/oauth/secret" - authorizeUrl: "https://dev-62129345.okta.com/oauth2/default/v1/authorize" - tokenUrl: "https://dev-62129345.okta.com/oauth2/default/v1/token" - callbackUrl: "http://localhost:8088/callback" - cookieHashKeyFile: "/Users/ytong/etc/secrets/hashkey/hashkey" - cookieBlockKeyFile: "/Users/ytong/etc/secrets/blockkey/blockkey" - redirectUrl: "/api/v1/projects" - thirdPartyConfig: - flyteClient: - clientId: 0oal62nxuD6OSFSRq5d6 - redirectUri: http://localhost:12345/callback - -After version `0.13.0 `__ of the platform, you can still use the IdP as -the Authorization Server if you wish. That configuration would now become: - -.. code-block:: yaml - - server: - # ... other settings - security: - secure: false - useAuth: true - allowCors: true - allowedOrigins: - - "*" - allowedHeaders: - - "Content-Type" - auth: - authorizedUris: - # This should point at your public http Uri. - - https://flyte.mycompany.com - # This will be used by internal services in the same namespace as flyteadmin - - http://flyteadmin:80 - # This will be used by internal services in the same cluster but different namespaces - - http://flyteadmin.flyte.svc.cluster.local:80 - userAuth: - openId: - # Put the URL of the OpenID Connect provider. - baseUrl: https://dev-62129345.okta.com/oauth2/default # Okta with a custom Authorization Server - scopes: - - profile - - openid - - offline_access # Uncomment if OIdC supports issuing refresh tokens. - # Replace with the client id created for Flyte. - clientId: 0oal5rch46pVhCGF45d6 - appAuth: - # External delegates app auth responsibilities to an external authorization server, Internal means Flyte Admin does it itself - authServerType: External - thirdPartyConfig: - flyteClient: - clientId: 0oal62nxuD6OSFSRq5d6 - redirectUri: http://localhost:12345/callback - scopes: - - all - - offline - -Specifically, - -* The original **oauth** section has been moved two levels higher into its own section and renamed **auth** but enabling/disabling of authentication remains in the old location. -* Secrets by default will now be looked up in **/etc/secrets**. Use the following command to generate them: - - .. prompt:: bash - - flyteadmin secrets init -p /etc/secrets - - This will generate the new cookie hash/block keys, as well as other secrets Admin needs to run the Authorization server. - -* The **clientSecretFile** has been moved to **/etc/secrets/oidc_client_secret** so move that there. -* **claims** has been removed, just delete that. -* **authorizeUrl** and **tokenUrl** are no longer necessary. -* The **baseUrl** for the external Authorization Server is now in the **appAuth** section. -* The **thirdPartyConfig** has been moved to **appAuth** as well. -* **redirectUrl** has been defaulted to **/console**. If that's the value you want, then you no longer need this setting. - -Propeller Config Changes -======================== - -You might have a configuration section that looks like this: - -.. code-block:: yaml - - admin: - endpoint: dns:///mycompany.domain.com - useAuth: true - clientId: flytepropeller - clientSecretLocation: /etc/secrets/client_secret - tokenUrl: https://demo.nuclyde.io/oauth2/token - scopes: - - all - -This can now be simplified to: - -.. code-block:: yaml - - admin: - endpoint: dns:///mycompany.domain.com - # If you are using the built-in authorization server, you can delete the following two lines: - clientId: flytepropeller - clientSecretLocation: /etc/secrets/client_secret - -Specifically, - -* **useAuth** is deprecated and will be removed in a future version. Auth requirement will be discovered through an anonymous admin discovery call. -* **tokenUrl** and **scopes** will also be discovered through a metadata call. -* **clientId** and **clientSecretLocation** have defaults that work out of the box with the built-in authorization server (e.g. if you setup Google OpenID Connect). - -Flyte-cli Config Changes -======================== - -You need these two configs set: - -.. prompt:: bash - - FLYTE_PLATFORM_HTTP_URL=http://localhost:8088 FLYTE_CREDENTIALS_CLIENT_ID=0oal62nxuD6OSFSRq5d6 flyte-cli ... - -**FLYTE_PLATFORM_HTTP_URL** is used because **flyte-cli** uses only gRPC to communicate with Admin. It needs to know the -HTTP port (which Admin hosts on a different port because of limitations of the grpc-gateway library). **flyte-cli** uses -this setting to talk to **/.well-known/oauth-authorization-server** to retrieve information regarding the auth endpoints -. Previously this redirected to the Okta Authorization Server's metadata endpoint. With this change, Admin now hosts its -own (even if still using the external Authorization Service). -""" diff --git a/cookbook/deployment/cluster/auth_setup.py b/cookbook/deployment/cluster/auth_setup.py deleted file mode 100644 index c63861bac6..0000000000 --- a/cookbook/deployment/cluster/auth_setup.py +++ /dev/null @@ -1,340 +0,0 @@ -""" -Authentication in Flyte ------------------------ - -Flyte ships with a canonical implementation of OpenIDConnect client and OAuth2 Server, integrating seamlessly into an -organization's existing identity provider. - -.. _auth-overview: - -######## -Overview -######## - -The Flyte system consists of multiple components. Securing communication between each components is crucial to ensure -the security of the overall system. - -In abstract, Flyte supports OAuth2 and OpenId Connect (built on top of OAuth2) to secure the various connections: - -* :ref:`OpenId Connect `: Used to secure user's authentication to flyteadmin service. -* :ref:`OAuth2 `: Used to secure communication between clients (i.e. flyte-cli, flytectl and - flytepropeller) and flyteadmin service. - -Identity Providers Support -========================== - -Support for these protocols varies per IdP, checkout the following table to understand the available support level for -your IdP. - -+----------------------+--------+-------------+---------------------+----------+-------+----------+--------+ -| Feature | Okta | Google free | GC Identity Service | Azure AD | Auth0 | KeyCloak | Github | -+======================+========+=============+=====================+==========+=======+==========+========+ -| OpenIdConnect | Yes | Yes | Yes | Yes | Yes | Yes | No | -+----------------------+--------+-------------+---------------------+----------+-------+----------+--------+ -| Custom Relying Party | Yes | No | Yes | Yes | ? | Yes | No | -+----------------------+--------+-------------+---------------------+----------+-------+----------+--------+ - -.. _auth-setup: - -#################### -Authentication Setup -#################### - -***************** -IdP Configuration -***************** -Flyte Admin requires that the application in your identity provider be configured as a web client (i.e. with a client secret). We recommend allowing the application to be issued a refresh token to avoid interrupting the user's flow by frequently redirecting to the IdP. - -**************************** -Example Flyte Configurations -**************************** - -Below are some canonical examples of how to set up some of the common IdPs to secure your Fyte services. OpenID Connect enables users to authenticate, in the -browser, with an existing IdP. Flyte also allows connecting to an external OAuth2 Authorization Server to allow centrally managed third party app access. - -OpenID Connect -=============== - -OpenID Connect allows users to authenticate to Flyte in their browser using a familiar authentication provider (perhaps an organization-wide configured IdP). -Flyte supports connecting with external OIdC providers. Here are some examples for how to set these up: - -.. tabs:: - - .. tab:: Google - - Follow `Google Docs `__ on how to configure the IdP for OpenIDConnect. - - .. note:: - - Make sure to create an OAuth2 Client Credential. The `client_id` and `client_secret` will be needed in the following - steps. - - .. tab:: Okta - - Okta supports OpenID Connect protocol and the creation of custom OAuth2 Authorization Servers, allowing it to act as both the user and apps IdP. - It offers more detailed control on access policies, user consent, and app management. - - 1. If you don't already have an Okta account, sign up for one `here `__. - 2. Create an app (choose Web for the platform) and OpenID Connect for the sign-on method. - 3. Add Login redirect URIs (e.g. http://localhost:30081/callback for sandbox or ``https:///callback``) - 4. *Optional*: Add logout redirect URIs (e.g. http://localhost:30081/logout for sandbox) - 5. Write down the Client ID and Client Secret - - .. tab:: KeyCloak - - `KeyCloak `__ is an open source solution for authentication, it supports both OpenID Connect and OAuth2 protocols (among others). - KeyCloak can be configured to be both the OpenID Connect and OAuth2 Authorization Server provider for Flyte. - -Apply Configuration -^^^^^^^^^^^^^^^^^^^ - -#. Store the `client_secret` in a k8s secret as follows: - - .. prompt:: bash - - kubectl edit secret -n flyte flyte-admin-auth - - Add a new key under `stringData`: - - .. code-block:: yaml - - stringData: - oidc_client_secret: - data: - ... - - Save and close your editor. - -#. Edit FlyteAdmin config to add `client_id` and configure auth as follows: - - .. prompt:: bash - - kubectl edit configmap -n flyte flyte-admin-config - - Follow the inline comments to make the necessary changes: - - .. code-block:: yaml - - server: - ... - security: - secure: false - # 1. Enable Auth by turning useAuth to true - useAuth: true - ... - auth: - userAuth: - openId: - # 2. Put the URL of the OpenID Connect provider. - # baseUrl: https://accounts.google.com # Uncomment for Google - baseUrl: https://dev-14186422.okta.com/oauth2/default # Okta with a custom Authorization Server - scopes: - - profile - - openid - # - offline_access # Uncomment if OIdC supports issuing refresh tokens. - # 3. Replace with the client ID created for Flyte. - clientId: 0oakkheteNjCMERst5d6 - - Save and exit your editor. - -#. Restart `flyteadmin` for the changes to take effect: - - .. prompt:: bash - - kubectl rollout restart deployment/flyteadmin -n flyte - -OAuth2 Authorization Server -=========================== - -An OAuth2 Authorization Server allows external clients to request to authenticate and act on behalf of users (or as their own identities). Having -an OAuth2 Authorization Server enables Flyte administrators control over which apps can be installed and what scopes they are allowed to request or be granted (i.e. what privileges can they assume). - -Flyte comes with a built-in authorization server that can be statically configured with a set of clients to request and act on behalf of the user. -The default clients are defined `here `__ -and the corresponding section can be modified through configs. - -To set up an external OAuth2 Authorization Server, please follow the instructions below: - -.. tabs:: - - .. tab:: Okta - - 1. Under security -> API, click `Add Authorization Server`. Set the audience to the public URL of flyte admin (e.g. ````). - 2. Under `Access Policies`, click `Add New Access Policy` and walk through the wizard to allow access to the authorization server. - 3. Under `Scopes`, click `Add Scope`. Set the name to `all` (required) and check `Require user consent for this scope` (recommended). - 4. Create 2 apps (for fltyectl and flytepropeller) to enable these clients to communicate with the service. - Flytectl should be created as a `native client`. - FlytePropeller should be created as an `OAuth Service` and note the client ID and client Secrets provided. - - .. tab:: KeyCloak - - `KeyCloak `__ is an open source solution for authentication, it supports both OpenID Connect and OAuth2 protocols (among others). - KeyCloak can be configured to be both the OpenID Connect and OAuth2 Authorization Server provider for flyte. - -Apply Configuration -^^^^^^^^^^^^^^^^^^^ - -#. It is possible to direct Flyte admin to use an external authorization server. To do so, edit the same config map once - more and follow these changes: - - .. code-block:: yaml - - auth: - appAuth: - # 1. Choose External if you will use an external Authorization Server (e.g. a Custom Authorization server in Okta) - # Choose Self (or omit the value) to use Flyte Admin's internal (albeit limited) Authorization Server. - authServerType: External - - # 2. Optional: Set external auth server baseUrl if different from OpenId baseUrl. - externalAuthServer: - baseUrl: https://dev-14186422.okta.com/oauth2/auskngnn7uBViQq6b5d6 - thirdPartyConfig: - flyteClient: - # 3. Replace with a new Native Client ID provisioned in the custom authorization server - clientId: flytectl - - redirectUri: https://localhost:53593/callback - - # 4. "all" is a required scope and must be configured in the custom authorization server - scopes: - - offline - - all - userAuth: - openId: - baseUrl: https://dev-14186422.okta.com/oauth2/auskngnn7uBViQq6b5d6 # Okta with a custom Authorization Server - scopes: - - profile - - openid - # - offline_access # Uncomment if OIdC supports issuing refresh tokens. - clientId: 0oakkheteNjCMERst5d6 - -#. Store flyte propeller's `client_secret` in a k8s secret as follows: - - .. prompt:: bash - - kubectl edit secret -n flyte flyte-propeller-auth - - Add a new key under `stringData`: - - .. code-block:: yaml - - stringData: - client_secret: from the previous step - data: - ... - - Save and close your editor. - -#. Edit FlytePropeller config to add `client_id` and configure auth as follows: - - .. prompt:: bash - - kubectl edit configmap -n flyte flyte-propeller-config - - Follow the inline comments to make the necessary changes: - - .. code-block:: yaml - - admin: - # 1. Replace with the client_id provided by the OAuth2 Authorization Server above. - clientId: flytepropeller - - Close the editor - -#. Restart `flytepropeller` for the changes to take effect: - - .. prompt:: bash - - kubectl rollout restart deployment/flytepropeller -n flyte - -Continuous Integration - CI -=========================== - -If your organization does any automated registration, then you'll need to authenticate with the `client credentials `_ flow. After retrieving an access token from the IDP, you can send it along to Flyte Admin as usual. - -.. tabs:: - - .. tab:: Flytectl - - Flytectl's `config.yaml `_ can be - configured to use either PKCE (`Proof key for code exchange `_) - or Client Credentials (`Client Credentials `_) flows. - - Update ``config.yaml`` as follows: - - .. code-block:: yaml - - admin: - # Update with the admin's endpoint. You must keep the 3 forward-slashe after dns: - endpoint:: dns:/// - - # Update auth type to `Pkce` or `ClientSecret` - authType: Pkce - - # Set to the clientId (will be used for both Pkce and ClientSecret flows) - # Leave empty to use the value discovered through flyteadmins Auth discovery endpoint. - clientId: - - # Set to the location where the client secret is mounted. - # Only needed/used for `ClientSecret` flow. - clientSecretLocation: - - # If required, set the scopes needed here. Otherwise, flytectl will discover scopes required for OpenID - # Connect through Admin's Auth discovery endpoint. - # scopes: [ "scope1", "scope2" ] - - To read further about the available config options, please - `visit here `_ - - .. tab:: Flytekit / Flyte-cli - - Flytekit configuration variables are automatically designed to look up values from relevant environment variables. - However, to aid with continuous integration use-cases, Flytekit configuration can also reference other environment - variables. - - For instance, if your CI system is not capable of setting custom environment variables like - ``FLYTE_CREDENTIALS_CLIENT_SECRET`` but does set the necessary settings under a different variable, you may use - ``export FLYTE_CREDENTIALS_CLIENT_SECRET_FROM_ENV_VAR=OTHER_ENV_VARIABLE`` to redirect the lookup. A - ``FLYTE_CREDENTIALS_CLIENT_SECRET_FROM_FILE`` redirect is available as well, where the value should be the full - path to the file containing the value for the configuration setting, in this case, the client secret. We found - this redirect behavior necessary when setting up registration within our own CI pipelines. - - The following is a listing of the Flytekit configuration values we set in CI, along with a brief explanation. - - .. code:: bash - - # When using OAuth2 service auth, this is the username and password. - export FLYTE_CREDENTIALS_CLIENT_ID= - export FLYTE_CREDENTIALS_CLIENT_SECRET= - - # This tells the SDK to use basic authentication. If not set, Flytekit will assume you want to use the - # standard OAuth based three-legged flow. - export FLYTE_CREDENTIALS_AUTH_MODE=basic - - # This value should be set to conform to this - # `header config `_ - # on the Admin side. - export FLYTE_CREDENTIALS_AUTHORIZATION_METADATA_KEY=
- - # When using basic authentication, you'll need to specify a scope to the IDP (instead of ``openid``, which is - # only for OAuth). Set that here. - export FLYTE_CREDENTIALS_OAUTH_SCOPES= - - # Set this to force Flytekit to use authentication, even if not required by Admin. This is useful as you're - # rolling out the requirement. - export FLYTE_PLATFORM_AUTH=True - -.. _auth-references: - -########## -References -########## - -This collection of RFCs may be helpful to those who wish to investigate the implementation in more depth. - -* `OAuth2 RFC 6749 `_ -* `OAuth Discovery RFC 8414 `_ -* `PKCE RFC 7636 `_ -* `JWT RFC 7519 `_ - -""" diff --git a/cookbook/deployment/cluster/auth_setup_appendix.py b/cookbook/deployment/cluster/auth_setup_appendix.py deleted file mode 100644 index 66f9f57740..0000000000 --- a/cookbook/deployment/cluster/auth_setup_appendix.py +++ /dev/null @@ -1,132 +0,0 @@ -""" -Authentication Appendix ------------------------- - -.. _auth-openid-appendix: - -############## -OpenID Connect -############## - -Flyte supports OpenID Connect. A defacto standard for user authentication. After configuring OpenID Connect, users -accessing flyte console or flytectl (or other 3rd party apps) will be prompted to authenticate using the configured -provider. - -.. mermaid:: - :alt: Flyte UI Swimlane - - sequenceDiagram - %%{config: { 'fontFamily': 'Menlo', 'fontSize': 10, 'fontWeight': 100} }%% - autonumber - User->>+Browser: /home - Browser->>+Console: /home - Console->>-Browser: 302 /login - Browser->>+Admin: /login - Admin->>-Browser: Idp.com/oidc - Browser->>+Idp: Idp.com/oidc - Idp->>-Browser: 302 /login - Browser->>-User: Enter user/pass - User->>+Browser: login - Browser->>+Idp: Submit username/pass - Idp->>-Browser: admin/?authCode= - Browser->>+Admin: admin/authCode= - Admin->>+Idp: Exchange Tokens - Idp->>-Admin: idt, at, rt - Admin->>+Browser: Write Cookies & Redirect to /console - Browser->>+Console: /home - Browser->>-User: Render /home - -.. _auth-oauth2-appendix: - -######## -OAuth2 -######## - -Flyte supports OAuth2 to control access to 3rd party and native apps. FlyteAdmin comes with a built in Authorization -Server that can perform 3-legged and 2-legged OAuth2 flows. It also supports delegating these responsibilities to an -external Authorization Server. - -Service Authentication using OAuth2 -=================================== - -Propeller (and potentially other non-user facing services) can also authenticate using client_credentials to the Idp and -be granted an access_token valid to be used with admin and other backend services. - -.. tabs:: - - .. tab:: FlyteAdmin's builtin Authorization Server - - .. mermaid:: - :alt: Service Authentication Swimlane - - sequenceDiagram - Propeller->>+Admin: /token?client_creds&scope=https://admin/ - Admin->>-Propeller: access_token - Propeller->>+Admin: /list_projects?token=access_token - - .. tab:: External Authorization Server - - .. mermaid:: - :alt: Service Authentication Swimlane - - sequenceDiagram - Propeller->>+External Authorization Server: /token?client_creds&scope=https://admin/ - External Authorization Server->>-Propeller: access_token - Propeller->>+Admin: /list_projects?token=access_token - -User Authentication in other clients (e.g. Cli) using OAuth2-Pkce -================================================================== - -Users accessing backend services through Cli should be able to use OAuth2-Pkce flow to authenticate (in a browser) to the Idp and be issued -an access_token valid to communicate with the intended backend service on behalf of the user. - -.. tabs:: - - .. tab:: FlyteAdmin's builtin Authorization Server - - .. mermaid:: - :alt: CLI Authentication with Admin's own Authorization Server - - sequenceDiagram - %%{config: { 'fontFamily': 'Menlo', 'fontSize': 10, 'fontWeight': 100} }%% - autonumber - User->>+Cli: flytectl list-projects - Cli->>+Admin: admin/client-config - Admin->>-Cli: Client_id=, ... - Cli->>+Browser: /oauth2/authorize?pkce&code_challenge,client_id,scope - Browser->>+Admin: /oauth2/authorize?pkce... - Admin->>-Browser: 302 idp.com/login - Note over Browser,Admin: The prior OpenID Connect flow - Browser->>+Admin: admin/logged_in - Note over Browser,Admin: Potentially show custom consent screen - Admin->>-Browser: localhost/?authCode= - Browser->>+Cli: localhost/authCode= - Cli->>+Admin: /token?code,code_verifier - Admin->>-Cli: access_token - Cli->>+Admin: /projects/ + access_token - Admin->>-Cli: project1, project2 - - .. tab:: External Authorization Server - - .. mermaid:: - :alt: CLI Authentication with an external Authorization Server - - sequenceDiagram - %%{config: { 'fontFamily': 'Menlo', 'fontSize': 10, 'fontWeight': 100} }%% - autonumber - User->>+Cli: flytectl list-projects - Cli->>+Admin: admin/client-config - Admin->>-Cli: Client_id=, ... - Cli->>+Browser: /oauth2/authorize?pkce&code_challenge,client_id,scope - Browser->>+ExternalIdp: /oauth2/authorize?pkce... - ExternalIdp->>-Browser: 302 idp.com/login - Note over Browser,ExternalIdp: The prior OpenID Connect flow - Browser->>+ExternalIdp: /logged_in - Note over Browser,ExternalIdp: Potentially show custom consent screen - ExternalIdp->>-Browser: localhost/?authCode= - Browser->>+Cli: localhost/authCode= - Cli->>+ExternalIdp: /token?code,code_verifier - ExternalIdp->>-Cli: access_token - Cli->>+Admin: /projects/ + access_token - Admin->>-Cli: project1, project2 -""" diff --git a/cookbook/deployment/cluster/config_flyte_deploy.py b/cookbook/deployment/cluster/config_flyte_deploy.py deleted file mode 100644 index b5b91f8fef..0000000000 --- a/cookbook/deployment/cluster/config_flyte_deploy.py +++ /dev/null @@ -1,191 +0,0 @@ -""" -Configuring Your Flyte Deployment ----------------------------------- - -*************************** -Configurable Resource Types -*************************** - -Flyte allows these custom settings along with the following combination of dimensions: - -- domain -- project and domain -- project, domain, and name (must be either the name of a workflow name or a launch plan) - -Please see the :doc:`Control Plane ` document to get to know about projects and domains. -Along these dimensions, the following settings are configurable. - -.. admonition:: Alert - Not all three of the combinations mentioned above are valid for each of these settings. - -- Defaults are used for task resource requests and limits (when not specified by the author of the task). -- Settings for project-namespaced cluster resource configuration that feeds into admin's cluster resource manager. -- Execution queues that are used for dynamic workflows. Effectively, they're meant to be used with constructs like AWS Batch. - -.. note:: - Execution queues are used to determine where tasks yielded by a :py:func:`flytekit:flytekit.dynamic` workflow or ``map task`` run. - -- Determining how workflow executions are assigned to clusters in a multi-cluster Flyte deployment. - -.. tip:: - The proto definition is the definitive source encapsulating which :ref:`Matchable Resource ` attributes can be customized. - -Each of the four above settings is discussed below. - -Task Resources -============== -Configuring task resources includes setting default values for the requests and limits for the following resources: - -- cpu -- gpu -- memory -- storage - -In the absence of an override, the global -`default values `__ -in the FlyteAdmin config are used. - -The override values from the database are assigned at execution time. - -To update individual project-domain attributes, use the following command for your reference. - -.. prompt:: bash - - curl --request PUT 'https://flyte.company.net/api/v1/project_domain_attributes/projectname/staging' \ - --header 'Content-Type: application/json' --data-raw \ - '{"attributes":{"matchingAttributes":{"taskResourceAttributes":{"defaults":{"cpu": "1000", "memory": "5000Gi"}, "limits": {"cpu": "4000"}}}}' - -.. tip:: - The equivalent ``flytectl`` command is: - - .. prompt:: bash - flytectl update task-resource-attribute - - Refer to the :ref:`docs ` to learn more about the command and its supported flag(s). - -Cluster Resources -================= -These are free-form key-value pairs that are used when filling in the templates that the admin feeds into its cluster manager. The keys represent templatized variables in `cluster resource template YAML `__ and the values are what you want to see filled in. - -In the absence of custom override values, ``templateData`` from the `FlyteAdmin config `__ is used as a default. - -.. note:: - The above-referenced settings can only take on domain, project, and domain specificity. Since Flyte has not tied in the notion of a workflow or a launch plan to any Kubernetes constructs, specifying a workflow or launch plan name doesn't make any sense. - -Running the following will ensure that when the admin fills in cluster resource templates, the Kubernetes namespace ``flyteexamples-development`` will have a resource quota of 1000 CPU cores and 5TB of memory. - -.. prompt:: bash - - flyte-cli -h localhost:30081 -p flyteexamples -d development update-cluster-resource-attributes \ - --attributes projectQuotaCpu 1000 --attributes projectQuotaMemory 5000Gi - -.. tip:: - The equivalent ``flytectl`` command is: - - .. prompt:: bash - flytectl update cluster-resource-attribute - - Refer to the :ref:`docs ` to learn more about the command and its supported flag(s). - -The above-updated values will, in turn, be used to fill in the template fields. - -.. rli:: https://raw.githubusercontent.com/flyteorg/flyte/master/kustomize/base/single_cluster/headless/config/clusterresource-templates/ab_project-resource-quota.yaml - -The values can now be copied from the base of this repository to the ``flyteexamples-development`` namespace only. - -For other namespaces, the `platform defaults `__ apply. - -.. note:: - The template values, e.g. ``projectQuotaCpu`` or ``projectQuotaMemory`` are freeform strings. You must ensure that they match the template placeholders in your `template file `__ - for your changes to take effect. - -Execution Queues -================ -Execution queues themselves are currently defined in the -`flyteadmin config `__. - -The **attributes** associated with an execution queue must match the **tags** for workflow executions. The tags are associated with configurable resources -stored in the admin database. - -.. prompt:: bash - - flyte-cli -h localhost:30081 -p flyteexamples -d development update-execution-queue-attributes \ - --tags critical --tags gpu_intensive - -.. tip:: - The equivalent command in ``flytectl`` is: - - .. prompt:: bash - flytectl update execution-queue-attribute - - Refer to the :ref:`docs ` to learn more about the command and its supported flag(s). - -You can view existing attributes for which tags can be assigned by visiting ``protocol:///api/v1/matchable_attributes?resource_type=3``. - -Execution Cluster Label -======================= -This allows forcing a matching execution to consistently execute on a specific Kubernetes cluster. - -You can set this using flyte-cli: - -.. prompt:: bash - - flyte-cli -h localhost:30081 -p flyteexamples -d development update-execution-cluster-label --value mycluster - -.. tip:: - The equivalent command in ``flytectl`` is: - - .. prompt:: bash - flytectl update execution-cluster-label - - Refer to the :ref:`docs ` to learn more about the command and its supported flag(s). - -********* -Hierarchy -********* -Increasing specificity defines how matchable resource attributes get applied. The available configurations, in order of decreasing specifity, are: - -#. Domain, project, workflow name, and launch plan. - -#. Domain, project, and workflow name - -#. Domain and project - -#. Domain - -Default values for all and per-domain attributes may be specified in the FlyteAdmin config as documented in the :ref:`config_resource_types`. - -Example -======= -If the database includes the following: - -+------------+--------------+----------+-------------+-----------+ -| Domain | Project | Workflow | Launch Plan | Tags | -+============+==============+==========+=============+===========+ -| production | widgetmodels | | | critical | -+------------+--------------+----------+-------------+-----------+ -| production | widgetmodels | Demand | | supply | -+------------+--------------+----------+-------------+-----------+ - -Any inbound ``CreateExecution`` requests with **[Domain: Production, Project: widgetmodels, Workflow: Demand]** for any launch plan will have a tag value of "supply". - -Any inbound ``CreateExecution`` requests with **[Domain: Production, Project: widgetmodels]** for any workflow other than ``Demand`` and any launch plan will have a tag value "critical". - -All other inbound CreateExecution requests will use the default values specified in the FlyteAdmin config (if any). - -********* -Debugging -********* -To get the matchable resources of :ref:`execution queue attributes `, run the command: - -.. prompt:: bash - flytectl get execution-queue-attribute - -.. note:: - Alternatively, you can also hit the URL: ``protocol://``. - -To get the global state of the world, list all endpoints. For example, visit ``protocol:///api/v1/matchable_attributes?resource_type=2``. - -The resource type enum (int) is defined in the :ref:`Matchable Resource `. - -""" diff --git a/cookbook/deployment/cluster/config_resource_mgr.py b/cookbook/deployment/cluster/config_resource_mgr.py deleted file mode 100644 index 580ac02823..0000000000 --- a/cookbook/deployment/cluster/config_resource_mgr.py +++ /dev/null @@ -1,8 +0,0 @@ -""" -Configuring Your Resource Manager --------------------------------------------- - -.. NOTE:: - - Coming soon 🛠 -""" \ No newline at end of file diff --git a/cookbook/deployment/cluster/monitoring.py b/cookbook/deployment/cluster/monitoring.py deleted file mode 100644 index e329e10a8f..0000000000 --- a/cookbook/deployment/cluster/monitoring.py +++ /dev/null @@ -1,19 +0,0 @@ -""" -Monitoring ----------- - -.. tip:: The Flyte core team publishes a maintains Grafana dashboards built using Prometheus data sources and can be found `here `__. - -Flyte Backend is written in Golang and exposes stats using Prometheus. The Stats themselves are labeled with the Workflow, Task, Project & Domain wherever appropriate. - -The dashboards are divided into primarily 2 types: - -- User facing dashboards. These are dashboards that can be used to triage/investigate/observe performance and characterisitics of Workflows and tasks. - The User facing dashboard is published under Grafana marketplace ID `13980 `_ - -- System Dashboards. These dashboards are useful for the system maintainer to maintain their Flyte deployments. These are further divided into - - DataPlane/FlytePropeller dashboards published @ `13979 `_ - - ControlPlane/Flyteadmin dashboards published @ `13981 `_ - -These are basic dashboards and do no include all the metrics that are exposed by Flyte. Please help us improve the dashboards by contributing to them. Refer to the build scripts `here `__. -""" diff --git a/cookbook/deployment/cluster/notifications.py b/cookbook/deployment/cluster/notifications.py deleted file mode 100644 index 7651f69cf8..0000000000 --- a/cookbook/deployment/cluster/notifications.py +++ /dev/null @@ -1,118 +0,0 @@ -""" -Notifications -------------- - -When a workflow completes, users can be notified by: - -* email -* `pagerduty `__ -* `slack `__ - -The content of these notifications is configurable at the platform level. - -***** -Usage -***** - -When a workflow reaches a specified `terminal workflow execution phase `__ -the :py:class:`flytekit:flytekit.Email`, :py:class:`flytekit:flytekit.PagerDuty`, or :py:class:`flytekit:flytekit.Slack` -objects can be used in the construction of a :py:class:`flytekit:flytekit.LaunchPlan`. - -For example - -.. code:: python - - from flytekit import Email, LaunchPlan - from flytekit.models.core.execution import WorkflowExecutionPhase - - # This launch plan triggers email notifications when the workflow execution it triggered reaches the phase `SUCCEEDED`. - my_notifiying_lp = LaunchPlan.create( - "my_notifiying_lp", - my_workflow_definition, - default_inputs={"a": 4}, - notifications=[ - Email( - phases=[WorkflowExecutionPhase.SUCCEEDED], - recipients_email=["admin@example.com"], - ) - ], - ) - - -See detailed usage examples in the :std:ref:`User Guide ` - -Notifications can be combined with schedules to automatically alert you when a scheduled job succeeds or fails. - -Future work -=========== - -Work is ongoing to support a generic event egress system that can be used to publish events for tasks, workflows and -workflow nodes. When this is complete, generic event subscribers can asynchronously process these vents for a rich -and fully customizable experience. - - -****************************** -Platform Configuration Changes -****************************** - -Setting Up Workflow Notifications -================================= - -The ``notifications`` top-level portion of the FlyteAdmin config specifies how to handle notifications. - -As with schedules, the notifications handling is composed of two parts. One handles enqueuing notifications asynchronously and the second part handles processing pending notifications and actually firing off emails and alerts. - -This is only supported for Flyte instances running on AWS. - -Config -======= - -To publish notifications, you'll need to set up an `SNS topic `_. - -In order to process notifications, you'll need to set up an `AWS SQS `_ queue to consume notification events. This queue must be configured as a subscription to your SNS topic you created above. - -In order to actually publish notifications, you'll need a `verified SES email address `_ which will be used to send notification emails and alerts using email APIs. - -The role you use to run FlyteAdmin must have permissions to read and write to your SNS topic and SQS queue. - -Let's look at the following config section and explain what each value represents: :: - - notifications: - type: "aws" - region: "us-east-1" - publisher: - topicName: "arn:aws:sns:us-east-1:{{ YOUR ACCOUNT ID }}:{{ YOUR TOPIC }}" - processor: - queueName: "{{ YOUR QUEUE NAME }}" - accountId: "{{ YOUR ACCOUNT ID }}" - emailer: - subject: "Notice: Execution \"{{ workflow.name }}\" has {{ phase }} in \"{{ domain }}\"." - sender: "flyte-notifications@company.com" - body: > - Execution \"{{ workflow.name }} [{{ name }}]\" has {{ phase }} in \"{{ domain }}\". View details at - - http://flyte.company.com/console/projects/{{ project }}/domains/{{ domain }}/executions/{{ name }}. {{ error }} - -* **type**: Because AWS is the only cloud back-end supported for executing scheduled workflows in this case, only ``"aws"`` is a valid value. By default, the no-op executor is used. -* **region**: This specifies which region AWS clients will use when creating SNS and SQS clients. -* **publisher**: This handles pushing notification events to your SNS topic. - * **topicName**: This is the arn of your SNS topic. -* **processor**: This handles the recording notification events and enqueueing them to be processed asynchronously. - * **queueName**: This is the name of the SQS queue which will capture pending notification events. - * **accountId**: Your AWS `account id `_ -* **emailer**: This section encloses config details for sending and formatting emails used as notifications. - * **subject**: Configurable subject line used in notification emails. - * **sender**: Your verified SES email sender. - * **body**: Configurable email body used in notifications. - -The full set of parameters which can be used for email templating are checked into `code `_. - -.. _admin-config-example: - -Example config -============== - -.. rli:: https://raw.githubusercontent.com/flyteorg/flyteadmin/master/flyteadmin_config.yaml - :lines: 66-80 - -""" diff --git a/cookbook/deployment/cluster/optimize_perf.py b/cookbook/deployment/cluster/optimize_perf.py deleted file mode 100644 index 764e90798d..0000000000 --- a/cookbook/deployment/cluster/optimize_perf.py +++ /dev/null @@ -1,8 +0,0 @@ -""" -Optimize Performance --------------------------------------------- - -.. NOTE:: - - Coming soon 🛠 -""" \ No newline at end of file diff --git a/cookbook/deployment/cluster/productionize_cluster.py b/cookbook/deployment/cluster/productionize_cluster.py deleted file mode 100644 index 935ce5e97c..0000000000 --- a/cookbook/deployment/cluster/productionize_cluster.py +++ /dev/null @@ -1,128 +0,0 @@ -""" -Productionize Your Flyte Cluster --------------------------------- - -In order to handle production load robustly, securely and with high availability, there are a number of important tasks that need to -be done independently from the sandbox deployment: - -* The kubernetes cluster needs to run securely and robustly -* The sandbox's object store must be replaced by a production grade storage system -* The sandbox's PostgreSQL database must be replaced by a production grade deployment of postgres -* A production grade task queueing system must be provisioned and configured -* A production grade notification system must be provisioned and configured -* All the above must be done in a secure fashion -* (Optionally) An official dns domain must be created -* (Optionally) A production grade email sending system must be provisioned and configured - -A Flyte user may provision and orchestrate this setup by themselves, but the Flyte team has partnered with the -`Opta `_ team to create a streamlined production deployment strategy for AWS with -ready-to-use templates provided in the `Flyte repo `_. The following -documentation specifies how to use and further configure them. - -Deploying Opta Environment and Service for Flyte -************************************************ -**The Environment** -To begin using Opta, please first `download the latest version `_ and all the listed -prerequisites and make sure that you have -`admin/fullwrite AWS credentials setup on your terminal `_. -With that prepared, go to the Opta subdirectory in the Flyte repo, and open up env.yaml in your editor. Please find and -replace the following values with your desired ones: - -* : your AWS account ID -* : your AWS region -* : your desired domain for your Flyte deployment (should be a domain which you own or a subdomain thereof - this environment will promptyly take ownership of the domain/subdomain so make sure it will only be used for this purpose) -* : a name for the new isolated cloud environment which is going to be created (e.g. flyte-prod) -* : your company or organization's name - -Once complete please run ``opta apply -c env.yaml`` and follow the prompts. - -**DNS Delegation** -Once Opta's apply for the environment is completed, you will need to complete dns delegation to fully setup public -traffic access. You may find instructions on `how to do so here `_. - -**The Flyte Deployment** -Once dns deployment delegation is complete, you may deploy the Flyte service and affiliated resources. Go to the Opta -subdirectory in the Flyte repo, and open up flyte.yaml in your editor. Please find and replace the following values with -your desired ones: - -* : your AWS account ID -* : your AWS region - -Once complete please run ``opta apply -c flyte.yaml`` and follow the prompts. - -Understanding the Opta Yamls -**************************** -The Opta yaml files - -**Production Grade Environment** -The Opta env.yaml is responsible for setting up the base infrastructure necessary for most cloud resources. The base -module sets up the VPC and subnets (both public and private) used by the environment as well as the shared KMS keys. -The dns sets up the hosted zone for domain and ssl certificates once completed. The k8s-cluster creates the -Kubernetes cluster and node pool (with encrypted disk storage). And lastly the k8s-base module sets up the resources -within Kubernetes like the autoscaler, metrics server, and ingress. - -**Production Grade Database** -The aws-postgres module in flyte.yaml creates an Aurora Postgresql database with disk encryption and regular snapshot -backups. You can read more about it `here `_ - -**Production Grade Object Store** -The aws-s3 module in flyte.yaml creates a new S3 bucket for Flyte, including disk encryption. You can read more about it -`here `_ - -**Production Grade Notification System** -Flyte uses a combination of the AWS Simple Notification Service (SNS) and Simple Queueing service for a notification -system. flyte.yaml creates both the SNS topic and SQS queue (via the notifcationsQueue and topic modules), which are -encrypted with unique KMS keys and only the flyte roles can access them. You can read more about the queues -`here `_ and the topics -`here `_. - -**Production Grade Queueing System** -Flyte uses SQS to power its task scheduling system, and flyte.yaml creates said queue (via the schedulesQueue -module) with encryption and principle of least privilege rbac access like the other SQS queue above. - -**Secure IAM Roles for Data and Control Planes** - - -**Flyte Deployment via Helm** -A Flyte deployment contains around 50 kubernetes resources. - -Additional Setup -**************** -By now you should be set up for most production deployments, but there are some extra steps which we recommend that -most users consider. - -**Email Setup** -Flyte has the power to send email notifications, which can be enabled in Opta via -`AWS' Simple Email Service `_ with a few extra steps (NOTE: make sure to have completed dns -delegation first): -1. Simply go to env.yaml and uncomment out the last line ( `- type: aws-ses` ) - -2. Run ``opta apply -c env.yaml`` again - -This will enable SES on your account and environment domain -- you may be prompted to fill in some user-specific input to take your account out of SES sandbox if not done already. -It may take a day for AWS to enable production SES on your account (you will be kept notified via the email addresses inputted on the user -prompt) but that should not prevent you from moving forward. - -3. Lastly, go ahead and uncomment out the 'Uncomment out for SES' line in the flyte.yaml and rerun ``opta apply -c flyte.yaml``. - -You will now be able to receive emails sent by Flyte as soon as AWS approves your account. You may also specify other -non-default email senders via the helm chart values. - -**Flyte Rbac** -All Flyte deployments are currently insecure on the application level by default (e.g. open/accessible to everyone) so it -is strongly recommended that users `add authentication `_. - -**Extra configuration** -It is possible to add extra configuration to your Flyte deployment by modifying the values passed in the helm chart -used by Opta. Please refer to the possible values allowed from the `Flyte helm chart `_ -and update the values field of the Flyte module in the flyte.yaml file accordingly. - - -Raw Helm Deployment -******************* -It is certainly possible to deploy a production Flyte cluster directly using the helm chart if a user does not wish to -use Opta. To do so properly, one will need to ensure they have completed the initial security/ha/robustness checklist -from above, and then use `helm `_ to deploy the `Flyte helm chart `_. - - -""" diff --git a/cookbook/deployment/workflow/customizing_resources.py b/cookbook/deployment/customizing_resources.py similarity index 100% rename from cookbook/deployment/workflow/customizing_resources.py rename to cookbook/deployment/customizing_resources.py diff --git a/cookbook/deployment/workflow/deploying_workflows.py b/cookbook/deployment/deploying_workflows.py similarity index 100% rename from cookbook/deployment/workflow/deploying_workflows.py rename to cookbook/deployment/deploying_workflows.py diff --git a/cookbook/deployment/workflow/fast_registration.py b/cookbook/deployment/fast_registration.py similarity index 99% rename from cookbook/deployment/workflow/fast_registration.py rename to cookbook/deployment/fast_registration.py index c670937f28..5b2b897ac6 100644 --- a/cookbook/deployment/workflow/fast_registration.py +++ b/cookbook/deployment/fast_registration.py @@ -17,7 +17,7 @@ Flytekit releases before the introduction of native typing have a slightly modified workflow for fast-registering. Please check the below-highlighted sections if your Flytkit version is < ``0.16.0``. Pre-requisites -============= +============== Upgrade your Flytekit dependency to ``>= 0.16.0`` and re-run pip-tools if necessary. .. admonition:: Flytkit < 0.16.0 diff --git a/cookbook/deployment/guides/README.rst b/cookbook/deployment/guides/README.rst deleted file mode 100644 index 3e981ccf21..0000000000 --- a/cookbook/deployment/guides/README.rst +++ /dev/null @@ -1,4 +0,0 @@ -.. _deployment_guides: - -Deployment Guides -------------------------------------------- diff --git a/cookbook/deployment/guides/__init__.py b/cookbook/deployment/guides/__init__.py deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/cookbook/deployment/guides/aws.py b/cookbook/deployment/guides/aws.py deleted file mode 100644 index ee906d9d3d..0000000000 --- a/cookbook/deployment/guides/aws.py +++ /dev/null @@ -1,8 +0,0 @@ -""" -AWS ------- - -.. NOTE:: - - Coming soon 🛠 -""" diff --git a/cookbook/deployment/guides/gcp.py b/cookbook/deployment/guides/gcp.py deleted file mode 100644 index e9e521d3b0..0000000000 --- a/cookbook/deployment/guides/gcp.py +++ /dev/null @@ -1,8 +0,0 @@ -""" -GCP ---- - -.. NOTE:: - - Coming soon 🛠 -""" \ No newline at end of file diff --git a/cookbook/deployment/guides/kubernetes.py b/cookbook/deployment/guides/kubernetes.py deleted file mode 100644 index 3b18fe6933..0000000000 --- a/cookbook/deployment/guides/kubernetes.py +++ /dev/null @@ -1,8 +0,0 @@ -""" -Kubernetes ----------- - -.. NOTE:: - - Coming soon 🛠 -""" \ No newline at end of file diff --git a/cookbook/deployment/workflow/lp_notifications.py b/cookbook/deployment/lp_notifications.py similarity index 100% rename from cookbook/deployment/workflow/lp_notifications.py rename to cookbook/deployment/lp_notifications.py diff --git a/cookbook/deployment/workflow/lp_schedules.py b/cookbook/deployment/lp_schedules.py similarity index 100% rename from cookbook/deployment/workflow/lp_schedules.py rename to cookbook/deployment/lp_schedules.py diff --git a/cookbook/deployment/workflow/multiple_k8s.py b/cookbook/deployment/multiple_k8s.py similarity index 100% rename from cookbook/deployment/workflow/multiple_k8s.py rename to cookbook/deployment/multiple_k8s.py diff --git a/cookbook/deployment/workflow/README.rst b/cookbook/deployment/workflow/README.rst deleted file mode 100644 index 80c9e17afb..0000000000 --- a/cookbook/deployment/workflow/README.rst +++ /dev/null @@ -1,7 +0,0 @@ -.. _deployment_workflow: - -Workflow Deployment -------------------- - -Once you've finished iterating on your tasks and workflows locally, you can serialize and register them to a Flyte -backend. This section covers how to configure and deploy your workflows for your particular needs. diff --git a/cookbook/deployment/workflow/__init__.py b/cookbook/deployment/workflow/__init__.py deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/cookbook/docs/_static/sphx_gallery_autogen.css b/cookbook/docs/_static/sphx_gallery_autogen.css index 14c6bad794..4cac8d9a38 100644 --- a/cookbook/docs/_static/sphx_gallery_autogen.css +++ b/cookbook/docs/_static/sphx_gallery_autogen.css @@ -1,19 +1,9 @@ #sphx-glr-download-auto-core-type-system-flyte-python-types-py, #sphx-glr-download-auto-core-containerization-workflow-labels-annotations-py, -#sphx-glr-download-auto-deployment-workflow-deploying-workflows-py, -#sphx-glr-download-auto-deployment-workflow-fast-registration-py, -#sphx-glr-download-auto-deployment-workflow-multiple-k8s-py, -#sphx-glr-download-auto-deployment-cluster-productionize-cluster-py, -#sphx-glr-download-auto-deployment-cluster-auth-migration-py, -#sphx-glr-download-auto-deployment-cluster-optimize-perf-py, -#sphx-glr-download-auto-deployment-cluster-auth-setup-appendix-py, -#sphx-glr-download-auto-deployment-cluster-access-cloud-resources-py, -#sphx-glr-download-auto-deployment-cluster-monitoring-py, -#sphx-glr-download-auto-deployment-cluster-auth-setup-py, -#sphx-glr-download-auto-deployment-cluster-config-resource-mgr-py, -#sphx-glr-download-auto-deployment-cluster-notifications-py, -#sphx-glr-download-auto-deployment-cluster-config-flyte-deploy-py, +#sphx-glr-download-auto-deployment-deploying-workflows-py, +#sphx-glr-download-auto-deployment-fast-registration-py, +#sphx-glr-download-auto-deployment-multiple-k8s-py, #sphx-glr-download-auto-core-extend-flyte-backend-plugins-py { height: 0px; visibility: hidden; diff --git a/cookbook/docs/conf.py b/cookbook/docs/conf.py index 32b24fcf3f..36ac6a8eda 100644 --- a/cookbook/docs/conf.py +++ b/cookbook/docs/conf.py @@ -234,9 +234,7 @@ def __call__(self, filename): "../case_studies/feature_engineering/sqlite_datacleaning", "../testing", "../core/containerization", - "../deployment/workflow", - "../deployment/cluster", - # "../deployment/guides", # TODO: add content to this section + "../deployment", # "../control_plane", # TODO: add content to this section # "../integrations/flytekit_plugins/sqllite3", # TODO: add content to this section "../integrations/flytekit_plugins/papermilltasks", @@ -263,8 +261,7 @@ def __call__(self, filename): "auto/case_studies/feature_engineering/sqlite_datacleaning", "auto/testing", "auto/core/containerization", - "auto/deployment/workflow", - "auto/deployment/cluster", + "auto/deployment", # "auto/deployment/guides", # TODO: add content to this section # "auto/control_plane", # TODO: add content to this section # "auto/integrations/flytekit_plugins/sqllite3", # TODO: add content to this section diff --git a/cookbook/docs/deployment.rst b/cookbook/docs/deployment.rst deleted file mode 100644 index 9cb40099a7..0000000000 --- a/cookbook/docs/deployment.rst +++ /dev/null @@ -1,37 +0,0 @@ -############### -Deployment -############### - -The aim of Flyte is to make it as easy as possible to deploy your data processing and machine learning workflows to -production. This section covers the journey of deploying workflows to a Flyte backend and productionizing a Flyte -cluster to handle non-trivial workloads. - -.. panels:: - :header: text-center - - .. link-button:: auto/deployment/workflow/index - :type: ref - :text: Workflow Deployment - :classes: btn-block stretched-link - ^^^^^^^^^^^^ - Schedule your workflows, customize task memory and CPU resources, and more. - - --- - - .. link-button:: auto/deployment/cluster/index - :type: ref - :text: Cluster Deployment - :classes: btn-block stretched-link - ^^^^^^^^^^^^ - Productionize your Flyte cluster, handle authentication, configure and monitor your cluster, and more. - -.. toctree:: - :maxdepth: 1 - :caption: Deployment - :hidden: - - Workflow Deployment - Cluster Deployment - -.. TODO: add deployment guides to toctree -.. Deployment Guides diff --git a/cookbook/docs/gcp.rst b/cookbook/docs/gcp.rst deleted file mode 100644 index 7c52cc0aa2..0000000000 --- a/cookbook/docs/gcp.rst +++ /dev/null @@ -1,9 +0,0 @@ -:nosearch: - -### -GCP -### - -.. NOTE:: - - Coming soon 🛠 \ No newline at end of file diff --git a/cookbook/docs/index.rst b/cookbook/docs/index.rst index 638eae91d0..3e65614edf 100644 --- a/cookbook/docs/index.rst +++ b/cookbook/docs/index.rst @@ -63,12 +63,12 @@ If this is your first time using Flyte, check out the --- - .. link-button:: deployment + .. link-button:: auto/deployment/index :type: ref - :text: 🚢 Deployment + :text: 🚢 Configuring Production Features :classes: btn-block stretched-link ^^^^^^^^^^ - Ship data processing and machine learning pipelines to production. + Ship and configure your machine learning pipelines on a production Flyte installation. --- @@ -97,6 +97,7 @@ If this is your first time using Flyte, check out the |book-reader| User Guide |chalkboard| Tutorials |project-diagram| Concepts + |rocket| Deployment |book| API Reference |hands-helping| Community @@ -114,7 +115,7 @@ If this is your first time using Flyte, check out the Type System Testing Containerization - deployment + Configuring Production Features integrations Extending flyte