From b1ea659632fead780619effc9f344e5438085a9e Mon Sep 17 00:00:00 2001 From: Kevin Su Date: Mon, 5 Feb 2024 12:06:41 -0800 Subject: [PATCH] [Doc] Snowflake agent deployment (#4832) Signed-off-by: Kevin Su --- docs/deployment/agents/snowflake.rst | 93 +++++++++++++++++++++++++++- 1 file changed, 92 insertions(+), 1 deletion(-) diff --git a/docs/deployment/agents/snowflake.rst b/docs/deployment/agents/snowflake.rst index 7b527469c6..0b9348383f 100644 --- a/docs/deployment/agents/snowflake.rst +++ b/docs/deployment/agents/snowflake.rst @@ -5,7 +5,98 @@ Snowflake agent This guide provides an overview of how to set up the Snowflake agent in your Flyte deployment. +1. Setup the key pair authentication in snowflake. For more details, you can refer to `here `__. +2. Create a secret with the group "snowflake" and the key "private_key". For more details, you can refer to `here `__. + +.. code-block:: bash + kubectl create secret generic snowflake-private-key --namespace=flytesnacks-development --from-file=your_private_key_above + Specify agent configuration ---------------------------- -TK \ No newline at end of file +.. tabs:: + + .. group-tab:: Flyte binary + + Edit the relevant YAML file to specify the agent. + + .. code-block:: bash + + kubectl edit configmap flyte-sandbox-config -n flyte + + .. code-block:: yaml + :emphasize-lines: 7,11,16 + + tasks: + task-plugins: + enabled-plugins: + - container + - sidecar + - k8s-array + - agent-service + default-for-task-types: + - container: container + - container_array: k8s-array + - snowflake: agent-service + + plugins: + agent-service: + supportedTaskTypes: + - snowflake + + .. group-tab:: Flyte core + + Create a file named ``values-override.yaml`` and add the following configuration to it. + + .. code-block:: yaml + + configmap: + enabled_plugins: + # -- Tasks specific configuration [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#GetConfig) + tasks: + # -- Plugins configuration, [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#TaskPluginConfig) + task-plugins: + # -- [Enabled Plugins](https://pkg.go.dev/github.com/flyteorg/flyteplugins/go/tasks/config#Config). Enable sagemaker*, athena if you install the backend + enabled-plugins: + - container + - sidecar + - k8s-array + - agent-service + default-for-task-types: + container: container + sidecar: sidecar + container_array: k8s-array + snowflake: agent-service + plugins: + agent-service: + supportedTaskTypes: + - snowflake + +Ensure that the propeller has the correct service account for BigQuery. + +Upgrade the Flyte Helm release +------------------------------ + +.. tabs:: + + .. group-tab:: Flyte binary + + .. code-block:: bash + + helm upgrade flyteorg/flyte-binary -n --values + + Replace ```` with the name of your release (e.g., ``flyte-backend``), + ```` with the name of your namespace (e.g., ``flyte``), + and ```` with the name of your YAML file. + + .. group-tab:: Flyte core + + .. code-block:: bash + + helm upgrade flyte/flyte-core -n --values values-override.yaml + + Replace ```` with the name of your release (e.g., ``flyte``) + + and ```` with the name of your namespace (e.g., ``flyte``). + +For Snowflake plugin on the Flyte cluster, please refer to `Snowflake Plugin Example `_