Skip to content

Commit

Permalink
[Doc] Snowflake agent deployment (#4832)
Browse files Browse the repository at this point in the history
Signed-off-by: Kevin Su <[email protected]>
  • Loading branch information
pingsutw authored Feb 5, 2024
1 parent 9aac0e8 commit b1ea659
Showing 1 changed file with 92 additions and 1 deletion.
93 changes: 92 additions & 1 deletion docs/deployment/agents/snowflake.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,98 @@ Snowflake agent

This guide provides an overview of how to set up the Snowflake agent in your Flyte deployment.

1. Setup the key pair authentication in snowflake. For more details, you can refer to `here <https://docs.snowflake.com/en/user-guide/key-pair-auth>`__.
2. Create a secret with the group "snowflake" and the key "private_key". For more details, you can refer to `here <https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/productionizing/use_secrets.html#secrets>`__.

.. code-block:: bash
kubectl create secret generic snowflake-private-key --namespace=flytesnacks-development --from-file=your_private_key_above
Specify agent configuration
----------------------------

TK
.. tabs::

.. group-tab:: Flyte binary

Edit the relevant YAML file to specify the agent.

.. code-block:: bash
kubectl edit configmap flyte-sandbox-config -n flyte
.. code-block:: yaml
:emphasize-lines: 7,11,16
tasks:
task-plugins:
enabled-plugins:
- container
- sidecar
- k8s-array
- agent-service
default-for-task-types:
- container: container
- container_array: k8s-array
- snowflake: agent-service
plugins:
agent-service:
supportedTaskTypes:
- snowflake
.. group-tab:: Flyte core

Create a file named ``values-override.yaml`` and add the following configuration to it.

.. code-block:: yaml
configmap:
enabled_plugins:
# -- Tasks specific configuration [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#GetConfig)
tasks:
# -- Plugins configuration, [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#TaskPluginConfig)
task-plugins:
# -- [Enabled Plugins](https://pkg.go.dev/github.com/flyteorg/flyteplugins/go/tasks/config#Config). Enable sagemaker*, athena if you install the backend
enabled-plugins:
- container
- sidecar
- k8s-array
- agent-service
default-for-task-types:
container: container
sidecar: sidecar
container_array: k8s-array
snowflake: agent-service
plugins:
agent-service:
supportedTaskTypes:
- snowflake
Ensure that the propeller has the correct service account for BigQuery.

Upgrade the Flyte Helm release
------------------------------

.. tabs::

.. group-tab:: Flyte binary

.. code-block:: bash
helm upgrade <RELEASE_NAME> flyteorg/flyte-binary -n <YOUR_NAMESPACE> --values <YOUR_YAML_FILE>
Replace ``<RELEASE_NAME>`` with the name of your release (e.g., ``flyte-backend``),
``<YOUR_NAMESPACE>`` with the name of your namespace (e.g., ``flyte``),
and ``<YOUR_YAML_FILE>`` with the name of your YAML file.

.. group-tab:: Flyte core

.. code-block:: bash
helm upgrade <RELEASE_NAME> flyte/flyte-core -n <YOUR_NAMESPACE> --values values-override.yaml
Replace ``<RELEASE_NAME>`` with the name of your release (e.g., ``flyte``)

and ``<YOUR_NAMESPACE>`` with the name of your namespace (e.g., ``flyte``).

For Snowflake plugin on the Flyte cluster, please refer to `Snowflake Plugin Example <https://docs.flyte.org/en/latest/flytesnacks/examples/snowflake_plugin/snowflake.html>`_

0 comments on commit b1ea659

Please sign in to comment.